JSON extractor putting _ instead of . as the Key separator

1. Describe your incident:

I’ve configured the JSON extractor to use a . as the Key separator but the logs end up with a _ the same!

2. Describe your environment:

  • OS Information:
    Linux Ubuntu Server
  • Package Version:
    Graylog v4.3.11+b48cf45

3. What steps have you already taken to try and solve the problem?

I’ve tried doing everything from the beginning with a fresh installation.

4. How can the community help?

Provide help troubleshooting the problem.

I am aware that this issue has already been reported and I’m also aware that in having dots in field names are possible in Elasticsearch but not recommended - they could cause issues with inner object fields. I imagine Graylog clears that up as it processes the message through to Elasticsearch.

But nevertheless, I need to have the Key separator as a . because of Wazuh and Kibana.

Have you tried something else besides a dot?

Have you tried to enable Flatten structures?

I did try the + sign. And the logs stop being JSON parsed and I receive the following error inside the log:

gl2_processing_error
Replaced invalid timestamp value in message <5c9c3e44-a15c-11ed-9312-024288ed94dc> with current time - Value <2023-01-31T11:42:37.338+0000> caused exception: Invalid format: "2023-01-31T11:42:37.338+0000" is malformed at "T11:42:37.338+0000".

When I flatten the structure I get this example log:

agent
id:043, name:WP-SVM-DC01, ip:10.10.90.13, labels:{group=Server}
data
win:{system={providerName=Microsoft-Windows-Security-Auditing, providerGuid={54849625-5478-4994-a5ba-3e3b0328c30d}, eventID=4624, version=2, level=0, task=12544, opcode=0, keywords=0x8020000000000000, systemTime=2023-01-31T11:46:05.2319794Z, eventRecordID=17381521, processID=848, threadID=1532, channel=Security, computer=WP-SVM-DC01.winprovit.pt, severityValue=AUDIT_SUCCESS, message="An account was successfully logged on.

Subject:
	Security ID:		S-1-0-0
	Account Name:		-
	Account Domain:		-
	Logon ID:		0x0

Logon Information:
	Logon Type:		3
	Restricted Admin Mode:	-
	Virtual Account:		No
	Elevated Token:		Yes

Impersonation Level:		Impersonation

New Logon:
	Security ID:		S-1-5-21-4086784526-1345709382-551355621-5174
	Account Name:		wppaloalto
	Account Domain:		WINPROVITAD
	Logon ID:		0x17F500744
	Linked Logon ID:		0x0
	Network Account Name:	-
	Network Account Domain:	-
	Logon GUID:		{00000000-0000-0000-0000-000000000000}

Process Information:
	Process ID:		0x0
	Process Name:		-

Network Information:
	Workstation Name:	CACEM-FW1
	Source Network Address:	10.20.95.254
	Source Port:		32950

Detailed Authentication Information:
	Logon Process:		NtLmSsp 
	Authentication Package:	NTLM
	Transited Services:	-
	Package Name (NTLM only):	NTLM V2
	Key Length:		128

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The impersonation level field indicates the extent to which a process in the logon session can impersonate.

The authentication information fields provide detailed information about this specific logon request.
	- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
	- Transited services indicate which intermediate services have participated in this logon request.
	- Package name indicates which sub-protocol was used among the NTLM protocols.
	- Key length indicates the length of the generated session key. This will be 0 if no session key was requested."}, eventdata={subjectUserSid=S-1-0-0, subjectLogonId=0x0, targetUserSid=S-1-5-21-4086784526-1345709382-551355621-5174, targetUserName=wppaloalto, targetDomainName=WINPROVITAD, targetLogonId=0x17f500744, logonType=3, logonProcessName=NtLmSsp, authenticationPackageName=NTLM, workstationName=CACEM-FW1, logonGuid={00000000-0000-0000-0000-000000000000}, lmPackageName=NTLM V2, keyLength=128, processId=0x0, ipAddress=10.20.95.254, ipPort=32950, impersonationLevel=%%1833, virtualAccount=%%1843, targetLinkedLogonId=0x0, elevatedToken=%%1842}}
decoder
name:windows_eventchannel
full_log
{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-a5ba-3e3b0328c30d}","eventID":"4624","version":"2","level":"0","task":"12544","opcode":"0","keywords":"0x8020000000000000","systemTime":"2023-01-31T11:46:05.2319794Z","eventRecordID":"17381521","processID":"848","threadID":"1532","channel":"Security","computer":"WP-SVM-DC01.winprovit.pt","severityValue":"AUDIT_SUCCESS","message":"\"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\t-\r\n\tAccount Domain:\t\t-\r\n\tLogon ID:\t\t0x0\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t3\r\n\tRestricted Admin Mode:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-21-4086784526-1345709382-551355621-5174\r\n\tAccount Name:\t\twppaloalto\r\n\tAccount Domain:\t\tWINPROVITAD\r\n\tLogon ID:\t\t0x17F500744\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x0\r\n\tProcess Name:\t\t-\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\tCACEM-FW1\r\n\tSource Network Address:\t10.20.95.254\r\n\tSource Port:\t\t32950\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tNtLmSsp \r\n\tAuthentication Package:\tNTLM\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\tNTLM V2\r\n\tKey Length:\t\t128\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\""},"eventdata":{"subjectUserSid":"S-1-0-0","subjectLogonId":"0x0","targetUserSid":"S-1-5-21-4086784526-1345709382-551355621-5174","targetUserName":"wppaloalto","targetDomainName":"WINPROVITAD","targetLogonId":"0x17f500744","logonType":"3","logonProcessName":"NtLmSsp","authenticationPackageName":"NTLM","workstationName":"CACEM-FW1","logonGuid":"{00000000-0000-0000-0000-000000000000}","lmPackageName":"NTLM V2","keyLength":"128","processId":"0x0","ipAddress":"10.20.95.254","ipPort":"32950","impersonationLevel":"%%1833","virtualAccount":"%%1843","targetLinkedLogonId":"0x0","elevatedToken":"%%1842"}}}
gl2_processing_error
Replaced invalid timestamp value in message <d9c82870-a15c-11ed-9312-024288ed94dc> with current time - Value <2023-01-31T11:46:06.299+0000> caused exception: Invalid format: "2023-01-31T11:46:06.299+0000" is malformed at "T11:46:06.299+0000".
id
1675165566.908494958
location
EventChannel
manager
name:srv-wazuh-server
message
{"true":1675165566.3113,"timestamp":"2023-01-31T11:46:06.299+0000","rule":{"level":6,"description":"Successful Remote Logon Detected - NTLM authentication, possible pass-the-hash attack.","id":"92652","mitre":{"id":["T1550.002"],"tactic":["Defense Evasion","Lateral Movement"],"technique":["Pass the Hash"]},"firedtimes":3636,"mail":false,"groups":["win_evt_channel","authentication_success"],"gdpr":["IV_32.2"],"gpg13":["7.1","7.2"],"hipaa":["164.312.b"],"nist_800_53":["AC.7","AU.14"],"pci_dss":["10.2.5"],"tsc":["CC6.8","CC7.2","CC7.3"]},"agent":{"id":"043","name":"WP-SVM-DC01","ip":"10.10.90.13","labels":{"group":"Server"}},"manager":{"name":"srv-wazuh-server"},"id":"1675165566.908494958","full_log":"{\"win\":{\"system\":{\"providerName\":\"Microsoft-Windows-Security-Auditing\",\"providerGuid\":\"{54849625-5478-4994-a5ba-3e3b0328c30d}\",\"eventID\":\"4624\",\"version\":\"2\",\"level\":\"0\",\"task\":\"12544\",\"opcode\":\"0\",\"keywords\":\"0x8020000000000000\",\"systemTime\":\"2023-01-31T11:46:05.2319794Z\",\"eventRecordID\":\"17381521\",\"processID\":\"848\",\"threadID\":\"1532\",\"channel\":\"Security\",\"computer\":\"WP-SVM-DC01.winprovit.pt\",\"severityValue\":\"AUDIT_SUCCESS\",\"message\":\"\\\"An account was successfully logged on.\\r\\n\\r\\nSubject:\\r\\n\\tSecurity ID:\\t\\tS-1-0-0\\r\\n\\tAccount Name:\\t\\t-\\r\\n\\tAccount Domain:\\t\\t-\\r\\n\\tLogon ID:\\t\\t0x0\\r\\n\\r\\nLogon Information:\\r\\n\\tLogon Type:\\t\\t3\\r\\n\\tRestricted Admin Mode:\\t-\\r\\n\\tVirtual Account:\\t\\tNo\\r\\n\\tElevated Token:\\t\\tYes\\r\\n\\r\\nImpersonation Level:\\t\\tImpersonation\\r\\n\\r\\nNew Logon:\\r\\n\\tSecurity ID:\\t\\tS-1-5-21-4086784526-1345709382-551355621-5174\\r\\n\\tAccount Name:\\t\\twppaloalto\\r\\n\\tAccount Domain:\\t\\tWINPROVITAD\\r\\n\\tLogon ID:\\t\\t0x17F500744\\r\\n\\tLinked Logon ID:\\t\\t0x0\\r\\n\\tNetwork Account Name:\\t-\\r\\n\\tNetwork Account Domain:\\t-\\r\\n\\tLogon GUID:\\t\\t{00000000-0000-0000-0000-000000000000}\\r\\n\\r\\nProcess Information:\\r\\n\\tProcess ID:\\t\\t0x0\\r\\n\\tProcess Name:\\t\\t-\\r\\n\\r\\nNetwork Information:\\r\\n\\tWorkstation Name:\\tCACEM-FW1\\r\\n\\tSource Network Address:\\t10.20.95.254\\r\\n\\tSource Port:\\t\\t32950\\r\\n\\r\\nDetailed Authentication Information:\\r\\n\\tLogon Process:\\t\\tNtLmSsp \\r\\n\\tAuthentication Package:\\tNTLM\\r\\n\\tTransited Services:\\t-\\r\\n\\tPackage Name (NTLM only):\\tNTLM V2\\r\\n\\tKey Length:\\t\\t128\\r\\n\\r\\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\\r\\n\\r\\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\\r\\n\\r\\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\\r\\n\\r\\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\\r\\n\\r\\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\\r\\n\\r\\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\\r\\n\\r\\nThe authentication information fields provide detailed information about this specific logon request.\\r\\n\\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\\r\\n\\t- Transited services indicate which intermediate services have participated in this logon request.\\r\\n\\t- Package name indicates which sub-protocol was used among the NTLM protocols.\\r\\n\\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\\\"\"},\"eventdata\":{\"subjectUserSid\":\"S-1-0-0\",\"subjectLogonId\":\"0x0\",\"targetUserSid\":\"S-1-5-21-4086784526-1345709382-551355621-5174\",\"targetUserName\":\"wppaloalto\",\"targetDomainName\":\"WINPROVITAD\",\"targetLogonId\":\"0x17f500744\",\"logonType\":\"3\",\"logonProcessName\":\"NtLmSsp\",\"authenticationPackageName\":\"NTLM\",\"workstationName\":\"CACEM-FW1\",\"logonGuid\":\"{00000000-0000-0000-0000-000000000000}\",\"lmPackageName\":\"NTLM V2\",\"keyLength\":\"128\",\"processId\":\"0x0\",\"ipAddress\":\"10.20.95.254\",\"ipPort\":\"32950\",\"impersonationLevel\":\"%%1833\",\"virtualAccount\":\"%%1843\",\"targetLinkedLogonId\":\"0x0\",\"elevatedToken\":\"%%1842\"}}}","decoder":{"name":"windows_eventchannel"},"data":{"win":{"system":{"providerName":"Microsoft-Windows-Security-Auditing","providerGuid":"{54849625-5478-4994-a5ba-3e3b0328c30d}","eventID":"4624","version":"2","level":"0","task":"12544","opcode":"0","keywords":"0x8020000000000000","systemTime":"2023-01-31T11:46:05.2319794Z","eventRecordID":"17381521","processID":"848","threadID":"1532","channel":"Security","computer":"WP-SVM-DC01.winprovit.pt","severityValue":"AUDIT_SUCCESS","message":"\"An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\t-\r\n\tAccount Domain:\t\t-\r\n\tLogon ID:\t\t0x0\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t3\r\n\tRestricted Admin Mode:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tS-1-5-21-4086784526-1345709382-551355621-5174\r\n\tAccount Name:\t\twppaloalto\r\n\tAccount Domain:\t\tWINPROVITAD\r\n\tLogon ID:\t\t0x17F500744\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x0\r\n\tProcess Name:\t\t-\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\tCACEM-FW1\r\n\tSource Network Address:\t10.20.95.254\r\n\tSource Port:\t\t32950\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tNtLmSsp \r\n\tAuthentication Package:\tNTLM\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\tNTLM V2\r\n\tKey Length:\t\t128\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.\""},"eventdata":{"subjectUserSid":"S-1-0-0","subjectLogonId":"0x0","targetUserSid":"S-1-5-21-4086784526-1345709382-551355621-5174","targetUserName":"wppaloalto","targetDomainName":"WINPROVITAD","targetLogonId":"0x17f500744","logonType":"3","logonProcessName":"NtLmSsp","authenticationPackageName":"NTLM","workstationName":"CACEM-FW1","logonGuid":"{00000000-0000-0000-0000-000000000000}","lmPackageName":"NTLM V2","keyLength":"128","processId":"0x0","ipAddress":"10.20.95.254","ipPort":"32950","impersonationLevel":"%%1833","virtualAccount":"%%1843","targetLinkedLogonId":"0x0","elevatedToken":"%%1842"}}},"location":"EventChannel"}
rule
level:6, description:Successful Remote Logon Detected - NTLM authentication, possible pass-the-hash attack., id:92652, mitre:{id=[T1550.002], tactic=[Defense Evasion, Lateral Movement], technique=[Pass the Hash]}, firedtimes:3636, mail:false, groups:[win_evt_channel, authentication_success], gdpr:[IV_32.2], gpg13:[7.1, 7.2], hipaa:[164.312.b], nist_800_53:[AC.7, AU.14], pci_dss:[10.2.5], tsc:[CC6.8, CC7.2, CC7.3]
source
172.16.20.11
timestamp
2023-01-31 11:46:08.122
true
1675165566.3113

Not sure about what to do with this…

Hello @joseraeiro

Well I see your getting Windows Security Events, If this is MS have you tried using GELF UDP input? and can I ask why your trying to use JSON extractor? Is this just windows events your shipping to Graylog?

From what I can see/understand It look like its replacing the Whitespaces with an underscore, have you tried adjusting this section?

I’m ingesting logs directly from Wazuh, coming from several different sources, including, Windows Servers, Linux Servers, Firewalls, Switches, etc.

I’m using a JSON extractor because it is the format in which the beat that I have shipping logs from Wazuh to Graylog uses.

And, yes, I believe I’ve tried adjusting that section to contain a full stop but nevertheless _ was outputted as shown above.

hey,

Oh I see all those devices are in the same stream of logs.

The reason I ask this was, this issue was something I had to go through a couple years ago. What I ended up doing was separating my device to a corresponding INPUT that was attached to a different index. What I found out was my Switches were sending data in RAW/PLAIN text and was not playing nice with my GELF input for Windows devices.

So basically, I have a INPUT for Windows, Firewall/security device. Then with a stream I route them to a different indices.

Since I do not make my index templates (i.e., they are Dynamic by default) it was easier to use extractors and/or pipelines to correct/modify logs. I’m not saying this maybe the case here, but it sounds very familiar.

I’m not sure where you are trying to get at to here. What do you mean by not making your index templates and being Dynamic by default?

Sorry but I’m new to graylog.

Hey,

So what happens is, you ingest logs/messages into 1 index. Let’s say Default index set.

Once those fields are mapped for example creates the data index, the _doc mapping type, and a field called count with data type long.

When you have another device sending logs/messages to the same index, the fields are mapped for example creates the data index, the _doc mapping type, and a field called count with data type string then there is a conflict for that field count.

Sometime this is a issue sometime its not.

So by defualt Elasticseach/Opensearch the "Default " index is dynamic mapping. On the other hand you have Static/Explicit mapping’s.

Unfortunately, using several inputs for different devices is not an option, since all logs must be processed by the Wazuh Manager (for it to apply its rule logic and raise the alerts).

Given that fact, what could I do to troubleshoot this?

I’m trying to use Pipelines to try and change the _ with a .

Is this a valid approach?

Hey,

Actually, yes. If all the messages/logs are coming into one INPUT you could separate those.
I had all my log comeing one input, I wanted to separate my Linux Device from Windows device. What I did was create a new stream and Index set for linux. So I routed my logs by source using the Stream Rule/s then attaching that stream to my perfered Index set.

A pipline could do the same.

Rule "route_to_stream"
when
    has_field("source") AND contains(to_string($message.source), "192.168.1.100")
then
    route_to_stream(id:"5efd23ee5a8ef67a8276dc39");
end
``

rule “replace_underscore”
when
has_field(“field_with_underscores”)
then
set_field(“field_with_underscores”, replace(“field_with_underscores”, “_”, “.”)); ~
end

Was as far as I can get at the moment. Any help would be appreciated.

I caught in my attention this solution:

Introducing Field Aliases in Elasticsearch | Elastic Blog

Do you think it would be appropriate to deal with this issue?

Thank you so very much for all your kind help!

Hey @joseraeiro

Short answer is yes. I get by with pipelines but @tmacgbay is very good at it. I normally refer to his expertise on this issue.

Not 100% sure but I believe a regex inside this pipeline maybe will do the trick.
In the forum I found something like this.

rule "R-DeviceID Regex Replacement"

when
    has_field("R-DeviceID")
then
    let N_DeviceID = regex_replace("(\\d+)\\/(\\d+)\\/(\\d+)\\/(\\d+)\\:(\\d+)\\.(\\d+)\\.(\\d+)",to_string($message."R-DeviceID"),"$3-$4-$5-$6-$7",true);
    set_field("R-DeviceID", N_DeviceID);
end

Graylog has two that i see.

  • regex_replace(pattern, value, replacement, [replace_all]) : String

  • replace(value, search, [replacement], [max]) : String

NOTE: I found this example.

Sorry this is out of my knowledge on modifing field data with Pipeline. honestly, I fix before it hits graylog and some instances I used a extractor.

hey just cuaght this

If you modify the Index template using PUT it may only hold till the index Rotation period, because of dynamic mapping.

Thanks for all your kind help @gsmith, it’s greatly appreciated!

1 Like

This has been around for a long time: Dots in field names are replaced silently · Issue #13043 · Graylog2/graylog2-server · GitHub
There is no easy fix that will make everybody happy. There is a good discussion in the issue comments which might be helpful to you.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.