I am trying to upload a log file which has json format using nc. I am interested in reading the timestamp from the log file rather than the time stamp of the upload itself. I even created an extractor as mentioned in the docs, but I still get the timestamp of the upload. Please find an example.
“Timestamp” and “timestamp” are two different fields. Graylog is using the lower case “timestamp” field as canonical message timestamp.
Can you please provide an example to overwrite the graylog Timestamp with my Timestamp from the uploaded logs?
Please refer to the documentation of the set_field() pipeline function.
1 Like
Without a representative event or two from your logfile it’s difficult to provide an exact example. But generally speaking here’s what you’re going to do:
- Create an extractor for the input that ingests your logfile
- Use a regex, a GROK expression, etc. to digest the portion of your input (often the message field) that contains the timestamp
- Store the parsed-out data as field timestamp (think of this as overwriting the Graylog-generated timestamp).
- Add a converter if necessary to format the extracted string in the correct manner.
FWIW I’m pretty new to Graylog as well and found the documentation for this kind of thing pretty straightforward. And I see that jochen just replied advising you check out the pipeline functionality, which long-term is definitely the “right” advice as I believe extractors are kind of the past and pipeline is the future.
Hey Matt,
I tried both methods, still not working and I am not able to figure out what I am doing wrong!
The json file is getting parsed correctly, but not able to overwrite the timestamp. Any hint would be much appreciated.
I was able to get the pipeline to work as I checked it through simulation processing
still not seeing any changes when I search for 30 days
FWIW I don’t think you’d want to cut (in that first example), you’d want to copy into “timestamp” (low case). That should overwrite the Graylog-provided timestamp with the one you parsed out. Also, don’t forget the time zone or your entries may end up in the future/past.
I can’t speak to the pipeline example as I’ve not gone there yet. But do remember that the extractor (or pipeline I’d guess) is not able to go back to info that’s already been indexed. You should narrow your search criteria to the period after you make changes to the extractor/pipeline to see if it’s now doing what you want. You may even have to force the current index to roll.
Thanks a lot Matt! It’s now working using pipeline
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.