Ingesting Log Files from JSON into graylog

Graylog Central (peer support)
1

Hi, Super Noob here so please pardon me if i dont get things on the fly. I have successfully installed Graylog Elasticsearch and Mongo as stated on the official download instruction. I also have filebeat installed and logstash.
I am trying to parse/read-in contents of this log file / json path into graylog. First Issue : When I use json input on graylog I am able to pull in ONLY the latest event entry or log. The previous entries and subsequent incoming ones are not being updated in live mode on graylog. So my question on this is … does graylog have to redownload the whole log file everytime is refreshes or does it know to pull only the recent log…

Secondly , I have download the contents of the log and curl it to output.txt file. Please I am serious difficulty is parsing the .txt file into graylog for analysis. Please can anyone help me with a solution to any of these two issues ?

I have done extensive research but due to my limit in linux language I am stuck . I tried apache - How to import old log files to graylog as input? - Stack Overflow
it didnt work. i get a bunch of errors about pipline config
I am running Graylog service on Linux Distro CentOs 7

2

Graylog does not download files. It just processes log messages that get sent to it.
Maybe this topic can be helpful for ingesting old entries.
If filebeat/logstash are not forwarding the incoming log messages, then something is misconfigured.

3

Thank you for the response.
I basically followed the instructions and i cant see anything coming into my graylog input

Please see my attached .

Screenshot from 2022-08-16 12-21-49
Screenshot from 2022-08-16 12-21-491876×469 59.1 KB

4

You need to try and narrow down the problem so we can help you effectively.
First determine if filebeats is working. Here are some tips.

Why do you have a TLS key file and password in your Beats input definition? You have TLS disabled. I would remove those.

5

Thank you again for the help.

I followed your link and the first command ‘filebeat -e’ threw an error Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'/etc/filebeat/filebeat.yml')

Screenshot from 2022-08-16 14-47-52
Screenshot from 2022-08-16 14-47-52890×278 21.6 KB

I ran this to read in from txt line by line and i get a connection peer closed error

while read x; do echo "$x" | nc 127.0.0.1 5554; done < output.txt
Ncat: Connection reset by peer.
6

netcat would need to send to a raw input, not to the beats input, since you said this was just plain JSON.

Sounds like you somehow have 2 output namespaces configured. Look at your filebeat.yml again.

7

Okay so basically i need to create a new input on graylog with Raw/Plainttext UDP parameters. Got it . As for the config …i cant tell where the error is or how I have 2 output namespaces

Screenshot from 2022-08-16 15-35-08
Screenshot from 2022-08-16 15-35-08892×779 57.2 KB

8

Following the example from this link Using FileBeat with GrayLog - Another Cyber Sec Blog
I noticed that theres been some traffic under the network section of the input. So should i assume that this worked to a certain degree? i cant see any log messages coming in but then again the log file i am trying to parse is a large one i believe. I did a curl to .txt from the GET link and the txt file is about 60 MB … will this also be an issue ?

9

Remove the elasticsearch output

10

There are actually quite a few threads in the forum already on how to install and troubleshoot beats input.
Here are some things to help troubleshoot:

1 Like
11

Okay will do that now

12

Thank you for this … ill definitely utilize for troubleshooting

13

So i removed elasticsearch output and ran filebeat -e

sudo filebeat -e
2022-08-17T11:35:01.060+0100	INFO	instance/beat.go:685	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-08-17T11:35:01.073+0100	INFO	instance/beat.go:693	Beat ID: 7b0c7041-82a1-4f51-9746-36623e50911c
2022-08-17T11:35:04.079+0100	WARN	[add_cloud_metadata]	add_cloud_metadata/provider_aws_ec2.go:79	read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-08-17T11:35:04.084+0100	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2022-08-17T11:35:04.084+0100	INFO	[beat]	instance/beat.go:1039	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "7b0c7041-82a1-4f51-9746-36623e50911c"}}}
2022-08-17T11:35:04.084+0100	INFO	[beat]	instance/beat.go:1048	Build info	{"system_info": {"build": {"commit": "05f73d937c955ef81eccacc460a161cf2e6ac123", "libbeat": "7.17.5", "time": "2022-06-23T22:06:51.000Z", "version": "7.17.5"}}}
2022-08-17T11:35:04.084+0100	INFO	[beat]	instance/beat.go:1051	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.2"}}}
2022-08-17T11:35:04.087+0100	INFO	[beat]	instance/beat.go:1055	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-08-12T10:48:10+01:00","containerized":false,"name":"firs-hq-gov-l","ip":["127.0.0.1/8","::1/128","10.2.1.110/24","fe80::a3c6:9f2:c693:925d/64","192.168.122.1/24"],"kernel_version":"3.10.0-1160.71.1.el7.x86_64","mac":["00:15:5d:0a:3b:3c","52:54:00:31:f1:1a","52:54:00:31:f1:1a"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"WAT","timezone_offset_sec":3600,"id":"6bf480c4f328457ba25b2d8abcf74139"}}}
2022-08-17T11:35:04.087+0100	INFO	[beat]	instance/beat.go:1084	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/home/mamman.muhammad/Downloads", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 127880, "ppid": 127877, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-08-17T11:35:00.450+0100"}}}
2022-08-17T11:35:04.087+0100	INFO	instance/beat.go:328	Setup Beat: filebeat; Version: 7.17.5
2022-08-17T11:35:04.088+0100	INFO	[publisher]	pipeline/module.go:113	Beat name: firs-hq-gov-l
2022-08-17T11:35:04.111+0100	WARN	beater/filebeat.go:202	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-08-17T11:35:04.111+0100	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
2022-08-17T11:35:04.112+0100	INFO	instance/beat.go:492	filebeat start running.
2022-08-17T11:35:04.124+0100	INFO	memlog/store.go:119	Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-08-17T11:35:04.136+0100	INFO	memlog/store.go:124	Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=23
2022-08-17T11:35:04.136+0100	WARN	beater/filebeat.go:411	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-08-17T11:35:04.137+0100	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 1
2022-08-17T11:35:04.137+0100	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 1
2022-08-17T11:35:04.137+0100	INFO	[crawler]	beater/crawler.go:117	starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]
2022-08-17T11:35:04.137+0100	WARN	[cfgwarn]	log/input.go:89	DEPRECATED: Log input. Use Filestream input instead.
2022-08-17T11:35:04.137+0100	INFO	[input]	log/input.go:171	Configured paths: [/home/mamman.muhammad/Downloads/output.log]	{"input_id": "f4136c1e-3ade-41f0-9306-e884c6552973"}
2022-08-17T11:35:04.137+0100	INFO	[crawler]	beater/crawler.go:148	Starting input (ID: 15281460985787299541)
2022-08-17T11:35:04.138+0100	INFO	[input.harvester]	log/harvester.go:309	Harvester started for paths: [/home/mamman.muhammad/Downloads/output.log]	{"input_id": "f4136c1e-3ade-41f0-9306-e884c6552973", "source": "/home/mamman.muhammad/Downloads/output.log", "state_id": "native::25237-64770", "finished": false, "os_id": "25237-64770", "harvester_id": "043895a0-6497-4c1d-9064-96a30c2aad6c"}
2022-08-17T11:35:04.138+0100	INFO	[crawler]	beater/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2022-08-17T11:35:04.138+0100	INFO	cfgfile/reload.go:164	Config reloader started
2022-08-17T11:35:04.138+0100	INFO	cfgfile/reload.go:224	Loading of config files completed.
2022-08-17T11:35:05.153+0100	WARN	[reader_line]	readfile/line.go:183	Exceeded 41943040 max bytes in line limit, skipped 49359162 bytes line
2022-08-17T11:35:07.107+0100	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:101	add_cloud_metadata: hosting provider type not detected.
2022-08-17T11:35:34.139+0100	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":290,"time":{"ms":292}},"total":{"ticks":530,"time":{"ms":534},"value":530},"user":{"ticks":240,"time":{"ms":242}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9ab4f43b-3526-47db-a9fc-49e11a259357","uptime":{"ms":33125},"version":"7.17.5"},"memstats":{"gc_next":163088824,"memory_alloc":81569000,"memory_sys":192633896,"memory_total":229605736,"rss":172253184},"runtime":{"goroutines":32}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1},"queue":{"max_events":4096}}},"registrar":{"states":{"current":2,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":8.47,"15":7.59,"5":8.23,"norm":{"1":2.1175,"15":1.8975,"5":2.0575}}}}}}
2022-08-17T11:36:04.161+0100	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":300,"time":{"ms":18}},"total":{"ticks":540,"time":{"ms":26},"value":540},"user":{"ticks":240,"time":{"ms":8}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9ab4f43b-3526-47db-a9fc-49e11a259357","uptime":{"ms":63122},"version":"7.17.5"},"memstats":{"gc_next":163088824,"memory_alloc":82379312,"memory_total":230416048,"rss":172601344},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":7.52,"15":7.54,"5":8.02,"norm":{"1":1.88,"15":1.885,"5":2.005}}}}}}
2022-08-17T11:36:34.149+0100	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":310,"time":{"ms":9}},"total":{"ticks":560,"time":{"ms":15},"value":560},"user":{"ticks":250,"time":{"ms":6}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9ab4f43b-3526-47db-a9fc-49e11a259357","uptime":{"ms":93112},"version":"7.17.5"},"memstats":{"gc_next":163088824,"memory_alloc":83112336,"memory_total":231149072,"rss":172527616},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":8.15,"15":7.58,"5":8.1,"norm":{"1":2.0375,"15":1.895,"5":2.025}}}}}}
2022-08-17T11:37:04.153+0100	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":340,"time":{"ms":23}},"total":{"ticks":600,"time":{"ms":27},"value":600},"user":{"ticks":260,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9ab4f43b-3526-47db-a9fc-49e11a259357","uptime":{"ms":123150},"version":"7.17.5"},"memstats":{"gc_next":163088824,"memory_alloc":83856904,"memory_total":231893640,"rss":172093440},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":7.37,"15":7.53,"5":7.9,"norm":{"1":1.8425,"15":1.8825,"5":1.975}}}}}}
2022-08-17T11:37:34.122+0100	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":370,"time":{"ms":32}},"total":{"ticks":640,"time":{"ms":51},"value":640},"user":{"ticks":270,"time":{"ms":19}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9ab4f43b-3526-47db-a9fc-49e11a259357","uptime":{"ms":153111},"version":"7.17.5"},"memstats":{"gc_next":8335720,"memory_alloc":3878832,"memory_total":232708096,"rss":22614016},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}},"system":{"load":{"1":8.33,"15":7.59,"5":8.06,"norm":{"1":2.0825,"15":1.8975,"5":2.015}}}}}}

When i run while read x; do echo "$x" | nc 127.0.0.1 5554; done < output.txt
I still get connection refused by peer error. Im really stuck

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.