Cannot see logs on graylog web ui

cghera · June 7, 2018, 7:16am

Hello,
I am trying to setup a graylog with collector-sidecar and filebeats for a weblogic server.

I have the weblogic server and the graylog server which I installed and configured based on the getting started guide (collector-sidecar, input, output etc)
However even though collector sidecar seems to start normally and my log is updated I see nothing on the graylog server part.
Shouldnt I see something on the sources tab of the graylog web interface?
On other question I have is what is the bind address. Should I have the address of the graylog server there or the address of the server that produces the log that I want to read? I tried both the suggestions 127.0.0.1 and 0.0.0.0 as well as the ip addresses of the two servers there but nothing appears to come on the graylog server.

I am not so experienced on these configurations. I can send you any file needed , screenshot etc.
Thank you

jochen · June 7, 2018, 8:27am

What’s the complete and unredacted configuration of all relevant components?

What’s in the logs of all relevant components, including the Graylog Collector Sidecar and Filebeat?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

It’s the local network socket on which Graylog will listen for incoming network traffic.
0.0.0.0 is the wildcard, meaning Graylog will listen on all network interfaces.

cghera · June 7, 2018, 8:47am

here are the last lines of the server.log

2018-06-04T15:38:49.848+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now RUNNING
2018-06-05T11:09:31.846+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now STOPPING
2018-06-05T11:09:31.849+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now STOPPED
2018-06-05T11:09:31.849+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now TERMINATED
2018-06-05T11:09:31.849+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now STARTING
2018-06-05T11:09:31.851+03:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=beats-input, type=org.graylog.plugins.beats.BeatsInput, nodeId=0c660a92-e96c-401a-b970-48dade2e736e} should be 1048576 but is 212992.
2018-06-05T11:09:31.853+03:00 INFO  [InputStateListener] Input [Beats/5b14e72c63cecd1f8e9a6e46] is now RUNNING

I will upload some screenshot also

also the collector-sidecar.yml on the weblogic server
is this

collector-id  collector_sidecar.yml  generated
[wlsuser@mtndevapp collector-sidecar]$ more collector_sidecar.yml
server_url: http://10.240.36.171:9000/api/
update_interval: 10
tls_skip_verify: false
send_status: true
list_log_files:
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - weblogic
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

and the filebeat.yml on the generated folder

filebeat:
  prospectors:
  - encoding: plain
    exclude_files: []
    fields:
      collector_node_id: graylog-collector-sidecar
      gl2_source_collector: 23ff088a-efc3-48b7-8834-0124d3bc4eba
      type: log
    ignore_older: 0
    paths:
    - /u01/app/oracle/product/fmw11g/user_projects/domains/mtndev/servers/SOA_server1/logs/
      SOA_server1.log
    scan_frequency: 10s
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - 10.240.36.171:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- weblogic

cghera · June 7, 2018, 8:47am

cghera · June 7, 2018, 8:47am

cghera · June 7, 2018, 8:47am

jochen · June 7, 2018, 8:52am

Please post the complete logs. Use a paste bin service such as https://gist.github.com or https://0bin.net to upload large log files and share the link.

Does Filebeat have appropriate permissions to read that file?
What’s the output of the following command?

# namei -l /u01/app/oracle/product/fmw11g/user_projects/domains/mtndev/servers/SOA_server1/logs/ SOA_server1.log

Also, please provide the logs of the Graylog Collector Sidecar and Filebeat.

cghera · June 7, 2018, 8:59am

the namei command returns this

namei -l /u01/app/oracle/product/fmw11g/user_projects/domains/mtndev/servers/SOA_server1/logs/SOA_server1.log
f: /u01/app/oracle/product/fmw11g/user_projects/domains/mtndev/servers/SOA_server1/logs/SOA_server1.log
dr-xr-xr-x root    root     /
drwxrwxr-x wlsuser oinstall u01
drwxrwxr-x wlsuser oinstall app
drwxrwxr-x wlsuser oinstall oracle
drwxrwxr-x wlsuser oinstall product
drwxr-x--- wlsuser oinstall fmw11g
drwxr-x--- wlsuser oinstall user_projects
drwxr-x--- wlsuser oinstall domains
drwxr-x--- wlsuser oinstall mtndev
drwxr----- wlsuser oinstall servers
drwxr-x--- wlsuser oinstall SOA_server1
drwxr-x--- wlsuser oinstall logs
-rw-r----- wlsuser oinstall SOA_server1.log

jochen · June 7, 2018, 9:01am

Unless Filebeat is running as “root”, it doesn’t have permissions to read the file.
Only “root” and “wlsuser” are able to read the file.

cghera · June 7, 2018, 9:08am

I ran ps axfo pid,euser,egroup,args | grep filebeats and returns this

12451 wlsuser  oinstall  |           \_ grep filebeats

so it seems that wlsuser is running filebeats but as you say wlsuser has permissions to read the file.
Isnt that correct?

jochen · June 7, 2018, 9:13am

No, that’s the grep process itself.

Hint: The Filebeat process is called filebeat.

cghera · June 7, 2018, 9:22am

# ps axfo pid,euser,egroup,args | grep filebeat
12541 wlsuser  oinstall  |           \_ grep filebeat
18527 root     root      \_ /usr/bin/filebeat -c /etc/graylog/collector-sidecar/generated/filebeat.yml

jochen · June 7, 2018, 9:27am

So now that you’ve established that Filebeat is running as “root”, you’ll have to check its logs.

cghera · June 7, 2018, 9:29am

OK thank you, you are very kind, as you understand I am not very experienced. I will try to find the filebeat log.
update:
I looked in the filebeat.yml and saw that logs are at /var/log/graylog/collector-sidecar
However the filebeat_stderr.log shows this which seems to be old records.
because the filebeat.yml exists in the path

filebeat2018/06/04 06:57:59.532474 beat.go:635: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/06/04 06:58:02.534182 beat.go:635: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/06/04 06:58:05.529910 beat.go:635: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/06/04 06:58:08.534623 beat.go:635: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
filebeat2018/06/04 07:03:53.576946 beat.go:635: CRIT Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory
Exiting: error loading config file: stat /etc/graylog/collector-sidecar/generated/filebeat.yml: no such file or directory

jochen · June 7, 2018, 10:18am

Is there any packet filter (firewall) on the machine running Graylog which might block network packets to port 5044/tcp?

You can also check with tcpdump or Wireshark whether any network packets arrive on the machine running Graylog at all. See https://hackertarget.com/tcpdump-examples/ for some examples.

Also, what’s in the logs of your Graylog and Elasticsearch nodes?

cghera · June 7, 2018, 1:13pm

Thank you, I will try to check what you suggest. I tried telnet from the weblogic server to the graylog server on the port 5044 and it opened. I dont know if this proves anything.
I will come back with the logs you ask for when I get how tcpdump works, and locate the rest of the logs.

cghera · June 8, 2018, 7:10am

Hello,
here is the elasticsearch log (i got it from the graylog server)

[2018-06-01 16:16:33,862][INFO ][node                     ] [Alchemy] version[2.4.0], pid[5720], build[ce9f0c7/2016-08-29T09:14:17Z]
[2018-06-01 16:16:33,863][INFO ][node                     ] [Alchemy] initializing ...
[2018-06-01 16:16:34,294][INFO ][plugins                  ] [Alchemy] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2018-06-01 16:16:34,311][INFO ][env                      ] [Alchemy] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [27.8gb],
 net total_space [34.9gb], spins? [unknown], types [rootfs]
[2018-06-01 16:16:34,311][INFO ][env                      ] [Alchemy] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-06-01 16:16:35,797][INFO ][node                     ] [Alchemy] initialized
[2018-06-01 16:16:35,797][INFO ][node                     ] [Alchemy] starting ...
[2018-06-01 16:16:35,865][INFO ][transport                ] [Alchemy] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.
0.1:9300}
[2018-06-01 16:16:35,869][INFO ][discovery                ] [Alchemy] graylog/vUNWf5JnQISMcoX9GcPO7w
[2018-06-01 16:16:38,890][INFO ][cluster.service          ] [Alchemy] new_master {Alchemy}{vUNWf5JnQISMcoX9GcPO7w}{127.0.0.1}{127.0.0.1:9300}
, reason: zen-disco-join(elected_as_master, [0] joins received)
[2018-06-01 16:16:38,899][INFO ][http                     ] [Alchemy] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.
0.1:9200}
[2018-06-01 16:16:38,899][INFO ][node                     ] [Alchemy] started
[2018-06-01 16:16:38,918][INFO ][gateway                  ] [Alchemy] recovered [0] indices into cluster_state
[2018-06-01 16:26:35,173][INFO ][cluster.metadata         ] [Alchemy] [graylog_0] creating index, cause [api], templates [graylog-internal],
shards [4]/[0], mappings [message]
[2018-06-01 16:26:35,438][INFO ][cluster.routing.allocation] [Alchemy] Cluster health status changed from [RED] to [GREEN] (reason: [shards s
tarted [[graylog_0][1], [graylog_0][0], [graylog_0][2], [graylog_0][1], [graylog_0][2], [graylog_0][0]] ...]).

cghera · June 8, 2018, 7:14am

here is the graylog server log
https://0bin.net/paste/92Y2QbaEbyV-08ih#QP1tgwYKFlkPDQYvZF2ywb9u54KM7xezeOsb8lLY3Lm

jochen · June 8, 2018, 7:45am

2018-06-07T14:21:11.817+03:00 ERROR [NettyTransport] Error in Input [Beats/5b14e72c63cecd1f8e9a6e46] (channel [id: 0x17c872f3, /10.240.36.166:40238 :> /10.240.36.171:5044])
java.lang.Exception: Unknown beats protocol version: -12

Are you sure that there’s no proxy or load-balancer between your Graylog node and the Filebeat client which mangles the network traffic?

cghera · June 8, 2018, 8:12am

OK , you catch me uninformed once more. My current knowledge is that there is no proxy. However I will have to ask and come back because my only info is the two servers and the credentials to do the configurations. I did the following as you suggested
sudo tcpdump -i ens32 -nn -s0 -v port 5044
tcpdump: listening on ens32, link-type EN10MB (Ethernet), capture size 262144 bytes

but after about an hour that I stopped tcpdump I got this.

0 packets captured
0 packets received by filter
0 packets dropped by kernel

I really appreciate your help, and I am very grateful.
I will come back.

Topic		Replies	Views
Graylog 3.2 sidecar collector/filebeat errors Graylog Central (peer support) sidecar	5	697	June 3, 2020
Graylog collector dont send any data Graylog Central (peer support) sidecar , nxlog , filebeat-linux , nodatanx , nosendlogfblx	1	817	November 13, 2018
Graylog setup not showing logs Graylog Central (peer support) sidecar , filebeat-linux , elastic	23	2053	December 21, 2022
How to use graylog to view logs from multiple machines [Mainly Windows] Graylog Central (peer support) sidecar , filebeat-windows , winlogbeat	2	1402	October 30, 2020
No Collectors to display Graylog Central (peer support) sidecar , nxlog , filebeat-windows , winlogbeat , nodatanx	24	6048	July 11, 2019

Cannot see logs on graylog web ui

Related topics