Graylog setup not showing logs

[parth-hb (Parth)]

Incident:

I am a new user to graylog open. I wanted to setup graylog to read server logs but I believe some link is missing in the setup. If anyone can help me setup the missing link.
Steps I followed till now:

1. Installed elasticsearch as per installation instructions.
2. Installed mongodb
3. Installed graylog
4. Installed graylog-sidecar
5. Installed filebeat from elasticsearch
6. Configured filebeat config
7. Created token for sidecar via graylog web UI and configured it in sidecar configuration via command line
8. Added filebeat collector with sidecar via graylog web UI
9. Updated default filebeat output to point to elasticsearch ip/port
10. Added input via web ui at system > inputs.
    After all this setup, all things are up and running but no logs/messages are seen.

Pls help me with the missing pieces to step.

OS Information:
My server is a centos based server.

[m_mlk]

Hi @parth-hb

can you please post your config files?

TIA

[parth-hb (Parth)]

hi @m_mlk
PFA the configs:

input config via web ui

bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 6
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password:********

sidecar config via web UI

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
    
  paths:
    - /opt/tomcat/latest/logs
  type: log
  enabled: true
output.elasticsearch:
    hosts: ["localhost:9200"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

filebeat.yml (minimised) via command line

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    enabled: true
    paths:
      - /opt/tomcat/latest/logs/hb_*.log
filebeat.config.modules:
  path: '${path.config}/modules.d/*.yml'
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana: null
output.elasticsearch:
  hosts:
    - 'localhost:9200'
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: null
  - add_docker_metadata: null
  - add_kubernetes_metadata: null

elasticsearch.yml (minimised) via command line

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: graylog
action.auto_create_index: false

Pls specify if any other config is needed.

PS: I am using tomcat for my application so the log I want to read are generated at /opt/tomcat/latest/logs/ folder which I have updated in the above config.

[gsmith]

Hey @parth-hb

Just chiming in for sec, Graylog-sidecar is a wrapper for Winlogbeat, FileBeat, etc… you should not need to install FileBeat and the configurations for FileBeat would be done through the Web UI. Only thing you need to do is install graylog-sidecar , edit the sidecar.yaml file to connect to Graylog server and edit the FileBeat configuration on the Web UI.

• Graylog Sidecar

[parth-hb (Parth)]

Hi @gsmith
Initially, I did what you suggested.
I installed graylog-sidecar and tried configuring the sidecar and filebeat. But I could not find filebeat installed. And I came across few article/blog/SO which suggested to install filebeat expliciltly as filebeat is no longer bundled with sidecar. I felt weird but then I did what I found on those articles.

Nonetheless, what do you suggest should I do now? Bcs at this point, I am clueless on how to proceed further.

[m_mlk]

Hello again!

I have some questions for you

a) if your clients are UNIX/Linux systems, why don’t you simply use rsyslog? it works out of the box, it’s pretty straightforward to configure and you don’t need to fiddle around with cumbersome filebeat options…

b) do you have a Graylog server ingesting logs from external clients/remove systems? In this sense, can you please describe your setup?

TIA

[parth-hb (Parth)]

Hi @m_mlk

1. The current scene is we are using tomcat based application server and we dont have any centralised logging system. We only log to local log files. So everytime a issue comes up, the team needs to access the log files to debug issues and the access to log files are with very limited stakeholders. So the larger actual usecase for setting up GrayLog is to ingest our tomcat based application logs to Graylog so that the developers and testers dont need to fiddle thru the log files to debug issues.
   So this setup is a POC and I am trying to make this work using the systemsLog or any logs available on server.
   Once this setup works, we will be then pushing our development and staging server logs to this particular instance (both servers are on different machines).
   Once we are good with the dev and staging server logs, we will also start pushing prod logs.
   So rsyslog is not our target.

2. The graylog-sidecar is currently trying to ingest logs from the same machine using filebeat. No other input or external system is configure yet to send data to this instance of graylog. Once this primary testing of ingesting local logs works, will start ingesting logs from external system.

I hope above points make it clear what we are trying to setup.

[m_mlk]

Hello @parth-hb

Thanks for the description.

When comparing the configuration shown here Ingest from files - Log sources, the relevant part looks like this:

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log

but yours looks like this:

filebeat.inputs:
- input_type: log
    
  paths:
    - /opt/tomcat/latest/logs
  type: log

What if you simply change your config to read:

  paths:
    - /opt/tomcat/latest/logs/*.log

and restart the service?

HTH

[parth-hb (Parth)]

Hello @m_mlk
I tried the changes in the log path previous, and I re-tried updating the log paths after your reply but it still does not work.
My sidecar show the files correctly as it was showing previously by I dont find any messages in the main search screen.

[gsmith]

Hello

@parth-hb

Nonetheless, what do you suggest should I do now? Bcs at this point, I am clueless on how to proceed further.

I would look back over this documentation here. When your stuck or have issues its better to get advice here or support instead of a third party documentation.

Steps:
Install graylog-sidecar
Configure sidecar.yaml file

Example:

root@keycloak:/home/greg# cat  /etc/graylog/sidecar/sidecar.yml  | egrep -v "^\s*(#|$)"
server_url: "http://8.8.8.8:9000/api/"
server_api_token: "1sdjiu5d6bjp2u44qrbvtjc74r2b79epb8us9n4vlp2bugrsr885"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "keycloak101"
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
    - "/var/log/"
root@keycloak:/home/greg#

NOTE For the server_api_token: you get that from Users & Teams section.

Graylog-sidecar is done.

The rest should be self-explanatory in the documents,

Click on configuration copy or modify FileBeat template and attach it to the host.

Don’t forget to check the logs.

[parth-hb (Parth)]

Hi @gsmith
I have already referred to the documentation link you shared.
And the necessary changes are also done to the sidecar.yml file.
Required filebeat config is also configured as per the docs.
Token for sidecar config is also generated as per the steps mentioned in the docs.

Even after setting up all the config, and things didn’t worked, I had to referred to different blogs and third party articles.

Sidecar.yml

[root@dev_server parth]# cat  /etc/graylog/sidecar/sidecar.yml  | egrep -v "^\s*(#|$)"
server_url: "http://127.0.0.1:9000/api/"
server_api_token: "some_generated_token"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "dev-server"
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: "/opt/tomcat/latest/logs"
log_path: "/var/log/graylog-sidecar"

This is the output of my sidecar.yml

PFA output of sidecar.log from /var/log/graylog-sidecar/sidecar.log

time="2022-12-02T19:18:55+05:30" level=info msg="Starting signal distributor"
time="2022-12-02T19:19:05+05:30" level=info msg="Adding process runner for: filebeat"
time="2022-12-02T19:19:05+05:30" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time="2022-12-02T19:19:05+05:30" level=info msg="[filebeat] Starting (exec driver)"
time="2022-12-05T19:25:04+05:30" level=info msg="Stopping signal distributor"
time="2022-12-05T19:25:04+05:30" level=info msg="[filebeat] Stopping"
time="2022-12-05T19:25:16+05:30" level=info msg="Starting signal distributor"
time="2022-12-05T19:25:26+05:30" level=info msg="Adding process runner for: filebeat"
time="2022-12-05T19:25:26+05:30" level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time="2022-12-05T19:25:27+05:30" level=info msg="[filebeat] Starting (exec driver)"

/var/log/graylog-sidecar/filebeat_stderr.log file is empty and no error is recorded here.

[gsmith]

Hello @parth-hb

@parth-hb

Even after setting up all the config, and things didn’t worked, I had to referred to different blogs and third party articles

That’s unfortunate and I understand.

On the dashboard you should be able to see the sidecar status Green/Running at this point, am I correct?
If this is correct then you should see the following.

Example:

Overview Section

Administration Section

Configuration Section

NOTE: This section is where you create the template needed for attaching it to the sidecar.

I also see your using Localhost in the FileBeat configuration and using 127.0.0.1 in the sidecar configuration. You may want to adjust the FileBeat configurations to use the loopback address.

On Graylog Web UI you should have something like this.

@parth-hb

PFA output of sidecar.log from /var/log/graylog-sidecar/sidecar.log

I see the the FileBeat service has started. So perhaps check Firewall and/or Input used.

[parth-hb (Parth)]

[Message 1/2]
Hi @gsmith
All items suggested by you are working fine.
And I also updated the filebeat config with 127.0.0.1 instead of localhost as per suggestion.

Due to forum restrictions,
unable to attach multiple screenshots so attaching links to the same and spliting this message in 2 parts

Streams Config:

Inputs stream config

Sidecar config running green

continued in next message…

[parth-hb (Parth)]

[Message 2/2]

List of file in sidecar details (0 in/ 0 out also visible)

Filebeat attached to sidecar

Filebeat Config (updated with loopback address)

Pls check if any other discrepancies in this configs.

[Wine_Merchant]

Should it be using the logstash output on 127.0.0.1:5044 as opposed to Elastic output on 9200?

[parth-hb (Parth)]

Hi @Wine_Merchant
The default was config was for logstash but I haven’t setup logstash.
And either way, even with the default logstash config initially, it did not worked.
And then I felt it should be updated from logstash to ES so I updated it to ES.

So I need to setup logstash as well?
Haven’t come across any instruction to setup up logstash so no idea on that.

[Wine_Merchant]

Hey Parth,

Perhaps I’ve misunderstood here but in the current setup of your filebeat you are attempting to send logs to 127.0.0.1 on port 9200. I can see you have an input listening on port 5044 on Graylog which I assume you want logs to arrive on, filebeat should be pointed at this port in order to deliver logs.

[parth-hb (Parth)]

Hey @Wine_Merchant
I had previously tried to update the inputs with port no to 9200 but that input failed to start. Let me change the port in filebeat and check if that works.

[parth-hb (Parth)]

Hey @Wine_Merchant
I tried by changing the filebeat config to push log to 5400 but still all systems are ups without any positive reslults.
And ideally, bcs the ES is running at 9200, i believe 9200 should be the actual value.

So the issue still persist.

[Wine_Merchant]

So you want to deliver these logs straight to Elasticsearch and not ingest via Graylog?