Graylog Cluster and Load Balancing

Graylog Central (peer support)
,
1

Hi everyone !

First of all, I would like to thank the whole Graylog team for creating and maintaining such a great product. Thanks also to all the people who take the time to come on this forum to propose solutions, it’s an excellent source of information ! And I hope that my questions will lead to answers that will in turn help other people as well !

I feel like I’m stuck / misunderstanding an important part about load balancing and I don’t feel like I’ve seen any solution online. Maybe there are, but I don’t feel like I’ve encountered “clear” answers to my questions. Or maybe I just don’t understand them !

I’ll start right here with my two questions :

  • How to make Graylog work with Apache2 as a load balancer
  • Should / Can I use a second load balancer for ingesting logs or should I use the same ?
  1. How to make Graylog work with Apache2 as a load balancer

So we’ve been playing around for few months with Graylog in minimum setup and decided to go with the Bigger Production Setup.

So far, everything works great in that bigger setup (everything Debian):

  • A cluster of three Elasticsearch nodes
  • A cluster of three MongoDB & Graylog nodes

I can go to the web interface of any Graylog nodes and check that everything is up and running.

I then went on and set up a server with just Apache as a load balancer with mod_proxy_balancer.

Here is the Apache load balancer configuration for now

<VirtualHost *:80>

        ServerName graylog.my.domain/

        Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

        ProxyPass "/" "balancer://graylog-cluster/"
        ProxyPassReverse "/" "balancer://graylog-cluster/"

        <Proxy "balancer://graylog-cluster">
                Order deny,allow
                Allow from all
                BalancerMember "gray01.my.domain/api" route=node1
                BalancerMember "gray02.my.domain/api" route=node2
                BalancerMember "gray03.my.domain/api" route=node3
                ProxySet stickysession=ROUTEID
        </Proxy>

</VirtualHost>

Works fine ! But from what I understand, load balancing is supposed to be transparent, but if I go to http://graylog.my.domain, I can see which node I am redirected to in the search bar…

And then I saw this page about load balancing web interfaces : since I’m new to the load balancing world, I do not really know what to do with that configuration file…
Should I make an Apache server running on each of the 3 Graylog nodes ? I tried, from what I understood using this how to, works fine but nothing changed.

  1. Should / Can I use a second load balancer for ingesting logs or should I use the same ?

We are using the following inputs :

  • ​Sidecars with Filebeat and Winlogbeat
  • Palo Alto
  • RAW UDP
  • Syslog

From what I read here, here and there :

  • If load balancing seems not to work that well with TCP, is it worth it to use one ?
  • Is it worth it to work with two load balancer : one for the web interface, another one for inputs ?
  • Should I go back and learn more about load balancing (I’m doing it anyway) ?

THANKS a LOT for reading this, I know it’s a long topic.
I’m a Sys Admin trainee and not a native English speaker, so I’m sorry if I’m saying anything stupid.

Have a great day ! :wave:

2

Hello && Welcome

There are a few ways to configure load balancers. First is just knowing what it does can be half the battle.

A simple explaination of Load balancers.

The purpose of the load balancer (in this case an HTTP load balancer) is to distribute all incoming requests to our backend web servers. The load balancer hides all our backend servers to the public, and from the outside it looks like a single server doing all the work.

Some firewalls can do this also.
Example:
https://docs.fortinet.com/document/fortigate/6.0.0/Handbook/154107/basic-load-balancing-configuration-example

The link below is just added info.

It would be a bad idea to have apache on the same nodes as Graylog/MongoDb. Personally, they should be on separate servers or another device.

As for

This might be an added configuration to apache config to hide those nodes it connects to, but I’m not 100% sure. I personally do not see mine. Not sure what search bar are you refering to. Could you explain or maybe a screen shot?

3

Hi there and thanks a lot for your answer !

Right, so that’s kind of what I thought : I don’t get the point of having an Apache server on top of Graylog on the same node.

Unfortunately I’m not sure that a screenshot would help to understand better…

Some architectural considerations (hope that helps) :

Load Balancer :

  • 192.168.20.10

Graylog Nodes :

  • 192.168.20.11
  • 192.168.20.12
  • 192.168.20.13

What I meant :

When I go to 192.168.20.10, I am sent back to one of the three nodes and that is exactly what I am looking for. However, from what I understand, I’m not supposed to know which node I’m on. Now in the browser search bar I can see which node I’m being redirected to because the address changes to alternately 192.168.20.12, or 192.168.20.11, etc.

Do you use Apache as a load balancer too ? Perhaps I should switch to HAProxy or something else.
And what about inputs ? I think about trying apache kafka on another machine.

Thanks again for your answer, really appreciate it.

4

So, it seems that I was doing things wrong because as always, I was trying to skip steps by mixing tutorials from the interweb without really knowing what I was doing. Rushing headlong into an unfamiliar technology was not as efficient as I thought. Who could have known ? :open_mouth:

Here is my new configuration that works just fine.

<VirtualHost *:80>

        ServerName graylog.my.domain
        ProxyRequests Off

        Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

        <Location /balancer-manager>
                SetHandler balancer-manager
                AuthType Basic
                AuthName "Load_Balancer_Manager"
                AuthBasicProvider file
                AuthUserFile "/path/to/passwords/files"
                Require user secretuser
        </location>

        ProxyPass /balancer-manager !

        <Proxy balancer://graylog>
                BalancerMember "http://node01.my.domain:9000" route=node1
                BalancerMember "http://node02.my.domain:9000" route=node2
                BalancerMember "http://node03.my.domain:9000" route=node3
                ProxySet lbmethod=byrequests
                ProxySet stickysession=ROUTEID
        </Proxy>

        RequestHeader set X-Graylog-Server-URL "http://graylog.my.domain"
        ProxyPass / balancer://graylog/
        ProxyPassReverse / balancer://graylog/

</VirtualHost>

I configured and secured the balancer manager by running htpasswd -c /path/to/passwords/files secretuser.

I’m going to go and dive more into load balancing inputs tomorrow.

Thanks again !

1 Like
5

New problem here : not sure if I should go on with this topic or open another one. Feel free to tell me if I’m doing this wrong, but I feel like it’s a Reverse Proxy / Load Balancing problem. I never had this kind of message on a Graylog-Server / MongoDB / Elasticsearch single instance.

I, from time to time, have got a 401 error from Sidecars with Filebeat.

Here are the typical type of logs from var/log/graylog-sidecar/sidecar.log (and I can also see them form Graylog Server UI).

[...]
time="2021-08-09T14:25:18+02:00" level=error msg="[RequestBackendList] Bad response status from Graylog server: 401 Unauthorized"
time="2021-08-09T14:25:18+02:00" level=error msg="Can't fetch collector list from Graylog API: GET http://graylog.my.domain/api/sidecar/collectors: 401 "
time="2021-08-09T14:25:28+02:00" level=error msg="[RequestBackendList] Bad response status from Graylog server: 401 Unauthorized"
time="2021-08-09T14:25:28+02:00" level=error msg="Can't fetch collector list from Graylog API: GET http://graylog.my.domain/api/sidecar/collectors: 401 "
time="2021-08-09T14:25:38+02:00" level=error msg="[RequestBackendList] Bad response status from Graylog server: 401 Unauthorized"
time="2021-08-09T14:25:38+02:00" level=error msg="Can't fetch collector list from Graylog API: GET http://graylog.my.domain/api/sidecar/collectors: 401 "
time="2021-08-09T14:25:48+02:00" level=error msg="[RequestBackendList] Bad response status from Graylog server: 401 Unauthorized"
time="2021-08-09T14:25:48+02:00" level=error msg="Can't fetch collector list from Graylog API: GET http://graylog.my.domain/api/sidecar/collectors: 401 "
time="2021-08-09T14:25:58+02:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 401 Unauthorized"
time="2021-08-09T14:25:58+02:00" level=error msg="Can't fetch configuration from Graylog API: GET http://graylog.my.domain/api/sidecar/configurations/render/15520d3d-b40e-4316-97ad-e7f200e61d40/61093806666bd536b0701b2g: 401 "
[...]

This happens after workdays, or at lunch. Feels like it happens some time after the UI is closed.
When I restart graylog.sidecar, it goes back up and running, till I get a 401 again.

My Graylog Cluster is behind an Apache Load balancer for the web interface whose configuration goes like this (as shown in a previous message) :

<VirtualHost *:80>

        ServerName graylog.my.domain
        ProxyRequests Off

        Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

        <Location /balancer-manager>
                SetHandler balancer-manager
                AuthType Basic
                AuthName "Load_Balancer_Manager"
                AuthBasicProvider file
                AuthUserFile "/path/to/passwords/files"
                Require user secretuser
        </location>

        ProxyPass /balancer-manager !

        <Proxy balancer://graylog>
                BalancerMember "http://node01.my.domain:9000" route=node1
                BalancerMember "http://node02.my.domain:9000" route=node2
                BalancerMember "http://node03.my.domain:9000" route=node3
                ProxySet lbmethod=byrequests
                ProxySet stickysession=ROUTEID
        </Proxy>

        RequestHeader set X-Graylog-Server-URL "http://graylog.my.domain"
        ProxyPass / balancer://graylog/
        ProxyPassReverse / balancer://graylog/

</VirtualHost>

And behind an Nginx LoadBalancer for inputs, whose configuration goes like this (inspired by Nginx Config Examples) :

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 1024;
}
stream {

  upstream graylog_beats {
    server 10.100.10.22:5044 max_fails=3 fail_timeout=30s;
    server 10.100.10.23:5044 max_fails=3 fail_timeout=30s;
    server 10.100.10.24:5044 max_fails=3 fail_timeout=30s;
  }

  server {
    listen 5044;
    proxy_pass graylog_beats;
    proxy_timeout 10s;
    error_log /var/log/nginx/graylog_beats.log;
  }

}

We only ingest Filebeats for now, but we will add more type of inputs soon.

The beat input, which is a global input is configured like this :

bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 1
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: true
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password:********

If anyone has any idea, it would be more than welcome.

Have a good day !

6

What does you Filebeat config look like?

Graylog sidecar error

EDIT:

HTTP 401 Unauthorized client error status response code indicates that the request has not been applied because it lacks valid authentication credentials for the target resource.

7

Thanks a lot for your answer !
I feel that this is not a very big issue…I’ts frustrating !

Here are the other configuration files you might need to understand what is happening.
I just read this post and double checked my configuration files : they seem to be ok.
What drives me bonkers is that this is working for an hour or two, and then, I get this 401 error again but the logs are still shipped and received !

Here is my sidecar.yml 

# The URL to the Graylog server API.
server_url: "http://graylog.my.domain/api/" # This is my Apache LoadBalancer for Graylog UI

# The API token to use to authenticate against the Graylog server API.
# This field is mandatory
server_api_token: "1o4l0vc8s7qdrm2uhk1[...]5dkps1dsaegtjc855f6jmjijafob"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:/etc/graylog/sidecar/node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
node_id: "file:/etc/graylog/sidecar/node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
#node_name: ""

# The update interval in seconds. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
#update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
#send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
list_log_files:
  - "/var/log/"

# Directory where the sidecar stores internal data.
#cache_path: "/var/cache/graylog-sidecar"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "/var/log/graylog-sidecar"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_whitelist:
#       - "/usr/bin/filebeat"
#       - "/opt/collectors/*"
#
# Example disable whitelisting:
#     collector_binaries_whitelist: []
#
# Default:
# collector_binaries_whitelist:
#  - "/usr/bin/filebeat"
#  - "/usr/bin/packetbeat"
#  - "/usr/bin/metricbeat"
#  - "/usr/bin/heartbeat"
#  - "/usr/bin/auditbeat"
#  - "/usr/bin/journalbeat"
#  - "/usr/share/filebeat/bin/filebeat"
#  - "/usr/share/packetbeat/bin/packetbeat"
#  - "/usr/share/metricbeat/bin/metricbeat"
#  - "/usr/share/heartbeat/bin/heartbeat"
#  - "/usr/share/auditbeat/bin/auditbeat"
#  - "/usr/share/journalbeat/bin/journalbeat"
#  - "/usr/bin/nxlog"
#  - "/opt/nxlog/bin/nxlog"

Here is my sidecar configuration on Graylog server's UI / collector_configuration_directory 

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log
output.logstash:
   hosts: ["graylog-inputs:5044"] # My Nginx LoadBalancer.
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

And here is my filebeat.yml 

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

# filestream is an input for collecting log messages from files. It is going to replace log input in the future.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"


# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

But from what I understand, filebeat.yml is rewritten by sidecar. So I didn’t even edit it. Do I get it wrong ?

EDIT: I guess yout were talking about /var/lib/graylog-sidecar/generated/filebeat.conf aka collector_configuration_directory !

Yes I know that… And since it seems to go up and down depending on whether someone is logged in or not, I feel that it my be something in my Apache loadbalancer.

Have a good day !

EDIT² : Looking at those configuration files, what is the difference between the log paths in those files ? Should they be the same ?

filebeat.conf

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log

and sidecar.yml

list_log_files:
  - "/var/log/"
8

Thank you for the added information.

You stated:

Then you stated

Do you have both nginx and apache working together?

I don’t much about at sidecar configuration. We use Ansible with Nxlog. I have been using FileBeat a little with Elasticsearch so I’m not sure what the GL sidecar does in the way of auto-configuring you log shipper files. Someone here might jump in to see if those configurations are correct.

States any file that is a .log it will try to access those files.

States anything it that directory it will try to access.

This could very well be. What I would do for troubleshooting this type of issue is check all your logs Elasticsearch, Apache, etc… If nothing else is found that would pertain to this issue (excluding your 401 error) my next step would be to go around your load balancer (If you can) and send logs straight to the master node. If you do not see these logs then it’s a positive that it’s you apache load balancer.

9

Yes : one for the web UI (Apache) and one for the inputs (Nginx).

I get it ! But my question was more about why do we configure two different lists in two configuration files : filebeat.conf and sidecar.yml. From what I understand now, what you edit in filebeat.conf is a of files that will be shipped into Graylog and what you edit ind sidecar.yml is a list of directories that you will see in Graylog UI > System > Sidecars > Sidecars status. They will be highlighted in blue when modified.

It seems like the 401s come at midnight and then stop whenever I log into the Graylog UI in the morning, around 09am.

image
image736×216 3.85 KB

Here are some relevant logs from the Apache Loadbalancer.

error.log.1 

[Mon Aug 16 14:04:15.943376 2021] [proxy_http:error] [pid 9324:tid 140699269838592] (104)Connection reset by peer: [client 10.100.11.21:37246] AH01102: error reading status line from remote server graylog03.my.domain:9000
[Mon Aug 16 14:04:15.943410 2021] [proxy:error] [pid 9324:tid 140699269838592] [client 10.100.11.21:37246] AH00898: Error reading from remote server returned by /api/sidecars/15520d3d-a40e-4316-97ad-e7f200e61d40
[Mon Aug 16 14:04:32.400841 2021] [proxy_http:error] [pid 9324:tid 140699390957312] (104)Connection reset by peer: [client 10.100.10.6:42706] AH01102: error reading status line from remote server graylog02.my.domain:9000
[Mon Aug 16 14:04:32.400868 2021] [proxy:error] [pid 9324:tid 140699390957312] [client 10.100.10.6:42706] AH00898: Error reading from remote server returned by /api/sidecars/1jq26cssvc6rj4qac4bt9oeeh0p4vt5u5kal9jocl1g9mdi4og3n
[Mon Aug 16 14:39:44.419806 2021] [proxy_http:error] [pid 9324:tid 140699219482368] (104)Connection reset by peer: [client 10.2.7.18:64401] AH01102: error reading status line from remote server graylog01.my.domain:9000, referer: http://graylog/search?q=&rangetype=relative&from=300
[Mon Aug 16 14:39:44.419841 2021] [proxy:error] [pid 9324:tid 140699219482368] [client 10.2.7.18:64401] AH00898: Error reading from remote server returned by /api/system/notifications, referer: http://graylog/search?q=&rangetype=relative&from=300
[Mon Aug 16 15:09:20.415570 2021] [proxy_http:error] [pid 9324:tid 140699152340736] (104)Connection reset by peer: [client 10.2.7.18:51777] AH01102: error reading status line from remote server graylog01.my.domain:9000, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:09:20.415601 2021] [proxy:error] [pid 9324:tid 140699152340736] [client 10.2.7.18:51777] AH00898: Error reading from remote server returned by /api/system/notifications, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:09:44.418375 2021] [proxy_http:error] [pid 9324:tid 140699118769920] (104)Connection reset by peer: [client 10.2.7.18:56489] AH01102: error reading status line from remote server graylog01.my.domain:9000, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:09:44.418406 2021] [proxy:error] [pid 9324:tid 140699118769920] [client 10.2.7.18:56489] AH00898: Error reading from remote server returned by /api/system/cluster/nodes, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:19:44.407087 2021] [proxy_http:error] [pid 9323:tid 140699169126144] (104)Connection reset by peer: [client 10.2.7.18:63746] AH01102: error reading status line from remote server graylog01.my.domain:9000, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:19:44.407124 2021] [proxy:error] [pid 9323:tid 140699169126144] [client 10.2.7.18:63746] AH00898: Error reading from remote server returned by /api/system/cluster/nodes, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:21:11.416050 2021] [proxy_http:error] [pid 9324:tid 140699278231296] (104)Connection reset by peer: [client 10.2.7.18:59528] AH01102: error reading status line from remote server graylog01.my.domain:9000, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 15:21:11.416096 2021] [proxy:error] [pid 9324:tid 140699278231296] [client 10.2.7.18:59528] AH00898: Error reading from remote server returned by /api/system/cluster/nodes, referer: http://graylog/search?q=&rangetype=relative&from=1200
[Mon Aug 16 17:28:45.285027 2021] [proxy_http:error] [pid 9324:tid 140699382564608] (20014)Internal error (specific information not available): [client 10.100.10.6:45262] AH01102: error reading status line from remote server graylog02.my.domain:9000
[Mon Aug 16 17:28:45.285057 2021] [proxy:error] [pid 9324:tid 140699382564608] [client 10.100.10.6:45262] AH00898: Error reading from remote server returned by /api/sidecars/1jq26cssvc6rj4qac4bt9oeeh0p4vt5u5kal9jocl1g9mdi4og3n
[Tue Aug 17 00:00:27.734730 2021] [mpm_event:notice] [pid 583:tid 140699404317824] AH00493: SIGUSR1 received.  Doing graceful restart

error.log 

[Tue Aug 17 00:00:27.738029 2021] [mpm_event:notice] [pid 583:tid 140699404317824] AH00489: Apache/2.4.38 (Debian) configured -- resuming normal operations
[Tue Aug 17 00:00:27.738043 2021] [core:notice] [pid 583:tid 140699404317824] AH00094: Command line: '/usr/sbin/apache2'
[Tue Aug 17 04:30:09.894673 2021] [proxy_http:error] [pid 11754:tid 140699390957312] (20014)Internal error (specific information not available): [client 10.100.11.20:40522] AH01102: error reading status line from remote server graylog03.my.domain:9000

Maybe it’s a misconfiguration in my Apache Load Balancer, or something that should be defined elsewhere… I anyone have any idea, that would be great !

Have a great day and thanks a lot !

10

Hello,
I took a brief look at your log files and what I noticed was connection issues. This could be a permission issue or something in the way like firewall maybe, but when you stated this.

I’m leaning more at permission issues for your Apache services.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.