Graylog Cluster behind HAproxy on HTTPS

(xorloader41) #1

Hi,

Currently, we are running single node Graylog that is running using the Self-signed certificate. We are planning to add one more graylog node and these node’s are going to sit behind HAProxy. I am trying with the following configuration, but didnt get this running.

Config:
frontend graylog_https
  bind *:443
  mode tcp
  default_backend graylog_server_cluster

backend graylog_server_cluster
  mode tcp
  balance roundrobin
  option httpchk HEAD /api/system/lbstatus
  http-request set-header X-Graylog-Server-URL https://graylog.example.com/api
  server g1 10.10.10.10:9000
  server g2 10.10.10.11:9000

with this configuration. I am able to see only the first node and complete cluster is not forming.

I started doubting whether HTTPS is possbile with graylog clusters ? If possible, Can someone point me what mistake I am making here. Thanks

(Jochen) #2

See http://docs.graylog.org/en/2.4/pages/configuration/web_interface.html#haproxy-1-6 for a working configuration of HAProxy with Graylog 2.4.x.

If you still have problems after reading the documentation, please provide the full configuration of both Graylog nodes and HAProxy.

(xorloader41) #3

Hi Jo,

I did follow the document in the link and I couldn’t make it work.

HAProxy:

frontend graylog_https
    bind *:443
    option forwardfor
    http-request add-header X-Forwarded-Host %[req.hdr(host)]
    http-request add-header X-Forwarded-Server %[req.hdr(host)]
    http-request add-header X-Forwarded-Port %[dst_port]
    acl is_graylog hdr_dom(host) -i -m str example.graylog.com
    use_backend     graylog

backend graylog
    description     The Graylog Web backend.
    balance roundrobin
    option httpchk HEAD /api/system/lbstatus
    http-request set-header X-Graylog-Server-URL https://example.graylog.com/api
    server graylog1 10.10.10.10:9000 maxconn 20 check
    server graylog2 10.10.10.11:9000 maxconn 20 check

GRAYLOG NODE 1:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = [redacted]
root_password_sha2 = [redacted]
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = https://10.10.10.10:9000/api/
rest_enable_tls = true
rest_tls_cert_file = /graylog/ssl-keys/Test-Certificate.pem
rest_tls_key_file = /graylog/ssl-keys/Test-PriKey.pem
web_listen_uri = http://10.10.10.10:9000/
web_enable_tls = true
web_tls_cert_file = /graylog/ssl-keys/Test-Certificate.pem
web_tls_key_file = /graylog/ssl-keys/Test-PriKey.pem
elasticsearch_hosts = http://10.10.10.10:9200,http://10.10.10.11:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 64
outputbuffer_processors = 64
processor_wait_strategy = blocking
ring_size = 131072
inputbuffer_ring_size = 131072
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://10.10.10.10,10.10.10.11/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = smtpio.testing.com
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog-alerts]
transport_email_from_email = graylog-alerts@testing.com
transport_email_web_interface_url = https://10.10.10.10:9000
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
allow_highlighting = true

GRAYLOG NODE 2:

is_master = false
node_id_file = /etc/graylog/server/node-id
password_secret = [redacted]
root_password_sha2 = [redacted]
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = https://10.10.10.11:9000/api/
rest_enable_tls = true
rest_tls_cert_file = /graylog/ssl-keys/Test-Certificate.pem
rest_tls_key_file = /graylog/ssl-keys/Test-PriKey.pem
web_listen_uri = http://10.10.10.11:9000/
web_enable_tls = true
web_tls_cert_file = /graylog/ssl-keys/Test-Certificate.pem
web_tls_key_file = /graylog/ssl-keys/Test-PriKey.pem
elasticsearch_hosts = http://10.10.10.10:9200,http://10.10.10.11:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 64
outputbuffer_processors = 64
processor_wait_strategy = blocking
ring_size = 131072
inputbuffer_ring_size = 131072
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 10gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://10.10.10.10,10.10.10.11/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = smtpio.testing.com
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_subject_prefix = [graylog-alerts]
transport_email_from_email = graylog-alerts@testing.com
transport_email_web_interface_url = https://10.10.10.11:9000
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
allow_highlighting = true

When I try to access https://example.graylog.com page is not opening. Can you help me what I am doing wrong. Thanks

(Jochen) #4

Are the contents of the node ID file unique to each Graylog node?

Have you created a MongoDB replica set or a sharded MongoDB cluster?

And last but not least: Since you’re doing HTTPS termination in HAProxy, you don’t necessarily need to set up HTTPS in Graylog itself.

If you’re using self-signed certificates, make sure to read http://docs.graylog.org/en/2.4/pages/configuration/https.html.

(xorloader41) #5

Hi Joe,
Thanks for the suggestion. And I am now terminating SSL on HAProxy itself. But Periodically ( every 2 seconds) I am seeing the below issue in the nodes and I am not able to fully form the cluster.

Node-Status
Node-Status.jpg1914x870 85.1 KB
Node-Status-1
Node-Status-1.jpg1914x870 63.8 KB

I am running HAPRoxy on the same host on which I am running graylog master node. Will it make some impact on the API connections?? Can you please suggest ? Thanks

(Jochen) #6

Please answer all of the questions.

Also please post the complete logs of your Graylog nodes.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

(Jan Doberstein) #7

If the configuration is still as shown in the above configuration for Graylog Node 1 and 2 you have a mixed HTTP/HTTPS Settings in your configuration what is not working and not supported.

You should re-read http://docs.graylog.org/en/2.4/pages/configuration/web_interface.html and http://docs.graylog.org/en/2.4/pages/configuration/https.html

In addition the settings that you are currently use should be posted here…

(xorloader41) #8

Thanks for your help Jan. I have found out that time is the culprit here. 2 modes are drifting apart 10 seconds and that was the cause.

After synching time now cluster is stable. Thanks for your assistance on this issue. much appericiated.

(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.