Graylog-server Failed

Kamsy · January 28, 2023, 12:48am

Hi,
My graylog server was enabled in http but when I configured in https it enabled for 1s and disabled after. I see in my logs and I see that I am told to increase the message_log_max. I did this but it is still disabled.
Thx for your help!

dpkg -l | grep -E “.(elasticsearch|graylog|mongo|java).”

ii  ca-certificates-java                  20190909                                all          Common CA certificates (JKS keystore)
ii  elasticsearch-oss                     7.10.2                                  amd64        Distributed RESTful search engine built for the cloud
ii  graylog-4.3-repository                1-6                                     all          Package to install Graylog 4.3 GPG key and repository
ii  graylog-server                        4.3.8-1                                 all          Graylog server
ii  graylog-sidecar                       1.2.0-1                                 amd64        Graylog collector sidecar
ii  graylog-sidecar-repository            1-2                                     all          Package to install Graylog Sidecar GPG key and repository
ii  java-common                           0.72build2                              all          Base package for Java runtimes
ii  mongodb-database-tools                100.6.0                                 amd64        mongodb-database-tools package provides tools for working with the MongoDB server:
ii  mongodb-mongosh                       1.6.0                                   amd64        MongoDB Shell CLI REPL Package
ii  mongodb-org                           6.0.2                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database                  6.0.2                                   amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-database-tools-extra      6.0.2                                   amd64        Extra MongoDB database tools
ii  mongodb-org-mongos                    6.0.2                                   amd64        MongoDB sharded cluster query router
ii  mongodb-org-server                    6.0.2                                   amd64        MongoDB database server
ii  mongodb-org-shell                     6.0.2                                   amd64        MongoDB shell client
ii  mongodb-org-tools                     6.0.2

This my server.conf:

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = WNJQ0jsf3lnqS2PVhECKi3vVUbHdjgDg93SfaF6vsMFfd5x1wsseSSOLaUCL9faQTSkoEmjiWQkO2Cgrrc2MzMpP1YzUd1DO
root_username = ad
root_password_sha2 = a757fb1f3d4bb65716a0725707dd28563e47472a2539c307185d7b759d611d68
root_timezone = Europe/Paris
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address =graylog-server:9000
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/cert.pem
http_tls_key_file = /etc/ssl/certs/graylog/pkcs8-encrypted.pem
http_tls_key_password = secret
elasticsearch_hosts = http://127.0.0.1:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
elasticsearch_index_optimization_timeout = 1h
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 100gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
http_connect_timeout = 10s
disable_native_system_stats_collector = true
proxied_requests_thread_pool_size = 32

● graylog-server.service - Graylog server
     Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2023-01-27 15:56:01 UTC; 8s ago
       Docs: http://docs.graylog.org/
    Process: 134980 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
   Main PID: 134980 (code=exited, status=1/FAILURE)
        CPU: 3.288s

janv. 27 15:56:01 srvlog systemd[1]: graylog-server.service: Consumed 3.288s CPU time.

This my elastisearch.yml:

root@srvlog:/etc/ssl/certs/graylog# cat /etc/elasticsearch/elasticsearch.yml    | egrep -v "^\s*(#|$)"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
cluster.name: graylog
action.auto_create_index: false
discovery.type: single-node

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-01-27 09:32:25 UTC; 6h ago
       Docs: https://www.elastic.co
   Main PID: 98272 (java)
      Tasks: 52 (limit: 4575)
     Memory: 1.3G
        CPU: 3min 57.602s
     CGroup: /system.slice/elasticsearch.service
             └─98272 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.e>

janv. 27 09:32:09 srvlog systemd[1]: Starting Elasticsearch...
janv. 27 09:32:25 srvlog systemd[1]: Started Elasticsearch.

● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-11-03 15:24:09 UTC; 2 months 24 days ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 36618 (mongod)
     Memory: 181.6M
        CPU: 3h 14min 52.287s
     CGroup: /system.slice/mongod.service
             └─36618 /usr/bin/mongod --config /etc/mongod.conf

nov. 03 15:24:09 srvlog systemd[1]: Started MongoDB Database Server.

gsmith · January 28, 2023, 1:14am

Hello @Kamsy

What does Graylog log show about this issue? And can you show them here? I dont understand what message_log_max has to do with HTTPS, doesnt make sense.

Should be located here.

/var/log/graylog-server/

I’m taking a guess, you need to install the correct Certificate in your keystore and/or Graylog needs access to them.

Add the following in the graylog config file.

http_bind_address = graylog-server:9000
http_publish_uri = https://graylog-server:9000/
.........

rest looks good.
Last idea is the certificate made correctly,

Kamsy · January 30, 2023, 11:45am

Hello @gsmith
I add the following in the graylog config file:

http_bind_address = graylog-server:9000
http_publish_uri = https://graylog-server:9000/

But my graylog server is not enable if i restart.
This my log in /var/log/graylog-server/server.log:

2023-01-30T11:32:12.383Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2023-01-30T11:32:12.407Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-30T11:32:12.433Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:82}] to localhost:27017
2023-01-30T11:32:12.436Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[6, 0, 2]}, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1906079}
2023-01-30T11:32:12.444Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:83}] to localhost:27017
2023-01-30T11:32:12.462Z INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:83}] to localhost:27017 because the pool has been closed.
2023-01-30T11:32:12.464Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.2
2023-01-30T11:32:12.546Z INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2023-01-30T11:32:12.548Z ERROR [PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (1702 MB) available. You need to provide additional 3337 MB to contain 'message_journal_max_size = 5120 MB'
2023-01-30T11:32:23.675Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2023-01-30T11:32:24.259Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.8 [org.graylog.aws.AWSPlugin]
2023-01-30T11:32:24.261Z INFO  [CmdLineTool] Loaded plugin: Collector 4.3.8 [org.graylog.plugins.collector.CollectorPlugin]
2023-01-30T11:32:24.263Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.8 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2023-01-30T11:32:24.263Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.8+8c4705e [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2023-01-30T11:32:24.264Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.8+8c4705e [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2023-01-30T11:32:24.286Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2023-01-30T11:32:24.532Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2023-01-30T11:32:24.576Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-30T11:32:24.600Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:84}] to localhost:27017
2023-01-30T11:32:24.605Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[6, 0, 2]}, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3309033}
2023-01-30T11:32:24.618Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:85}] to localhost:27017
2023-01-30T11:32:24.638Z INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:85}] to localhost:27017 because the pool has been closed.
2023-01-30T11:32:24.640Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.2
2023-01-30T11:32:24.768Z INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2023-01-30T11:32:24.771Z ERROR [PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (1702 MB) available. You need to provide additional 3337 MB to contain 'message_journal_max_size = 5120 MB'

Thanks !

gsmith · January 30, 2023, 10:22pm

Hey @Kamsy
Your logs tell you whats going on right here.

PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (1702 MB) available. You need to provide additional 3337 MB to contain ‘message_journal_max_size = 5120 MB’

You journal is full. You have two option I know of.

Add more space for your journal, by re-configuraing Graylog’s config file.

message_journal_max_size = 5gb

Delete everything inside the jouranal, remind you that you will lose data.

Kamsy · January 31, 2023, 9:29am

Hello @gsmith,
I have added more space for my journal, but i have same mistake.
I have added 25gb.
This my logs after modification

2023-01-31T09:01:29.714Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2023-01-31T09:01:29.967Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2023-01-31T09:01:30.002Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-31T09:01:30.036Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:546}] to localhost:27017
2023-01-31T09:01:30.040Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[6, 0, 2]}, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3181282}
2023-01-31T09:01:30.053Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:547}] to localhost:27017
2023-01-31T09:01:30.076Z INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:547}] to localhost:27017 because the pool has been closed.
2023-01-31T09:01:30.078Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.2
2023-01-31T09:01:30.179Z INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2023-01-31T09:01:30.181Z ERROR [PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (1698 MB) available. You need to provide additional 23820 MB to contain 'message_journal_max_size = 25600 MB'
2023-01-31T09:01:41.426Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2023-01-31T09:01:41.909Z INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.8 [org.graylog.aws.AWSPlugin]
2023-01-31T09:01:41.910Z INFO  [CmdLineTool] Loaded plugin: Collector 4.3.8 [org.graylog.plugins.collector.CollectorPlugin]
2023-01-31T09:01:41.911Z INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.8 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2023-01-31T09:01:41.911Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.8+8c4705e [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2023-01-31T09:01:41.912Z INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.8+8c4705e [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2023-01-31T09:01:41.924Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2023-01-31T09:01:42.120Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2023-01-31T09:01:42.155Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-31T09:01:42.178Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:548}] to localhost:27017
2023-01-31T09:01:42.181Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[6, 0, 2]}, minWireVersion=0, maxWireVersion=17, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1901413}
2023-01-31T09:01:42.189Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:549}] to localhost:27017
2023-01-31T09:01:42.206Z INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:549}] to localhost:27017 because the pool has been closed.
2023-01-31T09:01:42.208Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 6.0.2
2023-01-31T09:01:42.297Z INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2023-01-31T09:01:42.299Z ERROR [PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (1698 MB) available. You need to provide additional 23820 MB to contain 'message_journal_max_size = 25600 MB'

In my test environnement, i use 22.04.1 Ubuntu on VM, 4GB RAM and 30GB of SSD. Is it enough to centralize AD logs?
Thanks for your help !

gsmith · January 31, 2023, 10:46pm

Hello
you can check by excuting this

root# df -hT

That will check how much space you have, but to be honest probably not, your just going to run into issue later one depeneding how long you keep logs + OS stuff.

Marvin1 · February 2, 2023, 9:29am

keep an eye on the size of the Path/data file of elastic.
To watch it grow run

du -hs /var/lib/elasticsearch

probably you need a better storage concept

Kamsy · February 2, 2023, 10:25am

Hello,
I have this storage in /var/lib/elasticsearch:

3,0G    /var/lib/elasticsearch/

How to increase the storage in elasticseach?

Marvin1 · February 3, 2023, 8:23am

To increase storage you need another hard disk.
After add a new hard disk you have to mount the new disk
At first run

lsblk

now you can see all disks

format hard disk and create partitions

sudo fdisk /dev/sdd

first command: g
second command: n
third command: w

sudo mkfs.ext4 /dev/sdd1

this command format the partition
now set the mount point

 sudo mount /dev/sdd1 /<what ever you want>

move the data to the new disk

sudo mv /var/lib/postgresql/* /mnt/

Take the UUID from your disk and post it to the fstab

sudo blkid

sudo nano /etc/fstab

Now you habe to chnage the yml file of Elastic

path.data: /new/mount/point

would also recommend to increase the heap space
for Elastic

sudo nano /etc/elasticsearch/jvm.options

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
 
-Xms4g
-Xmx4g

for Graylog

sudo nano /etc/default/graylog-server
GRAYLOG_SERVER_JAVA_OPTS="-Xms3g -Xmx3g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow"

Greetings

Kamsy · February 3, 2023, 8:40am

Thanks for your help !

system · February 17, 2023, 8:41am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog Server Started Successfully, But Web Page Not Opening Graylog Central (peer support)	6	940	June 30, 2023
Input failed to start after https Graylog Central (peer support)	13	1272	March 26, 2020
Server currently unavailable 2 Graylog Central (peer support) sidecar	10	934	January 28, 2021
Unable to run graylog server (Couldn't run validator method) Graylog Central (peer support)	1	498	February 20, 2021
Using HTTPS with graylog 4 Graylog Central (peer support)	2	342	October 14, 2021

Graylog-server Failed

Related Topics