Unable to start graylog

Sargu79 · August 16, 2017, 7:25am

Hi ,

I have recently tried building a graylog server on a ubuntu 16.04 machine using the guide document from (http://docs.graylog.org/en/2.3/pages/installation/os/ubuntu.html) . I have followed the exact steps , but the graylog doesn’t seem to start. I am a complete stranger to graylog. Appreciate for any guide / help in bring up the graylog

in the log i can see the following

ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[RestApiService [STARTING], IndexerSetupService [STARTING]], FAILED=[WebInterfaceService [FAILED]]}

GTownson · August 16, 2017, 12:01pm

Is the web interface set to the correct IP in the config file?

scampuza · August 16, 2017, 1:28pm

Would you share with us the configuration file ?

Sargu79 · August 17, 2017, 2:46am

Hi GTownson , yes i have set the public ip of my server

Sargu79 · August 17, 2017, 2:49am

hi scampuza ,
blow is my graylog server.conf

root@gray7:~# grep -v “#” /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = Ns38iyUHgTjkEA4WN7lNrWah66H0NiDbQCSmXD3OQfvThlylwQUpLzD32TsmTXmvZx3e2WrF4DlDIVwcSYvcFfA35cih0J44
root_password_sha2 = 7cb22fd137eb17f9a7eb214e0284af0186a8cff3a73adfa4c504eb8b27841d32
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://54.254.216.238:9000/api/
web_listen_uri = http://54.254.216.238:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Sargu79 · August 17, 2017, 3:45am

adding to the above…from the gralog server.log

root@gray7:/var/log/graylog-server# cat server.log | grep -i bootstrap
2017-08-17T11:42:02.409+08:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JerseyService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], OutputSetupService [TERMINATED], StreamCacheService [TERMINATED], ConfigurationEtagService [TERMINATED], JournalReader [TERMINATED], PeriodicalsService [TERMINATED], LookupTableService [TERMINATED], KafkaJournal [TERMINATED], BufferSynchronizerService [TERMINATED]]}
2017-08-17T11:42:02.423+08:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] Graylog server 2.3.0+81f8228 starting up
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_131 on Linux 4.4.0-1031-aws
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] Deployment: deb
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2017-08-17T11:43:31.823+08:00 INFO [ServerBootstrap] Arch: amd64

GTownson · August 17, 2017, 9:45am

Our config is:

rest_listen_uri = ‘internalIP’:9000/api/
web_listen_uri = ‘internalIP’:9000/

We also have rest_transport_uri = ‘externalIP’:9000/api/

Sargu79 · August 17, 2017, 9:47am

so i should disable
rest_listen_uri = ‘internalIP’:9000/api/
web_listen_uri = ‘internalIP’:9000/

and enable

rest_transport_uri = ‘externalIP’:9000/api/ ?

GTownson · August 17, 2017, 9:48am

No, we have all three enabled. In our case web and rest listen api are set to the internal address of our Graylog instance, but yours seem to be an external address. Also in your config you seem to be missing rest_transport_uri, which in our case is set to the external IP of our Graylog instance.

I’m not sure if this is where your issue is coming from, however I have had problems in the past when these fields were set incorrectly.

G

Sargu79 · August 17, 2017, 9:53am

hi ,

i have just tried with the internal and public ip, yet having issue.

root@gray7:/etc/graylog/server# grep -v “#” server.conf |grep uri
rest_listen_uri = http://172.31.25.191:9000/api/
rest_transport_uri = http://54.254.216.238:9000/api/
web_listen_uri = http://172.31.25.191:9000/
mongodb_uri = mongodb://localhost/graylog

GTownson · August 17, 2017, 9:55am

Oh, I’m not too sure then. Could you post more logs?

Sargu79 · August 17, 2017, 9:56am

looks like it’s working now…just tailed the server.log

root@gray7:/var/log/graylog-server# tail -f server.log
2017-08-17T17:53:00.415+08:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-08-17T17:53:03.418+08:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-08-17T17:54:31.017+08:00 INFO [NetworkListener] Started listener bound to [172.31.25.191:9000]
2017-08-17T17:54:31.018+08:00 INFO [HttpServer] [HttpServer] Started.
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started REST API at http://172.31.25.191:9000/api/
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started Web Interface at http://172.31.25.191:9000/
2017-08-17T17:54:31.032+08:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=75, KafkaJournal [RUNNING]=76, BufferSynchronizerService [RUNNING]=108, JournalReader [RUNNING]=506, InputSetupService [RUNNING]=707, StreamCacheService [RUNNING]=709, LookupTableService [RUNNING]=726, ConfigurationEtagService [RUNNING]=748, PeriodicalsService [RUNNING]=1740, JerseyService [RUNNING]=92377}
2017-08-17T17:54:31.052+08:00 INFO [ServiceManagerListener] Services are healthy
2017-08-17T17:54:31.052+08:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-08-17T17:54:31.126+08:00 INFO [ServerBootstrap] Graylog server up and running.

GTownson · August 17, 2017, 9:57am

Ah good, so that fixed it?

Sargu79 · August 17, 2017, 9:57am

haha yes, i guess i wasn’t playing right with the local and public ip…
my bad …thanks so much!!

Sargu79 · August 17, 2017, 10:00am

but after sometime…i am seing some error again…timeout errors

root@gray7:/var/log/graylog-server# tail -f server.log
2017-08-17T17:53:00.415+08:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-08-17T17:53:03.418+08:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-08-17T17:54:31.017+08:00 INFO [NetworkListener] Started listener bound to [172.31.25.191:9000]
2017-08-17T17:54:31.018+08:00 INFO [HttpServer] [HttpServer] Started.
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started REST API at http://172.31.25.191:9000/api/
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started Web Interface at http://172.31.25.191:9000/
2017-08-17T17:54:31.032+08:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=75, KafkaJournal [RUNNING]=76, BufferSynchronizerService [RUNNING]=108, JournalReader [RUNNING]=506, InputSetupService [RUNNING]=707, StreamCacheService [RUNNING]=709, LookupTableService [RUNNING]=726, ConfigurationEtagService [RUNNING]=748, PeriodicalsService [RUNNING]=1740, JerseyService [RUNNING]=92377}
2017-08-17T17:54:31.052+08:00 INFO [ServiceManagerListener] Services are healthy
2017-08-17T17:54:31.052+08:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-08-17T17:54:31.126+08:00 INFO [ServerBootstrap] Graylog server up and running.
2017-08-17T17:58:25.729+08:00 INFO [UsageStatsOptOutService] Transmission of anonymous usage stats: enabled (opt-in)
2017-08-17T17:58:43.543+08:00 WARN [ProxiedResource] Unable to call http://54.254.216.238:9000/api/system/jobs on node <308bb471-6c67-4363-a855-a7ceca48fb28>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2017-08-17T17:58:53.764+08:00 WARN [ProxiedResource] Unable to call http://54.254.216.238:9000/api/system/jobs on node <308bb471-6c67-4363-a855-a7ceca48fb28>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2017-08-17T17:59:28.393+08:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:265}] to localhost:27017

GTownson · August 17, 2017, 10:01am

Oh, I’m not sure now. You will have to wait for someone else with more experience to help you out, sorry!

regards,
G

Sargu79 · August 17, 2017, 10:03am

no worries GTownson, thank you so much… i will close this thread for now,
atleast the graylog is up and running now. will play a bit more to familiarize. thanks again!

system · August 31, 2017, 10:03am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog service active but application not running Graylog Central (peer support)	4	1086	December 7, 2022
Graylog problem after installation Graylog Central (peer support)	4	3284	September 24, 2022
Graylog-server.service fails to start / no logs Graylog Central (peer support)	3	6596	January 6, 2021
Trying to enable SSL Graylog Central (peer support)	7	663	March 4, 2020
Graylog-server service won't start Graylog Central (peer support)	4	5457	June 16, 2022

Unable to start graylog

Related topics