Unable to start graylog

Graylog Central (peer support)
1

Hi ,

I have recently tried building a graylog server on a ubuntu 16.04 machine using the guide document from (http://docs.graylog.org/en/2.3/pages/installation/os/ubuntu.html) . I have followed the exact steps , but the graylog doesn’t seem to start. I am a complete stranger to graylog. Appreciate for any guide / help in bring up the graylog :frowning:

in the log i can see the following

ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[RestApiService [STARTING], IndexerSetupService [STARTING]], FAILED=[WebInterfaceService [FAILED]]}

2

Is the web interface set to the correct IP in the config file?

3

Would you share with us the configuration file ?

4

Hi GTownson , yes i have set the public ip of my server

5

hi scampuza ,
blow is my graylog server.conf

root@gray7:~# grep -v “#” /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = Ns38iyUHgTjkEA4WN7lNrWah66H0NiDbQCSmXD3OQfvThlylwQUpLzD32TsmTXmvZx3e2WrF4DlDIVwcSYvcFfA35cih0J44
root_password_sha2 = 7cb22fd137eb17f9a7eb214e0284af0186a8cff3a73adfa4c504eb8b27841d32
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://54.254.216.238:9000/api/
web_listen_uri = http://54.254.216.238:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

6

adding to the above…from the gralog server.log

root@gray7:/var/log/graylog-server# cat server.log | grep -i bootstrap
2017-08-17T11:42:02.409+08:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JerseyService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], OutputSetupService [TERMINATED], StreamCacheService [TERMINATED], ConfigurationEtagService [TERMINATED], JournalReader [TERMINATED], PeriodicalsService [TERMINATED], LookupTableService [TERMINATED], KafkaJournal [TERMINATED], BufferSynchronizerService [TERMINATED]]}
2017-08-17T11:42:02.423+08:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] Graylog server 2.3.0+81f8228 starting up
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_131 on Linux 4.4.0-1031-aws
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] Deployment: deb
2017-08-17T11:43:31.822+08:00 INFO [ServerBootstrap] OS: Ubuntu 16.04.3 LTS (xenial)
2017-08-17T11:43:31.823+08:00 INFO [ServerBootstrap] Arch: amd64

7

Our config is:

rest_listen_uri = ‘internalIP’:9000/api/
web_listen_uri = ‘internalIP’:9000/

We also have rest_transport_uri = ‘externalIP’:9000/api/

8

so i should disable
rest_listen_uri = ‘internalIP’:9000/api/
web_listen_uri = ‘internalIP’:9000/

and enable

rest_transport_uri = ‘externalIP’:9000/api/ ?

9

No, we have all three enabled. In our case web and rest listen api are set to the internal address of our Graylog instance, but yours seem to be an external address. Also in your config you seem to be missing rest_transport_uri, which in our case is set to the external IP of our Graylog instance.

I’m not sure if this is where your issue is coming from, however I have had problems in the past when these fields were set incorrectly.

G

10

hi ,

i have just tried with the internal and public ip, yet having issue.

root@gray7:/etc/graylog/server# grep -v “#” server.conf |grep uri
rest_listen_uri = http://172.31.25.191:9000/api/
rest_transport_uri = http://54.254.216.238:9000/api/
web_listen_uri = http://172.31.25.191:9000/
mongodb_uri = mongodb://localhost/graylog

11

Oh, I’m not too sure then. Could you post more logs?

12

looks like it’s working now…just tailed the server.log

root@gray7:/var/log/graylog-server# tail -f server.log
2017-08-17T17:53:00.415+08:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-08-17T17:53:03.418+08:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-08-17T17:54:31.017+08:00 INFO [NetworkListener] Started listener bound to [172.31.25.191:9000]
2017-08-17T17:54:31.018+08:00 INFO [HttpServer] [HttpServer] Started.
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started REST API at http://172.31.25.191:9000/api/
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started Web Interface at http://172.31.25.191:9000/
2017-08-17T17:54:31.032+08:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=75, KafkaJournal [RUNNING]=76, BufferSynchronizerService [RUNNING]=108, JournalReader [RUNNING]=506, InputSetupService [RUNNING]=707, StreamCacheService [RUNNING]=709, LookupTableService [RUNNING]=726, ConfigurationEtagService [RUNNING]=748, PeriodicalsService [RUNNING]=1740, JerseyService [RUNNING]=92377}
2017-08-17T17:54:31.052+08:00 INFO [ServiceManagerListener] Services are healthy
2017-08-17T17:54:31.052+08:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-08-17T17:54:31.126+08:00 INFO [ServerBootstrap] Graylog server up and running.

13

Ah good, so that fixed it?

14

haha yes, i guess i wasn’t playing right with the local and public ip…
my bad …thanks so much!!

1 Like
15

but after sometime…i am seing some error again…timeout errors

root@gray7:/var/log/graylog-server# tail -f server.log
2017-08-17T17:53:00.415+08:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-08-17T17:53:03.418+08:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-08-17T17:54:31.017+08:00 INFO [NetworkListener] Started listener bound to [172.31.25.191:9000]
2017-08-17T17:54:31.018+08:00 INFO [HttpServer] [HttpServer] Started.
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started REST API at http://172.31.25.191:9000/api/
2017-08-17T17:54:31.031+08:00 INFO [JerseyService] Started Web Interface at http://172.31.25.191:9000/
2017-08-17T17:54:31.032+08:00 INFO [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=75, KafkaJournal [RUNNING]=76, BufferSynchronizerService [RUNNING]=108, JournalReader [RUNNING]=506, InputSetupService [RUNNING]=707, StreamCacheService [RUNNING]=709, LookupTableService [RUNNING]=726, ConfigurationEtagService [RUNNING]=748, PeriodicalsService [RUNNING]=1740, JerseyService [RUNNING]=92377}
2017-08-17T17:54:31.052+08:00 INFO [ServiceManagerListener] Services are healthy
2017-08-17T17:54:31.052+08:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-08-17T17:54:31.126+08:00 INFO [ServerBootstrap] Graylog server up and running.
2017-08-17T17:58:25.729+08:00 INFO [UsageStatsOptOutService] Transmission of anonymous usage stats: enabled (opt-in)
2017-08-17T17:58:43.543+08:00 WARN [ProxiedResource] Unable to call http://54.254.216.238:9000/api/system/jobs on node <308bb471-6c67-4363-a855-a7ceca48fb28>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2017-08-17T17:58:53.764+08:00 WARN [ProxiedResource] Unable to call http://54.254.216.238:9000/api/system/jobs on node <308bb471-6c67-4363-a855-a7ceca48fb28>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_131]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
… 28 more
2017-08-17T17:59:28.393+08:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:265}] to localhost:27017

16

Oh, I’m not sure now. You will have to wait for someone else with more experience to help you out, sorry! :confused:

regards,
G

17

no worries GTownson, thank you so much… i will close this thread for now,
atleast the graylog is up and running now. will play a bit more to familiarize. thanks again!

18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.