Installation on Ubuntu 18.04.5 LTS not working

mveerdonk · September 22, 2020, 8:28am

Hi,
trying to do a clean install of Graylog on a hosted/managed server. Used https://docs.graylog.org/en/2.5/pages/installation/os/ubuntu.html

After performing the complete install, graylog server isn’t accessible via it’s public IP.
Looking at the services I get;

ubuntu@graylog:/etc/init.d$ sudo ./graylog-server status
ΓùÅ graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2020-09-22 08:26:38 UTC; 7s ago
     Docs: http://docs.graylog.org/
  Process: 5428 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
 Main PID: 5428 (code=exited, status=1/FAILURE)
ubuntu@graylog:/etc/init.d$   

ubuntu@graylog:/etc/init.d$ sudo ./mongodb status
ΓùÅ mongodb.service - An object/document-oriented database
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2020-09-22 08:15:40 UTC; 11min ago
     Docs: man:mongod(1)
  Process: 1201 ExecStart=/usr/bin/mongod --unixSocketPrefix=${SOCKETPATH} --config ${CONF} $DAEMON_OPTS (code=exited, status=100)
 Main PID: 1201 (code=exited, status=100)

Sep 22 08:15:39 graylog systemd[1]: Started An object/document-oriented database.
Sep 22 08:15:40 graylog systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Sep 22 08:15:40 graylog systemd[1]: mongodb.service: Failed with result 'exit-code'.
ubuntu@graylog:/etc/init.d$

Elasticsearch is running okay.

Any ideas/tips how to fix this?
// Marty

Karlis · September 22, 2020, 8:56am

Are You installing version 2.5?

Ponet · September 22, 2020, 8:58am

If your mongod service is failing to start, the graylog-server service won’t start as it requires Mongodb to be running (and accessible).

mveerdonk · September 22, 2020, 9:00am

How do I fix the mongodb part?

Ponet · September 22, 2020, 9:03am

Troubleshoot it. Look at the logs.
All I know from the info you provided is that the service failed.

Karlis · September 22, 2020, 9:03am

mveerdonk · September 22, 2020, 9:08am

ubuntu@graylog:/etc/init.d$ /usr/bin/mongod --verbose --config /etc/mongod.conf &
[1] 18501
ubuntu@graylog:/etc/init.d$ 2020-09-22T09:07:35.364+0000 F CONTROL  [main] Failed global initialization: FileNotOpen: Failed to open "/var/log/mongodb/mongod.log"

Magneton · September 22, 2020, 3:35pm

Hi mveerdonk,

perhaps you can paste the output of the following command comands to help.

sudo dpkg -l | grep mongo - this will tell us the mongo version you are using

sudo journalctl -xe will tell us why the service is failing to start

You can also look in the logs in the /vat/log/ directory there should be mongodb files in there

Cheers

Magneton

mveerdonk · September 22, 2020, 4:28pm

sudo dpkg -l | grep mongo

ubuntu@graylog:/var/lib/mongodb$ sudo dpkg -l | grep mongo
ii mongo-tools 3.6.3-0ubuntu1 amd64 collection of tools for administering MongoDB servers
iU mongodb-org 4.4.1 amd64 MongoDB open source document-oriented database system (metapackage)
iU mongodb-org-database-tools-extra 4.4.1 amd64 Extra MongoDB database tools
iU mongodb-org-shell 4.4.1 amd64 MongoDB shell client
iU mongodb-org-tools 4.4.1 amd64 MongoDB tools
rc mongodb-server 1:3.6.3-0ubuntu1.1 all object/document-oriented database (managed server package)
ii mongodb-server-core 1:3.6.3-0ubuntu1.1 amd64 object/document-oriented database (server binaries package)

ubuntu@graylog:/var/lib/mongodb$ sudo journalctl -xe

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:25:56 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:25:56 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd

– Unit graylog-server.service has finished starting up.

– The start-up result is RESULT.
Sep 22 16:25:59 graylog systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 16:25:59 graylog systemd[1]: graylog-server.service: Failed with result ‘exit-code’.
Sep 22 16:26:09 graylog systemd[1]: graylog-server.service: Service hold-off time over, scheduling restart.
Sep 22 16:26:09 graylog systemd[1]: graylog-server.service: Scheduled restart job, restart counter is at 2269.
– Subject: Automatic restarting of a unit has been scheduled
– Defined-By: systemd

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:26:09 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:26:09 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd

– Unit graylog-server.service has finished starting up.

– The start-up result is RESULT.
Sep 22 16:26:12 graylog systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 16:26:12 graylog systemd[1]: graylog-server.service: Failed with result ‘exit-code’.
Sep 22 16:26:22 graylog systemd[1]: graylog-server.service: Service hold-off time over, scheduling restart.
Sep 22 16:26:22 graylog systemd[1]: graylog-server.service: Scheduled restart job, restart counter is at 2270.
– Subject: Automatic restarting of a unit has been scheduled
– Defined-By: systemd

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:26:22 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:26:22 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd

– Unit graylog-server.service has finished starting up.

– The start-up result is RESULT.
Sep 22 16:26:25 graylog systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 22 16:26:25 graylog systemd[1]: graylog-server.service: Failed with result ‘exit-code’.
Sep 22 16:26:28 graylog sudo[3609]: ubuntu : TTY=pts/0 ; PWD=/var/lib/mongodb ; USER=root ; COMMAND=/bin/journalctl -xe
Sep 22 16:26:28 graylog sudo[3609]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)

ubuntu@graylog:/var/lib/mongodb$
ubuntu@graylog:/var/lib/mongodb$

mveerdonk · September 22, 2020, 4:31pm

Strange thing about MongoDB is that it’s generating a log file.
But in mongod.conf it’s configured to generate a log file

Where to store the data.

dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongodb.log

logappend=true

bind_ip = 127.0.0.1
port = 27017

mveerdonk · September 22, 2020, 5:46pm

When I started mongod with specificly mentioned the DB path it seems to work.
But the db path is also mentioned in the conf file.

ubuntu@graylog:/var/log$ sudo mongod --dbpath /var/lib/mongodb
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] MongoDB starting : pid=27733 port=27017 dbpath=/var/lib/mongodb 64-bit host=graylog
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] db version v3.6.3
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-09-22T17:44:04.771+0000 I CONTROL  [initandlisten] modules: none
2020-09-22T17:44:04.772+0000 I CONTROL  [initandlisten] build environment:
2020-09-22T17:44:04.772+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-09-22T17:44:04.772+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-09-22T17:44:04.772+0000 I CONTROL  [initandlisten] options: { storage: { dbPath: "/var/lib/mongodb" } }
2020-09-22T17:44:04.772+0000 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-09-22T17:44:04.772+0000 I STORAGE  [initandlisten]
2020-09-22T17:44:04.772+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-09-22T17:44:04.772+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-09-22T17:44:04.772+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15544M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-09-22T17:44:04.996+0000 I STORAGE  [initandlisten] WiredTiger message [1600796644:996498][27733:0x7fb91d5680c0], txn-recover: Main recovery loop: starting at 4/5376
2020-09-22T17:44:05.075+0000 I STORAGE  [initandlisten] WiredTiger message [1600796645:75697][27733:0x7fb91d5680c0], txn-recover: Recovering log 4 through 5
2020-09-22T17:44:05.121+0000 I STORAGE  [initandlisten] WiredTiger message [1600796645:121762][27733:0x7fb91d5680c0], txn-recover: Recovering log 5 through 5
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten]
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten]
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2020-09-22T17:44:05.178+0000 I CONTROL  [initandlisten]
2020-09-22T17:44:05.185+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
2020-09-22T17:44:05.186+0000 I NETWORK  [initandlisten] waiting for connections on port 27017

makarands · September 25, 2020, 11:59am

@mveerdonk Check your Graylog logs which might help you to identify the cause of it. May I know is there any specific reason for install the 2.5 Graylog version not the latest one?

mveerdonk · October 1, 2020, 6:52am

fixed the installation with latest versions of MongoDB, Elastic and graylog.
Services are running, but webgui of graylog is empty/not showing.

I see some errors/warning, but not sure if these are blocking…

> 020-10-01T06:45:55.089Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
> 2020-10-01T06:45:55.089Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseManagerPeriodical].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseManagerPeriodical] complete, took <0ms>.
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.license.LicenseReportPeriodical].
> 2020-10-01T06:45:55.090Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.license.LicenseReportPeriodical] complete, took <0ms>.
> 2020-10-01T06:45:55.091Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.scheduler.ReportPeriodical].
> 2020-10-01T06:45:55.091Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.scheduler.ReportPeriodical] complete, took <0ms>.
> 2020-10-01T06:45:55.091Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical].
> 2020-10-01T06:45:55.091Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.report.service.ChromeDriverCleanupPeriodical] complete, took <0ms>.
> 2020-10-01T06:45:55.094Z INFO  [GracefulShutdownService] Initiate shutdown for <JobWorkerPool>
> 2020-10-01T06:45:55.095Z INFO  [GracefulShutdownService] Finished shutdown for <JobWorkerPool>, took 0 ms
> 2020-10-01T06:45:55.098Z INFO  [GracefulShutdownService] Initiate shutdown for <BrowserDriverService>
> 2020-10-01T06:45:55.098Z INFO  [GracefulShutdownService] Finished shutdown for <BrowserDriverService>, took 0 ms
> 2020-10-01T06:45:55.101Z INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
> 2020-10-01T06:45:55.109Z INFO  [LogManager] Shutdown complete.
> 2020-10-01T06:45:55.161Z INFO  [JournalReader] Stopping.
> 2020-10-01T06:45:58.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:03.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:08.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:13.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:18.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:23.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:25.101Z ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[BufferSynchronizerService [STOPPING], JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], UrlWhitelistService [TERMINATED], OutputSetupService [TERMINATED], StreamCacheService [TERMINATED], PeriodicalsService [TERMINATED], GracefulShutdownService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], EtagService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
> 2020-10-01T06:46:25.101Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
> java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
> 	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
> 	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
> 	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
> 	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) [graylog.jar:?]
> 	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
> 	at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
> 	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
> 	Caused by: java.net.BindException: Cannot assign requested address
> 		at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
> 		at sun.nio.ch.Net.bind(Net.java:461) ~[?:?]
> 		at sun.nio.ch.Net.bind(Net.java:453) ~[?:?]
> 		at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) ~[?:?]
> 		at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) ~[?:?]
> 		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:238) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:218) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:209) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:723) ~[graylog.jar:?]
> 		at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:277) ~[graylog.jar:?]
> 		at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:201) ~[graylog.jar:?]
> 		at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:147) ~[graylog.jar:?]
> 		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
> 		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
> 		at java.lang.Thread.run(Thread.java:834) ~[?:?]
> 2020-10-01T06:46:25.103Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
> 2020-10-01T06:46:25.104Z INFO  [Server] SIGNAL received. Shutting down.
> 2020-10-01T06:46:25.106Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
> 2020-10-01T06:46:25.106Z INFO  [GracefulShutdown] Graceful shutdown initiated.
> 2020-10-01T06:46:25.106Z INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
> 2020-10-01T06:46:25.188Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog Events" (5f757b044f5624007959fc25) doesn't exist yet
> 2020-10-01T06:46:25.191Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog System Events" (5f757b044f5624007959fc28) doesn't exist yet
> 2020-10-01T06:46:28.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:33.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:38.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:43.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:48.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:53.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:46:55.109Z ERROR [Cluster] Couldn't read cluster health for indices [graylog_deflector, gl-events_deflector, gl-system-events_deflector]
> 
>  (n/a)
> 2020-10-01T06:46:55.110Z WARN  [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
> 2020-10-01T06:46:55.110Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
> 2020-10-01T06:46:55.110Z INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
> 2020-10-01T06:46:55.111Z INFO  [GracefulShutdown] Goodbye.
> 2020-10-01T06:46:58.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:03.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:08.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:13.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:18.177Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:23.177Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:25.187Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog Events" (5f757b044f5624007959fc25) doesn't exist yet
> 2020-10-01T06:47:25.189Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog System Events" (5f757b044f5624007959fc28) doesn't exist yet
> 2020-10-01T06:47:28.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet
> 2020-10-01T06:47:33.178Z WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Default index set" (5f757b024f5624007959fb5e) doesn't exist yet

mveerdonk · October 1, 2020, 7:34am

removed/disabled the Enterprise plugins.

Looks like it’s fixing the issue

ubuntu@graylog:/var/log/graylog-server$ sudo systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-10-01 07:33:36 UTC; 14s ago
     Docs: http://docs.graylog.org/
 Main PID: 10160 (graylog-server)
    Tasks: 106 (limit: 4915)
   CGroup: /system.slice/graylog-server.service
           ├─10160 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─10192 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled

Oct 01 07:33:36 graylog systemd[1]: Started Graylog server.
Oct 01 07:33:36 graylog graylog-server[10160]: Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be rem
Oct 01 07:33:37 graylog graylog-server[10160]: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Oct 01 07:33:38 graylog graylog-server[10160]: WARNING: An illegal reflective access operation has occurred
Oct 01 07:33:38 graylog graylog-server[10160]: WARNING: Illegal reflective access by com.google.inject.assistedinject.FactoryProvider2$MethodHandleWrapper (file:/usr/sh
Oct 01 07:33:38 graylog graylog-server[10160]: WARNING: Please consider reporting this to the maintainers of com.google.inject.assistedinject.FactoryProvider2$MethodHan
Oct 01 07:33:38 graylog graylog-server[10160]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Oct 01 07:33:38 graylog graylog-server[10160]: WARNING: All illegal access operations will be denied in a future release

mveerdonk · October 1, 2020, 10:37am

Unfortunally the webgui still doesn’t show.

shoothub · October 1, 2020, 11:16am

Did you try to use OpenJDK 8?

mveerdonk · October 1, 2020, 11:44am

dont understand your question completly

ubuntu@graylog:~$ java -version
java version “11.0.8” 2020-07-14 LTS
Java™ SE Runtime Environment 18.9 (build 11.0.8+10-LTS)
Java HotSpot™ 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode)

shoothub · October 1, 2020, 12:47pm

Try to use java openjdk-8 instead of version 11. I mean to use package openjdk-8-jre-headless instead of 11.

mveerdonk · October 1, 2020, 1:15pm

after this change everything is running. In elastic log I don’t see any errors.

My graylog server.log

2020-10-01T15:12:22.663+02:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], LookupTableService [TERMINATED], GracefulShutdownService [TERMINATED], PeriodicalsService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], EtagService [TERMINATED], KafkaJournal [TERMINATED], BufferSynchronizerService [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2020-10-01T15:12:22.663+02:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
	Caused by: java.net.BindException: Cannot assign requested address
		at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
		at sun.nio.ch.Net.bind(Net.java:461) ~[?:?]
		at sun.nio.ch.Net.bind(Net.java:453) ~[?:?]
		at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) ~[?:?]
		at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) ~[?:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:238) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:218) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:209) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:723) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:277) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:201) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:147) ~[graylog.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
		at java.lang.Thread.run(Thread.java:834) ~[?:?]
2020-10-01T15:12:22.670+02:00 INFO  [Server] SIGNAL received. Shutting down.
2020-10-01T15:12:22.675+02:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2020-10-01T15:12:22.675+02:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2020-10-01T15:12:26.678+02:00 INFO  [GracefulShutdown] Goodbye.

mveerdonk · October 1, 2020, 6:09pm

But still no webgui working!?!?!?

Anyone ideas how to solve this?

Topic		Replies	Views
Graylog-server.service failed Graylog Central (peer support)	9	12958	August 6, 2020
Graylog installation issue in Ubuntu server 18.04 Graylog Central (peer support)	2	566	April 29, 2021
I can't run graylog Graylog Central (peer support) access-specific-log-	1	41	July 26, 2024
Graylog problem after installation Graylog Central (peer support)	4	3417	September 24, 2022
Problem when installing graylog Graylog Central (peer support)	6	683	August 8, 2021

Installation on Ubuntu 18.04.5 LTS not working

ubuntu@graylog:/var/lib/mongodb$ sudo journalctl -xe

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for – the configured Restart= setting for the unit. Sep 22 16:25:56 graylog systemd[1]: Stopped Graylog server. – Subject: Unit graylog-server.service has finished shutting down – Defined-By: systemd

– Unit graylog-server.service has finished shutting down. Sep 22 16:25:56 graylog systemd[1]: Started Graylog server. – Subject: Unit graylog-server.service has finished start-up – Defined-By: systemd

– Unit graylog-server.service has finished starting up.

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for – the configured Restart= setting for the unit. Sep 22 16:26:09 graylog systemd[1]: Stopped Graylog server. – Subject: Unit graylog-server.service has finished shutting down – Defined-By: systemd

– Unit graylog-server.service has finished shutting down. Sep 22 16:26:09 graylog systemd[1]: Started Graylog server. – Subject: Unit graylog-server.service has finished start-up – Defined-By: systemd

– Unit graylog-server.service has finished starting up.

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for – the configured Restart= setting for the unit. Sep 22 16:26:22 graylog systemd[1]: Stopped Graylog server. – Subject: Unit graylog-server.service has finished shutting down – Defined-By: systemd

– Unit graylog-server.service has finished shutting down. Sep 22 16:26:22 graylog systemd[1]: Started Graylog server. – Subject: Unit graylog-server.service has finished start-up – Defined-By: systemd

– Unit graylog-server.service has finished starting up.

Where to store the data.

Related topics

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:25:56 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:25:56 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:26:09 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:26:09 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd

– Automatic restarting of the unit graylog-server.service has been scheduled, as the result for
– the configured Restart= setting for the unit.
Sep 22 16:26:22 graylog systemd[1]: Stopped Graylog server.
– Subject: Unit graylog-server.service has finished shutting down
– Defined-By: systemd

– Unit graylog-server.service has finished shutting down.
Sep 22 16:26:22 graylog systemd[1]: Started Graylog server.
– Subject: Unit graylog-server.service has finished start-up
– Defined-By: systemd