New installation; howto use local and public IP addresses in config files

mveerdonk · October 1, 2020, 10:52am

Hi,
deploying a complete new instance of Graylog.
java, monodb, elasticsearch and graylog are all running plus I don’t see any errors in the logs. However the webgui isn’t shown.

I’m using a cloud VM instance for MongoDB, Elastic and Graylog. All on the same VM. This VM has an private ip address 172.x.x.x and has a public IP address a.b.c.d

How do I need to configure Elastic and Graylog with these ip addresses?

Currently I have.
elasticsearch.yml
network.host: 172.x.y.z
http.port: 9200

Graylog - server.conf
http_publish_uri = http://172.x.y.z:9000/
http_external_uri = a.b.c.d/
rest_listen_uri = http://172.x.y.z:9000/api/
elasticsearch_hosts= http://172.x.y.z:9200/

cawfehman · October 2, 2020, 3:39am

have you opened the firewall to allow traffic in to port 9000?

mveerdonk · October 2, 2020, 6:38am

firewall is completely open for my personal home IP address.
Is there a way to verify if the web server is running on graylog?

The only strange things in the logs that I can find is this. Can’t make up out of it what the issue can be.

2020-10-01T15:11:52.656+02:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2020-10-01T15:11:52.656+02:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2020-10-01T15:11:52.657+02:00 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2020-10-01T15:11:52.666+02:00 INFO  [LogManager] Shutdown complete.
2020-10-01T15:11:52.678+02:00 INFO  [Buffers] Waiting until all buffers are empty.
2020-10-01T15:11:52.678+02:00 INFO  [Buffers] All buffers are empty. Continuing.
2020-10-01T15:11:52.679+02:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2020-10-01T15:11:52.738+02:00 INFO  [JournalReader] Stopping.
2020-10-01T15:12:22.663+02:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], LookupTableService [TERMINATED], GracefulShutdownService [TERMINATED], PeriodicalsService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], EtagService [TERMINATED], KafkaJournal [TERMINATED], BufferSynchronizerService [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2020-10-01T15:12:22.663+02:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:773) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:585) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:316) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
	Caused by: java.net.BindException: Cannot assign requested address
		at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
		at sun.nio.ch.Net.bind(Net.java:461) ~[?:?]
		at sun.nio.ch.Net.bind(Net.java:453) ~[?:?]
		at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) ~[?:?]
		at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) ~[?:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:238) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:218) ~[graylog.jar:?]
		at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:209) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:723) ~[graylog.jar:?]
		at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:277) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:201) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:147) ~[graylog.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
		at java.lang.Thread.run(Thread.java:834) ~[?:?]
2020-10-01T15:12:22.670+02:00 INFO  [Server] SIGNAL received. Shutting down.
2020-10-01T15:12:22.675+02:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2020-10-01T15:12:22.675+02:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2020-10-01T15:12:26.678+02:00 INFO  [GracefulShutdown] Goodbye.
2020-10-02T05:13:22.612+02:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:40}] to localhost:27017

Cato · October 2, 2020, 6:50am

Hi,

Comparing a little with our setup (it’s behind a loadbalancer, do you use that ?).
We don’t use the http_external_uri parameter.
What’s your http_bind_address parameter set to ?

mveerdonk · October 2, 2020, 6:55am

I don’ use a loadbalancer

Graylog - server.conf
http_publish_uri = http://172.x.y.z:9000/
http_external_uri = a.b.c.d/
rest_listen_uri = http://172.x.y.z:9000/api/
elasticsearch_hosts= http://172.x.y.z:9200/
http_bind_interface = a.b.c.d:9000

Cato · October 2, 2020, 7:17am

Just a thought, shouldn’t http_external_uri be httpd://a.b.c.d/ ?

http_bind_address 127.0.0.1:9000
The network interface used by the Graylog HTTP interface.

http_publish_uri If not set, http://$http_bind_address will be used.
The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all clients using the Graylog web interface.

http_external_uri If not set, $http_publish_uri will be used.
The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API. Graylog web interface.

mveerdonk · October 2, 2020, 7:20am

So you mean this setup?
is httpd an typo of you?

http_external_uri = httpd://a.b.c.d/
http_bind_interface = 127.0.0.1:9000
http_publish_uri = http://$http_bind_interface

rest_listen_uri = http://172.x.y.z:9000/api/
elasticsearch_hosts= http://172.x.y.z:9200/

Cato · October 2, 2020, 7:35am

Yes, that was a typo

mveerdonk · October 2, 2020, 7:48am

After restart of graylog services, log file is nice and clean.
Webgui is still not shown.

ubuntu@graylog:~$ curl -i http://127.0.0.1:9000/
HTTP/1.1 200 OK
X-UA-Compatible: IE=edge
X-Graylog-Node-ID: df24e902-b125-456e-9969-261dcdf0fe24
Content-Type: text/html
Date: Fri, 02 Oct 2020 07:48:21 GMT
Content-Length: 1502

Graylog Web Interface

<script src="http://130.61.140.229/assets/vendor.043dd426065882df527b.js"></script>

<script src="http://130.61.140.229/assets/polyfill.a7a8cf384e9fd1ed8a55.js"></script>

<script src="http://130.61.140.229/assets/builtins.a7a8cf384e9fd1ed8a55.js"></script>

<script src="http://130.61.140.229/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.8228d2d10b9d804b5723.js"></script>

<script src="http://130.61.140.229/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.a23ad41bb0e8db55d366.js"></script>

<script src="http://130.61.140.229/assets/plugin/org.graylog.integrations.IntegrationsPlugin/plugin.org.graylog.integrations.IntegrationsPlugin.1132f3911f4c9535bce2.js"></script>

<script src="http://130.61.140.229/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.95e803c80f4abddb515e.js"></script>

<script src="http://130.61.140.229/assets/app.a7a8cf384e9fd1ed8a55.js"></script>

ubuntu@graylog:~$

2020-10-02T09:43:59.445+02:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 3.3.6 [org.graylog.aws.AWSPlugin]
2020-10-02T09:43:59.448+02:00 INFO [CmdLineTool] Loaded plugin: Integrations 3.3.6 [org.graylog.integrations.IntegrationsPlugin]
2020-10-02T09:43:59.449+02:00 INFO [CmdLineTool] Loaded plugin: Collector 3.3.6 [org.graylog.plugins.collector.CollectorPlugin]
2020-10-02T09:43:59.450+02:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.3.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2020-10-02T09:43:59.658+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2020-10-02T09:43:59.878+02:00 INFO [Version] HV000001: Hibernate Validator null
2020-10-02T09:44:02.453+02:00 INFO [InputBufferImpl] Message journal is enabled.
2020-10-02T09:44:02.471+02:00 INFO [NodeId] Node ID: df24e902-b125-456e-9969-261dcdf0fe24
2020-10-02T09:44:02.654+02:00 INFO [LogManager] Loading logs.
2020-10-02T09:44:02.722+02:00 INFO [LogManager] Logs loading complete.
2020-10-02T09:44:02.725+02:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2020-10-02T09:44:02.746+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2020-10-02T09:44:02.789+02:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2020-10-02T09:44:02.806+02:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:41}] to localhost:27017
2020-10-02T09:44:02.814+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 9]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4110534}
2020-10-02T09:44:02.830+02:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:42}] to localhost:27017
2020-10-02T09:44:03.033+02:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2020-10-02T09:44:03.322+02:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://172.16.0.145:9200/]
2020-10-02T09:44:03.323+02:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2020-10-02T09:44:03.413+02:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2020-10-02T09:44:03.413+02:00 INFO [JestClientFactory] Node Discovery disabled…
2020-10-02T09:44:03.413+02:00 INFO [JestClientFactory] Idle connection reaping disabled…
2020-10-02T09:44:03.928+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-02T09:44:04.039+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-02T09:44:04.051+02:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2020-10-02T09:44:04.065+02:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:43}] to localhost:27017
2020-10-02T09:44:04.085+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-02T09:44:04.108+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-02T09:44:04.135+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-02T09:44:04.156+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-02T09:44:04.599+02:00 INFO [ServerBootstrap] Graylog server 3.3.6+92fb41e starting up
2020-10-02T09:44:04.600+02:00 INFO [ServerBootstrap] JRE: Oracle Corporation 11.0.8 on Linux 5.4.0-1025-oracle
2020-10-02T09:44:04.600+02:00 INFO [ServerBootstrap] Deployment: deb
2020-10-02T09:44:04.600+02:00 INFO [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2020-10-02T09:44:04.600+02:00 INFO [ServerBootstrap] Arch: amd64
2020-10-02T09:44:04.624+02:00 INFO [PeriodicalsService] Starting 30 periodicals …
2020-10-02T09:44:04.624+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2020-10-02T09:44:04.629+02:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2020-10-02T09:44:04.633+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2020-10-02T09:44:04.636+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2020-10-02T09:44:04.659+02:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:44}] to localhost:27017
2020-10-02T09:44:04.659+02:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2020-10-02T09:44:04.660+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2020-10-02T09:44:04.660+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2020-10-02T09:44:04.660+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2020-10-02T09:44:04.668+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2020-10-02T09:44:04.669+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2020-10-02T09:44:04.678+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2020-10-02T09:44:04.680+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2020-10-02T09:44:04.685+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2020-10-02T09:44:04.687+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2020-10-02T09:44:04.689+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2020-10-02T09:44:04.698+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2020-10-02T09:44:04.706+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2020-10-02T09:44:04.708+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2020-10-02T09:44:04.710+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2020-10-02T09:44:04.710+02:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:45}] to localhost:27017
2020-10-02T09:44:04.714+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2020-10-02T09:44:04.714+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2020-10-02T09:44:04.716+02:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:46}] to localhost:27017
2020-10-02T09:44:04.730+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2020-10-02T09:44:04.734+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2020-10-02T09:44:04.735+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2020-10-02T09:44:04.740+02:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2020-10-02T09:44:04.740+02:00 INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2020-10-02T09:44:04.741+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2020-10-02T09:44:04.742+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2020-10-02T09:44:04.748+02:00 INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2020-10-02T09:44:04.756+02:00 INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2020-10-02T09:44:04.759+02:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2020-10-02T09:44:05.173+02:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2020-10-02T09:44:35.074+02:00 INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2020-10-02T09:44:35.075+02:00 INFO [ServiceManagerListener] Services are healthy
2020-10-02T09:44:35.076+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2020-10-02T09:44:35.076+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, GracefulShutdownService [RUNNING]=9, JobSchedulerService [RUNNING]=13, BufferSynchronizerService [RUNNING]=14, OutputSetupService [RUNNING]=17, JournalReader [RUNNING]=21, UrlWhitelistService [RUNNING]=23, ConfigurationEtagService [RUNNING]=23, EtagService [RUNNING]=28, KafkaJournal [RUNNING]=37, MongoDBProcessingStatusRecorderService [RUNNING]=45, LookupTableService [RUNNING]=51, StreamCacheService [RUNNING]=63, PeriodicalsService [RUNNING]=142, JerseyService [RUNNING]=30451}
2020-10-02T09:44:35.081+02:00 INFO [ServerBootstrap] Graylog server up and running.
ubuntu@graylog:~$

mveerdonk · October 2, 2020, 9:57am

I do see incoming TCP packets on port 9000.

buntu@graylog:~$ sudo tcpdump -i ens3 port 9000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
11:56:03.465118 IP 62.58.XX.XX.63036 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 244867207, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0
11:56:03.708636 IP 62.58.XX.XX.63037 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 583754037, win 65280, options [mss 1360,nop,wscale 8,nop,nop,sackOK], length 0

cawfehman · October 5, 2020, 3:02pm

I find the simplest way to troubleshoot situations like this is to comment out your publish and external settings, and with just the http_bind_address set to not localhost, try to connect to it. The other settings when commented out will default to what’s in the bind.

Don’t bind to local host unless you only want to login to the gui from the machine you are running graylog on. Use whatever other IP address you have on the system.

http_bind_address = 172.x.y.z :9000
#http_publish_address =
#http_external_uri =

mveerdonk · October 5, 2020, 3:50pm

thanks, comment out everything but http_bind_address

Server logs looks okay, but still no webgui shown.

2020-10-05T17:48:05.289+02:00 INFO [ServerBootstrap] Graylog server 3.3.6+92fb41e starting up
2020-10-05T17:48:05.290+02:00 INFO [ServerBootstrap] JRE: Oracle Corporation 11.0.8 on Linux 5.4.0-1025-oracle
2020-10-05T17:48:05.290+02:00 INFO [ServerBootstrap] Deployment: deb
2020-10-05T17:48:05.290+02:00 INFO [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2020-10-05T17:48:05.290+02:00 INFO [ServerBootstrap] Arch: amd64
2020-10-05T17:48:05.316+02:00 INFO [PeriodicalsService] Starting 30 periodicals …
2020-10-05T17:48:05.317+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2020-10-05T17:48:05.326+02:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2020-10-05T17:48:05.330+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2020-10-05T17:48:05.330+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2020-10-05T17:48:05.347+02:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2020-10-05T17:48:05.334+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2020-10-05T17:48:05.351+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2020-10-05T17:48:05.351+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2020-10-05T17:48:05.354+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2020-10-05T17:48:05.357+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2020-10-05T17:48:05.362+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2020-10-05T17:48:05.363+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2020-10-05T17:48:05.364+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2020-10-05T17:48:05.366+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2020-10-05T17:48:05.366+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2020-10-05T17:48:05.367+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2020-10-05T17:48:05.368+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2020-10-05T17:48:05.368+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2020-10-05T17:48:05.368+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2020-10-05T17:48:05.373+02:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:41}] to localhost:27017
2020-10-05T17:48:05.373+02:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:43}] to localhost:27017
2020-10-05T17:48:05.374+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2020-10-05T17:48:05.375+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2020-10-05T17:48:05.380+02:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:44}] to localhost:27017
2020-10-05T17:48:05.385+02:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:42}] to localhost:27017
2020-10-05T17:48:05.392+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2020-10-05T17:48:05.402+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2020-10-05T17:48:05.403+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2020-10-05T17:48:05.412+02:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2020-10-05T17:48:05.412+02:00 INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2020-10-05T17:48:05.413+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2020-10-05T17:48:05.415+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2020-10-05T17:48:05.417+02:00 INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2020-10-05T17:48:05.421+02:00 INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2020-10-05T17:48:05.423+02:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2020-10-05T17:48:05.822+02:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2020-10-05T17:48:35.462+02:00 INFO [JerseyService] Started REST API at <127.0.0.1:9000>
2020-10-05T17:48:35.463+02:00 INFO [ServiceManagerListener] Services are healthy
2020-10-05T17:48:35.464+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2020-10-05T17:48:35.464+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=2, InputSetupService [RUNNING]=2, UrlWhitelistService [RUNNING]=12, BufferSynchronizerService [RUNNING]=13, JournalReader [RUNNING]=17, JobSchedulerService [RUNNING]=20, OutputSetupService [RUNNING]=21, ConfigurationEtagService [RUNNING]=31, KafkaJournal [RUNNING]=33, EtagService [RUNNING]=33, MongoDBProcessingStatusRecorderService [RUNNING]=66, StreamCacheService [RUNNING]=88, LookupTableService [RUNNING]=93, PeriodicalsService [RUNNING]=115, JerseyService [RUNNING]=30147}
2020-10-05T17:48:35.469+02:00 INFO [ServerBootstrap] Graylog server up and running.

mveerdonk · October 5, 2020, 3:55pm

when I do a curl command from my private host i get

curl -i http://130.x.y.z:9000
curl: (7) Failed to connect to 130.x.y.z port 9000: Connection refused

But I see via tcpdump that the request is hitting the graylog server

mveerdonk · October 5, 2020, 3:57pm

ubuntu@graylog:~$ curl -i http://172.16.0.145:9000/api
curl: (7) Failed to connect to 172.16.0.145 port 9000: Connection refused
ubuntu@graylog:~$ curl -i http://172.16.0.145:9000
curl: (7) Failed to connect to 172.16.0.145 port 9000: Connection refused

cawfehman · October 5, 2020, 4:35pm

I’m not sure which of these 2 attempts would qualify as your personal home IP space, but are either of them the ones you’ve permited through the firewall on the graylog server?

mveerdonk · October 5, 2020, 5:21pm

is external public ip
is local ip address on graylog server. = http_bind_address

Where in the config do i permit this IP address for the FW?

mveerdonk · October 5, 2020, 6:06pm

this is my current server.conf

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = not shown
root_password_sha2 = not shown
root_timezone = Europe/Amsterdam
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 172.16.0.145:9000
elasticsearch_hosts= http://172.16.0.145:9200/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

cawfehman · October 5, 2020, 6:58pm

sorry, you say 131 is your graylog IP, but I see no mention of that anywhere. you have a private address configured, so are you NATing .131 to 172.16.0.145 at the edge firewall?

The firewall I was referring to was the ubuntu firewall running on the graylog server itself.

I believe the command to check it is

ufw status verbose (sudo may be needed)

mveerdonk · October 5, 2020, 7:27pm

correct, I’m NATing.
131. is my public IP, 172. is my local interface IP address.

ubuntu@graylog:~$ sudo ufw status verbose
Status: inactive

cawfehman · October 5, 2020, 7:39pm

ok, so… 2 interfaces on 1 server?

nic1 = 131
nic2 = 172

are you trying to setup graylog to be accessible from 1 or the other or both?

Topic		Replies	Views
Graylog2 on AWS Graylog Central (peer support)	6	1277	October 11, 2017
Virtualizing my Graylog Server for Testing Purposes Graylog Central (peer support)	4	750	May 28, 2019
Graylog server not accessible on ip address Graylog Central (peer support)	6	1704	December 15, 2020
Graylog Web UI Blank using public IP Graylog Central (peer support)	7	2288	February 10, 2020
Graylog VM running on AWS Graylog Central (peer support)	1	493	April 29, 2017

New installation; howto use local and public IP addresses in config files

Related topics