Webgui not reacting via port 9000

Graylog Central (peer support)
1

I’m using a cloud VM via public IP address.
All processes are up and running. I do get inbound TCP request on my VM via port 9000.
Howeven my graylog webgui isn’t shown.

Any ideas what the issue can be?

This is a part of my graylog conf file

http_bind_address = 127.0.0.1:9000

Default: http://$http_bind_address/

#http_publish_uri = http://172.16.0.145:9000/

Default: $http_publish_uri

#http_external_uri = 172.16.0.145/
http_external_uri = http://130.61.140.229:9000/
rest_listen_uri = http://172.16.0.145:9000/api/

elasticsearch_hosts= http://172.16.0.145:9200/

ubuntu@graylog:~$ sudo tcpdump -i ens3 port 9000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
16:36:48.657833 IP 84-85-28-128.fixed.kpn.net.49218 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 996530733, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3538838243 ecr 0,sackOK,eol], length 0
16:36:48.677747 IP 84-85-28-128.fixed.kpn.net.49219 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 3100284998, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3538838262 ecr 0,sackOK,eol], length 0

2

You have configured Graylog to listen on the 127.0.0.1 loop-back address.

Do you have anything listening on TCP/9000 on a reachable IP address that proxies the requests through to 127.0.0.1:9000?

3

This in my current (non-working) config.
I can access this server only via its public IP address (a.b.c.d) which get NATted to its local 172.x.y.z address.

What should be the right config to gets this working??

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =
root_password_sha2 =
root_timezone = Europe/Amsterdam
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 127.0.0.1:9000
web_enable = true
elasticsearch_hosts= http://172.16.0.145:9200/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

4

You need to http_bind_address to the ip of your server to make it reachable for outside.

5

changed it into my external/public IP.

I see traffic entering the server via tcpdump, but nut webgui is shown.
How can I trace/verify if the mapping between external IP and internal IP is working well?

output of tcpdump on eth interface:

sudo tcpdump -i ens3 > capture3.log
8:37:16.923662 IP graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.ssh > 84-85-28-128.fixed.kpn.net.57629: Flags [P.], seq 185956404:185956592, ack 3788520686, win 466, options [nop,nop,TS val 2989230078 ecr 3$
08:37:16.941650 IP 84-85-28-128.fixed.kpn.net.57629 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.ssh: Flags [.], ack 188, win 2045, options [nop,nop,TS val 3876241763 ecr 2989230078], length 0
08:37:17.926939 IP graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.32831 > 169.254.169.254.domain: 14055+ [1au] PTR? 128.28.85.84.in-addr.arpa. (54)
08:37:17.927277 IP 169.254.169.254.domain > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.32831: 14055 1/0/1 PTR 84-85-28-128.fixed.kpn.net. (94)
08:37:17.927616 IP graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.35680 > 169.254.169.254.domain: 44138+ [1au] PTR? 145.0.16.172.in-addr.arpa. (54)
08:37:17.929152 IP 169.254.169.254.domain > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.35680: 44138 1/0/1 PTR graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com. (124)
08:37:20.067468 IP 84-85-28-128.fixed.kpn.net.57641 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 1442887202, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3876244857 ecr 0,$
08:37:20.067514 IP graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com > 84-85-28-128.fixed.kpn.net: ICMP host graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com unreachable - admin prohibited, length 72
08:37:20.085897 IP 84-85-28-128.fixed.kpn.net.57642 > graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com.9000: Flags [S], seq 3076196636, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 3876244874 ecr 0,$
08:37:20.085919 IP graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com > 84-85-28-128.fixed.kpn.net: ICMP host graylog-759158.esbcmanagement.esbcnetworks.oraclevcn.com unreachable - admin prohibited, length 72

6

What is the output of sudo netstat -lntp and ip a ?

8

ubuntu@graylog:~$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 774/rpcbind
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 941/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1274/sshd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1143/mongod
tcp6 0 0 :::111 :::* LISTEN 774/rpcbind
tcp6 0 0 172.16.0.145:9200 :::* LISTEN 14534/java
tcp6 0 0 172.16.0.145:9300 :::* LISTEN 14534/java
tcp6 0 0 :::22 :::* LISTEN 1274/sshd

ubuntu@graylog:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 02:00:17:00:f0:ba brd ff:ff:ff:ff:ff:ff
inet 172.16.0.145/27 brd 172.16.0.159 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe00:f0ba/64 scope link
valid_lft forever preferred_lft forever
ubuntu@graylog:~$

9

Nothing is listening on TCP/9000.

Have you set the http_bind_address to 172.16.0.145 or your external IP?
It should be set to the IP address of your ens3 interface.

10

Was set to external IP address (was advise of others). Now changed it back to IP address of ens3.

After restarting graylog service, still the same result.

ubuntu@graylog:~$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 774/rpcbind
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 941/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1274/sshd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1143/mongod
tcp6 0 0 :::111 :::* LISTEN 774/rpcbind
tcp6 0 0 172.16.0.145:9200 :::* LISTEN 14534/java
tcp6 0 0 172.16.0.145:9300 :::* LISTEN 14534/java
tcp6 0 0 :::22 :::* LISTEN 1274/sshd

11

Check the graylog-server log file for errors.
Should be at: /var/log/graylog-server/server.log

12

I don’t have any errors in this log.

2020-10-07T08:13:59.842+02:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:107}] to localhost:27017
2020-10-07T11:13:57.181+02:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 3.3.6 [org.graylog.aws.AWSPlugin]
2020-10-07T11:13:57.184+02:00 INFO [CmdLineTool] Loaded plugin: Integrations 3.3.6 [org.graylog.integrations.IntegrationsPlugin]
2020-10-07T11:13:57.186+02:00 INFO [CmdLineTool] Loaded plugin: Collector 3.3.6 [org.graylog.plugins.collector.CollectorPlugin]
2020-10-07T11:13:57.187+02:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.3.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2020-10-07T11:13:57.423+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2020-10-07T11:13:57.680+02:00 INFO [Version] HV000001: Hibernate Validator null
2020-10-07T11:14:00.233+02:00 INFO [InputBufferImpl] Message journal is enabled.
2020-10-07T11:14:00.251+02:00 INFO [NodeId] Node ID: df24e902-b125-456e-9969-261dcdf0fe24
2020-10-07T11:14:00.435+02:00 INFO [LogManager] Loading logs.
2020-10-07T11:14:00.503+02:00 INFO [LogManager] Logs loading complete.
2020-10-07T11:14:00.507+02:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2020-10-07T11:14:00.531+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2020-10-07T11:14:00.571+02:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2020-10-07T11:14:00.591+02:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:108}] to localhost:27017
2020-10-07T11:14:00.597+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 9]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2813540}
2020-10-07T11:14:00.609+02:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:109}] to localhost:27017
2020-10-07T11:14:00.805+02:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2020-10-07T11:14:01.024+02:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://172.16.0.145:9200/]
2020-10-07T11:14:01.024+02:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2020-10-07T11:14:01.088+02:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2020-10-07T11:14:01.089+02:00 INFO [JestClientFactory] Node Discovery disabled…
2020-10-07T11:14:01.089+02:00 INFO [JestClientFactory] Idle connection reaping disabled…
2020-10-07T11:14:01.483+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-07T11:14:01.585+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-07T11:14:01.602+02:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2020-10-07T11:14:01.609+02:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:110}] to localhost:27017
2020-10-07T11:14:01.648+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-07T11:14:01.679+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-07T11:14:01.703+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-07T11:14:01.731+02:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2020-10-07T11:14:02.209+02:00 INFO [ServerBootstrap] Graylog server 3.3.6+92fb41e starting up
2020-10-07T11:14:02.209+02:00 INFO [ServerBootstrap] JRE: Oracle Corporation 11.0.8 on Linux 5.4.0-1025-oracle
2020-10-07T11:14:02.210+02:00 INFO [ServerBootstrap] Deployment: deb
2020-10-07T11:14:02.210+02:00 INFO [ServerBootstrap] OS: Ubuntu 18.04.5 LTS (bionic)
2020-10-07T11:14:02.211+02:00 INFO [ServerBootstrap] Arch: amd64
2020-10-07T11:14:02.238+02:00 INFO [PeriodicalsService] Starting 30 periodicals …
2020-10-07T11:14:02.238+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2020-10-07T11:14:02.245+02:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2020-10-07T11:14:02.251+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2020-10-07T11:14:02.253+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2020-10-07T11:14:02.254+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2020-10-07T11:14:02.255+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2020-10-07T11:14:02.256+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2020-10-07T11:14:02.270+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2020-10-07T11:14:02.271+02:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2020-10-07T11:14:02.273+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2020-10-07T11:14:02.275+02:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:111}] to localhost:27017
2020-10-07T11:14:02.278+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2020-10-07T11:14:02.282+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2020-10-07T11:14:02.283+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2020-10-07T11:14:02.284+02:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:112}] to localhost:27017
2020-10-07T11:14:02.287+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2020-10-07T11:14:02.289+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2020-10-07T11:14:02.290+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2020-10-07T11:14:02.291+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2020-10-07T11:14:02.291+02:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:113}] to localhost:27017
2020-10-07T11:14:02.292+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2020-10-07T11:14:02.299+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2020-10-07T11:14:02.307+02:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:114}] to localhost:27017
2020-10-07T11:14:02.312+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2020-10-07T11:14:02.312+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2020-10-07T11:14:02.317+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2020-10-07T11:14:02.327+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2020-10-07T11:14:02.329+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2020-10-07T11:14:02.330+02:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2020-10-07T11:14:02.334+02:00 INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2020-10-07T11:14:02.337+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2020-10-07T11:14:02.345+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2020-10-07T11:14:02.348+02:00 INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2020-10-07T11:14:02.350+02:00 INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2020-10-07T11:14:02.354+02:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2020-10-07T11:14:02.788+02:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2020-10-07T11:14:32.589+02:00 INFO [JerseyService] Started REST API at <172.16.0.145:9000>
2020-10-07T11:14:32.590+02:00 INFO [ServiceManagerListener] Services are healthy
2020-10-07T11:14:32.591+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2020-10-07T11:14:32.591+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=2, InputSetupService [RUNNING]=3, JournalReader [RUNNING]=15, JobSchedulerService [RUNNING]=17, OutputSetupService [RUNNING]=18, BufferSynchronizerService [RUNNING]=19, EtagService [RUNNING]=20, ConfigurationEtagService [RUNNING]=24, UrlWhitelistService [RUNNING]=30, KafkaJournal [RUNNING]=39, LookupTableService [RUNNING]=52, MongoDBProcessingStatusRecorderService [RUNNING]=67, StreamCacheService [RUNNING]=72, PeriodicalsService [RUNNING]=119, JerseyService [RUNNING]=30352}
2020-10-07T11:14:32.597+02:00 INFO [ServerBootstrap] Graylog server up and running.
ubuntu@graylog:~$

13

OK, check to see if the service is listening on the port now.
Maybe you just checked before the service had started up fully.

14

now port 9000 is listing

ubuntu@graylog:~$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 774/rpcbind
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 941/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1274/sshd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1143/mongod
tcp6 0 0 :::111 :::* LISTEN 774/rpcbind
tcp6 0 0 172.16.0.145:9200 :::* LISTEN 14534/java
tcp6 0 0 172.16.0.145:9300 :::* LISTEN 14534/java
tcp6 0 0 :::22 :::* LISTEN 1274/sshd
tcp6 0 0 172.16.0.145:9000 :::* LISTEN 3053/java

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.