Messages getting to server but nothing showing in input

Graylog Central (peer support)
1

I’ve verified as far as I can that messages from my server are hitting the graylog server, but I’m not seeing that inputs are processing any messages.

The first input stream I worked on did this same thing. After messing with it for three hours (playing with switch ACLs, playing with/reloading firewalls, Picking different port numbers, picking different input types) it just started working.

This server was installed following the instructions for Debian.

graylog server.conf: (any entries not included are at their defaults)

jmoffitt@graylog:/etc/graylog/server$ cat server.conf
############################
# GRAYLOG CONFIGURATION FILE
############################
#
# This is the Graylog configuration file. The file has to use ISO 8859-1/Latin-1 character encoding.
# Characters that cannot be directly represented in this encoding can be written using Unicode escapes
# as defined in https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.3, using the \u prefix.
# For example, \u002c.
#
# * Entries are generally expected to be a single line of the form, one of the following:
#
# propertyName=propertyValue
# propertyName:propertyValue
#
# * White space that appears between the property name and property value is ignored,
#   so the following are equivalent:
#
# name=Stephen
# name = Stephen
#
# * White space at the beginning of the line is also ignored.
#
# * Lines that start with the comment characters ! or # are ignored. Blank lines are also ignored.
#
# * The property value is generally terminated by the end of the line. White space following the
#   property value is not ignored, and is treated as part of the property value.
#
# * A property value can span several lines if each line is terminated by a backslash (‘\’) character.
#   For example:
#
# targetCities=\
#         Detroit,\
#         Chicago,\
#         Los Angeles
#
#   This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).
#
# * The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.
#
# * The backslash character must be escaped as a double backslash. For example:
#
# path=c:\\docs\\doc1
#

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = true

# The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /etc/graylog/server/node-id

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = XXX

# The default root user is named 'admin'
root_username = admin

# You MUST specify a hash password for the root user (which you only need to initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 = XXX

# The email address of the root user.
# Default is empty
root_email = ""

# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
root_timezone = America/Chicago

# Set plugin directory here (relative or absolute)
plugin_dir = /usr/share/graylog-server/plugin

# REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
# When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
rest_listen_uri = http://192.168.216.17:9000/api/

# REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri
# is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
# If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on
# this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
# You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting
# the scheme, host name or URI.
# This must not contain a wildcard address (0.0.0.0).
#rest_transport_uri = XXX

# Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
# This is enabled by default. Uncomment the next line to disable it.
#rest_enable_cors = false

# Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#rest_enable_gzip = false

# Enable HTTPS support for the REST API. This secures the communication with the REST API with
# TLS to prevent request forgery and eavesdropping. This is disabled by default. Uncomment the
# next line to enable it.
#rest_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the REST API.
#rest_tls_cert_file = /path/to/graylog.crt

# The PKCS#8 private key file in PEM format to use for securing the REST API.
#rest_tls_key_file = /path/to/graylog.key

# The password to unlock the private key used for securing the REST API.
#rest_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#rest_max_header_size = 8192

# The size of the thread pool used exclusively for serving the REST API.
#rest_thread_pool_size = 16

# Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For
# header. May be subnets, or hosts.
#trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

# Enable the embedded Graylog web interface.
# Default: true
#web_enable = false

# Web interface listen URI.
# Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement
# for the application.context configuration parameter in pre-2.0 versions of the Graylog web interface.
web_listen_uri = http://192.168.216.17:9000/

# Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
# Default: $rest_transport_uri
#web_endpoint_uri =

# Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
#web_enable_cors = false

# Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#web_enable_gzip = false

# Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface
# using TLS to prevent request forgery and eavesdropping.
# This is disabled by default. Uncomment the next line to enable it and see the other related configuration settings.
#web_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the web interface.
#web_tls_cert_file = /path/to/graylog-web.crt

# The PKCS#8 private key file in PEM format to use for securing the web interface.
#web_tls_key_file = /path/to/graylog-web.key

# The password to unlock the private key used for securing the web interface.
#web_tls_key_password = secret

# The maximum size of the HTTP request headers in bytes.
#web_max_header_size = 8192

# The size of the thread pool used exclusively for serving the web interface.
#web_thread_pool_size = 16

/var/log/graylog/server.log:

2018-11-16T15:48:49.036-06:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.6 [org.graylog.aws.plugin.AWSPlugin]
2018-11-16T15:48:49.039-06:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.6 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-11-16T15:48:49.041-06:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.6 [org.graylog.plugins.cef.CEFInputPlugin]
2018-11-16T15:48:49.042-06:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.4.6 [org.graylog.plugins.collector.CollectorPlugin]
2018-11-16T15:48:49.043-06:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.6 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-11-16T15:48:49.044-06:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.6 [org.graylog.plugins.map.MapWidgetPlugin]
2018-11-16T15:48:49.045-06:00 INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.6 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-11-16T15:48:49.055-06:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.6 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-11-16T15:48:49.056-06:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-11-16T15:48:49.498-06:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-11-16T15:48:49.754-06:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-11-16T15:48:52.611-06:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-11-16T15:48:52.637-06:00 INFO  [NodeId] Node ID: 6d377582-107e-4d5f-a43a-a990af0552ee
2018-11-16T15:48:52.869-06:00 INFO  [LogManager] Loading logs.
2018-11-16T15:48:52.967-06:00 INFO  [LogManager] Logs loading complete.
2018-11-16T15:48:52.968-06:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-11-16T15:48:52.984-06:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2018-11-16T15:48:53.009-06:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2018-11-16T15:48:53.059-06:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-11-16T15:48:53.095-06:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:31}] to localhost:27017
2018-11-16T15:48:53.100-06:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 9]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=2128795}
2018-11-16T15:48:53.108-06:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:32}] to localhost:27017
2018-11-16T15:48:53.627-06:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2018-11-16T15:48:53.628-06:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-11-16T15:48:53.706-06:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2018-11-16T15:48:53.706-06:00 INFO  [JestClientFactory] Node Discovery disabled...
2018-11-16T15:48:53.706-06:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2018-11-16T15:48:54.079-06:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-11-16T15:48:55.860-06:00 INFO  [RulesEngineProvider] No static rules file loaded.
2018-11-16T15:48:56.063-06:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-11-16T15:48:56.072-06:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:33}] to localhost:27017
2018-11-16T15:48:56.077-06:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2018-11-16T15:48:56.102-06:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-11-16T15:48:56.126-06:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-11-16T15:48:56.148-06:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-11-16T15:48:56.176-06:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2018-11-16T15:48:56.518-06:00 INFO  [ServerBootstrap] Graylog server 2.4.6+ceaa7e4 starting up
2018-11-16T15:48:56.519-06:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_181 on Linux 4.9.0-8-amd64
2018-11-16T15:48:56.519-06:00 INFO  [ServerBootstrap] Deployment: deb
2018-11-16T15:48:56.519-06:00 INFO  [ServerBootstrap] OS: Debian GNU/Linux 9 (stretch) (debian)
2018-11-16T15:48:56.519-06:00 INFO  [ServerBootstrap] Arch: amd64
2018-11-16T15:48:56.556-06:00 INFO  [PeriodicalsService] Starting 25 periodicals ...
2018-11-16T15:48:56.556-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-11-16T15:48:56.558-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2018-11-16T15:48:56.558-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-11-16T15:48:56.559-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2018-11-16T15:48:56.559-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2018-11-16T15:48:56.559-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-11-16T15:48:56.560-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2018-11-16T15:48:56.560-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2018-11-16T15:48:56.561-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2018-11-16T15:48:56.568-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-11-16T15:48:56.568-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2018-11-16T15:48:56.569-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-11-16T15:48:56.571-06:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-11-16T15:48:56.576-06:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2018-11-16T15:48:56.581-06:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:34}] to localhost:27017
2018-11-16T15:48:56.585-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2018-11-16T15:48:56.585-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2018-11-16T15:48:56.589-06:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:36}] to localhost:27017
2018-11-16T15:48:56.593-06:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:35}] to localhost:27017
2018-11-16T15:48:56.595-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2018-11-16T15:48:56.597-06:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:37}] to localhost:27017
2018-11-16T15:48:56.612-06:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2018-11-16T15:48:56.613-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2018-11-16T15:48:56.615-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2018-11-16T15:48:56.625-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2018-11-16T15:48:56.632-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-11-16T15:48:56.633-06:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2018-11-16T15:48:56.641-06:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2018-11-16T15:48:56.654-06:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2018-11-16T15:48:56.666-06:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2018-11-16T15:48:56.714-06:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5beaf67be1ff7a1c8f24e39d [@6dc69d86] STARTING
2018-11-16T15:48:56.715-06:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-11-16T15:48:56.733-06:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5beaf67be1ff7a1c8f24e39d [@6dc69d86] RUNNING
2018-11-16T15:48:56.744-06:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-11-16T15:48:56.751-06:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5beaf67be1ff7a1c8f24e3a3 [@44dce06c] STARTING
2018-11-16T15:48:56.752-06:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5beaf67be1ff7a1c8f24e3a3 [@44dce06c] RUNNING
2018-11-16T15:48:56.771-06:00 INFO  [LookupTableService] Data Adapter whois/5beaf67be1ff7a1c8f24e39f [@761d508c] STARTING
2018-11-16T15:48:56.774-06:00 INFO  [LookupTableService] Data Adapter whois/5beaf67be1ff7a1c8f24e39f [@761d508c] RUNNING
2018-11-16T15:48:56.775-06:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5beaf67be1ff7a1c8f24e39e [@4d523d2c] STARTING
2018-11-16T15:48:56.776-06:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5beaf67be1ff7a1c8f24e3a0 [@70e9e4f4] STARTING
2018-11-16T15:48:56.777-06:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5beaf67be1ff7a1c8f24e3a2 [@16297f0a] STARTING
2018-11-16T15:48:56.777-06:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5beaf67be1ff7a1c8f24e3a1 [@1cf50f06] STARTING
2018-11-16T15:48:56.767-06:00 ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/5beaf67be1ff7a1c8f24e3a2/@16297f0a>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
        at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-11-16T15:48:56.768-06:00 ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/5beaf67be1ff7a1c8f24e3a0/@70e9e4f4>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
        at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-11-16T15:48:56.761-06:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/5beaf67be1ff7a1c8f24e3a1/@1cf50f06>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
        at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-11-16T15:48:56.771-06:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/5beaf67be1ff7a1c8f24e39e/@4d523d2c>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
        at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
        at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-11-16T15:48:56.790-06:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5beaf67be1ff7a1c8f24e3a2 [@16297f0a] RUNNING
2018-11-16T15:48:56.794-06:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5beaf67be1ff7a1c8f24e3a1 [@1cf50f06] RUNNING
2018-11-16T15:48:56.795-06:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5beaf67be1ff7a1c8f24e3a0 [@70e9e4f4] RUNNING
2018-11-16T15:48:56.805-06:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5beaf67be1ff7a1c8f24e39e [@4d523d2c] RUNNING
2018-11-16T15:48:56.828-06:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5beaf67be1ff7a1c8f24e397 [@1dcd519f] STARTING
2018-11-16T15:48:56.840-06:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5beaf67be1ff7a1c8f24e397 [@1dcd519f] RUNNING
2018-11-16T15:48:56.847-06:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5beaf67be1ff7a1c8f24e398 [@1f17646] STARTING
2018-11-16T15:48:56.850-06:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5beaf67be1ff7a1c8f24e39b [@16fc84bb] STARTING
2018-11-16T15:48:56.852-06:00 INFO  [LookupTableService] Cache whois-cache/5beaf67be1ff7a1c8f24e399 [@20ec8ef4] STARTING
2018-11-16T15:48:56.853-06:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5beaf67be1ff7a1c8f24e39a [@1e37573c] STARTING
2018-11-16T15:48:56.853-06:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5beaf67be1ff7a1c8f24e39b [@16fc84bb] RUNNING
2018-11-16T15:48:56.854-06:00 INFO  [LookupTableService] Cache whois-cache/5beaf67be1ff7a1c8f24e399 [@20ec8ef4] RUNNING
2018-11-16T15:48:56.854-06:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5beaf67be1ff7a1c8f24e398 [@1f17646] RUNNING
2018-11-16T15:48:56.855-06:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5beaf67be1ff7a1c8f24e39a [@1e37573c] RUNNING
2018-11-16T15:48:56.867-06:00 INFO  [LookupTableService] Starting lookup table otx-api-domain/5beaf67be1ff7a1c8f24e3a5 [@6ac5ddd0] using cache otx-api-domain-cache/5beaf67be1ff7a1c8f24e39a [@1e37573c], data adapter otx-api-domain/5beaf67be1ff7a1c8f24e39d [@6dc69d86]
2018-11-16T15:48:56.867-06:00 INFO  [LookupTableService] Starting lookup table tor-exit-node-list/5beaf67be1ff7a1c8f24e3a6 [@5263ac9c] using cache threat-intel-uncached-adapters/5beaf67be1ff7a1c8f24e397 [@1dcd519f], data adapter tor-exit-node/5beaf67be1ff7a1c8f24e3a0 [@70e9e4f4]
2018-11-16T15:48:56.867-06:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-ip/5beaf67be1ff7a1c8f24e3a7 [@72a0fef9] using cache threat-intel-uncached-adapters/5beaf67be1ff7a1c8f24e397 [@1dcd519f], data adapter abuse-ch-ransomware-ip/5beaf67be1ff7a1c8f24e3a1 [@1cf50f06]
2018-11-16T15:48:56.868-06:00 INFO  [LookupTableService] Starting lookup table whois/5beaf67be1ff7a1c8f24e3a8 [@55d28433] using cache whois-cache/5beaf67be1ff7a1c8f24e399 [@20ec8ef4], data adapter whois/5beaf67be1ff7a1c8f24e39f [@761d508c]
2018-11-16T15:48:56.868-06:00 INFO  [LookupTableService] Starting lookup table spamhaus-drop/5beaf67be1ff7a1c8f24e3a9 [@113de804] using cache spamhaus-e-drop-cache/5beaf67be1ff7a1c8f24e39b [@16fc84bb], data adapter spamhaus-drop/5beaf67be1ff7a1c8f24e3a2 [@16297f0a]
2018-11-16T15:48:56.868-06:00 INFO  [LookupTableService] Starting lookup table otx-api-ip/5beaf67be1ff7a1c8f24e3aa [@5e75a1f3] using cache otx-api-ip-cache/5beaf67be1ff7a1c8f24e398 [@1f17646], data adapter otx-api-ip/5beaf67be1ff7a1c8f24e3a3 [@44dce06c]
2018-11-16T15:48:56.868-06:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-domains/5beaf67be1ff7a1c8f24e3ab [@2bd7cbf2] using cache threat-intel-uncached-adapters/5beaf67be1ff7a1c8f24e397 [@1dcd519f], data adapter abuse-ch-ransomware-domains/5beaf67be1ff7a1c8f24e39e [@4d523d2c]
2018-11-16T15:48:57.107-06:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-11-16T15:49:11.353-06:00 INFO  [NetworkListener] Started listener bound to [192.168.216.17:9000]
2018-11-16T15:49:11.355-06:00 INFO  [HttpServer] [HttpServer] Started.
2018-11-16T15:49:11.355-06:00 INFO  [JerseyService] Started REST API at <http://192.168.216.17:9000/api/>
2018-11-16T15:49:11.356-06:00 INFO  [JerseyService] Started Web Interface at <http://192.168.216.17:9000/>
2018-11-16T15:49:11.357-06:00 INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=22, BufferSynchronizerService [RUNNING]=23, KafkaJournal [RUNNING]=32, InputSetupService [RUNNING]=38, StreamCacheService [RUNNING]=91, ConfigurationEtagService [RUNNING]=91, JournalReader [RUNNING]=92, PeriodicalsService [RUNNING]=121, LookupTableService [RUNNING]=315, JerseyService [RUNNING]=14803}
2018-11-16T15:49:11.359-06:00 INFO  [ServiceManagerListener] Services are healthy
2018-11-16T15:49:11.360-06:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-11-16T15:49:11.363-06:00 INFO  [ServerBootstrap] Graylog server up and running.
2018-11-16T15:49:11.386-06:00 INFO  [InputStateListener] Input [Syslog UDP/5bef2e7ed3cd9902305708a7] is now STARTING
2018-11-16T15:49:11.387-06:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5bef22dbd3cd9907d93e7c96] is now STARTING
2018-11-16T15:49:11.447-06:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=ISP Core Cisco 3850, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=6d377582-107e-4d5f-a43a-a990af0552ee} should be 262144 but is 212992.
2018-11-16T15:49:11.449-06:00 INFO  [InputStateListener] Input [Syslog UDP/5bef2e7ed3cd9902305708a7] is now RUNNING
2018-11-16T15:49:11.451-06:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5bef22dbd3cd9907d93e7c96] is now RUNNING

logs hitting this server

jmoffitt@graylog:/var/log/graylog-server$ sudo tcpdump -i ens160 -n udp port 10004
15:56:38.489845 IP x.x.x.x.45914 > 192.168.216.17.10004: UDP, length 67
15:56:38.491689 IP x.x.x.x.45914 > 192.168.216.17.10004: UDP, length 97
15:56:38.492917 IP x.x.x.x.45914 > 192.168.216.17.10004: UDP, length 90

Netstat:

Proto Recv-Q Send-Q Local Address                 Foreign Address         State       PID/Program name
udp6            0           0 192.168.216.17:10004    :::*                                              3416/java

iptable rule
-A ufw-user-input -s x.x.x.x/32 -d 192.168.216.17/32 -p udp -m udp --dport 10004 -j ACCEPT

Configuration of the input:

image
image.png599×896 44.2 KB

2

He @moffitj

the only option you have is to check all single moving parts. Did you see the messages in the stats of the input, but not in the search? Did you see a test message send with Netcat from localhost? Did you see a test message send from the sending network or switch in the search?

3

I see that messages are hitting the server with tcpdump and the firewall configured to allow them in, but no incrementing counters on the input stats, and of course nothing in search for it either.

Per your response I tried netcat both from localhost and from another server that is in the same range as the server that I am actually trying to send from. To allow it through I expanded the firewall rule to include the entire /24. Both of the test messages went through.

I next tried from the server that I’m trying to export from. Once again I could see the attempt in tcpdump, but not in the input counters.

I broke down the firewall a bit more to include all ports on that /24 and got the same results with a test again.

Is there any other logging that might point to a cause? Graylog failing to parse or something? This is from an ISC DHCP server by the say, forwarded by Rsyslog if that helps.

4

So my one working input I was able to break by doing the following:

turn off input for 10004 (not working, from DHCP server)
Change port for working input to 10004 to see if it could collect messages on that port.
After failure, change port back to 10002, where it was previously.

Input is no longer collecting messages on port 10002 where it was just a few moments prior

5

you could always check the Graylog server.log if you see parsing errors.

6

Well, I’ve resolved this, but I still don’t know how.

I disabled UFW and set iptables to default allow all, and no change.
I flushed all rules and chains put into iptables by UFW and no change.
At this point my iptables -S was as follows and messages were still not getting through:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

I rebooted the VM, and now messages were successfully getting through. I tried re-enabling UFW, and messages stopped. I attempted ufw reset to clear all of the UFW settings that I entered and messages were blocked.

At this point I was at default iptables and default UFW rules and chains with everything set to default allow, but these UDP messages weren’t getting through. This is the part that I don’t understand.

To resolve this I disabled UFW and used iptables directly after a reboot. Messages are now getting through as expected.

7

Try to run tcpdump with -AAAA parameter, and check the content.
First try it with plain text input format.
I’m not sure you can send all necessary syslog fields in 90 characters.
How do you try to search the messages? Eg. if you have problem with the message format maybe the graylog puts the source ip or another field in the source field.

You also can try loggen (part of Balabit’s syslog-ng) try to send a lot of messages, and check the in and out numbers at the top of graylog web interface.

8

Thanks for your response Macko.

The messages may very well not be to the sysog standard. They are from isc-dhcp-server so I imagine they would be though.

In any case, the issue ended up being something with UFW. When I disabled UFW completely and entered the lines into IPtables directly, everything works fine.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.