New Graylog Install - Inputs Not Generating Any Messages


(Steven) #1

Hi, I have just installed Graylog for the first time and am having difficulty getting Inputs to generate any messages. I am focusing on Rsyslog TCP at the moment. Just want to see anything get generated. Here is my config below.

Graylog:
Graylog v2.3.2+3df951e

server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = 9Cmqo6p0QVB0YgC9NHDEBKqjjhNGDgv9sEeBuHvogC7NbTBi782mPWlgpUyyig1Jp4byKWYNw0en3oRIoxUhPEMe6803E3yG
root_username = xxxxxxxx
root_password_sha2 = xxxxxxx
root_timezone = -07:00
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://162.70.27.206:9000/api/
web_listen_uri = http://162.70.27.206:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32

Linux:
Red Hat Enterprise Linux Server release 7.0 (Maipo)

Java:
java version "1.8.0_151"
Java™ SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot™ 64-Bit Server VM (build 25.151-b12, mixed mode)

Elasticsearch:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 426

{
“name” : “9mn9imw”,
“cluster_name” : “graylog”,
“cluster_uuid” : “vNWjHm-mTZOW249O-ZR-cw”,
“version” : {
“number” : “6.1.0”,
“build_hash” : “c0c1ba0”,
“build_date” : “2017-12-12T12:32:54.550Z”,
“build_snapshot” : false,
“lucene_version” : “7.1.0”,
“minimum_wire_compatibility_version” : “5.6.0”,
“minimum_index_compatibility_version” : “5.0.0”
},
“tagline” : “You Know, for Search”
}

/etc/elasticsearch/elasticsearch.yml
cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0

Mongodb:
db version v3.6.0
git version: a57d8e71e6998a2d0afde7edc11bd23e5661c915
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64

Graylog Server rsyslog.conf:
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.
/var/log/secure
mail.
-/var/log/maillog
cron.* /var/log/cron
.emerg :omusrmsg:
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
. @@127.0.0.1:5140

Target Server rsyslog.d graylog.conf
. @@graylog.ams.com:514;RSYSLOG_SyslogProtocol23Format

Graylog Input Properties:
Editing Input sat62

Global
Should this input start on all nodes
Node
On which node should this input start
Title
sat62

Bind address
127.0.0.1
Address to listen on. For example 0.0.0.0 or 127.0.0.1.

Port
1514
Port to listen on.

Receive Buffer Size(optional)
1048576
The size in bytes of the recvBufferSize for network connections to this input.

TLS cert file(optional)
Path to the TLS certificate file

TLS private key file(optional)
Path to the TLS private key file

Enable TLS(optional)
Accept TLS connections

TLS key password(optional)
The password for the encrypted key file.

TLS client authentication(optional)
Whether clients need to authenticate themselves in a TLS connection

Expand structured data?(optional)
Expand structured data elements by prefixing attributes with their SD-ID?

TCP keepalive(optional)
Enable TCP keepalive packets

Null frame delimiter?(optional)
Use null byte as frame delimiter? Otherwise newline delimiter is used.

Maximum message size(optional)
2097152
The maximum length of a message.

Override source(optional)
The source is a hostname derived from the received packet by default. Set this if you want to override it with a custom string.

Force rDNS?(optional)
Force rDNS resolution of hostname? Use if hostname cannot be parsed. (Be careful if you are sending DNS logs into this input because it can cause a feedback loop.)

Allow overriding date?(optional)
Allow to override with current date if date could not be parsed?

Store full message?(optional)
Store the full original syslog message as full_message?

TLS Client Auth Trusted Certs(optional)
TLS Client Auth Trusted Certs (File or Directory)

Graylog Server Netstat Output:
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN
tcp6 0 0 :::514 :::* LISTEN
tcp6 0 0 127.0.0.1:1514 :::* LISTEN
tcp6 0 0 127.0.0.1:5140 :::* LISTEN
udp 0 0 0.0.0.0:514 0.0.0.0:*
udp6 0 0 :::514 :::*
udp6 0 0 127.0.0.1:5140 :::*


(Jan Doberstein) #2

Graylog is not yet compatible with Elasticsearch 6.


(Steven) #3

Hi Thank you for the reply. I have removed elasticsearch version 6 and installed version 5. Services have been restarted and are running. The issue still persists - no messages appear from Inputs.
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: active (running) since Tue 2017-12-19 10:10:53 MST; 25s ago
Docs: http://www.elastic.co
Process: 28109 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 28111 (java)
CGroup: /system.slice/elasticsearch.service
└─28111 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -serv…

Dec 19 10:10:53 graylog.ams.com systemd[1]: Starting Elasticsearch…
Dec 19 10:10:53 graylog.ams.com systemd[1]: Started Elasticsearch.

graylog.ams.com>>/etc/elasticsearch>>curl -i http://127.0.0.1:9200/?pretty
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 321

{
“name” : “9mn9imw”,
“cluster_name” : “graylog”,
“cluster_uuid” : “vNWjHm-mTZOW249O-ZR-cw”,
“version” : {
“number” : “5.6.5”,
“build_hash” : “6a37571”,
“build_date” : “2017-12-04T07:50:10.466Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.1”
},
“tagline” : “You Know, for Search”
}

cluster.name: graylog


(Steven) #4

Elasticsearch log does not seem to contain anything alarming:
[2017-12-19T10:10:53,092][INFO ][o.e.n.Node ] [9mn9imw] stopping …
[2017-12-19T10:10:53,136][INFO ][o.e.n.Node ] [9mn9imw] stopped
[2017-12-19T10:10:53,137][INFO ][o.e.n.Node ] [9mn9imw] closing …
[2017-12-19T10:10:53,159][INFO ][o.e.n.Node ] [9mn9imw] closed
[2017-12-19T10:10:55,898][INFO ][o.e.n.Node ] [] initializing …
[2017-12-19T10:10:56,014][INFO ][o.e.e.NodeEnvironment ] [9mn9imw] using [1] data paths, mounts [[/var (/dev/mapper/rhel-var)]], net usable_space [881mb], net total_space [1.9gb], spins? [possibly], types [xfs]
[2017-12-19T10:10:56,014][INFO ][o.e.e.NodeEnvironment ] [9mn9imw] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-12-19T10:10:56,031][INFO ][o.e.n.Node ] node name [9mn9imw] derived from node ID [9mn9imwSQ2uQ-kQ3NK3Pcw]; set [node.name] to override
[2017-12-19T10:10:56,031][INFO ][o.e.n.Node ] version[5.6.5], pid[28111], build[6a37571/2017-12-04T07:50:10.466Z], OS[Linux/3.10.0-123.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot™ 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-12-19T10:10:56,031][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [aggs-matrix-stats]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [ingest-common]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [lang-expression]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [lang-groovy]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [lang-mustache]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [lang-painless]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [parent-join]
[2017-12-19T10:10:57,408][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [percolator]
[2017-12-19T10:10:57,409][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [reindex]
[2017-12-19T10:10:57,409][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [transport-netty3]
[2017-12-19T10:10:57,409][INFO ][o.e.p.PluginsService ] [9mn9imw] loaded module [transport-netty4]
[2017-12-19T10:10:57,409][INFO ][o.e.p.PluginsService ] [9mn9imw] no plugins loaded
[2017-12-19T10:10:59,698][INFO ][o.e.d.DiscoveryModule ] [9mn9imw] using discovery type [zen]
[2017-12-19T10:11:00,912][INFO ][o.e.n.Node ] initialized
[2017-12-19T10:11:00,912][INFO ][o.e.n.Node ] [9mn9imw] starting …
[2017-12-19T10:11:01,380][INFO ][o.e.t.TransportService ] [9mn9imw] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2017-12-19T10:11:04,587][INFO ][o.e.c.s.ClusterService ] [9mn9imw] new_master {9mn9imw}{9mn9imwSQ2uQ-kQ3NK3Pcw}{WVZj1Z9bRQWeWW46wpS39A}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-12-19T10:11:04,664][INFO ][o.e.h.n.Netty4HttpServerTransport] [9mn9imw] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2017-12-19T10:11:04,665][INFO ][o.e.n.Node ] [9mn9imw] started
[2017-12-19T10:11:04,922][INFO ][o.e.g.GatewayService ] [9mn9imw] recovered [1] indices into cluster_state

However the graylog-server/server.log contains the following:
2017-12-19T11:00:57.359-07:00 ERROR [Messages] Failed to index [4] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2017-12-19T11:01:08.377-07:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-12-19T11:01:15.360-07:00 ERROR [Messages] Failed to index [8] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.


(Jan Doberstein) #5

[2017-12-19T10:10:56,014][INFO ][o.e.e.NodeEnvironment ] [9mn9imw] using [1] data paths, mounts [[/var (/dev/mapper/rhel-var)]], net usable_space [881mb], net total_space [1.9gb], spins? [possibly], types [xfs]

I think that you will find high watermark warnings in your elasticsearch logfiles. Additional your Clusterstatus should be not green …

Fix your Elasticsearch and everything will work.


(Steven) #6

Hi, thanks for the info. I have added about 10G to /var and i still see the net usable_space entry in the elacticsearch/graylog.log in red - is that not enough? Also the GET / does not show the expected result. Could you please clarify on what else I need to do specifically to fix elasticsearch?


(Steven) #7

Hi, I reverted elastic search down to version 2.4.6 and now Input messages are coming through. Thanks for your help!


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.