Greylog does not show results (elasticsearch manual query works)

antonio.cucca · February 20, 2018, 11:24am

Hi all!
I configured a fairly standard installation of greylog2, using:

Elastic search 5.6.7-1
Mongo 3.2.19-1
Graylog-server 2.4.3-1

Rsyslog listens on *:514 and forwards all logs to localhost:1514.
I configured an Input (UDP Syslog) listening on localhost:1514.
When I send the logs, from a remote host, using:

logger -d -n IP_ADDRESS “pizza”

I see the logs arriving in elasticsearch:

curl -XGET 'http://localhost:9200/_search?q=pizza

This shows all logs.
The counters on the web interfaces (messages/s) show the icoming logs.
But when I query them through the web interface, I do not see anything, both using “keyword”, “relative”, “absolute”…

Tried to enable logging in elasticsearch (setting slow queries to 0s) but no luck.

Any idea?

jochen · February 20, 2018, 11:57am

Try searching a few hours in the past or the future (using an absolute time range).
This sounds like a typical timezone problem with log sources which don’t provide any timezone information in their timestamps.

antonio.cucca · February 20, 2018, 12:35pm

Hello,
we tried but no results appear from the interface.

We have also tried the time zone but the result does not change, from the interface the logs are not displayed.

jochen · February 20, 2018, 12:39pm

What’s the complete configuration of rsyslog?
What’s the configuration of the Syslog UDP input in Graylog?
What’s the complete output of the following cURL command?

# curl 'http://localhost:9200/_search?q=pizza&pretty'

antonio.cucca · February 20, 2018, 2:09pm

Below is our configuration of the rsyslog
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
.emerg :omusrmsg:

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin … end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
# ### end of the forwarding rule ###

and the Syslog UDP

\llow_override_date:
\true
\bind_address:
\ 0.0.0.0
\expand_structured_data:
\false
\force_rdns:
\ false
\override_source:
\
\port:
\1514
\recv_buffer_size:
\262144
\store_full_message:
\true

and the CURL output

{
\ “took” : 5,
“timed_out” : false,
"_shards" : {
\ “total” : 4,
“successful” : 4,
“skipped” : 0,
“failed” : 0
\ },
\ “hits” : {
\ “total” : 616,
\ “max_score” : 1.981038,
\ “hits” : [
\ {
\ “_index” : “graylog_0”,
\ “_type” : “message”,
\ “_id” : “382a55e0-1630-11e8-a1be-00505698db98”,
\ “_score” : 1.981038,
\ “_source” : {
\ “level” : -1,
\ “gl2_remote_ip” : “192.168.10.6”,
\ “gl2_remote_port” : 37277,
“streams” : [
\ “000000000000000000000001”
],
\ “source” : “192.168.10.6”,
\ “message” : “1 2018-02-20T12:21:38.752719+01:00 darkung d3k - pizza”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T12:21:38.752719+01:00 darkung d3k - pizza”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 11:21:39.132”
\ }
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e054f8b0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:33.426892+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:33.426892+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:33.785”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e10ed2d0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:34.645194+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:34.645194+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:35.003”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “d7a51d40-162a-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:43:09.272198+01:00 darkung d3k - ciao tony w la pizza 11:24”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:43:09.272198+01:00 darkung d3k - ciao tony w la pizza 11:24”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:43:09.650”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “db075b10-162a-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:43:15.023662+01:00 darkung d3k - ciao tony w la pizza 11:43”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:43:15.023662+01:00 darkung d3k - ciao tony w la pizza 11:43”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:43:15.390”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e33d72a0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:38.305789+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:38.305789+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:38.664”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e39aadd0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:38.916563+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:38.916563+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:39.275”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e4267310-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:39.832770+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:39.832770+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.191”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e4552430-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:40.138345+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:40.138345+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.496”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e483d550-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.802”
}
}
]
}
}
\
thanks

jochen · February 20, 2018, 2:17pm

I don’t see anything for sending the syslog messages to Graylog on port 1514/udp. Maybe that part is in /etc/rsyslog.d/?

antonio.cucca · February 20, 2018, 2:33pm

sorry, I was wrong to copy I forgot the last line below you can find the correct config

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
.emerg :omusrmsg:

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin … end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
# ### end of the forwarding rule ###

*.* @@127.0.0.1:1514;RSYSLOG_SyslogProtocol23Format

jochen · February 20, 2018, 2:37pm

You can properly format your configuration snippets by surrounding them with triple back ticks:
http://commonmark.org/help/

Example:

```
Some text
More text
```

jochen · February 20, 2018, 2:38pm

That’s using TCP to send messages to Graylog, not UDP.

Please refer to the syslog guide for more details:

github.com

Graylog2/graylog-guide-syslog-linux/blob/master/README.md#rsyslog

## Sending syslog from Linux systems into Graylog

The two most popular syslog deamons (the programs that run in the background to accept and write or forward logs) are [rsyslog](https://www.rsyslog.com/) and [syslog-ng](https://syslog-ng.com/). One of these will most likely be running on your Linux distribution.

Please refer to the documentation of your distribution if you are not sure about this.

### ⚠️ Warning ⚠️

**These instructions configure rsyslog and syslog-ng to send log messages _unencrypted_ over the network. This is generally not recommended on public networks.**

### rsyslog

Forwarding syslog messages with rsyslog is easy. The only important thing to get the most out of your logs is following
[RFC 5424](http://www.ietf.org/rfc/rfc5424.txt). The following examples configure your `rsyslog` daemon to send
RFC 5424 date to Graylog syslog inputs:

##### UDP:

    *.* @graylog.example.org:514;RSYSLOG_SyslogProtocol23Format

This file has been truncated. show original

antonio.cucca · February 20, 2018, 2:48pm

I modified the config file as per your suggestion but in the web interface I do not see any log, I see the log only if I do the query from curl.

jochen · February 20, 2018, 3:08pm

Are you seeing any messages in Graylog?
What’s in the logs of your Graylog and Elasticsearch nodes?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

antonio.cucca · February 20, 2018, 3:59pm

I would say the screenshots below

and the result of curl

{
        "_index" : "graylog_0",
        "_type" : "message",
        "_id" : "e483d550-162b-11e8-a1be-00505698db98",
        "_score" : 8.580914,
        "_source" : {
          "level" : -1,
          "gl2_remote_ip" : "192.168.10.6",
          "gl2_remote_port" : 37277,
          "streams" : [
            "000000000000000000000001"
          ],
          "source" : "192.168.10.6",
          "message" : "1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop",
          "gl2_source_input" : "5a8bed77419a0a03c4f3c629",
          "full_message" : "1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop",
          "gl2_source_node" : "ce0492ba-71eb-4274-ad22-2bd9588b521f",
          "facility" : "Unknown",
          "timestamp" : "2018-02-20 10:50:40.802"
        }

the graylo and elasticsearch logs are clean

jochen · February 20, 2018, 5:00pm

The “message” and “full_message” fields, as well as “level” and “facility” look very strange and not like they’ve been received via Syslog input.

Please upload them. Maybe you’ve missed something.

antonio.cucca · February 20, 2018, 5:05pm

below the elasticsearch logs

[2018-02-20T09:37:10,337][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T09:37:10,469][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T09:37:10,469][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] version[5.6.7], pid[967], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T09:37:12,265][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T09:37:15,596][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T09:37:16,872][INFO ][o.e.n.Node               ] initialized
[2018-02-20T09:37:16,872][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T09:37:17,115][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T09:37:20,293][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{1PwGCWa0RSGOqdZftB51Ww}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T09:37:20,463][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T09:37:20,463][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T09:37:20,664][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T09:37:21,243][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2], [graylog_0][0]] ...]).
[2018-02-20T09:46:13,954][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T09:46:14,044][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T09:46:14,044][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T09:46:14,053][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T09:50:28,422][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T09:50:28,582][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T09:50:28,582][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] version[5.6.7], pid[968], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T09:50:30,276][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T09:50:33,446][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T09:50:34,827][INFO ][o.e.n.Node               ] initialized
[2018-02-20T09:50:34,827][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T09:50:35,142][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T09:50:38,365][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{mZEpANQMRKSgU-yFM4mSSA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T09:50:38,576][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T09:50:38,577][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T09:50:38,878][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T09:50:39,302][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][1]] ...]).
[2018-02-20T10:30:04,110][INFO ][o.e.c.m.MetaDataMappingService] [Nz277Vr] [graylog_0/pAvJV2pTQvaOPZ2FbU6uSA] update_mapping [message]
[2018-02-20T10:58:31,842][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T10:58:31,988][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T10:58:31,989][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T10:58:32,008][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T10:59:23,269][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T10:59:23,409][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T10:59:23,410][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T10:59:23,429][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T10:59:23,429][INFO ][o.e.n.Node               ] version[5.6.7], pid[969], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T10:59:23,430][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T10:59:27,807][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T10:59:28,907][INFO ][o.e.n.Node               ] initialized
[2018-02-20T10:59:28,907][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T10:59:29,186][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T10:59:32,297][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{_GtnUbL5RByOgvuZyQ-Ozw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T10:59:32,464][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T10:59:32,464][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T10:59:32,754][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T10:59:33,255][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2], [graylog_0][0]] ...]).
[2018-02-20T15:34:42,613][INFO ][o.e.c.m.MetaDataMappingService] [Nz277Vr] [graylog_0/pAvJV2pTQvaOPZ2FbU6uSA] update_mapping [message]
[2018-02-20T15:44:15,739][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T15:44:16,017][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T15:44:16,017][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T15:44:16,032][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T15:45:07,562][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T15:45:07,680][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [47.7gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T15:45:07,680][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] version[5.6.7], pid[971], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T15:45:09,576][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T15:45:09,576][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T15:45:12,512][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T15:45:13,610][INFO ][o.e.n.Node               ] initialized
[2018-02-20T15:45:13,611][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T15:45:13,786][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T15:45:16,969][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{OaO0NtGmTqKeDZ0fHoGcuQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T15:45:17,070][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T15:45:17,071][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T15:45:17,387][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T15:45:17,956][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][2], [graylog_0][3], [graylog_0][0]] ...]).

antonio.cucca · February 27, 2018, 11:19am

Hi everyone,
Any News?

Amine-elhijazi · February 27, 2018, 11:48am

what’s the output of

rsyslogd -N1

jochen · February 27, 2018, 12:34pm

Not until you provide the information we’ve asked for.

Also, please use a pastebin-like service for sharing configuration and log files (e. g. https://0bin.net/ or https://gist.github.com/) or at least properly format your posts: Markdown Reference

Example:

```
Some text...
More text...
```

jebucha · March 1, 2018, 7:35pm

This thought may be way off the mark, but if you look at System -> Indices -> (Default index set, or name of the index set you are searching), are there any indices for which there is no calculated index (time) range? I use Curator to close older indices as a means of managing heap usage on my Elasticsearch nodes, and forgot to run a “recalculate index ranges” when I reopened some of the closed indices for one of my users. Once I did so they were able to query the time ranges/indices they needed.

system · March 15, 2018, 7:35pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logs not showing in web ui Graylog Central (peer support)	9	13809	May 4, 2018
Ubuntu 16.04 Rsyslog syslogs to Graylog 2.4 not showing in GUI Graylog Central (peer support)	6	784	May 17, 2018
New user needing some help Graylog Central (peer support)	5	836	February 11, 2019
Missing Data in Graylog, but in ElasticSearch Graylog Central (peer support)	4	672	May 20, 2020
Search is empty even elasticsearch has data in index Graylog Central (peer support)	10	4167	November 6, 2018

Greylog does not show results (elasticsearch manual query works)

Related topics