Greylog does not show results (elasticsearch manual query works)


(Antonio) #1

Hi all!
I configured a fairly standard installation of greylog2, using:

  • Elastic search 5.6.7-1
  • Mongo 3.2.19-1
  • Graylog-server 2.4.3-1

Rsyslog listens on *:514 and forwards all logs to localhost:1514.
I configured an Input (UDP Syslog) listening on localhost:1514.
When I send the logs, from a remote host, using:

logger -d -n IP_ADDRESS “pizza”

I see the logs arriving in elasticsearch:

curl -XGET 'http://localhost:9200/_search?q=pizza

This shows all logs.
The counters on the web interfaces (messages/s) show the icoming logs.
But when I query them through the web interface, I do not see anything, both using “keyword”, “relative”, “absolute”…

Tried to enable logging in elasticsearch (setting slow queries to 0s) but no luck.

Any idea?


(Jochen) #2

Try searching a few hours in the past or the future (using an absolute time range).
This sounds like a typical timezone problem with log sources which don’t provide any timezone information in their timestamps.


(Antonio) #3

Hello,
we tried but no results appear from the interface.

We have also tried the time zone but the result does not change, from the interface the logs are not displayed.


(Jochen) #4

What’s the complete configuration of rsyslog?
What’s the configuration of the Syslog UDP input in Graylog?
What’s the complete output of the following cURL command?

# curl 'http://localhost:9200/_search?q=pizza&pretty'

(Antonio) #5

Below is our configuration of the rsyslog
# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
.emerg :omusrmsg:

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin … end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
# ### end of the forwarding rule ###

and the Syslog UDP

\llow_override_date:
\true
\bind_address:
\ 0.0.0.0
\expand_structured_data:
\false
\force_rdns:
\ false
\override_source:
\
\port:
\1514
\recv_buffer_size:
\262144
\store_full_message:
\true

and the CURL output

{
\ “took” : 5,
“timed_out” : false,
"_shards" : {
\ “total” : 4,
“successful” : 4,
“skipped” : 0,
“failed” : 0
\ },
\ “hits” : {
\ “total” : 616,
\ “max_score” : 1.981038,
\ “hits” : [
\ {
\ “_index” : “graylog_0”,
\ “_type” : “message”,
\ “_id” : “382a55e0-1630-11e8-a1be-00505698db98”,
\ “_score” : 1.981038,
\ “_source” : {
\ “level” : -1,
\ “gl2_remote_ip” : “192.168.10.6”,
\ “gl2_remote_port” : 37277,
“streams” : [
\ “000000000000000000000001”
],
\ “source” : “192.168.10.6”,
\ “message” : “1 2018-02-20T12:21:38.752719+01:00 darkung d3k - pizza”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T12:21:38.752719+01:00 darkung d3k - pizza”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 11:21:39.132”
\ }
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e054f8b0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:33.426892+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:33.426892+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:33.785”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e10ed2d0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:34.645194+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:34.645194+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:35.003”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “d7a51d40-162a-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:43:09.272198+01:00 darkung d3k - ciao tony w la pizza 11:24”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:43:09.272198+01:00 darkung d3k - ciao tony w la pizza 11:24”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:43:09.650”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “db075b10-162a-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:43:15.023662+01:00 darkung d3k - ciao tony w la pizza 11:43”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:43:15.023662+01:00 darkung d3k - ciao tony w la pizza 11:43”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:43:15.390”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e33d72a0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:38.305789+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:38.305789+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:38.664”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e39aadd0-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:38.916563+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:38.916563+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:39.275”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e4267310-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:39.832770+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:39.832770+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.191”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e4552430-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:40.138345+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:40.138345+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.496”
}
},
{
"_index" : “graylog_0”,
"_type" : “message”,
"_id" : “e483d550-162b-11e8-a1be-00505698db98”,
"_score" : 1.9421341,
"_source" : {
“level” : -1,
“gl2_remote_ip” : “192.168.10.6”,
“gl2_remote_port” : 37277,
“streams” : [
“000000000000000000000001”
],
“source” : “192.168.10.6”,
“message” : “1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_input” : “5a8bed77419a0a03c4f3c629”,
“full_message” : “1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop”,
“gl2_source_node” : “ce0492ba-71eb-4274-ad22-2bd9588b521f”,
“facility” : “Unknown”,
“timestamp” : “2018-02-20 10:50:40.802”
}
}
]
}
}
\
thanks


(Jochen) #6

I don’t see anything for sending the syslog messages to Graylog on port 1514/udp. Maybe that part is in /etc/rsyslog.d/?


(Antonio) #7

sorry, I was wrong to copy I forgot the last line below you can find the correct config

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog

# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
.emerg :omusrmsg:

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin … end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#. @@remote-host:514
# ### end of the forwarding rule ###

*.* @@127.0.0.1:1514;RSYSLOG_SyslogProtocol23Format


(Jochen) #8

You can properly format your configuration snippets by surrounding them with triple back ticks:
http://commonmark.org/help/

Example:

```
Some text
More text
```

(Jochen) #9

That’s using TCP to send messages to Graylog, not UDP.

Please refer to the syslog guide for more details:


(Antonio) #10

I modified the config file as per your suggestion but in the web interface I do not see any log, I see the log only if I do the query from curl.


(Jochen) #11

Are you seeing any messages in Graylog?
What’s in the logs of your Graylog and Elasticsearch nodes?
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Antonio) #12

I would say the screenshots below

and the result of curl

{
        "_index" : "graylog_0",
        "_type" : "message",
        "_id" : "e483d550-162b-11e8-a1be-00505698db98",
        "_score" : 8.580914,
        "_source" : {
          "level" : -1,
          "gl2_remote_ip" : "192.168.10.6",
          "gl2_remote_port" : 37277,
          "streams" : [
            "000000000000000000000001"
          ],
          "source" : "192.168.10.6",
          "message" : "1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop",
          "gl2_source_input" : "5a8bed77419a0a03c4f3c629",
          "full_message" : "1 2018-02-20T11:50:40.444390+01:00 darkung d3k - ciao tony w la pizza loop",
          "gl2_source_node" : "ce0492ba-71eb-4274-ad22-2bd9588b521f",
          "facility" : "Unknown",
          "timestamp" : "2018-02-20 10:50:40.802"
        }

the graylo and elasticsearch logs are clean


(Jochen) #13

The “message” and “full_message” fields, as well as “level” and “facility” look very strange and not like they’ve been received via Syslog input.

Please upload them. Maybe you’ve missed something.


(Antonio) #14

below the elasticsearch logs

[2018-02-20T09:37:10,337][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T09:37:10,469][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T09:37:10,469][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] version[5.6.7], pid[967], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T09:37:10,511][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T09:37:12,264][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T09:37:12,265][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T09:37:15,596][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T09:37:16,872][INFO ][o.e.n.Node               ] initialized
[2018-02-20T09:37:16,872][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T09:37:17,115][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T09:37:20,293][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{1PwGCWa0RSGOqdZftB51Ww}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T09:37:20,463][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T09:37:20,463][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T09:37:20,664][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T09:37:21,243][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2], [graylog_0][0]] ...]).
[2018-02-20T09:46:13,954][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T09:46:14,044][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T09:46:14,044][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T09:46:14,053][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T09:50:28,422][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T09:50:28,582][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T09:50:28,582][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] version[5.6.7], pid[968], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T09:50:28,609][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T09:50:30,274][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T09:50:30,275][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T09:50:30,276][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T09:50:33,446][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T09:50:34,827][INFO ][o.e.n.Node               ] initialized
[2018-02-20T09:50:34,827][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T09:50:35,142][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T09:50:38,365][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{mZEpANQMRKSgU-yFM4mSSA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T09:50:38,576][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T09:50:38,577][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T09:50:38,878][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T09:50:39,302][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][1]] ...]).
[2018-02-20T10:30:04,110][INFO ][o.e.c.m.MetaDataMappingService] [Nz277Vr] [graylog_0/pAvJV2pTQvaOPZ2FbU6uSA] update_mapping [message]
[2018-02-20T10:58:31,842][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T10:58:31,988][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T10:58:31,989][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T10:58:32,008][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T10:59:23,269][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T10:59:23,409][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [48gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T10:59:23,410][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T10:59:23,429][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T10:59:23,429][INFO ][o.e.n.Node               ] version[5.6.7], pid[969], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T10:59:23,430][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T10:59:25,130][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T10:59:25,131][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T10:59:27,807][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T10:59:28,907][INFO ][o.e.n.Node               ] initialized
[2018-02-20T10:59:28,907][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T10:59:29,186][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T10:59:32,297][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{_GtnUbL5RByOgvuZyQ-Ozw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T10:59:32,464][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T10:59:32,464][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T10:59:32,754][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T10:59:33,255][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2], [graylog_0][0]] ...]).
[2018-02-20T15:34:42,613][INFO ][o.e.c.m.MetaDataMappingService] [Nz277Vr] [graylog_0/pAvJV2pTQvaOPZ2FbU6uSA] update_mapping [message]
[2018-02-20T15:44:15,739][INFO ][o.e.n.Node               ] [Nz277Vr] stopping ...
[2018-02-20T15:44:16,017][INFO ][o.e.n.Node               ] [Nz277Vr] stopped
[2018-02-20T15:44:16,017][INFO ][o.e.n.Node               ] [Nz277Vr] closing ...
[2018-02-20T15:44:16,032][INFO ][o.e.n.Node               ] [Nz277Vr] closed
[2018-02-20T15:45:07,562][INFO ][o.e.n.Node               ] [] initializing ...
[2018-02-20T15:45:07,680][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [47.7gb], net total_space [49.9gb], spins? [unknown], types [rootfs]
[2018-02-20T15:45:07,680][INFO ][o.e.e.NodeEnvironment    ] [Nz277Vr] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] node name [Nz277Vr] derived from node ID [Nz277VrOSg67aQ_WCdvd_w]; set [node.name] to override
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] version[5.6.7], pid[971], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T15:45:07,703][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [aggs-matrix-stats]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [ingest-common]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-expression]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-groovy]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-mustache]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [lang-painless]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [parent-join]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [percolator]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [reindex]
[2018-02-20T15:45:09,575][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty3]
[2018-02-20T15:45:09,576][INFO ][o.e.p.PluginsService     ] [Nz277Vr] loaded module [transport-netty4]
[2018-02-20T15:45:09,576][INFO ][o.e.p.PluginsService     ] [Nz277Vr] no plugins loaded
[2018-02-20T15:45:12,512][INFO ][o.e.d.DiscoveryModule    ] [Nz277Vr] using discovery type [zen]
[2018-02-20T15:45:13,610][INFO ][o.e.n.Node               ] initialized
[2018-02-20T15:45:13,611][INFO ][o.e.n.Node               ] [Nz277Vr] starting ...
[2018-02-20T15:45:13,786][INFO ][o.e.t.TransportService   ] [Nz277Vr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-02-20T15:45:16,969][INFO ][o.e.c.s.ClusterService   ] [Nz277Vr] new_master {Nz277Vr}{Nz277VrOSg67aQ_WCdvd_w}{OaO0NtGmTqKeDZ0fHoGcuQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-20T15:45:17,070][INFO ][o.e.h.n.Netty4HttpServerTransport] [Nz277Vr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-02-20T15:45:17,071][INFO ][o.e.n.Node               ] [Nz277Vr] started
[2018-02-20T15:45:17,387][INFO ][o.e.g.GatewayService     ] [Nz277Vr] recovered [1] indices into cluster_state
[2018-02-20T15:45:17,956][INFO ][o.e.c.r.a.AllocationService] [Nz277Vr] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][2], [graylog_0][3], [graylog_0][0]] ...]).


(Antonio) #15

Hi everyone,
Any News?


(EL HIJAZI) #16

what’s the output of

rsyslogd -N1


(Jochen) #17

Not until you provide the information we’ve asked for.

Also, please use a pastebin-like service for sharing configuration and log files (e. g. https://0bin.net/ or https://gist.github.com/) or at least properly format your posts: http://commonmark.org/help/

Example:

```
Some text...
More text...
```

(John Buchanan) #18

This thought may be way off the mark, but if you look at System -> Indices -> (Default index set, or name of the index set you are searching), are there any indices for which there is no calculated index (time) range? I use Curator to close older indices as a means of managing heap usage on my Elasticsearch nodes, and forgot to run a “recalculate index ranges” when I reopened some of the closed indices for one of my users. Once I did so they were able to query the time ranges/indices they needed.


(system) #19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.