Graylog search question

[nickgeo (Nick Geovanis)]

Hi -
I have a search query result which I am unable to explain, perhaps due to ignorance
The platform is Debian 8.5, Graylog is version 2.1.1. I am querying for log messages produced
by Salt minions, specifically records which contain the text (enclosed here in double-quotes): “salt.loaded.int.module.cmdmod”.
If I search for that entire string (see below), I retrieve 12 results and each contains that string.
But if I search for “salt.loaded.int.module.cmdmo” (dropping the final “d”), I retrieve 12 results,
NONE OF WHICH contain the string “salt.loaded.int.module.cmdmo”. In fact the results
returned contain no part of that string whatsoever. Why so?

Here are the invocations, grepping for the string “salt” in results:

root@graylog01:/var/log/graylog-server# curl -XGET 'localhost:9200/graylog*/_search?q="salt.loaded.int.module.cmdmo"?&pretty' | grep salt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12013  100 12013    0     0  14337      0 --:--:-- --:--:-- --:--:-- 14335
root@graylog01:/var/log/graylog-server# curl -XGET 'localhost:9200/graylog*/_search?q="salt.loaded.int.module.cmdmod"?&pretty' | grep salt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11526  100 11526    0     0  13435      0 --:--:-- --:--:-- --:--:-- 13433
        "msg" : "2017-08-09 09:55:17,244 [salt.loaded.int.module.cmdmod            ][ERROR   ][4942] Command 'ls -l /mail' failed with return code: 2",
        "msg" : "2017-08-09 09:55:17,245 [salt.loaded.int.module.cmdmod            ][ERROR   ][4942] output: ls: cannot access /mail: No such file or directory",
        "msg" : "2017-08-09 14:38:52,416 [salt.loaded.int.module.cmdmod            ][ERROR   ][13276] Command 'ls -l /etc/redhat-release' failed with return code: 2",
        "msg" : "2017-08-09 11:44:18,186 [salt.loaded.int.module.cmdmod            ][ERROR   ][8027] output: ls: cannot access /etc/redhat-release: No such file or directory",
        "msg" : "2017-08-09 14:38:52,417 [salt.loaded.int.module.cmdmod            ][ERROR   ][13276] output: ls: cannot access /etc/redhat-release: No such file or directory",
        "msg" : "2017-08-09 14:48:20,160 [salt.loaded.int.module.cmdmod            ][ERROR   ][13394] output: ls: cannot access /etc/debian-version: No such file or directory",
        "msg" : "2017-08-10 10:39:08,921 [salt.loaded.int.module.cmdmod            ][ERROR   ][25643] Command 'ls -l /hoopla' failed with return code: 2",
        "msg" : "2017-08-09 09:54:52,516 [salt.loaded.int.module.cmdmod            ][ERROR   ][4918] Command 'ls -l /var/log/mail' failed with return code: 2",
        "msg" : "2017-08-09 09:54:52,517 [salt.loaded.int.module.cmdmod            ][ERROR   ][4918] output: ls: cannot access /var/log/mail: No such file or directory",
        "msg" : "2017-08-09 11:44:18,185 [salt.loaded.int.module.cmdmod            ][ERROR   ][8027] Command 'ls -l /etc/redhat-release' failed with return code: 2",
root@graylog01:/var/log/graylog-server#

[jochen]

@nickgeo

But if I search for “salt.loaded.int.module.cmdmo” (dropping the final “d”), I retrieve 12 results,

NONE OF WHICH contain the string “salt.loaded.int.module.cmdmo”

What messages did you receive with that search query?

[nickgeo (Nick Geovanis)]

Here are the results from that search:

root@graylog01:/var/log# curl -XGET 'localhost:9200/graylog*/_search?q="salt.loaded.int.module.cmdmo"?&pretty'
{
  "took" : 1076,
  "timed_out" : false,
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "failed" : 0
  },
  "hits" : {
    "total" : 43943913,
    "max_score" : 0.033698954,
    "hits" : [ {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e73d972-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:13 portaonelab04 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/ntp.pl --offset_c 1 --hosts pool.ntp.org",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:14.000",
        "Location" : "(portaonelab04.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e73d974-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:13 portaonelab04 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/disk_health.pl -skip_bbu=N -skip_wcp=N -temp_w=40 -temp_c=45 -pending_w=0 -pending_c=0 -wearout_w=85 -wearout_c=95 -error_w=0 -error_c=20 -other_error_w=0 -other_error_c=20",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:14.000",
        "Location" : "(portaonelab04.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e742797-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:14 portaonelab01 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/watchdog.pl -service=cron,syslog,sshd,chrony -mail=1 -sip=0 -log=0 -xdrm=0",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ "58012b0db2ab7c7cacd2f466" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:15.000",
        "Location" : "(portaonelab01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e742793-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:15 poconfig01 sshd[48236]: Accepted publickey for porta-one from 10.59.10.19 port 39650 ssh2: DSA 8f:b3:11:8a:e0:6b:54:30:bc:8a:4d:ce:67:35:3d:30",
        "device_product" : "OSSEC HIDS",
        "src" : "10.59.10.19",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5715, LOW] SSHD authentication success.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "porta-one",
        "event_class_id" : "5715",
        "severity_number" : 3,
        "name" : "SSHD authentication success.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:15.000",
        "Location" : "(poconfig01.chi5.prlss.net)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e744ea4-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:14 portaonelab01 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/disk_health.pl -skip_bbu=N -skip_wcp=N -temp_w=40 -temp_c=45 -pending_w=0 -pending_c=0 -wearout_w=85 -wearout_c=95 -error_w=0 -error_c=20 -other_error_w=0 -other_error_c=20",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ "58012b0db2ab7c7cacd2f466" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:15.000",
        "Location" : "(portaonelab01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e744ea5-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:14 portaonelab01 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/home/porta-billing/scripts/disconnector_monitor.pl -delay_c=10 -delay_w=5 -pending_c=100 -pending_w=50",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ "58012b0db2ab7c7cacd2f466" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:15.000",
        "Location" : "(portaonelab01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e7475b0-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:15 portaonelab02 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/watchdog.pl -service=cron,syslog,sshd,chrony,task-queue,converter-queue,g729-converter-queue,billsoft,kairosdb,gearmand,pum-mwid,pum-mfd,dbmail-imapd,gearman_worker -mail=1 -sip=1 -log=0 -xdrm=1",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:16.000",
        "Location" : "(portaonelab02.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e7475b1-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:14 portaonelab03 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/netif.pl -errors_w=0.1% -errors_c=1% -exclude=none -ignore_drops=none",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:15.000",
        "Location" : "(portaonelab03.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e7475b6-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:15 portaonelab02 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/libexec/nagios/portaone/ntp.pl --offset_c 1 --hosts pool.ntp.org",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:16.000",
        "Location" : "(portaonelab02.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "9e74c3d5-4877-11e7-89eb-005056bb66df",
      "_score" : 0.033698954,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "Jun  3 16:16:15 portaonelab02 sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/home/porta-configurator/bin/crontabs-mon.pl",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 35119,
        "streams" : [ ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [5402, LOW] Successful sudo to ROOT executed",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "suser" : "nagios",
        "event_class_id" : "5402",
        "severity_number" : 3,
        "name" : "Successful sudo to ROOT executed",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-06-03 16:16:16.000",
        "Location" : "(portaonelab02.chi5.prlss.int)"
      }
    } ]
  }
}

[jochen]

This is the magic of analyzers and tokenizers.

The dot character (.) serves as a stop character in the tokenizer used by Graylog by default, so the query “salt.loaded.int.module.cmdmo” is actually for “salt”, “loaded”, “int”, “module”, and “cmdmo”.
The same is true for all fields in the messages (unless you have configured a custom Elasticsearch index mapping with a custom analyzer or tokenizer setting), so the “Location” field contains not a single string, but multiple terms (example: “(portaonelab04.chi5.prlss.int)” → “portaonelab04”, “chi5”, “prlss”, and “int”).

As you can see, the query will thus find the “int” inside the “Location” field.

If you want to change this behavior, you’ll have to create a custom Elasticsearch index mapping with custom analyzer settings.

Check out the following references for more details:

• https://www.elastic.co/guide/en/elasticsearch/guide/current/analysis-intro.html
• https://www.elastic.co/guide/en/elasticsearch/guide/current/configuring-analyzers.html
• https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-analyzers.html
• https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-tokenizers.html
• http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings

[nickgeo (Nick Geovanis)]

Thanks for your reply. But in light of it, I do not understand why this query produces no results…
curl -XGET ‘localhost:9200/_search?q=“cmdmod”&pretty&results=10000’
…while the following query finds records containing “cmdmod”…
curl -XGET 'localhost:9200/_search?q=“salt.loaded.int.module.cmdmod”&pretty&results=10000’
The second result set shows that the string “cmdmod” definitely exists in the data, but
curl -XGET 'localhost:9200/_search?q=“cmdmod”&pretty&results=10000’
does not find it. Why not?

root@graylog01:~# curl -XGET 'localhost:9200/_search?q="cmdmod"&pretty&results=10000'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}root@graylog01:~# curl -XGET 'localhost:9200/_search?q="salt.loaded.int.module.cmdmod"&pretty&results=10000'
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "failed" : 0
  },
  "hits" : {
    "total" : 12,
    "max_score" : 1.608919,
    "hits" : [ {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "be61c019-7d12-11e7-bc6e-005056bb66df",
      "_score" : 1.608919,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 09:55:17,244 [salt.loaded.int.module.cmdmod            ][ERROR   ][4942] Command 'ls -l /mail' failed with return code: 2",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 14:55:22.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "be61c01a-7d12-11e7-bc6e-005056bb66df",
      "_score" : 1.5450578,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 09:55:17,245 [salt.loaded.int.module.cmdmod            ][ERROR   ][4942] output: ls: cannot access /mail: No such file or directory",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 14:55:22.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "5b76ffb2-7d3a-11e7-bc6e-005056bb66df",
      "_score" : 1.5006579,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 14:38:52,416 [salt.loaded.int.module.cmdmod            ][ERROR   ][13276] Command 'ls -l /etc/redhat-release' failed with return code: 2",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 19:38:56.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "f9de07c6-7d21-11e7-bc6e-005056bb66df",
      "_score" : 1.5006579,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 11:44:18,186 [salt.loaded.int.module.cmdmod            ][ERROR   ][8027] output: ls: cannot access /etc/redhat-release: No such file or directory",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 16:44:23.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "5b7726c0-7d3a-11e7-bc6e-005056bb66df",
      "_score" : 1.4691975,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 14:38:52,417 [salt.loaded.int.module.cmdmod            ][ERROR   ][13276] output: ls: cannot access /etc/redhat-release: No such file or directory",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 19:38:56.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "af95c4e0-7d3b-11e7-bc6e-005056bb66df",
      "_score" : 1.4691975,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 14:48:20,160 [salt.loaded.int.module.cmdmod            ][ERROR   ][13394] output: ls: cannot access /etc/debian-version: No such file or directory",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2", "598b66b8b2ab7c088bae6d85" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 19:48:26.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "08265422-7de2-11e7-bc6e-005056bb66df",
      "_score" : 1.4691975,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-10 10:39:08,921 [salt.loaded.int.module.cmdmod            ][ERROR   ][25643] Command 'ls -l /hoopla' failed with return code: 2",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2", "598b66b8b2ab7c088bae6d85" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-10 15:39:13.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "af76a0c0-7d12-11e7-bc6e-005056bb66df",
      "_score" : 1.4078041,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 09:54:52,516 [salt.loaded.int.module.cmdmod            ][ERROR   ][4918] Command 'ls -l /var/log/mail' failed with return code: 2",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 14:54:58.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "af76a0c1-7d12-11e7-bc6e-005056bb66df",
      "_score" : 1.4078041,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 09:54:52,517 [salt.loaded.int.module.cmdmod            ][ERROR   ][4918] output: ls: cannot access /var/log/mail: No such file or directory",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 14:54:58.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    }, {
      "_index" : "graylog_2",
      "_type" : "message",
      "_id" : "f9de07c0-7d21-11e7-bc6e-005056bb66df",
      "_score" : 1.4078041,
      "_source" : {
        "device_version" : "v2.8",
        "severity" : "LOW",
        "msg" : "2017-08-09 11:44:18,185 [salt.loaded.int.module.cmdmod            ][ERROR   ][8027] Command 'ls -l /etc/redhat-release' failed with return code: 2",
        "device_product" : "OSSEC HIDS",
        "gl2_remote_ip" : "172.16.159.203",
        "gl2_remote_port" : 58541,
        "streams" : [ "5988a130b2ab7c088bab83f2" ],
        "source" : "ossec01",
        "message" : "OSSEC HIDS: [1002, LOW] Unknown problem somewhere in the system.",
        "gl2_source_input" : "57fbc518b2ab7c425214314d",
        "dvc" : "ossec01",
        "event_class_id" : "1002",
        "severity_number" : 2,
        "name" : "Unknown problem somewhere in the system.",
        "gl2_source_node" : "06c5aaf1-30e9-4546-9dbd-474a752bf2dd",
        "device_vendor" : "Trend Micro Inc.",
        "timestamp" : "2017-08-09 16:44:23.000",
        "Location" : "(anistage-proxy01.chi5.prlss.int)"
      }
    } ]
  }
}
root@graylog01:~#

[jochen]

Please properly format your text snippets. This will make them much easier to read: http://commonmark.org/help/

Example:

```
Text
```