After import of extractor no fields show in stream

Graylog Central (peer support)
1

hello,

I imported graylog extractor in a few imputs. one is this ;

{
  "extractors": [
    {
      "title": "SSO_Server_IP_Address",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Server_IP_Address",
      "extractor_config": {
        "regex_value": "^.*SERVER IP ADDRESS: (.+)#012="
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Application",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Application",
      "extractor_config": {
        "regex_value": "^.*#012APPLICATION: (.+)#012WHEN"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Client_IP_Address",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Client_IP_Address",
      "extractor_config": {
        "regex_value": "^.*CLIENT IP ADDRESS: (.+)#012SERVER"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Action",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Action",
      "extractor_config": {
        "regex_value": "^.*ACTION:(.+)#012APPLICATION:"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Supplied_Credentials",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Supplied_Credentials",
      "extractor_config": {
        "regex_value": "^.*Supplied credentials: \\[(.+)\\]#012ACTION"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_WHO",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "full_message",
      "target_field": "SSO_WHO",
      "extractor_config": {
        "regex_value": "^.*WHO:(.+)#012WHAT:"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "2.4.3"
}

it says the extractors were imported successfully but I don’t see any fields in stream ? Is this a bug ?

2

I also installed the extractors manually but the fields don’t show up in the streams.

3

2018-03-30T12:23:29.752-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:24:54.004-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:25:59.279-04:00 INFO [ExtractorsResource] Added extractor <07c189d0-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:27:31.477-04:00 INFO [ExtractorsResource] Added extractor <3eb5fd40-3437-11e8-8681-005056b7d9d7> of type [split_and_index] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:29:04.166-04:00 INFO [ExtractorsResource] Added extractor <75f50e40-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:29:55.665-04:00 INFO [ExtractorsResource] Added extractor <94a72df0-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:30:44.763-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:32:17.073-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:33:10.589-04:00 INFO [ExtractorsResource] Added extractor <08d611a0-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:34:17.736-04:00 INFO [ExtractorsResource] Added extractor <30dc0a60-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:35:18.613-04:00 INFO [ExtractorsResource] Added extractor <55252230-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:36:08.128-04:00 INFO [ExtractorsResource] Added extractor <72a8acf0-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:37:01.172-04:00 INFO [ExtractorsResource] Added extractor <92466520-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:38:53.279-04:00 INFO [ExtractorsResource] Added extractor of type [grok] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:39:45.104-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:41:08.403-04:00 INFO [ExtractorsResource] Added extractor <25a2da10-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:42:24.112-04:00 INFO [ExtractorsResource] Added extractor <52c347f0-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:43:09.211-04:00 INFO [ExtractorsResource] Added extractor <6da4b090-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:43:58.948-04:00 INFO [ExtractorsResource] Added extractor <8b4a1b30-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:44:43.186-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.

4

Hej @unixinthebox

without knowing what you ingest nobody is able to help you.

Did you check if the messages are that kind of messages that are expected by the creator of the pattern?

5

Hello Graylog Guru,

Attached are the logs and configuration files. Hope that works. Thank you !

-Bob

Thanks,

Bob Gizynski
Technology Services
© The School District of Philadelphia
440 N. Broad Street, Philadelphia, PA 19130
cell : (215) 279-0787

Love what you do. Love where you do it. Apply to teach in Philly today!

6

Nope, no attachment here.

7

server.log

2018-04-08T13:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T14:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T15:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T16:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T17:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T18:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T19:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T20:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T21:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T22:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T23:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T00:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T01:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T02:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T03:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T04:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T05:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T06:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T07:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T07:35:20.027-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2018-04-09T08:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
mac-D25MD0F7F8JC:Graylog rgizynski$

graylog.log

[2018-04-06T14:27:45,703][INFO ][o.e.n.Node ] [qDUPSsw] stopping …
[2018-04-06T14:27:45,726][INFO ][o.e.n.Node ] [qDUPSsw] stopped
[2018-04-06T14:27:45,726][INFO ][o.e.n.Node ] [qDUPSsw] closing …
[2018-04-06T14:27:45,736][INFO ][o.e.n.Node ] [qDUPSsw] closed
[2018-04-06T14:27:56,078][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna183854902005335039.tmp: /tmp/jna–1985354563/jna183854902005335039.tmp: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at com.sun.jna.Native.(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-04-06T14:27:56,085][WARN ][o.e.b.Natives ] cannot check if running as root because JNA is not available
[2018-04-06T14:27:56,085][WARN ][o.e.b.Natives ] cannot register console handler because JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-04-06T14:27:56,303][INFO ][o.e.n.Node ] [] initializing …
[2018-04-06T14:27:56,572][INFO ][o.e.e.NodeEnvironment ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [746.2mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-04-06T14:27:56,573][INFO ][o.e.e.NodeEnvironment ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] version[5.6.8], pid[16240], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-06T14:27:58,627][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-04-06T14:27:58,627][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [ingest-common]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-expression]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-groovy]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-mustache]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-painless]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [parent-join]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [percolator]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [reindex]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [transport-netty3]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [transport-netty4]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] no plugins loaded
[2018-04-06T14:28:02,717][INFO ][o.e.d.DiscoveryModule ] [qDUPSsw] using discovery type [zen]
[2018-04-06T14:28:04,322][INFO ][o.e.n.Node ] initialized
[2018-04-06T14:28:04,322][INFO ][o.e.n.Node ] [qDUPSsw] starting …
[2018-04-06T14:28:04,633][INFO ][o.e.t.TransportService ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-04-06T14:28:04,645][INFO ][o.e.b.BootstrapChecks ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-06T14:28:08,147][INFO ][o.e.c.s.ClusterService ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300}, added {{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300},{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{48a_TUrCRfm3RY1WkE6vcA}{graylog2-mgmt.philasd.net}{172.16.5.13:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300} committed version [68]])
[2018-04-06T14:28:08,257][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-04-06T14:28:08,258][INFO ][o.e.n.Node ] [qDUPSsw] started
mac-D25MD0F7F8JC:Graylog rgizynski$

8

rsyslog.conf

local5.* @@172.16.5.11:5519

input(type=“imfile”
File="/var/log/audit/audit.log"
Tag=“linux_auditd”
Facility=“local5”
Severity=“info”
readMode=“1”)

9

i personal would say that the linux auditlog does not match the regex values you try to get out.

10

here is one of the extractors

graylog1 audit_log type=CRYPTO_KEY_USER msg=audit(1523364260.738:117483): pid=25895 uid=0 auid=4294967295 ses=4294967295 msg=‘op=destroy kind=server fp=SHA256:15:08:c2:82:dc:fa:13:bd:86:d7:f4:06:12:9a:e7:bc:5c:11:ae:24:aa:04:13:fc:8a:e7:9c:ee:65:80:67:30 direction=? spid=25895 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’

regular expression -> ^.*

for HOSTNAME

11

sorry @unixinthebox I do net get want the intention of your last message is.

as you can clearly see the regex that is included in the first presented extractor rules does not match the logline of a audit.log on linux. You can’t squeeze apple juice out of a lemon …

12

I think that regex ^.* should be “graylog1”

13

what would ^.* return in this case ? shouldn’t it return the first word which is graylog1 ?

14

The regular expression ^.* matches everything from the beginning of the line.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.