After import of extractor no fields show in stream


(Bob Gizynski) #1

hello,

I imported graylog extractor in a few imputs. one is this ;

{
  "extractors": [
    {
      "title": "SSO_Server_IP_Address",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Server_IP_Address",
      "extractor_config": {
        "regex_value": "^.*SERVER IP ADDRESS: (.+)#012="
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Application",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Application",
      "extractor_config": {
        "regex_value": "^.*#012APPLICATION: (.+)#012WHEN"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Client_IP_Address",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Client_IP_Address",
      "extractor_config": {
        "regex_value": "^.*CLIENT IP ADDRESS: (.+)#012SERVER"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Action",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Action",
      "extractor_config": {
        "regex_value": "^.*ACTION:(.+)#012APPLICATION:"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_Supplied_Credentials",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "SSO_Supplied_Credentials",
      "extractor_config": {
        "regex_value": "^.*Supplied credentials: \\[(.+)\\]#012ACTION"
      },
      "condition_type": "none",
      "condition_value": ""
    },
    {
      "title": "SSO_WHO",
      "extractor_type": "regex",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "full_message",
      "target_field": "SSO_WHO",
      "extractor_config": {
        "regex_value": "^.*WHO:(.+)#012WHAT:"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "2.4.3"
}

it says the extractors were imported successfully but I don’t see any fields in stream ? Is this a bug ?


(Bob Gizynski) #2

I also installed the extractors manually but the fields don’t show up in the streams.


(Bob Gizynski) #3

2018-03-30T12:23:29.752-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:24:54.004-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:25:59.279-04:00 INFO [ExtractorsResource] Added extractor <07c189d0-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:27:31.477-04:00 INFO [ExtractorsResource] Added extractor <3eb5fd40-3437-11e8-8681-005056b7d9d7> of type [split_and_index] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:29:04.166-04:00 INFO [ExtractorsResource] Added extractor <75f50e40-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:29:55.665-04:00 INFO [ExtractorsResource] Added extractor <94a72df0-3437-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:30:44.763-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:32:17.073-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:33:10.589-04:00 INFO [ExtractorsResource] Added extractor <08d611a0-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:34:17.736-04:00 INFO [ExtractorsResource] Added extractor <30dc0a60-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:35:18.613-04:00 INFO [ExtractorsResource] Added extractor <55252230-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:36:08.128-04:00 INFO [ExtractorsResource] Added extractor <72a8acf0-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:37:01.172-04:00 INFO [ExtractorsResource] Added extractor <92466520-3438-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:38:53.279-04:00 INFO [ExtractorsResource] Added extractor of type [grok] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:39:45.104-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:41:08.403-04:00 INFO [ExtractorsResource] Added extractor <25a2da10-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:42:24.112-04:00 INFO [ExtractorsResource] Added extractor <52c347f0-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:43:09.211-04:00 INFO [ExtractorsResource] Added extractor <6da4b090-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:43:58.948-04:00 INFO [ExtractorsResource] Added extractor <8b4a1b30-3439-11e8-8681-005056b7d9d7> of type [regex] to input <5abe2cc24f617d71f6002cc4>.
2018-03-30T12:44:43.186-04:00 INFO [ExtractorsResource] Added extractor of type [regex] to input <5abe2cc24f617d71f6002cc4>.


When I add an extractor successfully. the field doesn't show in streams all fields
(Jan Doberstein) #4

Hej @unixinthebox

without knowing what you ingest nobody is able to help you.

Did you check if the messages are that kind of messages that are expected by the creator of the pattern?


(Bob Gizynski) #5

Hello Graylog Guru,

Attached are the logs and configuration files. Hope that works. Thank you !

-Bob

Thanks,

Bob Gizynski
Technology Services
© The School District of Philadelphia
440 N. Broad Street, Philadelphia, PA 19130
cell : (215) 279-0787

Love what you do. Love where you do it. Apply to teach in Philly today!


(Jochen) #6

Nope, no attachment here.


(Bob Gizynski) #7

server.log

2018-04-08T13:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T14:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T15:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T16:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T17:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T18:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T19:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T20:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T21:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T22:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-08T23:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T00:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T01:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T02:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T03:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T04:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T05:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T06:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T07:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2018-04-09T07:35:20.027-04:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2018-04-09T08:28:12.239-04:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
mac-D25MD0F7F8JC:Graylog rgizynski$

graylog.log

[2018-04-06T14:27:45,703][INFO ][o.e.n.Node ] [qDUPSsw] stopping …
[2018-04-06T14:27:45,726][INFO ][o.e.n.Node ] [qDUPSsw] stopped
[2018-04-06T14:27:45,726][INFO ][o.e.n.Node ] [qDUPSsw] closing …
[2018-04-06T14:27:45,736][INFO ][o.e.n.Node ] [qDUPSsw] closed
[2018-04-06T14:27:56,078][WARN ][o.e.b.Natives ] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna–1985354563/jna183854902005335039.tmp: /tmp/jna–1985354563/jna183854902005335039.tmp: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_161]
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_161]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_161]
at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_161]
at java.lang.System.load(System.java:1086) ~[?:1.8.0_161]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at com.sun.jna.Native.(Native.java:190) ~[jna-4.4.0-1.jar:4.4.0 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_161]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_161]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.6.8.jar:5.6.8]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.6.8.jar:5.6.8]
[2018-04-06T14:27:56,085][WARN ][o.e.b.Natives ] cannot check if running as root because JNA is not available
[2018-04-06T14:27:56,085][WARN ][o.e.b.Natives ] cannot register console handler because JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_AS beacuse JNA is not available
[2018-04-06T14:27:56,087][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-04-06T14:27:56,303][INFO ][o.e.n.Node ] [] initializing …
[2018-04-06T14:27:56,572][INFO ][o.e.e.NodeEnvironment ] [qDUPSsw] using [1] data paths, mounts [[/var (/dev/mapper/centos-var)]], net usable_space [746.2mb], net total_space [1.4gb], spins? [possibly], types [xfs]
[2018-04-06T14:27:56,573][INFO ][o.e.e.NodeEnvironment ] [qDUPSsw] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] node name [qDUPSsw] derived from node ID [qDUPSswHQVWUbdZLO-knyQ]; set [node.name] to override
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] version[5.6.8], pid[16240], build[688ecce/2018-02-16T16:46:30.010Z], OS[Linux/3.10.0-693.17.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-04-06T14:27:56,606][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-04-06T14:27:58,627][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [aggs-matrix-stats]
[2018-04-06T14:27:58,627][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [ingest-common]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-expression]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-groovy]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-mustache]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [lang-painless]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [parent-join]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [percolator]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [reindex]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [transport-netty3]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] loaded module [transport-netty4]
[2018-04-06T14:27:58,628][INFO ][o.e.p.PluginsService ] [qDUPSsw] no plugins loaded
[2018-04-06T14:28:02,717][INFO ][o.e.d.DiscoveryModule ] [qDUPSsw] using discovery type [zen]
[2018-04-06T14:28:04,322][INFO ][o.e.n.Node ] initialized
[2018-04-06T14:28:04,322][INFO ][o.e.n.Node ] [qDUPSsw] starting …
[2018-04-06T14:28:04,633][INFO ][o.e.t.TransportService ] [qDUPSsw] publish_address {172.16.5.11:9300}, bound_addresses {172.16.5.11:9300}
[2018-04-06T14:28:04,645][INFO ][o.e.b.BootstrapChecks ] [qDUPSsw] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-06T14:28:08,147][INFO ][o.e.c.s.ClusterService ] [qDUPSsw] detected_master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300}, added {{SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300},{ueq7c6I}{ueq7c6IVStKlNtZhePNEOg}{48a_TUrCRfm3RY1WkE6vcA}{graylog2-mgmt.philasd.net}{172.16.5.13:9300},}, reason: zen-disco-receive(from master [master {SxhDwH3}{SxhDwH3HSg-UBlcxPpDU5Q}{N-od2C4rQQqqawrtXlXQDA}{graylog3-mgmt.philasd.net}{172.16.5.14:9300} committed version [68]])
[2018-04-06T14:28:08,257][INFO ][o.e.h.n.Netty4HttpServerTransport] [qDUPSsw] publish_address {172.16.5.11:9200}, bound_addresses {172.16.5.11:9200}
[2018-04-06T14:28:08,258][INFO ][o.e.n.Node ] [qDUPSsw] started
mac-D25MD0F7F8JC:Graylog rgizynski$


(Bob Gizynski) #8

rsyslog.conf

local5.* @@172.16.5.11:5519

input(type=“imfile”
File="/var/log/audit/audit.log"
Tag=“linux_auditd”
Facility=“local5”
Severity=“info”
readMode=“1”)


(Jan Doberstein) #9

i personal would say that the linux auditlog does not match the regex values you try to get out.


(Bob Gizynski) #10

here is one of the extractors

graylog1 audit_log type=CRYPTO_KEY_USER msg=audit(1523364260.738:117483): pid=25895 uid=0 auid=4294967295 ses=4294967295 msg=‘op=destroy kind=server fp=SHA256:15:08:c2:82:dc:fa:13:bd:86:d7:f4:06:12:9a:e7:bc:5c:11:ae:24:aa:04:13:fc:8a:e7:9c:ee:65:80:67:30 direction=? spid=25895 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’

regular expression -> ^.*

for HOSTNAME


(Jan Doberstein) #11

sorry @unixinthebox I do net get want the intention of your last message is.

as you can clearly see the regex that is included in the first presented extractor rules does not match the logline of a audit.log on linux. You can’t squeeze apple juice out of a lemon …


(Bob Gizynski) #12

I think that regex ^.* should be “graylog1”


(Bob Gizynski) #13

what would ^.* return in this case ? shouldn’t it return the first word which is graylog1 ?


(Jochen) #14

The regular expression ^.* matches everything from the beginning of the line.


(system) #15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.