Geolocation Processor empty geo fields

Graylog Add-ons
1

Hello,
Bringing this from github #37

Trying to set this up. Follow the doc to the letter. I was able to get the geo fields, <field>_geolocation, when i ran the command:

nc -w1 <graylog_host> 5555 <<< '8.8.8.8'

Unfortunately it does not work with normal logs that come in. On my input i have an extractor that copies the IP from the message into gl2_remote_ip field, which works.

Trying to extract data from message into gl2_remote_ip, leaving the original intact.
Condition
     Will only attempt to run if the message matches the regular expression from ([0-9]+.[0-9]+.[0-9]+.[0-9]+|\S+)(\s|:\s)
Configuration 
     regex_value: from ([0-9]+.[0-9]+.[0-9]+.[0-9]+|\S+)(\s|:\s)

Also set GeoIP Resolver to last (3) in “Message Processors Configuration”.

#	Processor	Status
1	Pipeline Processor	active
2	Message Filter Chain	active
3	GeoIP Resolver	active

This is how the Plugin is configured:

Enabled: yes
Database type: City database
Database path: /etc/graylog/server/GeoLite2-City.mmdb

The DB exists and has the correct permissions:

ll /etc/graylog/server/GeoLite2-City.mmdb
-rw-r--r-- 1 root root 53531421 Mar 29 19:46 /etc/graylog/server/GeoLite2-City.mmdb

Not really to sure what is going on. Any input is appreciated.
Hope this makes sense.

CentOS Linux release 7.3.1611 (Core)
Graylog 2.2.2+691b4b7

Thanks
dave

2

Someone had asked about the gl2_remote_ip field

curl -XGET 'http://localhost:9200/graylog2_126/_search?q=gl2_remote_ip:<ip-redacted>' | python -m json.tool

            {
                "_id": "c948b594-1572-11e7-9862-d8d38561c4a0",
                "_index": "graylog2_126",
                "_score": 13.408007,
                "_source": {
                    "application_name": "sshd2",
                    "facility": "user-level",
                    "gl2_remote_ip": "<ip-redacted>",
                    "gl2_remote_port": 43854,
                    "gl2_source_input": "58a4b2b948fdda09423bcced",
                    "gl2_source_node": "c5092753-5f87-4e7c-b42c-ec91b4b0f731",
                    "level": 6,
                    "message": "Accepted keyboard-interactive/pam for dvmacias from <ip-redacted> port 40417 ssh2",
                    "process_id": "5032",
                    "source": "h-txa-1ter-ecr-1",
                    "streams": [
                        "58803ca348fdda55811e9ca0",
                        "000000000000000000000001",
                        "58803ca348fdda55811e9c92"
                    ],
                    "timestamp": "2017-03-30 18:00:40.552"
                },
                "_type": "message"
            },
3

Is your <ip-redacted> in one of these networks?

  • 127.0.0.1/8
  • 10.0.0.0/8
  • 172.16.0.0/12
  • 192.168.0.0/16
  • 192.0.2.0/24
  • 198.51.100.0/24
  • 203.0.113.0/24

Does the gl2_remote_ip field contain only the IP address and no whitespaces or other characters around it?

4

Thanks for replying @jochen

Is your in one of these networks?

No. It is a public IP on the internet.

Does the gl2_remote_ip field contain only the IP address and no white-spaces or other characters around it?

Yes, only the IP. No special chars or white-spaces. Copied field from exactly from terminal just replaced IP.

   "gl2_remote_ip": "1.1.1.1",

I tried again for sanity’s sake the command nc -w1 localhost 5555 <<< '91.200.12.76' and it works but it goes to the message field which then creates the message_geo* fields. (that’s not my public IP, it’s an ip that was trying to get in :slight_smile: from ukrain)

5

Please use another field name than gl2_remote_ip to store the IP addresses you want to run the GeoIP processor on.

Upon inspecting the code of the Map Widget plugin, I realized that the GeoIP processor will ignore any “internal” field (i. e. any field starting with gl2_). We should probably document that better. :wink:

6

I changed the field. Just created one called ip_address and works as expected.

Should i submit a PR for documentation sake?

Not sure how to close “this” issue

thank you for your support
Dave

7

I’ve already added it to the documentation: http://docs.graylog.org/en/2.2/pages/geolocation.html#configure-the-message-processor

8

Im back with issues again…hurray!

Now running graylog2.3.1 and elasticsearch5.5

I cannot get geoip to work anymore. I have this log:

curl -XGET ‘http://localhost:9200/graylog2_172/_search?q=message:1.1.1.1’ | jq
{
“_index”: “graylog2_172”,
“_type”: “message”,
“_id”: “b6b66441-9fab-11e7-961b-68b599c95aa0”,
“_score”: 24.335264,
“_source”: {
“level”: 5,
“gl2_remote_ip”: “127.0.0.1”,
“gl2_remote_port”: 59397,
“streams”: [
“000000000000000000000001”,
“58803ca348fdda55811e9ca0”
],
“source”: “datacom-test-ecr-server”,
“message”: “1.1.1.1”,
“gl2_source_input”: “59c3c4fc8ac0d46170835fa2”,
“application_name”: “root”,
“gl2_source_node”: “0b268e84-b342-4f79-8bda-0ce7bc969869”,
“facility”: “user-level”,
“timestamp”: “2017-09-22 15:35:51.171”
}

Which i created with

logger “1.1.1.1”

Now when i have GeoIP “Message Processors” Enabled I get this error under /system/indices/failures

{“type”:“mapper_parsing_exception”,“reason”:“failed to parse”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“Can’t parse [index] value [not_analyzed] for field [message_country_code], expected [true] or [false]”}}

What am i doing wrong?

I have the exact same settings as those mentioned on first comment

Thank you for the support
-dave

9

Please post the existing index templates in your ES cluster.

10

Thank you @jochen for replying. I hope this is what you looking for:

curl -XGET ‘http://localhost:9200/graylog2_0’ | jq
{
“graylog2_0”: {
“aliases”: {
“graylog2_deflector”: {}
},
“mappings”: {
“message”: {
“dynamic_templates”: [
{
“internal_fields”: {
“match”: “gl2_",
“mapping”: {
“type”: “keyword”
}
}
},
{
“store_generic”: {
“match”: "”,
“mapping”: {
“index”: “not_analyzed”
}
}
}
],
“properties”: {
“application_name”: {
“type”: “keyword”
},
“facility”: {
“type”: “keyword”
},
“full_message”: {
“type”: “text”,
“analyzer”: “standard”
},
“gl2_remote_ip”: {
“type”: “keyword”
},
“gl2_remote_port”: {
“type”: “keyword”
},
“gl2_source_input”: {
“type”: “keyword”
},
“gl2_source_node”: {
“type”: “keyword”
},
“ip_address”: {
“type”: “keyword”
},
“level”: {
“type”: “keyword”
},
“message”: {
“type”: “text”,
“analyzer”: “standard”
},
“process_id”: {
“type”: “keyword”
},
“source”: {
“type”: “text”,
“analyzer”: “analyzer_keyword”,
“fielddata”: true
},
“streams”: {
“type”: “keyword”
},
“timestamp”: {
“type”: “date”,
“format”: “yyyy-MM-dd HH:mm:ss.SSS”
}
}
}
},
“settings”: {
“index”: {
“number_of_shards”: “1”,
“provided_name”: “graylog2_0”,
“creation_date”: “1506104165340”,
“analysis”: {
“analyzer”: {
“analyzer_keyword”: {
“filter”: “lowercase”,
“tokenizer”: “keyword”
}
}
},
“number_of_replicas”: “0”,
“uuid”: “HYW4go4ZRZOzVbwve0fqdw”,
“version”: {
“created”: “5060199”
}
}
}
}
}

Thanks!

11

No, I was referring to the index templates, not the mapping of a single index.

See Index Templates | Elasticsearch Reference [5.6] | Elastic for details.

12

Aaahh

curl -XGET 'http://<localhost>:9200/template’ | jq
{
“graylog-internal”: {
“order”: -1,
“template”: "graylog2",
“settings”: {
“index”: {
“analysis”: {
“analyzer”: {
“analyzer_keyword”: {
“filter”: “lowercase”,
“tokenizer”: “keyword”
}
}
}
}
},
“mappings”: {
“message”: {
"source": {
“enabled”: true
},
“dynamic_templates”: [
{
“internal_fields”: {
“mapping”: {
“type”: “keyword”
},
“match”: "gl2"
}
},
{
“store_generic”: {
“mapping”: {
“index”: “not_analyzed”
},
“match”: “*”
}
}
],
“properties”: {
“full_message”: {
“fielddata”: false,
“analyzer”: “standard”,
“type”: “text”
},
“streams”: {
“type”: “keyword”
},
“source”: {
“fielddata”: true,
“analyzer”: “analyzer_keyword”,
“type”: “text”
},
“message”: {
“fielddata”: false,
“analyzer”: “standard”,
“type”: “text”
},
“timestamp”: {
“format”: “yyyy-MM-dd HH:mm:ss.SSS”,
“type”: “date”
}
}
}
},
“aliases”: {}
}
}

13

Ok i think i got it…

So… when i rebuild graylog2.3 and elasticv5.5 on another box geoip worked like a charm.
I decided to remove and reinstall the applications on the offending server and geoip works as expected…

thanks!