No proposal chosen?

[jendamozna]

Hello i’ve found following error in elasticsearch.log:

failed to execute bulk item (index) index {[srxfw_deflector][message][d52195f0-227b-11e8-a9d5-005056993b45], source[{
"process_id":"1431",
"data":"1.2.3.4",
"gl2_remote_ip":"2.3.4.5",
"gl2_remote_port":41367,
"streams": ["...","..."],
"source":"some.hostname",
"message":"empty",
"gl2_source_input":"...",
"receivedat":"2018-03-08T02:53:08.688Z",
"source-address":"No proposal chosen",
"application_name":"kmd",
"error-message":"3.4.5.6",
"destination-address":"(null)",
"gl2_source_node":"box73",
"facility":"system daemon",
"timestamp":"2018-03-08 02:14:19.997"}]}

MapperParsingException[failed to parse [source-address]]; nested: IllegalArgumentException[failed to parse ip [No proposal chosen], not a valid ip address];

Is there any possibility that Graylog parses “No proposal chosen” into “source-address” or “(null)” into “destination-address” from SRX Key-Value structured FW log message. Or this is most likely scrambled by SRX box?

[jan (Jan Doberstein)]

without knowing how the logs are ingested into graylog and what kind of extractor or pipeline is used to work with the messages, but the error message from elasticsearch is that no proposal chosen is not a valid IP - what elasticsearch expect to be - as of the given index template.

so what exactly is your question?

[jendamozna]

Input is Syslog UDP, fields are parsed automatically by GL. Pipeline only empties message for easy indexing.

SRX messages are usually following:
<14>1 2016-11-04T17:08:34.555+02:00 HOSTNAME RT_FLOW - RT_FLOW_SESSION_DENY [junos@1.1.1.2.40 source-address=“1.2.3.4” source-port=“11” destination-address=“2.3.4.5” destination-port=“1” service-name=“junos-telnet” protocol-id=“1” icmp-type=“0” policy-name=“GLOBAL-DENY(global)” source-zone-name=“zone1” destination-zone-name=“zone2” application=“UNKNOWN” nested-application=“UNKNOWN” username=“N/A” roles=“N/A” packet-incoming-interface=“reth0.0” encrypted=“UNKNOWN” reason=“policy deny”]

I wonder if Syslog UDP parser is able to set some constant “no proposal chosen” into “source-address” if sth fails inside SyslogUDP input processing…

Ive never seen it before in IndexFailures… I dont exclude bug in SRX… I am just curious about GL SyslogUDP parser.

[jochen]

You’ll have to create a custom index mapping for the fields for which the Elasticsearch dynamic mapping fails, e. g. “source-address” and “destination-address”.

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings