Getting a unique list of a field with the latest occurance of another


(Stephen) #1

Hi,

First let me start with this is my first post here and I wasnt sure where to post this, this category seemed the most likely place, but apologies if it isnt. Also to add, we are barely starting to use graylog at our organization so if there is a way to do this that I havent found, it is entirely possible.

This is the situation. We are gathering Cisco Catalyst logs into our server. It would be useful to have a list of ports and their status based on the last updown message received per port. if this works, it could be useful for other purposes such as palo alto etc

what we’ve done:

  • created a custom extractor to create an “interface” field:
    regex_value: (GigabitEthernet\d[/\d]+)
  • created a custom extractor to create a “state” field:
    grok_pattern: .*changed state to %{GREEDYDATA:state}
  • created a stream that takes in messages coming from the cisco input, has a source field, and an interface field
  • created a pipeline to process those messages and create a custom “s-i_touple” field with the source and interface:
    rule "create touple field" when true then let iface = concat("-GE",substring(to_string($message.interface),15)); let touple = to_string(concat(to_string($message.source),iface)); set_field("s-i_touple",touple); end

This creates the s-i_touple field in those messages that looks something like “192.168.1.55-GE2/22”

Now here’s the part that I’m not sure how to accomplish. I have messages coming in with a host-interface field and an action field (up, down, administratively down, etc). Does graylog/elasticsearch have a way to aggregate those two by getting a unique set of the “s-i_touple” field (such as what Quick Values provides) and getting the latest “state” message for each distinct “s-i_touple”?

Thanks,
Stephen


(Stephen) #2

bump! any takers on this one? I’ve learned much since this was first posted but how to do this still eludes me.


(Jan Doberstein) #3

hej @sgarciam

currently those kind of aggregation is not possible with vanilla Graylog. As you can see some issues in github are present but no solution until now.

regards
Jan


(Stephen) #4

Thanks for the response @jan

-Stephen