I have several sites of a dozen or so hosts each behind routers doing NAT; these log via syslog/TLS/TCP to a set of Graylog frontends in a cloud compute environment. Each host has its own client certificate; a single server certificate is shared by all the Graylog frontends.
My problem is that I’d like to be able to use the logs in Graylog for forensics should one of the servers become compromised, but I am having difficulty figuring out how to ascertain that any particular log record actually came from a given server at a given time, as opposed to being spoofed by an attacker.
Graylog appears to record the source IP address of the connection that delivered the log message in the gl2_remote_ip
field, but this lets me distinguish only between sites (since all servers at a site appear to have the same source address due to NAT), and it’s not even clear to me that a client might not be able to override this if I enable, e.g., a GELF input.
Graylog also, in the default configuration, appears to record receipt timestamp data only if it can’t parse a timestamp out of the message; if it can, it seems to quietly throw away the receipt timestamp and record only the (potentially attacker-manipulated) timestamp parsed from the message.
The attack here is obvious: once you manage to get enough access to one host to get hold of the Graylog client key, create a large number of messages claiming to be from other hosts with timestamps some time in the past, inject them into Graylog, and use the extra time you’ve gained by sending the forensics guys off on a wild goose chase to do nefarious things.
I think I might be able to solve the received timestamp issue with an extractor that uses the now
function to fill in a field (though I’m still not clear on whether that might be able to be overridden), but that doesn’t help with client authentication. Is there any way to get and log information about the TLS authentication, such as data from the matching client certificate?
As well, if anybody has pointers to resources on configuring Graylog to make it more suitable for forensic analysis, I’d appreciate them.