Basic understanding of ELK / Graylog

Hello,

I’m new around ELK and Graylog world, and trying to understand the basics on this SIEM.

Actually i went into a manual install of :

  • Elasticsearch for index
  • Kibana for direct search and visualization on index datas
  • Logstash for parsing / stashing (??)
  • Graylog seems to be doing the same work as Kibana, with search, visualization …
  • Collector-sidecar seems to contains all listeners for clients like NXlog and Filebeats.
  • Failed install of NXlog for now, i’ll solve this later.

However i cannot point out about how to send extracted log files as Splunk does on Graylog, i must use NXlog and Filebeats ? I wish to send any log files (no matter the type of logs) and sort out by regex filter and standard delimiters myself. Isn’t there a easier method to add my own logs on elasticsearch or graylog ?

So i actually have installed the above parts, and graylog can connect to elasticsearch index / cluster named graylog2 or graylog. However Kibana can’t see the elasticsearch index while graylog can see it and is in green state … very strange.

Is Kibana needed ? Graylog seems to do the same kind of work.

Everything is installed on the same Debian server, since i wish to put as a one offline analysis platform for education/testing purposes.

Thanks for guidance and pointing out things.

Let’s take things slowly :slight_smile:

http://docs.graylog.org/en/2.5/pages/architecture.html

That article explains the most important parts…

  • Graylog serves multiple purposes:
    • Receiving data through its “inputs
    • Parsing, massaging and processing data through “pipelines” and more.
    • Pushing data into Elastic for storage.
    • User interface for querying and visualization.
  • ElasticSearch is purely for (log)data storage.
  • MongoDB is purely for configuration storage.
  • The Sidecar Collector is agent software that you can install on systems that have logs that cannot be forwarded to Graylog automatically.
    • For example, where Linux can use RSyslog/Syslog to send logs to Graylog,
    • Windows EventLog cannot forward logs by itself.
    • Ditto for many applications, which simply log into their own files in /var/log/whatever, or in random places on C:, D:\ etc.
    • The Sidecar Collector can be configured to tail specific eventlogs and files, to forward all that data into a Graylog “input”.

Thanks you for clarification !

So that means Kibana does same work as Graylog, can they both be connected on the same elasticsearch server ?

By the way is there any way to send my own log files to parse, easily ?
I was thinking about mounting a remote folder (host is elasticsearch server) on a windows client, and then putting logs automatically on elasticsearch by a simple copy of log file on the remote accessible folder. There is no syslog involved, or any network communication needed.

But it seems there is no local storage for this and i got to use NXlog or Filebeats, right ?

Kibana mostly is used for querying and visualization, correct. So yes, it would be a duplicate of functionality if you already have Graylog. On the other hand, some people prefer Kibana, as supposedly it offers more in the way of visualization. And yes, you can let Kibana query the Graylog indexes, as long as the relevant user account is provided with access.

With regards to sending in your logfiles: that’s the whole point of Graylog and also what I’ve described above. You can use the Sidecar Collector (or another implementation of NXLog or BEATS) to track log files and eventlogs, to forward them into Graylog.

There is no way of simply uploading an existing file, because all data is inside the ElasticSearch data lake. There are no files accessible through the file system. Also, as Jan has pointed out in other threads: in order for Graylog to be able to work with log data, it needs to have gone through Graylog. You cannot simply chuck info into Elastic in hopes that Graylog finds it. Graylog does a bit of massaging and parsing on all data that comes in, adding some required metadata along with it.

Kibana query the Graylog indexes, as long as the relevant user account is provided with access.

It’s ok now, i see graylog_0 as index, solved.

Actually i’m using nxlog and i see data on graylog, however i don’t see my log file data when searching * for all timestamp date. I’m putting a file inside “C:\Program Files (x86)\nxlog\Files_Logs\*.log” but i’m only getting logs from my computer which is polluting, and isn’t relevant.

So i think i’m wrong here, below the example on nxlog manual :

This configuration will read from a file and forward messages via TCP.
No additional processing is done.
nxlog.conf
<Input messages>
Module im_file
File “/var/log/messages”
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.1
Port 514
</Output>
<Route messages_to_tcp>
Path messages => tcp
</Route>

Below the my configuration file of nxlog :

Module im_file File 'C:\Program Files (x86)\nxlog\Files_Logs\\*.log' PollInterval 1 SavePos True ReadFromLast True Recursive False RenameCheck False Exec $FileName = file_name(); # Send file name with each message Module om_tcp Host 192.168.1.103 Port 12201 OutputType GELF_TCP # These fields are needed for Graylog $gl2_source_collector = '${sidecar.nodeId}'; $collector_node_id = '${sidecar.nodeName}';

Unfortunately I don’t have prior experience with NXLog, so I wouldn’t know whether it’s enough to just chuck log files into an NXLog directory. I dunno ¯\(°_o)/¯

The example from the NXLog manual that you quoted does not seem relevant because it:

  1. Is for Linux
  2. Seemingly points towards a Syslog input on a receiver host (port 514)

Personally, I have only worked with the Sidecar (for Linux files, Windows files and Windows Eventlogs).

When working with the Sidecar, there’s a stack of configs that you need to make.

  1. Install the Sidecar on the sending-side.
  2. Configure the Sidecar with the right URI for your Graylog API.
  3. Configure the Sidecar with the right tags (see below).
  4. In Graylog, under Collector Configuration define tags. Each tag can be used to define any number of files or eventlogs that need to be monitored. For each of these tags, define the desired files and the desired output-to-input.
  5. In Graylog, under Inputs define the required input.

In my case I mostly use the FileBEAT / WinLogBEAT parts of the Sidecar, thus I’ve made a BEATS input on the Graylog hosts. Each of the tags is configured to trace a bunch of files/logs and to forward them through FileBEAT / WinLogBEAT to the input on the Graylog side.

No, Kibana is not needed, nor is Logstash since Graylog can handle both the tasks. Kibana does offer a higher level of visualisation support, but Graylog isn’t bad at it either.

I’ve been trying to wrap my head around NXLog. From what I understand it’s YALM: Yet Another Log Manager, i.e. another competitor to Syslog/RSyslog/Syslog-NG/BEATS/etc. Correct? So if you’re already installing the Sidecar, you technically don’t need NXLog either, right?

Trying to do that right now, any advice on configuration files for direct log files sending ?
As i want to install client on graylog server and access to an empty “log folder” to send my extracted log files to.

Uhm… I’m having a hard time picturing that, because what you describe is utterly not how Graylog works. Could you please provide some more details on what you think you’re building?

Yours words are absolute ! :grin:
Either one of the two solutions are fine to me.

Sidecar goes on the system where you want to collect the logs from and then are shipped to Graylog via beats input (if you use filebeat)

So, i only need filebeats and specify a local folder on my computer. When i drop any log file into that folder, filebeats will send to graylog.

Correct ?

CORRECT! :slight_smile:

correct

And the easiest way to set that up is with the Sidecar.

Alright so, checking about Sidecar configuration for this !

  1. Install the Sidecar on the sending-side.
  2. Configure the Sidecar with the right URI for your Graylog API.
  3. Configure the Sidecar with the right tags (see below).
  4. In Graylog, under Collector Configuration define tags. Each tag can be used to define any number of files or eventlogs that need to be monitored. For each of these tags, define the desired files and the desired output-to-input.
  5. In Graylog, under Inputs define the required input.

FileBEAT :

So graylog will recognize the log type i’ll send ? Since i’ll throw a lot of various logs files in the folder, can be anything from windows logs to application or linux logs.

There is configuration file to edit i guess … i’ll have to look into this afternoon.

1 Like

You won’t have to edit any configuration files manually, beyond setting the URI and the tags for each sidecar. That’s it. The rest is: Magic!

Yes, Graylog will work it all out under the hood once you’ve gotten the Sidecar to send in the files and eventlogs that you need.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.