Step 8: Confirm that your data arrives and is stored

[dscryber (David Sciuto)]

Can you believe we’re already up to Step 8 (of 12) ?
One participant will win a $100 Amazon Gift Certificate . Our top contenders continue to be @shoothub and @gsmith currently vying for the most chances to win. But you can still jump in and play! For each step you respond to, you’ll get another change to win. Click here to join in!

This week, I want to know if after your initial Graylog install and setup, you confirmed that your data arrived and was properly stored. Did you get the expected results? Did the data arrive? Did it go to the proper storage? Did you need to tweak anything? If so, what did you need to change? Be specific.
What did you learn in this step that might help other community members?

Thank you to the community members who have been playing the User Journey Game. Keep it going! Remember, each submission is an entry in the User Journey raffle. The winner whose name is drawn will receive a $100 Amazon Gift Certificate.

Posting a valid++ response to this question is worth 1 chance to win.

For EACH STEP (there are 12 in all) in Graylog User’s Journey in which you post a valid++ response, you get a chance to win a $100 Amazon Gift Certificate! One lucky winner will have up to 12 chances to win. Go to “From the Graylog Book” to find the steps.

++Validity of response is subject to the community manager’s approval.
TO play, respond to this post with your response to this week’s questions.

[gsmith]

“My Struggles and Accomplishments of Graylog”.

@dscryber

I want to know if after your initial Graylog install and setup, you confirmed that your data arrived and was properly stored.

For this post I went back looking at old work logs of installing and testing Graylog. I choose Graylog for our testing environment.
My first Graylog installment was on 11/30/2016 10:59 PM (Happy New year’s) I believe it was version 1.3 and I had multiple problems. Not much was said accept I had problems with the configuration file. I believe it was from copying & pasting in Graylog configuration file. There also was a problem with getting these setting correct.

rest_listen_uri = http://your-server-ip:9000/api , web_listen_uri = http://your-server-ip:9000/ .

Noticing my messages shown on the Web UI had the wrong timestamp, I had to resolve this by configuring this root_timezone = America/Chicago.

A new version of Graylog called Graylog2 came out. I installed GrayLog 2.1 on CentOS 7. I was able to log into the GUI with default credentials. Only problem I did noticed were permission issues and elasticsearch is in red. I found this out by executing the following command

Root # curl -XGET http://localhost:9200/_cluster/health?pretty

Then I found the problem it was an index being stuck (i.e. “initializing_shards” : 1,).
Once the issue was identified I remove the stuck index, I know this was not the right way of fixing this issue because later I found out this was a bad idea. I was a n00b at this point.

Root # curl -XDELETE 'http://localhost:9200/graylog_1  /' 

My first data received was from a configured Cisco Switch 3750 to send log to GrayLog Server. GrayLog Web UI did show these messages without any problems. I was using Raw/Plaintext UDP port 5140 as an input, and no problems occurred. Then I created a new input Syslog UDP for Windows machines using just Nxlog-ce shipper and at this time I was only using one or two nodes. Later, fine tuning my Graylog skills I started using GELF TCP/TLS for my inputs.

As time passed, I ran into some more problems like Graylog Default Java options for heap and garbage collection, Set ES_HEAP_SIZE to 50% of available RAM, and my favorite problem was “How to Configure Self-Signed certificates for HTTPS”.

This is what I wrote about trying to use/configure https on Graylog 2.1:

”Since I have trouble configuring https on graylog config files, I need to do some more testing and configuration to get this to work. So far Graylog server becomes unresponsive after configuring Graylog.conf file as stated in the direction. I was trying to use Self-Sign-On certs but with no luck.”.

These posts helped resolve all my issues from that.

Graylog2 Using https

Hello All, I am having a problem with https on graylog2.2.3, Currently using CentOS-7.3 all in one graylog2 server. This is a Virtual machine with 8 cores, 10 GB of ram and 1 Tb storage. I created an Input for Localhost and tested to received messages, completed. No problems. I attempted to use the following: http://docs.graylog.org/en/2.2/pages/configuration/https.html Unfortunately, toward the end of the page I was having problems with JVM to pick up the new trust store. So, I attempted t…

Graylog2 https Error on Chrome

Hello All, Question about self-signed Certs for Graylog using HTTPS. I’m using CentOS 7 Virtual machine, all in one Graylog Server, My installed Certificates work from the instruction use from here: https://github.com/3vi1john/graylog Chrome states “Not Secure” on the URL I opened Developer Tools on Chrome and noticed the following; [Graylog-cert-error] The errors “There are issues with the site’s certificate chain (net::ERR_CERT_COMMON_NAME_INVALID)” And “The certificate for this sit…

Graylog HTTPS Redirect Using Apache

All, I’m trying to figure out how to redirect my IP Address to my FQDN. I have HTTPS enable and working with FQDN . https://graylog.enseva-labs.net:9000/search I also have the IP address. https://10.200.6.70:9000/search How do I get only the FQDN and not the IP Address? Basically either I redircet the IP Address or I block it. My Evironment: CentOS 7 Latest Version graylog-server-3.0.0-12.noarch elasticsearch-6.6.1-1.noarch mongodb-org-4.0.6-1.el7.x86_64 I have looked here Web-Inte…

Once those issues were done, I tried to install GeoIP plugin. I went through Graylog server making sure everything was functioning and working correctly. Notice that a couple were not, such as GeoIP Locator.
What I needed was to add the GeoIP Lite Database. Ooops . All of our storage, containing logs were always ok. Never really had a problem since we use Virtual machine it’s very easy to add storage.

About a year later I have accomplished Installing Graylog on CentOS 7, configured HTTPS for Graylog, also using GELF TCP/TLS Inputs. All plugins, INPUTs, and connections were working fine. I kind of mastered the Self-Signed certificates for Graylog. The Certs and pipelines were and are the hardest thing to overcome.

My last graylog adventure was creating a cluster for our production environment, you can check it out here.

Testing Elasticsearch Cluster

Hello All, Problem With elasticsearch Cluster. When shutting down ES-Node 1 (testing failover), Graylog+ MongoDb Cluster is unable to use index. NOTE: At this point I do not have Load balancer in place for Graylog+MDB Servers. I was performing tests on redundancy with Elasticsearch Servers. Environment; Total of 6 CentOS 7.3 Servers minimal install. 3 Servers with Graylog version 2.3 and Mongo version 3.4 ‘Clustered’ 3 Servers with Elasticsearch 5.6.4 ‘Clustered’ Graylog Steps install/co…

That same single Graylog server now have version 4.0.11, elasticsearch 7.10.2 and MongoDb 4.4.
I have a ton of streams configured on it from a user login/failed, DNS errors/warnings, AD group policies, Veeam Backup, Cluster server/s warnings and errors, etc… I probably have almost 70 -100 streams for testing out alerts in our environment at this point. Since I posted in the forum of any problems or issues having with Graylog its now easy to comback to find fixes for my current issues That is, if I have any.

@dscryber

What did you learn in this step that might help other community members?

• Have patience, read the documents thorough and don’t skip parts because you may think it might not help or pertain to your issue at that point, It will later

• Start with basic installments and work from there. By doing this you can find your issue quicker and understand how parts and pieces of Graylog work/function.

• Since Graylog is made up of different components for example (Elasticsearch, MongoDb) search there forums also. Sometimes its not Graylog problem it’s your elasticsearch and/or MongoDb problem.

[shoothub]

@dscryber

I want to know if after your initial Graylog install and setup, you confirmed that your data arrived and was properly stored. Did you get the expected results? Did the data arrive? Did it go to the proper storage? Did you need to tweak anything? If so, what did you need to change? Be specific.

In my experience data usually arrived fine. But from time to time, I had problems, so this steps helped me:

1. First step is to verify that data arrived at all.
   • If not it’s usually problem with firewall blocking connection, or you defined wrong port number on sending device or input, or wrong protocol TCP or UDP. So first check if your remove device is sending data to graylog.
   • You can run tcpdump on graylog box to sniff sending logs on specific port to verify log is comming to graylog, examples below.
2. Second step if data were stored is to check timestamps.
   • If you can’t see messages in Search section but you see them in Input - Show received messages section, it’s usually problem with timestamps.
   • Sometimes you forgot to setup correct timezone on sending device or graylog box (OVA has configured UTC system timezone by default).
   • Also don’t forgot to setup NTP synchronization, so time is up to date.
   • Graylog store timestamp in UTC in ElasticSearch, and display in timezone defined in user’s profile. Note, that admin user has to setup correct timezone in /etc/graylog/server/server.conf file in parameter root_timezone.
3. If sending device doesn’t send correct timezone (usualy syslog with RFC 3164 format, which doesn’t send timezone at all), graylog could store wrong timestamp.
   • In this situation, you can use either extractor with converter Date, or Flexible. Or pipeline rule with function parse_date() to fix timestamps.
4. If you find, that graylog stored unnecessary field (usually if you use winlogbeat or filebeat), you can use pipeline rule to delete these fields, to save storage space using simple snippet:

Remove unused fields pipeline rule

rule "Remove unused fields"
when
  has_field("unnecessary_field")
then
  remove_field("unnecessary_field");
  remove_field("unnecessary_field2");
end

5. If you find, that field names are not correct, and you want to normalize them (e.g change source_ip to src_ip), use pipeline rule:

Normalize fields pipeline rule

rule "Normalize fields"
when
  has_field("source_ip")
then
  rename_field("source_", "src_ip");
  rename_field("destination", "dst_ip");
end

How to debug:
Best way to debug if message received to graylog input is to either sniff logs using tcpdump or send test message to graylog using logger, nc or curl.

1. Tcpdump:
   Run tcpdump sniffer on graylog box to verify that syslog traffic is send by remote syslog IP:
   tcpdump -s0 -A -nni <interface> host <remote_syslog_IP> and port <graylog_input_port>

Parameters:

• -A print each packet as ASCII, so you can view log content
• -nn don’t convert protocols and ports numbers to name
• -i <interface> interface to capture (interface on which graylog’s input listens)
• <remote_syslog_IP> IP of remote device sending logs
• <graylog_input_port> graylog’s port that listens for logs

2. Logger:
   Use logger command to send test message to graylog’s input.
   logger --tcp --server <graylog_server_IP> -P <graylog_input_port> --rfc5424=notq -p <priority> "Test message"

Parameters:

• --udp Use UDP protocol for connection
• --tcp Use TCP protocol for connection
• --rfc3164 Use the RFC 3164 BSD syslog protocol
• --rfc5424=notq Use the RFC 5424 syslog protocol, notq means don’t sent time quality data
• -p <priority> Priority of the message. The priority may be specified numerically or as a facility_name.level. The default is user.notice
  • The numeric priority value is calculated using the formula (Priority = Facility code * 8 + Level).
  • Valid facilities:
    • auth
    • authpriv - for security information of a sensitive nature
    • cron
    • daemon
    • ftp
    • kern - cannot be generated from userspace process, automatically converted to user
    • lpr
    • mail
    • news
    • syslog
    • user
    • uucp
    • local0 - local7172.28.128.19
    • security - deprecated synonym for auth
  • Valid level names are:
    • emerg
    • alert
    • crit
    • err
    • warning
    • notice
    • info
    • debug
    • panic - deprecated synonym for emerg
    • error - deprecated synonym for err
    • warn - deprecated synonym for warning

3. Netcat
   Send test message from command line using nc command.

• Syslog (RFC 3164):
  • For TCP syslog input:
    echo "<priority>$(env LANG=us_US.UTF-8 date "+%b %d %H:%M:%S") $(hostname -s) TESTMESSAGE from TCP" | nc <graylog_server_IP> <graylog_input_port>
  • For UDP syslog input:
    echo -n "<priority>$(env LANG=us_US.UTF-8 date "+%b %d %H:%M:%S") $(hostname -s) TESTMESSAGE from TCP" | nc -u <graylog_server_IP> <graylog_input_port>
• GELF:
  • For GELF TCP input:
    echo -n -e '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }'"\0" | nc <graylog_server_IP> <graylog_input_port>
  • For GELF UDP input:
    echo -n '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' | nc -u <graylog_server_IP> <graylog_input_port>

4. Curl
   Send test message for GELF HTTP input using curl command:
   curl -X POST -H 'Content-Type: application/json' -d '{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_some_info": "foo" }' 'http://graylog.example.com:12201/gelf'