“My Struggles and Accomplishments of Graylog”.
For this post I went back looking at old work logs of installing and testing Graylog. I choose Graylog for our testing environment.
My first Graylog installment was on 11/30/2016 10:59 PM (Happy New year’s) I believe it was version 1.3 and I had multiple problems. Not much was said accept I had problems with the configuration file. I believe it was from copying & pasting in Graylog configuration file. There also was a problem with getting these setting correct.
rest_listen_uri = http://your-server-ip:9000/api , web_listen_uri = http://your-server-ip:9000/ .
Noticing my messages shown on the Web UI had the wrong timestamp, I had to resolve this by configuring this root_timezone = America/Chicago.
A new version of Graylog called Graylog2 came out. I installed GrayLog 2.1 on CentOS 7. I was able to log into the GUI with default credentials. Only problem I did noticed were permission issues and elasticsearch is in red. I found this out by executing the following command
Root # curl -XGET http://localhost:9200/_cluster/health?pretty
Then I found the problem it was an index being stuck (i.e. “initializing_shards” : 1,).
Once the issue was identified I remove the stuck index, I know this was not the right way of fixing this issue because later I found out this was a bad idea. I was a n00b at this point.
Root # curl -XDELETE 'http://localhost:9200/graylog_1 /'
My first data received was from a configured Cisco Switch 3750 to send log to GrayLog Server. GrayLog Web UI did show these messages without any problems. I was using Raw/Plaintext UDP port 5140 as an input, and no problems occurred. Then I created a new input Syslog UDP for Windows machines using just Nxlog-ce shipper and at this time I was only using one or two nodes. Later, fine tuning my Graylog skills I started using GELF TCP/TLS for my inputs.
As time passed, I ran into some more problems like Graylog Default Java options for heap and garbage collection, Set ES_HEAP_SIZE to 50% of available RAM, and my favorite problem was “How to Configure Self-Signed certificates for HTTPS”.
This is what I wrote about trying to use/configure https on Graylog 2.1:
”Since I have trouble configuring https on graylog config files, I need to do some more testing and configuration to get this to work. So far Graylog server becomes unresponsive after configuring Graylog.conf file as stated in the direction. I was trying to use Self-Sign-On certs but with no luck.”.
These posts helped resolve all my issues from that.
Once those issues were done, I tried to install GeoIP plugin. I went through Graylog server making sure everything was functioning and working correctly. Notice that a couple were not, such as GeoIP Locator.
What I needed was to add the GeoIP Lite Database. Ooops 
. All of our storage, containing logs were always ok. Never really had a problem since we use Virtual machine it’s very easy to add storage.
About a year later I have accomplished Installing Graylog on CentOS 7, configured HTTPS for Graylog, also using GELF TCP/TLS Inputs. All plugins, INPUTs, and connections were working fine. I kind of mastered the Self-Signed certificates for Graylog. The Certs and pipelines were and are the hardest thing to overcome.
My last graylog adventure was creating a cluster for our production environment, you can check it out here.
That same single Graylog server now have version 4.0.11, elasticsearch 7.10.2 and MongoDb 4.4.
I have a ton of streams configured on it from a user login/failed, DNS errors/warnings, AD group policies, Veeam Backup, Cluster server/s warnings and errors, etc… I probably have almost 70 -100 streams for testing out alerts in our environment at this point. Since I posted in the forum of any problems or issues having with Graylog its now easy to comback to find fixes for my current issues That is, if I have any. 
-
Have patience, read the documents thorough and don’t skip parts because you may think it might not help or pertain to your issue at that point, It will later
-
Start with basic installments and work from there. By doing this you can find your issue quicker and understand how parts and pieces of Graylog work/function.
-
Since Graylog is made up of different components for example (Elasticsearch, MongoDb) search there forums also. Sometimes its not Graylog problem it’s your elasticsearch and/or MongoDb problem.