Graylog installation manual for ubuntu improvements - elasticsearch issue

Hello,
i am working on Ubuntu 16.04.
I followed very carefully, step-by-step manual from here:
http://docs.graylog.org/en/3.0/pages/installation/os/ubuntu.html?fbclid=IwAR2X3Cz1mvgIlTnnPJ7X05B52mDAI3pV55fF49UfsoqiSxG0Y5dLwFICUDY

Elastic config: https://pastebin.com/bgUb0NB0
It seems that elasticsearch wont start. I stumbled upon the issue after logging in to Graylog web interface and seeing the elastisearch erros. curl to 127.0.0.1:9200 got me connection refused.

service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-05-01 23:47:24 CEST; 14min ago
Docs: http://www.elastic.co
Process: 3691 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1
Main PID: 3691 (code=exited, status=1/FAILURE)

May 01 23:47:23 Graylog-ubuntu systemd[1]: Stopped Elasticsearch.
May 01 23:47:24 Graylog-ubuntu systemd[1]: Started Elasticsearch.
May 01 23:47:24 Graylog-ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
May 01 23:47:24 Graylog-ubuntu systemd[1]: elasticsearch.service: Unit entered failed state.
May 01 23:47:24 Graylog-ubuntu systemd[1]: elasticsearch.service: Failed with result ‘exit-code’.
lines 1-12/12 (END)

/var/log/elasticsearch does not contain any files

Here is the same issue but all of those i searched just end without solving the issue, example:

This is why I think the documentation on installing graylog3 must be missing something important.

@Graylog-ubuntu:/usr/share/elasticsearch/bin$ ./elasticsearch
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
./elasticsearch-env: line 71: /etc/default/elasticsearch: Permission denied

What are the permissions on that file?

ls -l /etc/default/elasticsearch

1 Like

-rw-rw---- 1 root root 1645 May 1 23:06 /etc/default/elasticsearch

@mattimac

That may be your problem. I imagine the other files in /etc/default/ have permissions set to 664.

Set the permissions of /etc/default/elasticsearch to 664 and try to start the service again. I encountered this issue when replacing elasticsearch with elasticsearch-oss

Command for reference: sudo chmod 664 /etc/default/elasticsearch

@Ponet After tried to restart elasticsearch service after chmod:

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-05-10 00:21:14 CEST; 3s ago
Docs: http://www.elastic.co
Process: 51773 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited,
     Main PID: 51773 (code=exited, status=1/FAILURE)
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at org.elasticsearch.common.xcontent.json.JsonXContentParser.n
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at org.elasticsearch.common.settings.Settings.fromXContent(Set
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at org.elasticsearch.common.settings.Settings.fromXContent(Set
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at org.elasticsearch.common.settings.Settings.access$500(Setti
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         at org.elasticsearch.common.settings.Settings$Builder.loadFrom
    May 10 00:21:14 Graylog-ubuntu elasticsearch[51773]:         ... 8 more
    May 10 00:21:14 Graylog-ubuntu systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
    May 10 00:21:14 Graylog-ubuntu systemd[1]: elasticsearch.service: Unit entered failed state.
    May 10 00:21:14 Graylog-ubuntu systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

I believe i got it working, elasticsearch cluster in graylog is green. I did the following:

sudo apt-get purge elasticsearch-oss
sudo apt-get update && sudo apt-get install elasticsearch-oss

followed Elasticsearch part of manual
http://docs.graylog.org/en/3.0/pages/installation/os/ubuntu.html

then the cluster was red so
curl -X DELETE 'http://localhost:9200/_all'
Now the cluster status is green in graylog.

The issue i am facing now is on the search tab in graylog:
Index not found for query: graylog_14. Try recalculating your index ranges

I tried recalculating indexes but it did not help. I searched and it seems i should purge index_ranges in mongo but no idea how to to it after seraching google. Any help please?
When i go to mongo shell and issue
db
I just see database named “test” only.

How can i clear the mongo database for graylog and then restart graylog/rebuild the indexes so everything works fine again?

I haven’t ever had to clear out the indexes from the MongoDB so, ‘WARNING: HERE BE DRAGONS’ for what I am going to say below, I cannot confirm or deny whether the below will resolve the issue or whether additional issues will be encountered however, this is what I would try…

First, the MongoDB command you want to display all the databases is show dbs – Just running db will show you the database you are currently ‘using’.

Configuration for the index sets is located in graylog.index_sets
Configuration for the index ranges is located in graylog.index_ranges

I am wary to advise that you clear the index_sets collection, Graylog may recreate the default contents of that collection when it starts up however, I cannot confirm or deny whether that will happen.

From looking at the contents of the collections relating to the indexes and looking at the error message you have show above, I would try clearing out the graylog.index_ranges collection.

Steps I would take:

  • Stop the graylog-server Service
  • Connect to the MongoDB shell and run the following
    – use graylog
    – db.index_ranges.remove({}) # This will delete all documents from the index_ranges collection.
    – db.index_ranges.find({}) # Execute this to confirm that the collection is empty.
    – exit
  • Start the graylog-server Service
  • Watch the graylog-server log file ( tail -f /var/log/graylog-server/server.log )

If you have deleted the indexes from ES, I think this should be fine.

GL!

1 Like

Alternatively, my colleague pointed out that, since you actually have access to the web interface, you should be able to delete those indexes from there. I wouldn’t delete the index set(s) themselves, just the indexes within them.

If you go into: System > Indices and then go into your index set(s), you can delete the indexes individually from there.

Lack of caffeine causes me to forget the simple solutions, apparently.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.