Elastic.service fails to start; [Warning] Can't connect to display manager

Graylog Central (peer support)
1

Hello
I am looking for help to get elasticsearch.service running again. It fails at startup and if I try to restart. I had a working environment, only that I had to restart elasticsearch.service manually after a reboot. So I tried to fix this by re-installing the service as per the installation guide:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl restart elasticsearch.service
sudo systemctl --type=service --state=active | grep elasticsearch

After that, the elasticsearch.service fails permanently.


status elastisearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: signal) since Sat 2022-02-26 20:40:39 CET; 1min 46s ago
     Docs: https://www.elastic.co
  Process: 31924 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
 Main PID: 31924 (code=killed, signal=KILL)

Feb 26 20:40:30 ubuntu_1804 systemd[1]: Starting Elasticsearch...
Feb 26 20:40:39 ubuntu_1804 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Feb 26 20:40:39 ubuntu_1804 systemd[1]: elasticsearch.service: Failed with result 'signal'.
Feb 26 20:40:39 ubuntu_1804 systemd[1]: Failed to start Elasticsearch.


journalctl -xe
-- The start-up result is RESULT.
Feb 26 20:40:34 ubuntu_1804 runuser[32108]: pam_unix(runuser-l:session): session opened for user admin by (uid=0)
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: No protocol specified
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: xhost:  unable to open display ":1"
Feb 26 20:40:34 ubuntu_1804 runuser[32108]: pam_unix(runuser-l:session): session closed for user admin
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: [Info] display-manager: gdm3
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: [Info] users: "[120, 1000]"
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: [Info] XAUTHORITY: /run/user/120/gdm/Xauthority; DISPLAY: :1
Feb 26 20:40:34 ubuntu_1804 qnap-x.py[251]: [Warning] Can't connect to display manager. Maybe, it's stil in greeter phase. (Can't connect to display ":1": b'No protocol specified\n')
Feb 26 20:40:39 ubuntu_1804 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Feb 26 20:40:39 ubuntu_1804 systemd[1]: elasticsearch.service: Failed with result 'signal'.
Feb 26 20:40:39 ubuntu_1804 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit elasticsearch.service has failed.
--
-- The result is RESULT.
Feb 26 20:40:40 ubuntu_1804 sudo[31921]: pam_unix(sudo:session): session closed for user root
Feb 26 20:40:41 ubuntu_1804 systemd[1]: Started Session c33460 of user admin.
-- Subject: Unit session-c33460.scope has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit session-c33460.scope has finished starting up.
--
-- The start-up result is RESULT.
Feb 26 20:40:41 ubuntu_1804 runuser[32199]: pam_unix(runuser-l:session): session opened for user admin by (uid=0)
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: No protocol specified
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: xhost:  unable to open display ":1"
Feb 26 20:40:42 ubuntu_1804 runuser[32199]: pam_unix(runuser-l:session): session closed for user admin
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: [Info] display-manager: gdm3
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: [Info] users: "[120, 1000]"
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: [Info] XAUTHORITY: /run/user/120/gdm/Xauthority; DISPLAY: :1
Feb 26 20:40:42 ubuntu_1804 qnap-x.py[251]: [Warning] Can't connect to display manager. Maybe, it's stil in greeter phase. (Can't connect to display ":1": b'No protocol specified\n')
Feb 26 20:40:42 ubuntu_1804 python3[1178]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Feb 26 20:40:42 ubuntu_1804 python3[1178]:                                  Dload  Upload   Total   Spent    Left  Speed
Feb 26 20:40:42 ubuntu_1804 python3[1178]: [158B blob data]
Feb 26 20:40:44 ubuntu_1804 python3[1178]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Feb 26 20:40:44 ubuntu_1804 python3[1178]:                                  Dload  Upload   Total   Spent    Left  Speed
Feb 26 20:40:44 ubuntu_1804 python3[1178]: [158B blob data]


graylog-server.log
2022-02-26T20:45:11.402+01:00 ERROR [IndexFieldTypePollerPeriodical] Couldn't update field types for index set <General-Index/5fa696294ad26c1360d75dca>
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:140) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:100) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:93) ~[?:?]
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.resolveAlias(IndicesAdapterES7.java:139) ~[?:?]
        at org.graylog2.indexer.indices.Indices.aliasTarget(Indices.java:145) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getActiveWriteIndex(MongoIndexSet.java:202) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$schedule$4(IndexFieldTypePollerPeriodical.java:249) ~[graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_312]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: java.net.ConnectException: Connection refused
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.getAlias(IndicesClient.java:1315) ~[?:?]
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$resolveAlias$2(IndicesAdapterES7.java:139) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:98) ~[?:?]
        ... 12 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_312]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_312]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[?:?]
        ... 1 more

graylog version 4.2
elasticsearch version 7.11.2

2

This looks like its trying to start up elasticsearch with a GUI in gdm3 via qnap-x.py which doesn’t sound right… pretty sure if you want a GUI on Elasticsearch you would use Kibana. Could you talk a little more about your install? Of note: The highest supported version of Elasticsearch with Graylog is 7.10.

1 Like
3

I have a simple installation and followed the installation guide. No additional tools like Kibana.

4

What is your Elasticsearch config?

cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"

Anything interesting to post from /var/log/elasticsearch ?

5

Elasticsearch config

cluster.name: graylog
action.auto_create_index: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Elasticsearch log

[2022-03-02T17:23:40.089+0000][26365][gc] Using G1
[2022-03-02T17:23:40.129+0000][26365][gc,init] Version: 17.0.2+8 (release)
[2022-03-02T17:23:40.129+0000][26365][gc,init] CPUs: 4 total, 4 available
[2022-03-02T17:23:40.129+0000][26365][gc,init] Memory: 7849M
[2022-03-02T17:23:40.129+0000][26365][gc,init] Large Page Support: Disabled
[2022-03-02T17:23:40.129+0000][26365][gc,init] NUMA Support: Disabled
[2022-03-02T17:23:40.129+0000][26365][gc,init] Compressed Oops: Enabled (32-bit)
[2022-03-02T17:23:40.129+0000][26365][gc,init] Heap Region Size: 1M
[2022-03-02T17:23:40.129+0000][26365][gc,init] Heap Min Capacity: 1954M
[2022-03-02T17:23:40.129+0000][26365][gc,init] Heap Initial Capacity: 1954M
[2022-03-02T17:23:40.129+0000][26365][gc,init] Heap Max Capacity: 1954M
[2022-03-02T17:23:40.129+0000][26365][gc,init] Pre-touch: Disabled
[2022-03-02T17:23:40.129+0000][26365][gc,init] Parallel Workers: 4
[2022-03-02T17:23:40.129+0000][26365][gc,init] Concurrent Workers: 1
[2022-03-02T17:23:40.129+0000][26365][gc,init] Concurrent Refinement Workers: 4
[2022-03-02T17:23:40.129+0000][26365][gc,init] Periodic GC: Disabled
[2022-03-02T17:23:40.129+0000][26365][gc,metaspace] CDS archive(s) not mapped
[2022-03-02T17:23:40.129+0000][26365][gc,metaspace] Compressed class space mapped at: 0x0000000100000000-0x0000000140000000, reserved size: 1073741824
[2022-03-02T17:23:40.129+0000][26365][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x140000000
[2022-03-02T17:23:40.444+0000][26365][gc,heap,exit] Heap
[2022-03-02T17:23:40.444+0000][26365][gc,heap,exit]  garbage-first heap   total 2000896K, used 1538K [0x0000000085e00000, 0x0000000100000000)
[2022-03-02T17:23:40.444+0000][26365][gc,heap,exit]   region size 1024K, 2 young (2048K), 0 survivors (0K)
[2022-03-02T17:23:40.444+0000][26365][gc,heap,exit]  Metaspace       used 3432K, committed 3520K, reserved 1056768K
[2022-03-02T17:23:40.444+0000][26365][gc,heap,exit]   class space    used 270K, committed 320K, reserved 1048576K
6

Check out this post - a lot of information packed in there:

2 Likes
7

Thank you @tmacgbay ! configuring a JVM heap size in /etc/elasticsearch/jvm.options did it:

-Xms1024m
-Xmx1024m

I saw other posts recommending this but they had out-of-memory error messages.

2 Likes
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.