Graylog web interface problem!


(Rıdvan) #1

Hi. I’ve a problem. Elasticsearch, mongodb is working. But graylog service has problem.

/etc/elasticsearch/elasticsearch.yml

cluster.name: graylog
network.host: 192.168.1.222
http.port: 9200

[root@localhost]# curl -XGET ‘http://192.168.1.222:9200/_cluster/health?pretty=true

{
“cluster_name” : “graylog”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 0,
“active_shards” : 0,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}

/etc/graylog/server/server.conf

password_secret = <my_pass_secret>
root_password_sha2 = <my_pass_sha2>
rest_listen_uri = http://192.168.1.222:12900/
web_listen_uri = http://192.168.1.222:9000/

[root@localhost]# systemctl status graylog-server.service
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Pzt 2019-02-04 05:05:46 +03; 3s ago

Process: 20798 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
Main PID: 20798 (code=exited, status=1/FAILURE)

Şub 04 05:05:46 localhost.localdomain systemd[1]: graylog-server.service: main process exited, code=exited, status=1/FAILURE
Şub 04 05:05:46 localhost.localdomain systemd[1]: Unit graylog-server.service entered failed state.
Şub 04 05:05:46 localhost.localdomainsystemd[1]: graylog-server.service failed.

Logs: /var/log/graylog-server - last lines -

2019-02-05T00:09:51.663+03:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 2.5.1 [org.graylog.aws.plugin.AWSPlugin]
2019-02-05T00:09:51.665+03:00 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.5.1 [org.graylog.plugins.beats.BeatsInputPlugin]
2019-02-05T00:09:51.666+03:00 INFO [CmdLineTool] Loaded plugin: CEF Input 2.5.1 [org.graylog.plugins.cef.CEFInputPlugin]
2019-02-05T00:09:51.667+03:00 INFO [CmdLineTool] Loaded plugin: Collector 2.5.1 [org.graylog.plugins.collector.CollectorPlugin]
2019-02-05T00:09:51.668+03:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.5.1 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2019-02-05T00:09:51.669+03:00 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.5.1 [org.graylog.plugins.map.MapWidgetPlugin]
2019-02-05T00:09:51.670+03:00 INFO [CmdLineTool] Loaded plugin: NetFlow Plugin 2.5.1 [org.graylog.plugins.netflow.NetFlowPlugin]
2019-02-05T00:09:51.676+03:00 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.5.1 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2019-02-05T00:09:51.677+03:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.5.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2019-02-05T00:09:52.008+03:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2019-02-05T00:09:52.218+03:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2019-02-05T00:09:55.218+03:00 INFO [InputBufferImpl] Message journal is enabled.
2019-02-05T00:09:55.246+03:00 INFO [NodeId] Node ID: a80a69d0-0d07-434d-aef1-3bd81e8afc37
2019-02-05T00:09:55.547+03:00 INFO [LogManager] Loading logs.
2019-02-05T00:09:55.595+03:00 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index…
2019-02-05T00:09:55.649+03:00 INFO [LogManager] Logs loading complete.
2019-02-05T00:09:55.649+03:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2019-02-05T00:09:55.681+03:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2019-02-05T00:09:55.717+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2019-02-05T00:09:55.795+03:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2019-02-05T00:09:55.813+03:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:22}] to localhost:27017
2019-02-05T00:09:55.817+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 10]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, roundTripTimeNanos=867808}
2019-02-05T00:09:55.831+03:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:23}] to localhost:27017
2019-02-05T00:09:56.348+03:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://192.168.1.222:92000]
2019-02-05T00:09:56.349+03:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2019-02-05T00:09:56.454+03:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2019-02-05T00:09:56.454+03:00 INFO [JestClientFactory] Node Discovery disabled…
2019-02-05T00:09:56.455+03:00 INFO [JestClientFactory] Idle connection reaping disabled…
2019-02-05T00:09:56.837+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:58.950+03:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:58.999+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:59.227+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:59.455+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:59.631+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:09:59.805+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:00.045+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:00.217+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:00.377+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:00.735+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:00.748+03:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:24}] to localhost:27017
2019-02-05T00:10:00.973+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:01.101+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-02-05T00:10:01.314+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.315+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.315+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.315+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.316+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.316+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.316+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.316+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.317+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.317+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.317+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.317+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.318+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]
2019-02-05T00:10:01.318+03:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: Unable to fetch module from resource: ByteArrayResource[bytes=[80, 75, 3, 4, 20, 0, 8, 8, 8, 0, …], encoding=null]


(Jan Doberstein) #2
  1. do not use a aboned version of Graylog - current stable Version is 2.5 and your 2.3 will not get any updates (not even security updates)
  2. check the JAVA Version you have installed on the server (JAVA 8 is the only supported)

(Rıdvan) #3

@jan Thanks for your reply. Again i installed graylog 2.5 . But same error.

Information :

java version   "1.8.0_191"
elasticsearch versi  :  "version" : {
       "number" :            " 6.6.0"
       "lucene_version" : "7.6.0"

Source I’m using >>> http://docs.graylog.org/en/2.5/pages/installation/os/centos.html.


#4

It can’t cause your errors, but it will another :slight_smile:
check netstat -ntlp the elasticsearch’s IP.
In ES config you bind for private IP, but in the log GL try to work with loopback IP.
If you have one node only, maybe change the ES config back to LO IP.

Also after you fixed your problems as @jan wrote


(Rıdvan) #5

@macko003

netstat -ntlp output:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      7511/mongod         
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      7692/dnsmasq        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      7121/sshd           
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      7119/cupsd          
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      7427/master         
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 192.168.1.222:9200      :::*                    LISTEN      20523/java          
tcp6       0      0 192.168.1.222:9300      :::*                    LISTEN      20523/java          
tcp6       0      0 :::22                   :::*                    LISTEN      7121/sshd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      7119/cupsd          
tcp6       0      0 ::1:25                  :::*                    LISTEN      7427/master

(Rıdvan) #6

[root@localhost ~]# systemctl status elasticsearch.service`

> ● elasticsearch.service - Elasticsearch
>        Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
>        Active: active (running) since Pzt 2019-02-04 22:57:20 +03; 3min 58s ago
>          Docs: http://www.elastic.co
>      Main PID: 22362 (java)
>         Tasks: 40
>        CGroup: /system.slice/elasticsearch.service
>                └─22362 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkad...
> 
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:127)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:302)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.cli.Command.main(Command.java:90)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
>     Şub 04 22:57:26 localhost.localdomain elasticsearch[22362]: 2019-02-04 22:57:26,614 main ERROR Null object returned for Delete in DefaultRolloverStrategy.

[root@localhost ~]# systemctl status graylog-server.service

> ● graylog-server.service - Graylog server
>    Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
>    Active: active (running) since Pzt 2019-02-04 22:57:33 +03; 5min ago
>      Docs: http://docs.graylog.org/
>  Main PID: 22575 (graylog-server)
>     Tasks: 63
>    CGroup: /system.slice/graylog-server.service
>            ├─22575 /bin/sh /usr/share/graylog-server/bin/graylog-server
>            └─22576 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -X...
> 
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.graylog2.bootstrap.Main.main(Main.java:44)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: Caused by: java.lang.RuntimeException: XSD validation failed against the new schema (cvc-enumeration-valid: Value 'ıdentıty' …).
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kproject.models.KieModuleModelImpl$KieModuleValidator.validate(KieModuleModelImpl.java:309)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kproject.models.KieModuleModelImpl$KieModuleValidator.validate(KieModuleModelImpl.java:274)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kproject.models.KieModuleModelImpl$KieModuleValidator.access$100(KieModuleModelImpl.java:249)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kproject.models.KieModuleModelImpl$kModuleMarshaller.fromXML(KieModuleModelImpl.java:196)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kproject.models.KieModuleModelImpl.fromXML(KieModuleModelImpl.java:146)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: at org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:243)
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: ... 119 more
> Şub 04 22:57:57 localhost.localdomain graylog-server[22575]: Caused by: org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 194; cvc-enumeration-valid: Value 'ıdentıty'…enumeration.
> Hint: Some lines were ellipsized, use -l to show in full.

#7

check your graylog’s XML config’s third lines…

eg.


(Rıdvan) #8

Yes i checked but same. I didn’t change anything in there before.

Same problem as in there but not working for me.


#9

any do you see any connection between you issue and the linked one?
did you try what @jan wrote?


(Rıdvan) #10

Yes. I tried . Versions Graylog 2.5 Java 1.8.0_191.

I changed last log in /var/log/graylog-server/server.log in first post.


#11

I have no more idea.
You can try to remove the package, and reinstall it from an official repo.


(Rıdvan) #12

Problem is foreign language. I installed Centos ENG and it worked.

Thanks for support.