Can’t search messages on Syslog UDP via rsylog (Loading without end )

MDJ · July 11, 2018, 3:51pm

Hi

Ms,Mrs

I have some problems to searching syslogs messages on my rules Syslog UDP .
I done a inputs for send to my log of my server Rsyslog from my server Graylog .
My server Rsylog listen on the port 514 , My server Rsylog receive my log of my Firewall.
My rules work , I have a message (msg/s ) and throughput ( Network IO) when I try to show received messages my page loading without end .
And I have a other problem , I think to lie .I have a notification " Deflector exists as an index and is not an alias"
I try to change “elasticsearch_discovery_enabled = false” it’s not work
I try to stop graylog and delete garylog_deflector and start graylog It’s not work
I have a mental block

I’m using Centos7 ,Graylog version2,4

Can you help me ?
Thanks

PS: I am French I hope which you understand my message and I am sorry in order to my fault spelling.

jochen · July 12, 2018, 7:01am

Which version of Elasticsearch are you using?
Take note that Graylog 2.x doesn’t work with Elasticsearch 6.x.

What’s in the logs of your Graylog and Elasticsearch nodes?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#rpm-package

See http://docs.graylog.org/en/2.4/pages/faq.html#how-do-i-fix-the-deflector-exists-as-an-index-and-is-not-an-alias-error-message for details about this message and how to fix it.

MDJ · July 12, 2018, 8:43am

Hi,

Jochen,

I use a version 6.3.0 of Elasticsearch. i done a curl -XGET http://localhost:9200 .

i found a files log of Graylog and Elasticsearch nodes.I have this messages ( look attached documents )

In order to my notification , I should use this line "curl -X DELETE http://127.0.0.1:9300/graylog_deflector or curl -XDELETE http://127.0.0.1:9000/api/system/deflector/cycle ?

What’s a good lines use ?

jochen · July 12, 2018, 8:47am

Graylog 2.x is not compatible with Elasticsearch 6.x as mentioned multiple times in the documentation.

You’ll have to downgrade to Elasticsearch 5.6.10.

MDJ · July 12, 2018, 8:51am

I will try to downgrade elasticsearch and I will see if i can show my messages.

MDJ · July 12, 2018, 1:10pm

I have installed rpm Elasticsearch 5X.
I have a other notification.
First : when I try check my version of Elasticsearch it’s not working it’s write connexion refused .
I try to change network.host :0.0.0.0 and 127.0.0.1 in /etc/elastcsearch/elasticsearch.yml it’s not works
Second: I have this messages :

when I execute

lsof -i

netstat -tplen

Elasticsearch has dissapear . it’s good or bad ?

jochen · July 12, 2018, 1:36pm

Check the logs of your Graylog and Elasticsearch nodes.

MDJ · July 12, 2018, 1:54pm

I have this messages error.

var/log/garylog-server/server.log

 2018-07-12T15:46:09.484+02:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
    2018-07-12T15:46:09.484+02:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
    2018-07-12T15:46:09.902+02:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
    2018-07-12T15:46:10.943+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
    2018-07-12T15:46:11.878+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
    2018-07-12T15:46:11.893+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).

/var/log/elasticsearch/elasticsearch.log

Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
  at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayMetaState
    for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:196)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:335)
	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:113)
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57)
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:92)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:176)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
	at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
	at org.elasticsearch.node.Node.<init>(Node.java:499)
	at org.elasticsearch.node.Node.<init>(Node.java:245)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:328)
	... 46 more
Caused by: java.lang.IllegalArgumentException: Template must not be null
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:94)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:374)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:502)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1164)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1211)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1202)
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:203)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:323)
	... 46 more

I think it’s java the problems ?
My Index by default is disappear too

jochen · July 12, 2018, 2:11pm

You have to delete the data directory of Elasticsearch.
Elasticsearch 5.x is not able to read data files created by Elasticsearch 6.x.

MDJ · July 12, 2018, 2:24pm

How i did that ?
With the line "curl -X DELETE http://127.0.0.1:9300/graylog_deflector …
So I should delete all my configuration (mongodb, elasticsearch and graylog ) .
And I do an installation again

jochen · July 12, 2018, 2:49pm

You have to delete all files in the Elasticsearch data directory /var/lib/elasticsearch/ while Elasticsearch is not running.

See https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-node.html#_node_data_path_settings for details.

MDJ · July 13, 2018, 8:02am

I deleted all files in the Elasticsearch /var/lib/elasticsearch , I have a same error in my files log Graylog and Elasticsearch .nodes
I have note in my Nodes I receive message in but not out , it’s normal?
My line curl -XGET 'http://localhost:9200 it’s not work befor i can see my version and my status Elasticsearch Cluster

jochen · July 13, 2018, 8:17am

What was the command you’ve executed and what was its complete output?

Which error message is that exactly?

That’s because your Elasticsearch node is still not starting. The reason is pretty certainly mentioned in its log files.

MDJ · July 16, 2018, 7:14am

Jochen : What was the command you’ve executed and what was its complete output?

I deleted my files via WinSCP .I dont’ understand “What was its complete output?”

Which error message is that exactly?

I have a same error which my fifth post , I rewrite you

var/log/graylog-server/server.log

2018-07-12T15:46:09.484+02:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2018-07-12T15:46:09.484+02:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2018-07-12T15:46:09.902+02:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
2018-07-12T15:46:10.943+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
2018-07-12T15:46:11.878+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
2018-07-12T15:46:11.893+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).

var/log/elasticsearch/elasticsearch.log

Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
  at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayMetaState
    for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:196)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:335)
	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:113)
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57)
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:92)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:176)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
	at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
	at org.elasticsearch.node.Node.<init>(Node.java:499)
	at org.elasticsearch.node.Node.<init>(Node.java:245)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:328)
	... 46 more
Caused by: java.lang.IllegalArgumentException: Template must not be null
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:94)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:374)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:502)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1164)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1211)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1202)
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:203)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:323)
	... 46 more

jochen · July 16, 2018, 8:50am

In that case you didn’t delete the Elasticsearch data files.

MDJ · July 19, 2018, 7:04am

i’m sorry to not reply yesterday , i was busy,

Before i execute your command i have to delete all file in /var/lib/elasticsearch.
This is my step if i understand:

systemctl stop graylog-server.service
systemctl stop elasticsearch
delete file nodes in /var/lib/elastcisearch
image.png793×95 5.59 KB
systemctl restart elasticsearch and graylog

it’s good ?

MDJ · July 27, 2018, 2:02pm

Sorry jochen i tninking I will write you a private .
A other solution in order to my issue ?
Thanks in advance for the help,

MDJ · August 7, 2018, 9:07am

I thinking a my problem of research messages on my rules Syslog UDP .
With my old post ( eslasticsearch.log and server.log ),the problem can come of my server rsyslog version ?
I explain me ,
I should do a template in my file config rsyslog in order to that work or not ?
I look on my server my version of rsyslog is 8.24.0 .
If i read this url https://github.com/Graylog2/graylog-guide-syslog-linux
i write a good instruction which is “. @XXX.YYYY.WWW.ZZZZ:5140;RSYSLOG_SyslogProtocol23Format” but I doubt .
It’s good ?

MDJ · August 17, 2018, 8:14am

Hello,

A Answer at my problem?

jan · August 17, 2018, 9:12am

Your question is not clear - maybe rephrase that and someone can help you

Topic		Replies	Views
Don't receive log / graylog deflector Graylog Central (peer support)	2	526	May 14, 2019
Deflector exists as an index and is not an alias. Again Graylog Central (peer support)	7	13773	July 24, 2018
Deflector exists as an index and is not an alias. Reopen Graylog Central (peer support)	5	3248	December 27, 2017
Graylog just gives 'Loading' message for infinite time Graylog Central (peer support)	4	3019	January 23, 2018
Graylog_deflector exists as an indexer and is not an alias Graylog Central (peer support)	4	5124	November 12, 2018

Can’t search messages on Syslog UDP via rsylog (Loading without end )

lsof -i

netstat -tplen

Related Topics