Can’t search messages on Syslog UDP via rsylog (Loading without end )


#1

Hi

Ms,Mrs

I have some problems to searching syslogs messages on my rules Syslog UDP .
I done a inputs for send to my log of my server Rsyslog from my server Graylog .
My server Rsylog listen on the port 514 , My server Rsylog receive my log of my Firewall.
My rules work , I have a message (msg/s ) and throughput ( Network IO) when I try to show received messages my page loading without end .
And I have a other problem , I think to lie .I have a notification " Deflector exists as an index and is not an alias"
I try to change “elasticsearch_discovery_enabled = false” it’s not work
I try to stop graylog and delete garylog_deflector and start graylog It’s not work
I have a mental block

I’m using Centos7 ,Graylog version2,4

Can you help me ?
Thanks

PS: I am French I hope which you understand my message and I am sorry in order to my fault spelling.


(Jochen) #2

Which version of Elasticsearch are you using?
Take note that Graylog 2.x doesn’t work with Elasticsearch 6.x.

What’s in the logs of your Graylog and Elasticsearch nodes?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#rpm-package

See http://docs.graylog.org/en/2.4/pages/faq.html#how-do-i-fix-the-deflector-exists-as-an-index-and-is-not-an-alias-error-message for details about this message and how to fix it.


#3

Hi,

Jochen,

I use a version 6.3.0 of Elasticsearch. i done a curl -XGET 'http://localhost:9200 .

i found a files log of Graylog and Elasticsearch nodes.I have this messages ( look attached documents )

In order to my notification , I should use this line "curl -X DELETE http://127.0.0.1:9300/graylog_deflector or curl -XDELETE http://127.0.0.1:9000/api/system/deflector/cycle ?

What’s a good lines use ?


(Jochen) #4

Graylog 2.x is not compatible with Elasticsearch 6.x as mentioned multiple times in the documentation.

You’ll have to downgrade to Elasticsearch 5.6.10.


#5

I will try to downgrade elasticsearch and I will see if i can show my messages.


#6

I have installed rpm Elasticsearch 5X.
I have a other notification.
First : when I try check my version of Elasticsearch it’s not working it’s write connexion refused .
I try to change network.host :0.0.0.0 and 127.0.0.1 in /etc/elastcsearch/elasticsearch.yml it’s not works
Second: I have this messages :

when I execute

lsof -i

netstat -tplen

Elasticsearch has dissapear . it’s good or bad ?


(Jochen) #7

Check the logs of your Graylog and Elasticsearch nodes.


#8

I have this messages error.

var/log/garylog-server/server.log

 2018-07-12T15:46:09.484+02:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
    2018-07-12T15:46:09.484+02:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
    2018-07-12T15:46:09.902+02:00 WARN  [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.
    2018-07-12T15:46:10.943+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
    2018-07-12T15:46:11.878+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).
    2018-07-12T15:46:11.893+02:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #135).

/var/log/elasticsearch/elasticsearch.log

Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
  at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayMetaState
    for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source)
  while locating org.elasticsearch.gateway.GatewayService
Caused by: ElasticsearchException[java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IOException[failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]]; nested: IllegalArgumentException[Template must not be null];
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:196)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:335)
	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:113)
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57)
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:92)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
	at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:176)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96)
	at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
	at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42)
	at org.elasticsearch.node.Node.<init>(Node.java:499)
	at org.elasticsearch.node.Node.<init>(Node.java:245)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Caused by: java.io.IOException: failed to read [id:13, legacy:false, file:/var/lib/elasticsearch/nodes/0/_state/global-13.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:328)
	... 46 more
Caused by: java.lang.IllegalArgumentException: Template must not be null
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData.<init>(IndexTemplateMetaData.java:94)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.build(IndexTemplateMetaData.java:374)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:502)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1164)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1211)
	at org.elasticsearch.cluster.metadata.MetaData$2.fromXContent(MetaData.java:1202)
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:203)
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:323)
	... 46 more

I think it’s java the problems ?
My Index by default is disappear too


(Jochen) #9

You have to delete the data directory of Elasticsearch.
Elasticsearch 5.x is not able to read data files created by Elasticsearch 6.x.


#10

How i did that ?
With the line "curl -X DELETE http://127.0.0.1:9300/graylog_deflector
So I should delete all my configuration (mongodb, elasticsearch and graylog ) .
And I do an installation again


(Jochen) #11

You have to delete all files in the Elasticsearch data directory /var/lib/elasticsearch/ while Elasticsearch is not running.

See https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-node.html#_node_data_path_settings for details.


#12

I deleted all files in the Elasticsearch /var/lib/elasticsearch , I have a same error in my files log Graylog and Elasticsearch .nodes
I have note in my Nodes I receive message in but not out , it’s normal?
My line curl -XGET 'http://localhost:9200 it’s not work befor i can see my version and my status Elasticsearch Cluster


(Jochen) #13

What was the command you’ve executed and what was its complete output?

Which error message is that exactly?

That’s because your Elasticsearch node is still not starting. The reason is pretty certainly mentioned in its log files.