Problem - unprocessed messages

oshri1091 · October 31, 2019, 1:43pm

Hi.

I have a graylog server, with one node.
there is 4 different inputs .
every couple hour, i get a thousands of unprocessed messages, and not get logs.
how i can track and find The source of the problem.

tmacgbay · October 31, 2019, 1:52pm

What version are you on? What have you done so far?

Have you watched the log file? Did you see anything?
tail -f /var/log/graylog-server/server.log

Have you looked at the details on all the related extractors?
System->inputs-> manage Extractors -> details (on Graylog v3)
Are any of those taking a long time? (I think extractors don’t time out while pipeline rules do)

Any other symptoms or anomalies that you can thing of that would help diagnose?

Post up anything odd - make sure you use formatting properly…

oshri1091 · October 31, 2019, 2:18pm

Hi .

this the log when it’s return:

2019-10-31T13:46:29.577Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPED
2019-10-31T13:46:29.577Z INFO [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2019-10-31T13:46:29.579Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now TERMINATED
2019-10-31T13:46:29.579Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STARTING
2019-10-31T13:46:29.588Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now RUNNING
2019-10-31T13:47:00.374Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPING
2019-10-31T13:47:00.379Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPED
2019-10-31T13:47:00.380Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now TERMINATED
2019-10-31T13:47:08.214Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STARTING
2019-10-31T13:47:08.241Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now RUNNING
ubuntu@Td-GrayLog:/var/log/graylog-server$
ubuntu@Td-GrayLog:/var/log/graylog-server$
ubuntu@Td-GrayLog:/var/log/graylog-server$ tail -f server.log
2019-10-31T13:46:29.577Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPED
2019-10-31T13:46:29.577Z INFO [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2019-10-31T13:46:29.579Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now TERMINATED
2019-10-31T13:46:29.579Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STARTING
2019-10-31T13:46:29.588Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now RUNNING
2019-10-31T13:47:00.374Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPING
2019-10-31T13:47:00.379Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STOPPED
2019-10-31T13:47:00.380Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now TERMINATED
2019-10-31T13:47:08.214Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now STARTING
2019-10-31T13:47:08.241Z INFO [InputStateListener] Input [Raw/Plaintext UDP/5d0793d60835ce0218b2ed5b] is now RUNNING
2019-10-31T14:11:16.693Z ERROR [InputServiceImpl] Extractor not found.
2019-10-31T14:11:53.203Z ERROR [InputServiceImpl] Extractor not found.
2019-10-31T14:13:22.886Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
2019-10-31T14:13:22.896Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
2019-10-31T14:13:22.897Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
2019-10-31T14:13:22.923Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
2019-10-31T14:13:22.923Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 1.2.840.113556.1.4.319
2019-10-31T14:13:22.924Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
2019-10-31T14:13:22.924Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
2019-10-31T14:13:22.925Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 1.2.840.113556.1.4.473
2019-10-31T14:13:22.932Z INFO [DefaultLdapCodecService] Registered pre-bundled control factory: 1.2.840.113556.1.4.474
2019-10-31T14:13:22.952Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
2019-10-31T14:13:22.953Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
2019-10-31T14:13:22.961Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
2019-10-31T14:13:22.962Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
2019-10-31T14:13:22.988Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.319
2019-10-31T14:13:22.988Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
2019-10-31T14:13:22.988Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
2019-10-31T14:13:22.989Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.42.2.27.8.5.1
2019-10-31T14:13:22.989Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.9
2019-10-31T14:13:22.997Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 2.16.840.1.113730.3.4.10
2019-10-31T14:13:22.997Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.3
2019-10-31T14:13:22.998Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.4
2019-10-31T14:13:22.998Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.1
2019-10-31T14:13:22.999Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.2
2019-10-31T14:13:22.999Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.473
2019-10-31T14:13:22.999Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.474
2019-10-31T14:13:23.035Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.841
2019-10-31T14:13:23.042Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.417
2019-10-31T14:13:23.043Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.1413
2019-10-31T14:13:23.043Z INFO [CodecFactoryUtil] Registered pre-bundled control factory: 1.2.840.113556.1.4.528
2019-10-31T14:13:23.044Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.1.8
2019-10-31T14:13:23.052Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.8
2019-10-31T14:13:23.070Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.3
2019-10-31T14:13:23.071Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.6
2019-10-31T14:13:23.078Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.5
2019-10-31T14:13:23.088Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.1
2019-10-31T14:13:23.098Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.3
2019-10-31T14:13:23.099Z INFO [CodecFactoryUtil] Registered pre-bundled extended operation factory: 1.3.6.1.4.1.1466.20037
2019-10-31T14:13:25.580Z ERROR [LdapUserAuthenticator] Error during LDAP user account sync. Cannot log in user admin
java.lang.RuntimeException: MessageType : BIND_RESPONSE
Message ID : 4
BindResponse
Ldap Result
Result code : (INVALID_CREDENTIALS) invalidCredentials
Matched Dn : ‘’
Diagnostic message : ‘80090308: LdapErr: DSID-0C09041C, comment: AcceptSecurityContext error, data 52e, v4563’

    at org.graylog2.security.ldap.LdapConnector.authenticate(LdapConnector.java:402) ~[graylog.jar:?]
    at org.graylog2.security.realm.LdapUserAuthenticator.doGetAuthenticationInfo(LdapUserAuthenticator.java:117) [graylog.jar:?]
    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:219) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:269) [graylog.jar:?]
    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [graylog.jar:?]
    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [graylog.jar:?]
    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:274) [graylog.jar:?]
    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroSecurityContext.loginSubject(ShiroSecurityContext.java:107) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroAuthenticationFilter.filter(ShiroAuthenticationFilter.java:48) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

2019-10-31T14:14:22.553Z ERROR [LdapUserAuthenticator] Error during LDAP user account sync. Cannot log in user admin
java.lang.RuntimeException: MessageType : BIND_RESPONSE
Message ID : 4
BindResponse
Ldap Result
Result code : (INVALID_CREDENTIALS) invalidCredentials
Matched Dn : ‘’
Diagnostic message : ‘80090308: LdapErr: DSID-0C09041C, comment: AcceptSecurityContext error, data 52e, v4563’

    at org.graylog2.security.ldap.LdapConnector.authenticate(LdapConnector.java:402) ~[graylog.jar:?]
    at org.graylog2.security.realm.LdapUserAuthenticator.doGetAuthenticationInfo(LdapUserAuthenticator.java:117) [graylog.jar:?]
    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:219) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:269) [graylog.jar:?]
    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [graylog.jar:?]
    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [graylog.jar:?]
    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:274) [graylog.jar:?]
    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroSecurityContext.loginSubject(ShiroSecurityContext.java:107) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroAuthenticationFilter.filter(ShiroAuthenticationFilter.java:48) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

2019-10-31T14:14:22.588Z ERROR [InputServiceImpl] Extractor not found.
2019-10-31T14:16:13.031Z ERROR [LdapUserAuthenticator] Error during LDAP user account sync. Cannot log in user admin
java.lang.RuntimeException: MessageType : BIND_RESPONSE
Message ID : 4
BindResponse
Ldap Result
Result code : (INVALID_CREDENTIALS) invalidCredentials
Matched Dn : ‘’
Diagnostic message : ‘80090308: LdapErr: DSID-0C09041C, comment: AcceptSecurityContext error, data 52e, v4563’

    at org.graylog2.security.ldap.LdapConnector.authenticate(LdapConnector.java:402) ~[graylog.jar:?]
    at org.graylog2.security.realm.LdapUserAuthenticator.doGetAuthenticationInfo(LdapUserAuthenticator.java:117) [graylog.jar:?]
    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:219) [graylog.jar:?]
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:269) [graylog.jar:?]
    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [graylog.jar:?]
    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [graylog.jar:?]
    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:274) [graylog.jar:?]
    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroSecurityContext.loginSubject(ShiroSecurityContext.java:107) [graylog.jar:?]
    at org.graylog2.shared.security.ShiroAuthenticationFilter.filter(ShiroAuthenticationFilter.java:48) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:132) [graylog.jar:?]
    at org.glassfish.jersey.server.ContainerFilteringStage.apply(ContainerFilteringStage.java:68) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:318) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

2019-10-31T14:16:13.067Z ERROR [InputServiceImpl] Extractor not found

oshri1091 · October 31, 2019, 2:20pm

i thing this line it’s the problem:

2019-10-31T14:14:22.588Z ERROR [InputServiceImpl] Extractor not found.

but i don’t find this extractor…

system · November 14, 2019, 2:20pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Solved Need help unprocessed messages (message in elasticsearch but not appears in graylog) Graylog Central (peer support)	7	7844	August 7, 2018
Messages coming in, not processing Graylog Central (peer support)	2	395	August 11, 2020
Graylog terminology! Graylog Central (peer support)	1	384	August 21, 2017
Message processing - Graylog 3.1 Graylog Central (peer support)	3	405	July 1, 2020
Unprocessed Messages in Journal Graylog Central (peer support)	6	3445	August 24, 2017

Problem - unprocessed messages

Related topics