Graylog 2.5 to 3.0

celine · May 18, 2020, 2:24pm

Hi,

I’ve upgrade my graylog server to 3.0.

I’ve this events in the graylogs logs:

2020-05-18T16:17:08.452+02:00 ERROR [ConfigurationManagementPeriodical] Error while running migration <V20180924111644_AddDefaultGrokPatterns{2018-09-24T11:16:44Z}>
org.graylog2.contentpacks.exceptions.ContentPackException: Failed to install content pack <a3ce55ad-bdf3-7a50-305c-1e5bf3de6eca/1>
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:158) ~[graylog.jar:?]
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:99) ~[graylog.jar:?]
        at org.graylog2.migrations.V20180924111644_AddDefaultGrokPatterns.upgrade(V20180924111644_AddDefaultGrokPatterns.java:76) ~[graylog.jar:?]
        at org.graylog2.periodical.ConfigurationManagementPeriodical.doRun(ConfigurationManagementPeriodical.java:43) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.graylog2.contentpacks.exceptions.DivergingEntityConfigurationException: Expected Grok pattern for name "HOUR": <(?:2[0123]|[01]?[0-9])>; actual Grok pattern: <(?:[0-2][0-9])>
        at org.graylog2.contentpacks.facades.GrokPatternFacade.compareGrokPatterns(GrokPatternFacade.java:138) ~[graylog.jar:?]
        at org.graylog2.contentpacks.facades.GrokPatternFacade.lambda$findExisting$0(GrokPatternFacade.java:131) ~[graylog.jar:?]
        at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_232]
        at org.graylog2.contentpacks.facades.GrokPatternFacade.findExisting(GrokPatternFacade.java:131) ~[graylog.jar:?]
        at org.graylog2.contentpacks.facades.GrokPatternFacade.findExisting(GrokPatternFacade.java:119) ~[graylog.jar:?]
        at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:131) ~[graylog.jar:?]
        ... 5 more
		
		
[...]

2020-05-18T16:17:31.470+02:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=LINUX, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=null} (channel [id: 0x15d4248d, L:/0.0.0.0:61025]) should be 1048576 but is 425984.

[...]

2020-05-18T16:18:03.124+02:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
		
Caused by: java.io.IOException: Connection is closed
        at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:445) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:677) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1354) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1301) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:457) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:49) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
        at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
        ... 20 more
		
Caused by: java.io.IOException: Peer reset connection
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_232]
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_232]
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_232]
        at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_232]
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_232]
        at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeSimpleBuffer(TCPNIOUtils.java:133) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:126) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:260) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:169) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.write(AbstractNIOAsyncQueueWriter.java:71) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(TCPNIOTransportFilter.java:126) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(TransportFilter.java:191) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ~[graylog.jar:?]
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:890) ~[graylog.jar:?]
        at org.glassfish.grizzly.filterchain.FilterChainContext.write(FilterChainContext.java:858) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.io.OutputBuffer.flushBuffer(OutputBuffer.java:1059) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:709) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:83) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:233) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(UTF8JsonGenerator.java:853) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.NumberSerializers$IntegerSerializer.serialize(NumberSerializers.java:137) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1429) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951) ~[graylog.jar:?]
        at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) ~[graylog.jar:?]
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86) ~[graylog.jar:?]
        ... 20 more

After a second restart, I have no more errors. Weird ??

Also my received buffer is set to 1048576 but events say they are set to 425984???

On my second node I’ve:

2020-05-18T16:29:39.757+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:232) ~[graylog.jar:?]
        at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:355) ~[graylog.jar:?]
[...]
2020-05-18T16:29:44.851+02:00 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.

Someone could help me ?

Thanks

jan · May 19, 2020, 9:15am

he @celine

not every warning in the log is something to take care of. As I do not have any 2.5 Graylog to update to 3.0 (the 3.0 is more than a year old already … ) I can’t recall if that was a known issue or not.

Speak the installation of the default grok pattern.

On the second node you have the “did not find meta” information most likely because the time of both servers differ or is not in sync. That is mostly the reason for this.

celine · May 19, 2020, 9:40am

Hi @jan

I’ve upgrade to 3.1.4 since.

I’ve same error.

I force ntp resync with ntpd -gq on all nodes. I’m waiting a bit to see if it comes back.

I don’t understand what you mean for grok?

Also, I’ve these errors:

ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/...>
ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/...>
ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/...>
ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/...>

These modules are disabled (seen in configuration tab).

What should you do to avoid having alerts about buffers?

An other warn and error in logs I seen:

ERROR [CmdLineTool] Invalid configuration
WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
WARN  [ProxiedResource] Unable to call http://10.0.0.1:9000/api/system/metrics/multiple on node <ID>
ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource

What are these errors?

And the last for the moment was about Graylog events and system events indices:

WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog Events" (ID) doesn't exist yet
WARN  [IndexFieldTypePollerPeriodical] Active write index for index set "Graylog System Events" (ID) doesn't exist yet

I don’t understand this warning, because in indices tab, I seen:

Graylog Events 1 index, 202 documents - Index prefix:    gl-events
Graylog System Events 1 index, 0 documents - Index prefix:    gl-system-events

Thank you for your time spent

EDIT :

I still got the [NodePingThread] Did not find meta info of this node. Re-registering. message after NTP synchronization

makarands · May 20, 2020, 11:09am

@celine: Above error was pointing towards wrong/old configuration parameter configured and hence reporting multiple Error’s and warnings to you.

You can check configurational changes from Graylog 2.5 to 3.0 by checking out below documentation link which will help you lot to make corrective changes.

https://docs.graylog.org/en/3.2/pages/upgrade/graylog-3.0.html

celine · May 26, 2020, 7:32am

Thanks I debug it

system · June 9, 2020, 7:32am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issues with upgrading Graylog from 2.5 to 3.0 Graylog Central (peer support)	9	1430	April 30, 2019
Graylog 2.5 to 3.0 Upgrade Failing Graylog Central (peer support)	5	2462	February 28, 2019
Upgrade from 2.3.2 to 3.1.3 provides exceptions Graylog Central (peer support)	5	894	January 17, 2020
Upgrading from 2.2 -> 2.3 Graylog Central (peer support)	3	714	October 10, 2017
Broken following 3.0 upgrade Graylog Central (peer support)	8	905	March 28, 2019

Graylog 2.5 to 3.0

Related topics