I have an active subscription at alienvault otx.
most of the times the query is ok and i get the data.
But on some request the data-adapter having a timeout.
if i copy the failed ip into my test-lookup from lookup-table or data-adapter directly, the call works in unter 2 seconds.
Timeouts:
HTTP Connect Timeout 2000 ms
HTTP Write Timeout 3000 ms
HTTP Read Timeout 9000 ms
System Information:
Debian 12 (bookworm)
Graylog 6.3.1+7bd8532 (enterprise)
(updated today, on 6.1 i had the same problem)
for debug purposes i tried to reroute the call via dns reroute (via hosts-file entry on graylog-server) to my own nginx-server which did the api-call for graylog and returns the object.
This seems to work fine with much less timeouts.
All Request are non private IPs/Hostnames and the call works if i copy the ip/hostname to the lookup-table or data-adapter testpage.
2025-07-16T11:41:44.510+02:00 ERROR [OTXDataAdapter] OTX IPv4 request error for key <17.248.213.64>
java.net.SocketTimeoutException: timeout
at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:146) ~[graylog.jar:?]
at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:161) ~[graylog.jar:?]
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:339) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:430) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:323) ~[graylog.jar:?]
at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180) ~[graylog.jar:?]
at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at org.graylog2.shared.bindings.providers.OkHttpClientProvider$UserAgentInterceptor.intercept(OkHttpClientProvider.java:80) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[graylog.jar:?]
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[graylog.jar:?]
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[graylog.jar:?]
at org.graylog.plugins.threatintel.adapters.otx.OTXDataAdapter.doGet(OTXDataAdapter.java:217) ~[graylog.jar:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.get(LookupDataAdapter.java:143) ~[graylog.jar:?]
at org.graylog2.lookup.LookupTable.lambda$lookup$0(LookupTable.java:73) ~[graylog.jar:?]
at org.graylog2.lookup.caches.CaffeineLookupCache.lambda$get$2(CaffeineLookupCache.java:160) ~[graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:167) ~[graylog.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$0(BoundedLocalCache.java:2694) ~[graylog.jar:?]
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) [?:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2692) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2674) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63) [graylog.jar:?]
at org.graylog2.lookup.caches.CaffeineLookupCache.get(CaffeineLookupCache.java:187) [graylog.jar:?]
at org.graylog2.lookup.LookupTable.lookup(LookupTable.java:73) [graylog.jar:?]
at org.graylog2.lookup.LookupTableService$Function.lookup(LookupTableService.java:641) [graylog.jar:?]
at org.graylog.plugins.threatintel.functions.otx.AbstractOTXLookupFunction.lookupIntel(AbstractOTXLookupFunction.java:49) [graylog.jar:?]
at org.graylog.plugins.threatintel.functions.otx.AbstractOTXLookupFunction.lookupIP(AbstractOTXLookupFunction.java:41) [graylog.jar:?]
at org.graylog.plugins.threatintel.functions.otx.OTXIPLookupFunction.evaluate(OTXIPLookupFunction.java:59) [graylog.jar:?]
at org.graylog.plugins.threatintel.functions.otx.OTXIPLookupFunction.evaluate(OTXIPLookupFunction.java:31) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:70) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.expressions.Expression.evaluate(Expression.java:36) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:33) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:22) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStatement(PipelineInterpreter.java:401) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.executeRuleActions(PipelineInterpreter.java:385) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStage(PipelineInterpreter.java:337) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForResolvedPipelines(PipelineInterpreter.java:281) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:153) [graylog.jar:?]
at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:109) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:175) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:145) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:115) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:54) [graylog.jar:?]
at org.graylog2.shared.buffers.PartitioningWorkHandler.onEvent(PartitioningWorkHandler.java:52) [graylog.jar:?]
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) [graylog.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endRead(Unknown Source) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source) ~[?:?]
at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) ~[?:?]
at okio.InputStreamSource.read(JvmOkio.kt:93) ~[graylog.jar:?]
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:128) ~[graylog.jar:?]
... 56 more
I did not modify the threat-intel adapters/tables (except tried various timeout settings and increased the max cache items)
Cache hits and lookup-rate seems to be fine:
The lookups/s sometimes jumps up to 20/s but never seen higher.
The pipeline looks like this:
rule "First Threat Intel (dst ip)"
when
regex("([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)", to_string($message.message)).matches == true
AND
has_field("dst_ip") == true
AND
in_private_net(to_string($message.dst_ip)) == false
AND
// own lookuptable to prevent lookups on whitelisted ips or ranges in otx
lookup_has_value("blacklisted_ips", to_string($message.dst_ip)) == false
AND
lookup_has_value("blacklisted_ip_ranges", to_string($message.dst_ip)) == false
then
let src_addr_intel = threat_intel_lookup_ip(to_string($message.dst_ip), "dst_ip");
let intel = otx_lookup_ip(to_string($message.dst_ip));
let whois_data = whois_lookup_ip(to_string($message.dst_ip), "dst_ip");
let spamhaus = spamhaus_lookup_ip(to_string($message.dst_ip));
set_field("spamhaus_drop", spamhaus.threat_indicated);
set_fields(whois_data);
set_fields(src_addr_intel);
set_field("threat_key", "dst_ip");
set_field("threat_indicated", intel.otx_threat_indicated);
//if on of the lookups gets *threat_indicated = true, replace the "wrong" false
set_field("dst_ip_threat_indicated", replace(to_string(get_field("dst_ip_threat_indicated")),"false",to_string(intel.otx_threat_indicated)));
set_field("threat_ids", intel.otx_threat_ids);
set_field("threat_names", intel.otx_threat_names);
end
Any suggestions on how to resolve this random timeout-error?
PS: if i created this problem in the wrong section, please move it to the desired place. - Thanks
PS2: The created firewall-rule to allow all http(s) calls from graylog to otx was used about 15,6k times since 28th of May, so i dont think this is anywhere near api-limit.
best regards
coffee_is_life