Hello,
I have a Graylog who is processing between 15 000 and 20 000 msg/s, for years.
After changing the Message Processing order , we got an huge increase en MongoDB activity , which is slowing down everything.
(in the graph, the change has been made at 11h10)
I have no idea if this a common activity on MongoDB for Graylog, can someone share his MongoDB stats with the number of messages/s ?
MongoDB : 8.0.17
Graylog : 7.0.4
Thanks.
What did you change about the message processing order?
Are you using pipelines or extractors?
The main change was, I put the Pipeline processor before Illuminate processor.
**
Before**
After
Yes I have :
2 extractors on my main input
around 100 Pipelines rules in around 20 Pipelines.
Hey @benoitp,
I would suggest enabling profiling within MongoDB to catch slow queries and get a better understanding of which collections are being queried the most.
If slow queries are not the issue then raise profiling to 2 for a short amount of time, this will capture all traffic and again give some insight into which queries might be causing this. Don’t leave the profiling level at 2 for very long.
Changing Pipelines and Illuminate in the processing order let me think, that Illuminate did something, which was more efficient than your pipelines.
Do you do mongoDB Lookups (requires Graylog Enterprise) in your pipelines, or do you have other heavy queries on the MongoDB?
I will enable profiling in MongoDB as sugested by @Wine_Merchant
I currently learning how to do it properly.
I have some lookup tables (I have an Enterprise license), I tried to disable theses pipelines, with no change.
I did some profiling at level 2.
nearly all the requests are for the “cluster_config” collections, and they search “org.graylog2.shared.buffers.processors.TimeStampConfig”
{ "command.find": "cluster_config" }
{
"op": "query",
"ns": "graylog.cluster_config",
"command": {
"find": "cluster_config",
"filter": {
"type": "org.graylog2.shared.buffers.processors.TimeStampConfig"
},
"limit": 1,
"singleBatch": true,
"$db": "graylog",
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1773334759,
"i": 21
}
},
"signature": {
"hash": {
"$binary": {
"base64": "vMQrlpugmLIiM/x3YFV8BvlzL5M=",
"subType": "00"
}
},
"keyId": {
"$numberLong": "7560239835168899089"
}
}
},
"lsid": {
"id": {
"$uuid": "b7e8c810-e036-45e6-8564-59337ff14b8d"
}
}
},
"keysExamined": 1,
"docsExamined": 1,
"nBatches": 1,
"cursorExhausted": true,
"numYield": 0,
"nreturned": 1,
"queryShapeHash": "6D6613FCDD401EAA01CD4E9067006C645D073DB8736BF1F202A1E1966D9A4097",
"queryFramework": "classic",
"locks": {
"Global": {
"acquireCount": {
"r": {
"$numberLong": "1"
}
}
}
},
"flowControl": {},
"readConcern": {
"level": "local",
"provenance": "implicitDefault"
},
"responseLength": 420,
"protocol": "op_msg",
"cpuNanos": 49898,
"millis": 0,
"planSummary": "EXPRESS_IXSCAN { type: 1 }",
"planningTimeMicros": 23,
"execStats": {
"isCached": false,
"stage": "EXPRESS_IXSCAN",
"keyPattern": "{ type: 1 }",
"indexName": "unique_type",
"keysExamined": 1,
"docsExamined": 1,
"nReturned": 1
},
"ts": {
"$date": "2026-03-12T16:59:20.160Z"
},
"client": "10.66.85.5",
"allUsers": [
{
"user": "graylog",
"db": "graylog"
}
],
"user": "graylog@graylog"
}
And this document is empty :
{
"_id": {
"$oid": "69a956187eb4340e144364f0"
},
"type": "org.graylog2.shared.buffers.processors.TimeStampConfig",
"payload": {},
"last_updated": {
"$date": "2026-03-10T15:59:24.386Z"
},
"last_updated_by": "dc9aad51-4473-4a85-8d2d-90237ea0f58f"
}
For this moment I have no idea why this documents seems to be read for every message and why it’s empty.
Can someone check if this document on their side contain something ?
Hey @benoitp,
Nice work digging this up.
An initial stab in the dark, under system/configuration/message processors is the option for “Future Timestamp Normalization” enabled? If so could you try disabling briefly and seeing if Mongo utilization drops.
From my configuration : Future Timestamp Normalization: Disabled
I saw on a test platform, who was re-installed from scratch recently, that this document was not present in the MongoDB database.
So I did a test on my staging platform (who has the same problem, and the same document, like my production platform), I dropped from “cluster_config” the document “type”: “org.graylog2.shared.buffers.processors.TimeStampConfig”
I ingested a bunch of 100k messages, and the activity of MongoDB didn’t moved !!!
Before the deletion of this document, I if I did this, the operations rate was skyrocketing .
I don’t believed it, I think I found the problem.
But we are Friday, and I will not try this change on production. I will wait Monday, and be sure I will report if it’s the solution.
I also experienced this same issue and found the same cause. I think there is a bug in a recent version that must be changing the behavior surrounding Timestamp Normalization when you make a change to the message processors, even if you are not changing the Timestamp Normalization setting. I saw the performance of my Mongo nodes go back to normal after I enabled Timestamp Normalization, and after I disable it again the Mongo master node is back to being overloaded by calls fetching “org.graylog2.shared.buffers.processors.TimeStampConfig".I didn’t find any bug reports in github covering this issue.
