Hi,
We are testing the limit on the maximum number of streams one may create and the mongodb has been consuming very high CPU (Top shows close to 700% on an AWS t3a.2xlarge instance with 8 vCPUs) and we’ve tried restarting the containers and upgrading the instance from t3a.xlarge. We’d like to ask if it is expected? Also, the problem that we are facing is, with mongodb this busy, calls like ‘/api/streams’ is not responding.
A sample of the log is as followed (and it keeps on running):
mongodb_1 | 2020-02-26T22:11:06.070+0000 I COMMAND [conn21] command graylog.streams command: find { find: “streams”, filter: { disabled: false }, $db: “graylog”, lsid: { id: UUID(“d579e0a8-d15b-4846-8e2e-48f31639de6a”) }, $readPreference: { mode: “primaryPreferred” } } planSummary: COLLSCAN keysExamined:0 docsExamined:47750 cursorExhausted:1 numYields:375 nreturned:5 reslen:1485 locks:{ Global: { acquireCount: { r: 752 } }, Database: { acquireCount: { r: 376 } }, Collection: { acquireCount: { r: 376 } } } protocol:op_msg 272ms
mongodb_1 | 2020-02-26T22:11:06.092+0000 I COMMAND [conn37] command graylog.streams command: find { find: “streams”, filter: { disabled: false }, $db: “graylog”, lsid: { id: UUID(“d420baf6-2890-4b88-b6b7-e666449ebb25”) }, $readPreference: { mode: “primaryPreferred” } } planSummary: COLLSCAN keysExamined:0 docsExamined:47750 cursorExhausted:1 numYields:374 nreturned:5 reslen:1485 locks:{ Global: { acquireCount: { r: 750 } }, Database: { acquireCount: { r: 375 } }, Collection: { acquireCount: { r: 375 } } } protocol:op_msg 278ms
And I also see this:
mongodb_1 | 2020-02-26T23:40:32.820+0000 I COMMAND [conn14] warning: log line attempted (1772kB) over max size (10kB), printing beginning and end … command graylog.streamrules command: find { find: “streamrules”, filter: { stream_id: { $in: [ ObjectId(‘000000000000000000000001’), ObjectId(‘5e5558214756c60013b6441d’), ObjectId(‘5e56a5b0b7b6c60012457778’), ObjectId(‘5e56a5dcb7b6c600124577ad’), ObjectId(‘5e56a5dcb7b6c600124577b0’), ObjectId(‘5e56a5dcb7b6c600124577b3’), ObjectId(‘5e56a5dcb7b6c600124577b6’), ObjectId(‘5e56a5dcb7b6c600124577b9’), ObjectId(‘5e56a5dcb7b6c600124577bc’), ObjectId(‘5e56a5dcb7b6c600124577bf’), ObjectId(‘5e56a5dcb7b6c600124577c2’), ObjectId(‘5e56a630b7b6c6001245781f’), ObjectId(‘5e56a630b7b6c60012457822’), ObjectId(‘5e56a630b7b6c60012457825’), ObjectId(‘5e56a630b7b6c60012457828’), ObjectId(‘5e56a630b7b6c6001245782b’), ObjectId(‘5e56a630b7b6c6001245782e’), ObjectId(‘5e56a630b7b6c60012457831’), ObjectId(‘5e56a630b7b6c60012457834’), ObjectId(‘5e56a762b7b6c60012457983’), ObjectId(‘5e56a762b7b6c60012457986’), ObjectId(‘5e56a762b7b6c60012457989’), ObjectId(‘5e56a762b7b6c6001245798c’), ObjectId(‘5e56a762b7b6c6001245798f’), ObjectId(‘5e56a762b7b6c60012457993’), ObjectId(‘5e56a762b7b6c60012457996’), ObjectId(‘5e56a762b7b6c60012457999’), ObjectId(‘5e56a7b6b7b6c600124579f6’), ObjectId(‘5e56a7b6b7b6c600124579f9’), ObjectId(‘5e56a7b6b7b6c600124579fc’), ObjectId(‘5e56a7b6b7b6c600124579ff’), ObjectId(‘5e56a7b6b7b6c60012457a02’), ObjectId(‘5e56a7b6b7b6c60012457a05’), ObjectId(‘5e56a7b6b7b6c60012457a08’), ObjectId(‘5e56a7b6b7b6c60012457a0b’), ObjectId(‘5e56a817b7b6c60012457a77’), ObjectId(‘5e56a817b7b6c60012457a7a’), ObjectId(‘5e56a817b7b6c60012457a7d’), ObjectId(‘5e56a817b7b6c60012457a80’), ObjectId(‘5e56a817b7b6c60012457a83’), ObjectId(‘5e56a817b7b6c60012457a86’), ObjectId(‘5e56a817b7b6c60012457a89’), ObjectId(‘5e56a817b7b6c60012457a8c’), ObjectId(‘5e56a82eb7b6c60012457aa7’), ObjectId(‘5e56a82eb7b6c60012457aaa’), ObjectId(‘5e56a82eb7b6c60012457aad’), ObjectId(‘5e56a82eb7b6c60012457ab0’), ObjectId(‘5e56a82eb7b6c60012457ab3’), ObjectId(‘5e56a82eb7b6c60012457ab6’), ObjectId(‘5e56a82eb7b6c60012457ab9’), ObjectId(‘5e56a82eb7b6c60012457abc’), ObjectId(‘5e56a82eb7b6c60012457abf’), ObjectId(‘5e56a82eb7b6c60012457ac2’), ObjectId(‘5e56a85cb7b6c60012457af7’), ObjectId(‘5e56a85cb7b6c60012457afa’), ObjectId(‘5e56a85cb7b6c60012457afd’), ObjectId(‘5e56a85cb7b6c60012457b00’), ObjectId(‘5e56a85cb7b6c60012457b03’), ObjectId(‘5e56a85cb7b6c60012457b06’), ObjectId(‘5e56a85cb7b6c60012457b09’), ObjectId(‘5e56a85cb7b6c60012457b0c’), ObjectId(‘5e56a85cb7b6c60012457b10’), ObjectId(‘5e56a85cb7b6c60012457b13’), ObjectId(‘5e56a85cb7b6c60012457b16’), ObjectId(‘5e56a85cb7b6c60012457b19’), ObjectId(‘5e56a85cb7b6c60012457b1c’), ObjectId(‘5e56a85cb7b6c60012457b1f’), ObjectId(‘5e56a85cb7b6c60012457b22’), ObjectId(‘5e56a85cb7b6c60012457b25’), ObjectId(‘5e56a85cb7b6c60012457b28’), ObjectId(‘5e56a85cb7b6c60012457b2b’), ObjectId(‘5e56a85cb7b6c60012457b2e’), ObjectId(‘5e56a85cb7b6c60012457b31’), ObjectId(‘5e56a85cb7b6c60012457b34’), ObjectId(‘5e56a85cb7b6c60012457b37’), ObjectId(‘5e56a85cb7b6c60012457b3a’), ObjectId(‘5e56a85cb7b6c60012457b3d’), ObjectId(‘5e56a85cb7b6c60012457b40’), ObjectId(‘5e56a85cb7b6c60012457b43’), ObjectId(‘5e56a85cb7b6c60012457b46’), ObjectId(‘5e56a85cb7b6c60012457b49’), ObjectId(‘5e56a85cb7b6c60012457b4c’), ObjectId(‘5e56a85cb7b6c60012457b4f’), ObjectId(‘5e56a85cb7b6c60012457b52’), ObjectId(‘5e56a85cb7b6c60012457b55’), ObjectId(‘5e56a85cb7b6c60012457b58’), ObjectId(‘5e56a85cb7b6c60012457b5b’), ObjectId(‘5e … 6c600124df14c’), ObjectId(‘5e56d6d8efe6c600124df14f’), ObjectId(‘5e56d6d9efe6c600124df152’), ObjectId(‘5e56d6daefe6c600124df155’), ObjectId(‘5e56d6daefe6c600124df158’), ObjectId(‘5e56d6dbefe6c600124df15b’), ObjectId(‘5e56d6dbefe6c600124df15e’), ObjectId(‘5e56d6dcefe6c600124df161’), ObjectId(‘5e56d6ddefe6c600124df164’), ObjectId(‘5e56d6ddefe6c600124df167’), ObjectId(‘5e56d6deefe6c600124df16a’), ObjectId(‘5e56d6dfefe6c600124df16d’), ObjectId(‘5e56d6dfefe6c600124df170’), ObjectId(‘5e56d6e0efe6c600124df173’), ObjectId(‘5e56d6e0efe6c600124df176’), ObjectId(‘5e56d6e1efe6c600124df179’), ObjectId(‘5e56d6e2efe6c600124df17c’), ObjectId(‘5e56d6e2efe6c600124df17f’), ObjectId(‘5e56d6e3efe6c600124df182’), ObjectId(‘5e56d6e4efe6c600124df187’), ObjectId(‘5e56d6e4efe6c600124df18a’), ObjectId(‘5e56d6e5efe6c600124df18d’), ObjectId(‘5e56d6e6efe6c600124df190’), ObjectId(‘5e56d6e6efe6c600124df193’), ObjectId(‘5e56d6e7efe6c600124df196’), ObjectId(‘5e56d6e7efe6c600124df199’), ObjectId(‘5e56d6e8efe6c600124df19c’), ObjectId(‘5e56d6e9efe6c600124df19f’), ObjectId(‘5e56d6e9efe6c600124df1a2’), ObjectId(‘5e56d6eaefe6c600124df1a5’), ObjectId(‘5e56d6ebefe6c600124df1a8’), ObjectId(‘5e56d6ebefe6c600124df1ab’), ObjectId(‘5e56d6ecefe6c600124df1ae’), ObjectId(‘5e56d6ecefe6c600124df1b1’), ObjectId(‘5e56d6edefe6c600124df1b4’), ObjectId(‘5e56d6eeefe6c600124df1b7’), ObjectId(‘5e56d6eeefe6c600124df1ba’), ObjectId(‘5e56d6efefe6c600124df1bd’), ObjectId(‘5e56d6f0efe6c600124df1c0’), ObjectId(‘5e56d6f0efe6c600124df1c3’), ObjectId(‘5e56d6f1efe6c600124df1c6’), ObjectId(‘5e56d6f1efe6c600124df1c9’), ObjectId(‘5e56d6f2efe6c600124df1cc’), ObjectId(‘5e56d6f3efe6c600124df1cf’), ObjectId(‘5e56d6f3efe6c600124df1d3’), ObjectId(‘5e56d6f4efe6c600124df1d6’), ObjectId(‘5e56d6f5efe6c600124df1d9’), ObjectId(‘5e56d6f5efe6c600124df1dc’), ObjectId(‘5e56d6f6efe6c600124df1df’), ObjectId(‘5e56d6f6efe6c600124df1e2’), ObjectId(‘5e56d6f7efe6c600124df1e5’), ObjectId(‘5e56d6f8efe6c600124df1e8’), ObjectId(‘5e56d6f8efe6c600124df1eb’), ObjectId(‘5e56d6f9efe6c600124df1ee’), ObjectId(‘5e56d6faefe6c600124df1f1’), ObjectId(‘5e56d6faefe6c600124df1f4’), ObjectId(‘5e56d6fbefe6c600124df1f7’), ObjectId(‘5e56d6fbefe6c600124df1fa’), ObjectId(‘5e56d6fcefe6c600124df1fd’), ObjectId(‘5e56d6fdefe6c600124df200’), ObjectId(‘5e56d6fdefe6c600124df203’), ObjectId(‘5e56d6feefe6c600124df206’), ObjectId(‘5e56d6ffefe6c600124df209’), ObjectId(‘5e56d6ffefe6c600124df20c’), ObjectId(‘5e56d700efe6c600124df20f’), ObjectId(‘5e56d701efe6c600124df212’), ObjectId(‘5e56d701efe6c600124df216’), ObjectId(‘5e56d702efe6c600124df219’), ObjectId(‘5e56d702efe6c600124df21c’), ObjectId(‘5e56d703efe6c600124df21f’), ObjectId(‘5e56d704efe6c600124df222’), ObjectId(‘5e56d704efe6c600124df225’), ObjectId(‘5e56d705efe6c600124df228’), ObjectId(‘5e56d706efe6c600124df22b’), ObjectId(‘5e56d706efe6c600124df22e’), ObjectId(‘5e56d707efe6c600124df231’), ObjectId(‘5e56d707efe6c600124df234’), ObjectId(‘5e56d708efe6c600124df237’), ObjectId(‘5e56d709efe6c600124df23a’), ObjectId(‘5e56d709efe6c600124df23d’) ] } }, $db: “graylog”, lsid: { id: UUID(“f0770e9e-9cc1-449a-8b11-5c9bd2522e9d”) }, $readPreference: { mode: “primaryPreferred” } } planSummary: IXSCAN { stream_id: 1 } keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:0 nreturned:1 reslen:215 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_msg 174ms
We are using this:
https://docs.graylog.org/en/3.2/pages/installation/docker.html#persisting-data
, and the script that I am running to create the streams is basically a while loop with the curl command to create the stream and a sleep 1 (sec) inside. The payload for the curl is: {"title":"test-stream-${num_of_stream}", "index_set_id": "5e56a595b7b6c60012457756"}
Have we missed something? Please advise.