Got "Command failed with error 85: 'Index with name: compound_0 already exists with a different name' on server" and failed to start

#1

Graylog 3.1 docker container
Mongodb 4.2

It worked before I restarted the container but now it fails to start with the following messages.
I was running two mongodb instances with clustering and two graylog instances.

2019-09-04 02:35:21,620 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 85: ‘Index with name: compound_0 already exists with a different name’ on server mongodb1:27017. The full response is { “operationTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “ok” : 0.0, “errmsg” : “Index with name: compound_0 already exists with a different name”, “code” : 85, “codeName” : “IndexOptionsConflict”, “$clusterTime” : { “clusterTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “signature” : { “hash” : { “$binary” : “cYudYhmDdPbXU1WHfTCcoWaXL7s=”, “$type” : “00” }, “keyId” : { “$numberLong” : “6721582427640168450” } } } }

2019-09-04 02:35:21,621 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 85: ‘Index with name: compound_0 already exists with a different name’ on server mongodb1:27017. The full response is { “operationTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “ok” : 0.0, “errmsg” : “Index with name: compound_0 already exists with a different name”, “code” : 85, “codeName” : “IndexOptionsConflict”, “$clusterTime” : { “clusterTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “signature” : { “hash” : { “$binary” : “cYudYhmDdPbXU1WHfTCcoWaXL7s=”, “$type” : “00” }, “keyId” : { “$numberLong” : “6721582427640168450” } } } }

2019-09-04 02:35:21,622 ERROR: org.graylog2.commands.Server -

################################################################################

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Command failed with error 85: ‘Index with name: compound_0 already exists with a different name’ on server mongodb1:27017. The full response is { “operationTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “ok” : 0.0, “errmsg” : “Index with name: compound_0 already exists with a different name”, “code” : 85, “codeName” : “IndexOptionsConflict”, “$clusterTime” : { “clusterTime” : { “$timestamp” : { “t” : 1567564515, “i” : 1 } }, “signature” : { “hash” : { “$binary” : “cYudYhmDdPbXU1WHfTCcoWaXL7s=”, “$type” : “00” }, “keyId” : { “$numberLong” : “6721582427640168450” } } } }

################################################################################

#2

did you persist your mongoDB data store?

it looks like you have restarted your MongoDB and Graylog now think it is starting from the scratch but elasticsearch is already havin indices it tries to create.

(just a guess without knowing your setup)

#3

MongoDB store is persistent and I didn’t restart it.
BTW, I found processing_status has index with different name not ‘compound_0’.
Here’s my fix. Not sure this is the correct fix, though…

rs0:PRIMARY> db.processing_status.getIndexes()
[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "unique" : true,
        "key" : {
            "node_id" : 1
        },
        "name" : "node_id_1",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "key" : {
            "updated_at" : 1,
            "input_journal.uncommitted_entries" : 1,
            "input_journal.written_messages_1m_rate" : 1
        },
        "name" : "updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1",
        "ns" : "graylog.processing_status"
    }
]
rs0:PRIMARY> db.processing_status.createIndex(
...   { "updated_at": 1, "input_journal.uncommitted_entries": 1, "input_journal.written_messages_1m_rate": 1 } ,
...   { name: "compound_0" }
... )
{
    "operationTime" : Timestamp(1567576545, 1),
    "ok" : 0,
    "errmsg" : "Index with name: compound_0 already exists with a different name",
    "code" : 85,
    "codeName" : "IndexOptionsConflict",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1567576545, 1),
        "signature" : {
            "hash" : BinData(0,"qALPe89RllLmqeQ0lo75TMOWTGI="),
            "keyId" : NumberLong("6721582427640168450")
        }
    }
}
rs0:PRIMARY> db.processing_status.getIndexes()
[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "unique" : true,
        "key" : {
            "node_id" : 1
        },
        "name" : "node_id_1",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "key" : {
            "updated_at" : 1,
            "input_journal.uncommitted_entries" : 1,
            "input_journal.written_messages_1m_rate" : 1
        },
        "name" : "updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1",
        "ns" : "graylog.processing_status"
    }
]
rs0:PRIMARY>
rs0:PRIMARY> db.processing_status.dropIndex("updated_at_1_input_journal.uncommitted_entries_1_input_journal.written_messages_1m_rate_1")
{
    "nIndexesWas" : 3,
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1567576649, 1),
        "signature" : {
            "hash" : BinData(0,"Hwh9hiJOLtBXT6HBp2TMCUzXcnY="),
            "keyId" : NumberLong("6721582427640168450")
        }
    },
    "operationTime" : Timestamp(1567576649, 1)
}
rs0:PRIMARY> db.processing_status.getIndexes()
[
    {
        "v" : 2,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "unique" : true,
        "key" : {
            "node_id" : 1
        },
        "name" : "node_id_1",
        "ns" : "graylog.processing_status"
    },
    {
        "v" : 2,
        "key" : {
            "updated_at" : 1,
            "input_journal.uncommitted_entries" : 1,
            "input_journal.written_messages_1m_rate" : 1
        },
        "name" : "compound_0",
        "ns" : "graylog.processing_status"
    }
]
#4

I had the same situation waiting for me this morning, freaking me out a bit because I only upgraded to 3.1 yesterday…

Dropping the weirdly named index seems to have fixed it for me as well. :+1:

I’m running graylog+mongo+elastic under docker-compose, and have a nightly timer that pulls all images, and restarts any changed containers. This morning the graylog:3.1 image had been updated and restarted the graylog container, going into a crash loop.

It turns out that 3.1.1 was released 13 hours ago, so my best guess is that the naming of that index changed between 3.1.0 and 3.1.1, without proper migration logic.

#5

I created an issue for this: https://github.com/Graylog2/graylog2-server/issues/6383

#6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.