Importing index sets from a previous instance

Graylog Central (peer support)
1

Hello everyone !

I am trying to import some Elasticsearch indices from a previous instance of Graylog.

The previous instance was a production cluster architecture

  • 3 Graylog 4.2.11 & MongoDB 5.0.3 on Debian 10
  • 3 Elasticsearch 7.10.2 on Debian 10
  • 1 Apache2 loadbalancer on top of that

And the new one is a standalone instance

  • 1 node with Elasticsearch 7.10.2, Graylog 5.0.1-1, MongoDB 6.0.3.

The “old one” was left behind since September and seems to be kind of broken (I do not really know what happened but it looks like a MongoDB problem) and I can’t connect to the web interface anymore.

I tried importing the index from the previous instance to the new one by importing an Elastic snapshot, following these two threads :

And it looks like it’s working : I can clearly see them in the indice set from System > Indices : (index prefix has a double “__”)

image
image1454×417 31.7 KB

But sadly I can’t query the logs contained in these indices from the Search menu, no matter what timestamp I use. I recalculated the index range but still nothing.

I tried to edit the new instance configuration file server.conf to use the previous Elasticsearch cluster : result is the same for those indices but I do see logs from the default indices aka. graylog_**.

So my question is : is there anyway to query those indices ? Is it doable ? I feel this is kinda “hacky” but getting those logs back would be great. From what I understand it could be a matter of metadata ?

If you have any idea, feel free to answer ! :slight_smile:

Thanks everyone !

2

Try going into system/index then click on your Index, then Maintenance, then Recalculate Index Ranges…

image

1 Like
3

Yes, using cURL.

First I would do what @tmacgbay suggested.

Examples:

Check for issues:
curl -XGET http://ip_address:9200/_cluster/allocation/explain?pretty
Check for dangling indices:
curl -X GET http://ip_address:9200/_dangling?pretty
Check Shards:
curl -XGET http://ip_address:9200/_cat/shards
Check Indices
curl -XGET http://ip_address:9200/_cat/indices
4

Hi there and thanks for those answers !

I did what @tmacgbay suggested before trying anything else ago, and nothing happened. I just tryo ne more time, just in case…

  • curl -XGET http://ip_address:9200/_cluster/allocation/explain?pretty
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "graylog",
  "dangling_indices" : [ ]
}
  • curl -X GET http://ip_address:9200/_dangling?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}
  • The other commands gave me back what seems to be healthy indices and shards.

Thanks a lot !

5

Do the indexes you imported have the same naming scheme as existing indexes?, I had assumed yes, but after re-reading, I am not sure. Generally Graylog only recognizes Elastic indexes that it initiated and considers to be under it’s management… which you are seeing the new ones in the Graylog Index interface so… hmmmm. You have a Stream associated to the index, right? Can you search that Stream for all time for results?

6

Generally Graylog only recognizes Elastic indexes that it initiated and considers to be under it’s management

Yes, that’s what I thought too: I can browse default indexes, because they have the same naming pattern in all instances of Graylog.

I think that the indexes are named the same way… But I can’t confirm it, not having access to the web interface of the previous instance.

You have a Stream associated to the index, right?

Yes I do !

Can you search that Stream for all time for results?

No I can’t ! :frowning:

Thanks all !

7

It’s not clear why it stopped working from what I can see… not sure what else to ask for - It seems like this would be a major bug for 5.x! Perhaps it’s time to submit as a bug?

8

@Ginkgo_Balboa

I have a question for you. Stating that you had a Cluster of Three ES nodes and you reduced it to a standalone instance. I noticed in your screenshot above the shard routing.

image

Is this correct?

9

I tested two scenarii

  1. Standalone Graylog using 127.0.0.1 as Elasticsearch server.
    Elasticsearch snapshots were move to the new standalone Graylog.
  2. Standalone Graylog using the previous Elasticsearch cluster.
    In server.conf, elasticsearch_host is pointing towards the previous Elasticsearch nodes IP addresses.

I’m currently testing the second scenario here.

Hope I answered your question !

1 Like
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.