Trying to get my head around re-index

[Archie]

Hi Everyone,

First post and complete newbie Graylog having inherited the system from a previous admin, Looking for help…

I have a new cluster and an old cluster and I’m using the reindex API to bring old documents into a new Index on our new server. This works and I can see 20M+ documents under management in the Index once the re-index command finishes.

{"took":6156976,"timed_out":false,"total":20000085,"updated":0,"created":20000085,"deleted":0,"batches":20001,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}

However, I can only see those messages if I find out the gl2_source_input and search for that:

gl2_source_input : 5a58389f21394d0e92c8f4fd

The messages also report that they have been “Received by deleted input on stopped node” which I understand, because that is exactly what is happening, however, I thought that a re-index updated this data as it was imported.

How do I get this data to be searchable via the usual front-end search and not resorting to a search on gl2_source_input?

Is the re-index API the best way to migrate this old data to a new cluster?

Running 2 graylog nodes, Clustered MongoDB (3 hosts) and Clustered Elasticsearch (3 hosts)

Thank you.

Archie.

[jan (Jan Doberstein)]

Did you run the “recalculate index range” in the new Graylog after you have re-indexed the data?

[Archie]

Hi Jan, and thanks for getting back to me.

Yes - I ran the reindex and the dates came back as OK and said that time range was in 2 months and 1 week, which the “old” index was.

However when clicking on the stream that uses that index/indices to view data it does not appear in that stream.

Maybe my understanding is wrong, in that clicking on a stream to view the data is not the way and I must always search for the data from the main search console??

For example in the main search I can find IP source ranges that are in the migrated index and the data is present, but the deleted input/missing node text persists…

Hope that helps diagnose / explain.