Over the past week, I have been chasing an issue where I am unable to get an ‘Export to CSV’ to complete successfully on searches which contain 1+ million records in them. When the process completes for these large searches, an empty CSV file is downloaded. I was eventually able to track down the log message for this error:
2020-08-18T21:34:37.215Z ERROR [MessageBodyWriter] Encountered unspecified error when writing message result as CSV, result is likely malformed.
2020-08-18T21:34:37.230Z ERROR [ChunkedRunner] Error executing runnable
org.graylog2.indexer.ElasticsearchException: Failed to execute Scroll request with scroll id DnF1ZXJ5VGhlbkZldGNoWAAAAAAAAANzFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADchZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA3EWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAN1FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADdBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA3YWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAN3FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADeBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA3kWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAN6FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADexZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA3wWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAN9FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADfhZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA38WUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOAFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADgRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA4IWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAODFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADhBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA4UWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOGFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADhxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA4gWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOJFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADihZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA4sWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOMFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADjRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA44WUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOPFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADkBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA5EWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOSFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADkxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA5QWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOVFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADlhZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA5cWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOYFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADmRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA5oWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAObFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADnBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA50WUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOeFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADnxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA6AWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOhFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADohZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA6MWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOkFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADpRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA6YWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOnFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADqBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA6kWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOqFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADqxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA6wWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOtFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADrhZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA68WUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOwFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADsRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA7IWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAOzFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADtBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA7UWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAO2FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADtxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA7gWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAO5FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADuhZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA7sWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAO8FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADvRZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA74WUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAO_FlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADwBZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA8EWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAPCFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADwxZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA8QWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAPFFlBGQXJqNWZiVEJHZmhPcTE4VjdadWcAAAAAAAADxhZQRkFyajVmYlRCR2ZoT3ExOFY3WnVnAAAAAAAAA8cWUEZBcmo1ZmJUQkdmaE9xMThWN1p1ZwAAAAAAAAPIFlBGQXJqNWZiVEJHZmhPcTE4VjdadWc=
No search context found for id [883]No search context found for id [882]No search context found for id [881]No search context found for id [885]No search context found for id [884]No search context found for id [886]No search context found for id [887]No search context found for id [888]No search context found for id [889]No search context found for id [890]No search context found for id [891]No search context found for id [892]No search context found for id [893]No search context found for id [894]No search context found for id [895]No search context found for id [896]No search context found for id [897]No search context found for id [898]No search context found for id [899]No search context found for id [900]No search context found for id [901]No search context found for id [902]No search context found for id [903]No search context found for id [904]No search context found for id [905]No search context found for id [906]No search context found for id [907]No search context found for id [908]No search context found for id [909]No search context found for id [910]No search context found for id [911]No search context found for id [912]No search context found for id [913]No search context found for id [914]No search context found for id [915]No search context found for id [916]No search context found for id [917]No search context found for id [918]No search context found for id [919]No search context found for id [920]No search context found for id [921]No search context found for id [922]No search context found for id [923]No search context found for id [924]No search context found for id [925]No search context found for id [926]No search context found for id [927]No search context found for id [928]No search context found for id [929]No search context found for id [930]No search context found for id [931]No search context found for id [932]No search context found for id [933]No search context found for id [934]No search context found for id [935]No search context found for id [936]No search context found for id [937]No search context found for id [938]No search context found for id [939]No search context found for id [940]No search context found for id [941]No search context found for id [942]No search context found for id [943]No search context found for id [944]No search context found for id [945]No search context found for id [946]No search context found for id [947]No search context found for id [948]No search context found for id [949]No search context found for id [950]No search context found for id [951]No search context found for id [952]No search context found for id [953]No search context found for id [954]No search context found for id [955]No search context found for id [956]No search context found for id [957]No search context found for id [958]No search context found for id [959]No search context found for id [960]No search context found for id [961]No search context found for id [962]No search context found for id [963]No search context found for id [964]No search context found for id [965]No search context found for id [966]No search context found for id [967]No search context found for id [968]
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:110) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:60) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.JestWrapper.execute(JestWrapper.java:41) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.Scroll.continueScroll(Scroll.java:116) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.Scroll.retrieveHits(Scroll.java:76) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.Scroll.nextChunk(Scroll.java:60) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.ElasticsearchExportBackend.search(ElasticsearchExportBackend.java:99) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.es.ElasticsearchExportBackend.run(ElasticsearchExportBackend.java:75) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.DecoratingMessagesExporter.export(DecoratingMessagesExporter.java:37) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.AuditingMessagesExporter.export(AuditingMessagesExporter.java:48) ~[graylog.jar:?]
at org.graylog.plugins.views.search.rest.MessagesResource.lambda$retrieveForSearchType$3(MessagesResource.java:148) ~[graylog.jar:?]
at org.graylog.plugins.views.search.export.ChunkedRunner.lambda$run$0(ChunkedRunner.java:48) ~[graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
The server resources I need in order to make these multi-million line exports is not an issue. It seems that there’s a limit that I am hitting on something when running these large CSV export jobs on various different sizes of EC2 instances. The log message I copied occurred on a c5a.4xlarge instance which seems to be more than enough compute I need in order to do this. I have only tried this export from the GUI.
The Graylog version I am working with is 3.3.5+6436f1b installed via the Graylog repository. The server host is running Ubuntu 18.04.5 LTS with all packages up-to-date as of writing.
Any assistance on this issue would be very much appreciated.
Thank You.