My Environment
VM
16 Cors
12 GB RAM
Graylog Version: Graylog 3.3.8+e223f85
Elasticsearch Version: 6.8.12
MongoDB Version: 4.0.20
Operating System: Ubuntu 18.04
Browser version: 81.0.2 (64-Bit) under Windows 10
Graylog Start Options
/etc/default/graylog-server
GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true"
Elasticsearch Start Options
/etc/elasticsearch/jvm.options
-Xms3g
-Xmx3g
Changes to Orginal Server.conf
allow_leading_wildcard_searches = true
output_batch_size = 4000
processbuffer_processors = 7
outputbuffer_processors = 7
ring_size = 262144
inputbuffer_ring_size = 262144
Missing in my Server.conf
Set the default scheme when connecting to Elasticsearch discovered nodes
Default: http (available options: http, https)
#elasticsearch_discovery_default_scheme = http
Configures the prefix used for graylog event indices
Default: gl-events
#default_events_index_prefix = gl-events
Configures the prefix used for graylog system event indices
Default: gl-system-events
#default_system_events_index_prefix = gl-system-events
Automatically load content packs in “content_packs_dir” on the first start of Graylog.
#content_packs_loader_enabled = false
The directory which contains content packs which should be loaded on the first start of Graylog.
#content_packs_dir = data/contentpacks
A comma-separated list of content packs (files in “content_packs_dir”) which should be applied on
the first start of Graylog.
Default: empty
#content_packs_auto_install = grok-patterns.json
The allowed TLS protocols for system wide TLS enabled servers. (e.g. message inputs, http interface)
Setting this to an empty value, leaves it up to system libraries and the used JDK to chose a default.
Default: TLSv1.2,TLSv1.3 (might be automatically adjusted to protocols supported by the JDK)
#enabled_tls_protocols= TLSv1.2,TLSv1.3
===================================================================================================
Now to my Question,
like I wrote in CSV Export Progress bar or speed , the
export of the CSV file takes some time 10 minutes and other times over an hour.
Yesterday it took me two hours.
In the manuel for 3.3.8 there is written:
9.7.4 TroubleshootingDepending on the number of messages the export may take a while.
If the download never starts or the document doesnot contain the expected result, have a look at the server.log.
Warning:Exporting results to a CSV willnotpreserve sorting because Graylog is using the virtual_docfieldto “sort”
documents for performance reasons. If you need to have the exported data ordered you will need to eithermake a
scroll query to ElasticSearch and process it after, or to download the file and post process it via other means.
I checked the server.log /var/log/graylog-server/server.log,
/var/log/elasticsearch/gc.log,
/var/log/elasticsearch/graylog.log and
/var/log/elasticsearch/graylog_deprecation.log
and I can not find any entries for the export.
I’m interested to figure out why it some times takes 10 minutes and some times over an hour.
What can i do to figure that out and for what do i have to look.
Thanks for help SR.