No Messages In or Out after rebooting Graylog Server

rrmike · November 19, 2021, 3:23pm

For what it’s worth. Sometimes you need to look at the basics. These screen shots are taken after the reboot, when it’s broken.

Thanks @gsmith @tmacgbay for your continual help!

tmacgbay · November 19, 2021, 4:47pm

So many things to look back at… please post current graylog server.conf and elasticsearch.yml Check logs for both and post any warnings. Stumped.

rrmike · November 19, 2021, 7:18pm

The Problem

Graylog is working and collecting messages. However, after rebooting the server no message come in or out.

Operating system

Hyper-V Server 2019
Ubuntu 18.04

Package versions

Graylog 4.2.0+5adccc3 on graylog (Private Build 1.8.0_292 on Linux 4.15.0-159-generic)
MongoDB v4.0.27
Elasticsearch 7.10.2

What we’ve done

Ubuntu needs a restart. Linux-base will be updated.
We discovered that the data files are in a non-standard location and may be causing some confusion.

path.data: /mnt/sdb/data
path.logs: /mnt/sdb/logs

I updated some configuration settings.
/etc/graylog/server/server.conf

elasticsearch_hosts = https://127.0.0.1:9200

I’ve made some changes to Elasticsearch config.
/etc/elasticsearch/elasticsearch.yml

Config Files

Graylog

ldog@graylog:~$ cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.1:9000
elasticsearch_hosts = http://127.0.0.1:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Elasticsearch

ldog@graylog:~$ sudo cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /mnt/sdb/data
path.logs: /mnt/sdb/logs
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false
discovery.type: single-node

tmacgbay · November 19, 2021, 8:31pm

One more what are the results of:

curl "localhost:9200/_nodes/settings?pretty=true"

This will tell us what Elasticsearch has picked up from configurations (or show defaults)

I don’t think we have asked - in the Graylog UI, when you go to system/indices, and iterate through them, are they all happy… meaning Graylog thinks they are all happy?

rrmike · November 19, 2021, 9:50pm

Graylog seems happy to me.

ldog@graylog:~$ curl "localhost:9200/_nodes/settings?pretty=true"
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "graylog",
  "nodes" : {
    "Lfg5ABAgRtKaa-BepiwdMw" : {
      "name" : "graylog",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1",
      "version" : "7.10.2",
      "build_flavor" : "oss",
      "build_type" : "deb",
      "build_hash" : "xxx",
      "roles" : [
        "data",
        "ingest",
        "master",
        "remote_cluster_client"
      ],
      "settings" : {
        "cluster" : {
          "name" : "graylog"
        },
        "node" : {
          "name" : "graylog",
          "pidfile" : "/var/run/elasticsearch/elasticsearch.pid"
        },
        "path" : {
          "data" : [
            "/mnt/sdb/data"
          ],
          "logs" : "/mnt/sdb/logs",
          "home" : "/usr/share/elasticsearch"
        },
        "discovery" : {
          "type" : "single-node"
        },
        "action" : {
          "auto_create_index" : "false"
        },
        "client" : {
          "type" : "node"
        },
        "http" : {
          "type" : {
            "default" : "netty4"
          },
          "port" : "9200"
        },
        "transport" : {
          "type" : {
            "default" : "netty4"
          }
        },
        "network" : {
          "host" : "127.0.0.1"
        }
      }
    }
  }
}

tmacgbay · November 19, 2021, 10:12pm

What are the three things Graylog is whining about here:

and also, interestingly - graylog_13 says it has received messages only a few seconds ago

If you do a general search, what would those messages be?

I will try to keep an eye out and be available over the weekend - get out there and enjoy something outside!

gsmith · November 19, 2021, 11:10pm

Nice, seams like were getting closer

I have a couple of question to add to @tmacgbay suggestions.
This picture below (which I marked with a red box) shows my concerns that your plugin is not the right version.

You can either perform an upgrade to the plugin or navigate to the plugin directory and remove it.
Directory location.

/usr/share/graylog-server/plugin

Command for install plugin.

sudo apt-get install graylog-integrations-plugins

https://docs.graylog.org/docs/setup-intergrations

Have you checked permissions on Elasticsearch data directory?

ls -al /var/lib/elasticsearch

I believe in your case you have moved the data directory.

ls -al /mnt/sdb/data

Are you actually rebooting the server or restarting GL service?
If you rebooted the servers I’m concerned about the mount point /mnt in your fstab file.

EDIT: I re-read this post again and remembered a incident similar to this one. This is referring to your mount points. As you stated, you reconfigured your data/log directory when you had your server running. I have done the same thing before maybe I can shed some light on your fstab file configuration. Here is an example of what I would have done in your situation. Maybe it can help.

Stop graylog service using command: sudo systemctl stop graylog.service
Stop elasticsearch.service using command: sudo systemctl stop elasticsearch.service
Make a backup of your data !!! For example, simple copy to another destination with enough space using command: cp -av /var/lib/elasticsearch /media/backupdisk.
Check name for mounted volume.
- sudo fdisk -l

Create elasticsearch directory in /mnt
- sudo mkdir /mnt/elasticsearch
Mount /dev/sdb1 to /mnt/elasticsearch
- mount /dev/sdb1 /mnt/elasticsearch
Create new sub-directories for elastic data/logs using these commands:
- sudo mkdir -p /mnt/elasticsearch/es_data
- sudo mkdir -p /mnt/elasticsearch/es_log
Now make sure the mounts are good after reboot by adding it to fstab file
- /dev/sdb1 /mnt/elasticsearch ext4 defaults 0 0

Setup permissions for these directories using commands:
- sudo chown -R elasticsearch:elasticsearch /mnt/elasticsearch/es_data
- sudo chown -R elasticsearch:elasticsearch /mnt/elasticsearch/es_log
Move elasticsaerch db and logs to new directory.
- sudo mv -v /var/lib/elasticsearch/ /mnt/elasticsearch/es_data*
- sudo mv -v /var/log/elasticsearch/ /mnt/elasticsearch/es_log*
Start elasticsearch. service using command:
- sudo systemctl start elasticsearch.service
Wait few moments to elasticsearch and then start graylog using:
- sudo systemctl start graylog.service

If you noticed my second drive partition is named /dev/sdb1 my drive is named /dev/sdb.
using this command may help.

root # lsblk
The lsblk command lists all the block devices of your system along with their logical partitions.

I’m not 100% sure what you did before but maybe this would could give you some insight of what I did.

@tmacgbay

rrmike · November 22, 2021, 2:44pm

Thanks @tmacgbay, @gsmith. It was nice to spend a few days not thinking about Graylog. I hope you had a nice weekend.

An input has failed to start. I recently deleted the input. Not sure why it’s still complaining about it. I may not cleared the error. It was an input for the server itself, something the setup documentation had me do.
You are running an outdated Graylog version.
Email Transport Configuration is missing or invalid!
I can fix both of them.

Two things here.

After I reboot the server and I can log back on to Graylog’s UI I see a few messages being collected before it goes to 0 in a few seconds.
Graylog is not a critical application for us right now. So I’ve been trouble shooting on the production server. I make the changes you recommend, take a snapshot, reboot and see if it works. If it’s no go then I revert back (keeping the changes). At first I thought this was a good way to work on this, it gave me a chance to work snapshots (which I had not done much with before), but now almost a week into this and 26 posts on the forum maybe I should clone this VM! So prior to rebooting the server, it was collecting messages.

I initially installed the enterprise version but then realized we would never buy it, so I removed the plugins. Or at least thought I did.

I removed the files in /usr/share/graylog-server/plugin. However, Graylog did not like that at all. I restarted the Graylog services and the webpage does not come up now.

● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2021-11-22 09:26:43 EST; 7s ago
     Docs: http://docs.graylog.org/
  Process: 42782 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
 Main PID: 42782 (code=exited, status=1/FAILURE)


2021-11-22T09:21:20.750-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.indexer.fieldtypes.IndexFieldTypePollerAdapter>> was bound.
2021-11-22T09:21:20.751-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.indexer.indices.IndicesAdapter>> was bound.
2021-11-22T09:21:20.751-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.indexer.messages.MessagesAdapter>> was bound.
2021-11-22T09:21:20.752-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.indexer.searches.SearchesAdapter>> was bound.
2021-11-22T09:21:20.752-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.migrations.V20170607164210_MigrateReopenedIndicesToAliases$ClusterState>> was bound.
2021-11-22T09:21:33.391-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2021-11-22T09:21:34.926-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-11-22T09:21:35.223-05:00 INFO  [Version] HV000001: Hibernate Validator null
2021-11-22T09:21:39.413-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.events.search.MoreSearchAdapter>> was bound.
2021-11-22T09:21:39.414-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.plugins.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEvents$ElasticsearchAdapter>> was bound.
2021-11-22T09:21:39.415-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.plugins.views.search.engine.QueryBackend<? extends org.graylog.plugins.views.search.engine.GeneratedQueryContext>>> was bound.
2021-11-22T09:21:39.415-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.plugins.views.search.export.ExportBackend>> was bound.
2021-11-22T09:21:39.416-05:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog2.indexer.IndexToolsAdapter>> was bound.

I’m going to roll that one back! What did I do wrong?

My permissions look a little different than yours.

ldog@graylog:~$ ls -al /mnt/sdb/data
total 16
drwxrwxrwx 3 ldog          root          4096 Sep 21 15:33 .
drwxr-xr-x 5 ldog          root          4096 Sep 21 14:48 ..
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Sep 21 15:33 nodes
-rw-rw-r-- 1 ldog          ldog            76 Sep 21 15:30 test

Restarting Graylog and it’s related services is fine. It only breaks after rebooting the server.
This is what my fstab looks like:

ldog@graylog:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-n4UnpBz6N7UfCGajqOh2QmRanFXlt5f9j6uBLeba12wd9oXZqaF29G26T0YljUsw / ext4 defaults 0 0
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/d2c58bea-c775-443e-806b-b329116dc3f4 /boot ext4 defaults 0 0
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/889C-D099 /boot/efi vfat defaults 0 0
/swap.img       none    swap    sw      0       0
/dev/sdb    /mnt/sdb     ext4      defaults        0             0

Disk /dev/sdb: 700 GiB, 751619276800 bytes, 1468006400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

@gsmith Thanks for the rest of your instructions. You’ll have to give me some time to work through those!

gsmith · November 22, 2021, 11:09pm

Hello,

Rule #1 make sure all the plugins are the same version.

I see now you have some configuration issue. As you stated about cloning. I would totally clone your production server BUT remember you cant use the same IP/MAC address. Sometime you get nowhere just rolling it back.

When you execute this command what does it look like?

root # lsblk

Example:

[root@graylogl]#  lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk
sda               8:0    0  100G  0 disk
├─sda1            8:1    0  200M  0 part /boot
└─sda2            8:2    0 97.9G  0 part
  ├─centos-root 253:0    0 82.2G  0 lvm  /
  └─centos-swap 253:1    0 15.6G  0 lvm  [SWAP]
sdb               8:16   0  300G  0 disk
└─sdb1            8:17   0  300G  0 part /mnt/elasticsearch
[root@nextcloud-web1 html]#

rrmike · November 23, 2021, 2:41pm

Here is the output of lsblk.

root@graylog:/home/ldog# lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0  68.3M  1 loop /snap/powershell/189
loop2                       7:2    0  55.5M  1 loop /snap/core18/2246
loop3                       7:3    0  42.2M  1 loop /snap/snapd/13831
loop4                       7:4    0  32.5M  1 loop /snap/snapd/13640
loop5                       7:5    0  55.5M  1 loop /snap/core18/2253
loop6                       7:6    0  66.5M  1 loop /snap/powershell/185
sda                         8:0    0   127G  0 disk
├─sda1                      8:1    0   512M  0 part /boot/efi
├─sda2                      8:2    0     1G  0 part /boot
└─sda3                      8:3    0 125.5G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0 125.5G  0 lvm  /
sdb                         8:16   0   700G  0 disk /mnt/sdb

Next things to do:

Clone VM.
Upgrade plugins.
Work through instructions that @gsmith provided.

gsmith · November 23, 2021, 10:44pm

Hey @rrmike
If you can keep us posted, I would like to know how its going.

rrmike · November 24, 2021, 4:35pm

@tmacgbay @gsmith. I will keep you posted. However, it will be next week. My boss just pulled me off this, in order to figure out how to replicate our ldap server and add SSL. I think I’d rather work on Graylog!

rrmike · December 1, 2021, 2:47pm

Plugins are updated.

@gsmith Two questions about your instructions.

Where my data is stored seems to be one of the big problems here. Do I want to backup /var/lib/elasticsearch or should I backup /mnt/sdb/data and /mnt/sdb/logs where my data is actually being saved.
In your instructions, I don’t see any commands that tell GrayLog or Elasticsearch that we moved directories. How does it know where to look? I’m guessing I need to update /etc/elasticsearch/elasticsearch.yml with the changes. Is there anyplace else that needs to be updated?

gsmith · December 1, 2021, 11:31pm

Hello,

Good question, its hard for me to tell you but from what you stated I would go with where the data is actually being saved.

[root@graylog graylog_user]#  cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"
cluster.name: graylog
path.data: /var/lib/elasticsearch  <----- **DATA**
path.logs: /var/log/elasticsearch <---- **LOGS**
network.host: 8.8.8.8
http.port: 9200
action.auto_create_index: false
discovery.type: single-node
path.repo: ["/mnt/sdb1/my_repo"]
[root@graylog graylog_user]#

Note:
You need to modify the path.data setting in the elasticsearch.yml file to the new folder you want the data to.

Here is what you need to do:

You may want to shut down your services , like graylog and elasticsearch first.

In elasticsearch.yml modify path.data to:

path.data: /foo/bar

You’ll end up with your data being stored in /foo/bar/elasticsearch instead of /var/lib/elasticsearch.
Make sure that the elasticsearch process can access your new folder.

Once you configure Elasticsearch make sure you start elasticsearch service first, wait until its completely running then start Graylog.

Hope that helps

EDIT @rrmike I have a question for ya, I was wondering why you didn’t create a portion on you drive sdb ? Or did you format the whole drive?

rrmike · December 14, 2021, 8:15pm

Keeping this ticket alive. Had to jump on some other projects!

system · December 28, 2021, 8:16pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

tmacgbay · December 29, 2021, 5:53pm

We can go back to the Graylog server log - lets make sure we are addressing and fixing errors in there. If I recall correctly you were rolling back any changes if they did not fix the problem? We may have multiple issues going. The odd thing we found was you moved elasticsearch to store data on /mnt… it’s possible it’s not fully moved and it’s confused.

To your question about rebuilding and moving data from your old server to the new - I have never tried that - it would be complicated in that Graylog manages the indexes for you and needs to know about them… I… you would have to research how to copy them and make sure they are registered in Graylog… maybe a separate post for that.

To the single/split drives… either would work since it is just as easy to grow drives in a virtual environment… but in splitting them up it returns to the question of making sure that Elastic/graylog are looking properly at their non-default drives.

Not really definitive answers - if you want to keep troubleshooting lets focus on resolving anything that comes up in Graylog server logs. You could try upgrading Graylog to see if that clears any issues. long shot though.

rrmike · December 29, 2021, 7:44pm

Yeah, let’s not do that!

Makes sense.

I was going to clone the VM so I had one that was broken and could play with. However, I don’t have the hard drive space to either clone it or move to another machine. So I’m stuck making a change and if it does not work rolling the server back. If you think that simple will not work I’ll talk to my boss about getting more storage in the computer.

I figured it might make sense to look at the logs in it’s working state.

tail -f /var/log/graylog-server/server.log

2021-12-29T14:03:33.615-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2021-12-29T14:03:36.089-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.2.1 [org.graylog.aws.AWSPlugin]
2021-12-29T14:03:36.092-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.2.1 [org.graylog.integrations.IntegrationsPlugin]
2021-12-29T14:03:36.095-05:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.2.1 [org.graylog.plugins.collector.CollectorPlugin]
2021-12-29T14:03:36.101-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.2.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-12-29T14:03:36.102-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.2.1+5442e44 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-12-29T14:03:36.102-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.2.1+5442e44 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-12-29T14:03:36.194-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-12-29T14:03:36.819-05:00 INFO  [Version] HV000001: Hibernate Validator null
2021-12-29T14:03:43.513-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2021-12-29T14:03:43.553-05:00 INFO  [NodeId] Node ID: 0646dbed-0a28-49e5-bf71-00e9e67fcfd9
2021-12-29T14:03:44.095-05:00 INFO  [LogManager] Loading logs.
2021-12-29T14:03:44.200-05:00 INFO  [LogManager] Logs loading complete.
2021-12-29T14:03:44.205-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-12-29T14:03:44.295-05:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-12-29T14:03:44.368-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-12-29T14:03:44.431-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:107}] to localhost:27017
2021-12-29T14:03:44.445-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=12418853}
2021-12-29T14:03:44.476-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:108}] to localhost:27017
2021-12-29T14:03:44.876-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-12-29T14:03:45.977-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-12-29T14:03:47.027-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:109}] to localhost:27017
2021-12-29T14:03:47.449-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-12-29T14:03:48.599-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-12-29T14:03:50.922-05:00 INFO  [ServerBootstrap] Graylog server 4.2.1+5442e44 starting up
2021-12-29T14:03:50.923-05:00 INFO  [ServerBootstrap] JRE: Private Build 1.8.0_312 on Linux 4.15.0-159-generic
2021-12-29T14:03:50.924-05:00 INFO  [ServerBootstrap] Deployment: deb
2021-12-29T14:03:50.924-05:00 INFO  [ServerBootstrap] OS: Ubuntu 18.04.6 LTS (bionic)
2021-12-29T14:03:50.925-05:00 INFO  [ServerBootstrap] Arch: amd64
2021-12-29T14:03:51.078-05:00 INFO  [PeriodicalsService] Starting 29 periodicals ...
2021-12-29T14:03:51.079-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-12-29T14:03:51.113-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:110}] to localhost:27017
2021-12-29T14:03:51.114-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-12-29T14:03:51.122-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-12-29T14:03:51.122-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-12-29T14:03:51.130-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-12-29T14:03:51.154-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-12-29T14:03:51.158-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-12-29T14:03:51.157-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-12-29T14:03:51.186-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-12-29T14:03:51.189-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-12-29T14:03:51.194-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-12-29T14:03:51.196-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-12-29T14:03:51.203-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-12-29T14:03:51.232-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-12-29T14:03:51.235-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@3c554f33] STARTING
2021-12-29T14:03:51.265-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-12-29T14:03:51.282-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-12-29T14:03:51.296-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-12-29T14:03:51.302-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@3c554f33] RUNNING
2021-12-29T14:03:51.320-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/614e28029f8bf82a3736d378/@3c554f33> [interval=60000ms]
2021-12-29T14:03:51.326-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-12-29T14:03:51.331-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-12-29T14:03:51.374-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-12-29T14:03:51.375-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-12-29T14:03:51.382-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-12-29T14:03:51.383-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-12-29T14:03:51.384-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-12-29T14:03:51.389-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-12-29T14:03:51.430-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-12-29T14:03:51.430-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-12-29T14:03:51.436-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-12-29T14:03:51.449-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-12-29T14:03:51.450-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-12-29T14:03:51.686-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@37701764] STARTING
2021-12-29T14:03:51.687-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@37701764] RUNNING
2021-12-29T14:03:51.708-05:00 INFO  [LookupTableService] Starting lookup table geoip/614e288a9f8bf82a3736d40e [@371b0d52] using cache geoip/614e28469f8bf82a3736d3c2 [@37701764], data adapter geoip/614e28029f8bf82a3736d378 [@3c554f33]
2021-12-29T14:03:52.303-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:111}] to localhost:27017
2021-12-29T14:03:58.494-05:00 INFO  [NetworkListener] Started listener bound to [xxx.xxx.x.x:9000]
2021-12-29T14:03:58.496-05:00 INFO  [HttpServer] [HttpServer] Started.
2021-12-29T14:03:58.496-05:00 INFO  [JerseyService] Started REST API at <xxx.xxx.x.x:9000>
2021-12-29T14:03:58.498-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=31, InputSetupService [RUNNING]=59, UserSessionTerminationService [RUNNING]=63, UrlWhitelistService [RUNNING]=68, GracefulShutdownService [RUNNING]=68, LocalKafkaMessageQueueReader [RUNNING]=68, LocalKafkaMessageQueueWriter [RUNNING]=68, ConfigurationEtagService [RUNNING]=87, JobSchedulerService [RUNNING]=87, PrometheusExporter [RUNNING]=87, OutputSetupService [RUNNING]=88, EtagService [RUNNING]=88, BufferSynchronizerService [RUNNING]=104, LocalKafkaJournal [RUNNING]=105, MongoDBProcessingStatusRecorderService [RUNNING]=131, StreamCacheService [RUNNING]=274, PeriodicalsService [RUNNING]=405, LookupTableService [RUNNING]=656, JerseyService [RUNNING]=7443}
2021-12-29T14:03:58.503-05:00 INFO  [ServerBootstrap] Graylog server up and running.
2021-12-29T14:03:58.504-05:00 INFO  [ServiceManagerListener] Services are healthy
2021-12-29T14:03:58.512-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-12-29T14:03:58.709-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now STARTING
2021-12-29T14:03:59.986-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now RUNNING
2021-12-29T14:04:01.138-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:112}] to localhost:27017
2021-12-29T14:04:10.403-05:00 WARN  [LookupTableService] Lookup table <geoip-lookup> does not exist
2021-12-29T14:04:10.403-05:00 WARN  [LookupTableService] Lookup table <geoip-lookup> does not exist

sudo tail -f /mnt/elasticsearch/es_log/graylog.log

[2021-12-29T14:16:54,168][INFO ][o.e.n.Node               ] [graylog] version[7.10.2], pid[18978], build[oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.15.0-159-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-12-29T14:16:54,184][INFO ][o.e.n.Node               ] [graylog] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-12-29T14:16:54,185][INFO ][o.e.n.Node               ] [graylog] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-11512786683561730124, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-12-29T14:16:55,904][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [aggs-matrix-stats]
[2021-12-29T14:16:55,904][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [analysis-common]
[2021-12-29T14:16:55,905][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [geo]
[2021-12-29T14:16:55,906][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-common]
[2021-12-29T14:16:55,908][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-geoip]
[2021-12-29T14:16:55,909][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-user-agent]
[2021-12-29T14:16:55,910][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [kibana]
[2021-12-29T14:16:55,911][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-expression]
[2021-12-29T14:16:55,911][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-mustache]
[2021-12-29T14:16:55,912][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-painless]
[2021-12-29T14:16:55,913][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [mapper-extras]
[2021-12-29T14:16:55,913][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [parent-join]
[2021-12-29T14:16:55,914][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [percolator]
[2021-12-29T14:16:55,915][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [rank-eval]
[2021-12-29T14:16:55,916][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [reindex]
[2021-12-29T14:16:55,917][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [repository-url]
[2021-12-29T14:16:55,918][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [systemd]
[2021-12-29T14:16:55,919][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [transport-netty4]
[2021-12-29T14:16:55,920][INFO ][o.e.p.PluginsService     ] [graylog] no plugins loaded
[2021-12-29T14:16:55,998][INFO ][o.e.e.NodeEnvironment    ] [graylog] using [1] data paths, mounts [[/mnt/elasticsearch (/dev/sdb)]], net usable_space [409.3gb], net total_space [688gb], types [ext4]
[2021-12-29T14:16:55,999][INFO ][o.e.e.NodeEnvironment    ] [graylog] heap size [1gb], compressed ordinary object pointers [true]
[2021-12-29T14:16:56,306][INFO ][o.e.n.Node               ] [graylog] node name [graylog], node ID [Lfg5ABAgRtKaa-BepiwdMw], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]
[2021-12-29T14:17:03,518][INFO ][o.e.t.NettyAllocator     ] [graylog] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2021-12-29T14:17:03,645][INFO ][o.e.d.DiscoveryModule    ] [graylog] using discovery type [single-node] and seed hosts providers [settings]
[2021-12-29T14:17:04,057][WARN ][o.e.g.DanglingIndicesState] [graylog] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-12-29T14:17:04,343][INFO ][o.e.n.Node               ] [graylog] initialized
[2021-12-29T14:17:04,344][INFO ][o.e.n.Node               ] [graylog] starting ...
[2021-12-29T14:17:04,527][INFO ][o.e.t.TransportService   ] [graylog] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2021-12-29T14:17:04,946][INFO ][o.e.c.c.Coordinator      ] [graylog] cluster UUID [fdHrTb1WTaaK4sJgTBXS9A]
[2021-12-29T14:17:05,090][INFO ][o.e.c.s.MasterService    ] [graylog] elected-as-master ([1] nodes joined)[{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{IgItPMikQV-kxBhuZdfQ_Q}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 10, version: 1074, delta: master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{IgItPMikQV-kxBhuZdfQ_Q}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-12-29T14:17:05,249][INFO ][o.e.c.s.ClusterApplierService] [graylog] master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{IgItPMikQV-kxBhuZdfQ_Q}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 10, version: 1074, reason: Publication{term=10, version=1074}
[2021-12-29T14:17:05,296][INFO ][o.e.h.AbstractHttpServerTransport] [graylog] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2021-12-29T14:17:05,296][INFO ][o.e.n.Node               ] [graylog] started
[2021-12-29T14:17:05,679][INFO ][o.e.g.GatewayService     ] [graylog] recovered [28] indices into cluster_state
[2021-12-29T14:17:17,620][INFO ][o.e.c.r.a.AllocationService] [graylog] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[gl-events_0][1], [gl-events_0][2], [gl-events_0][0]]]).
[2021-12-29T14:19:13,904][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [graylog] uncaught exception in thread [elasticsearch[graylog][search][T#2]]
org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
        at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction.registerChildNode(TransportAction.java:56) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:545) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction.executeSearch(TransportMultiSearchAction.java:149) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.handleResponse(TransportMultiSearchAction.java:172) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onFailure(TransportMultiSearchAction.java:157) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:98) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$5.onFailure(ActionListener.java:258) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:594) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:568) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:100) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-12-29T14:19:14,162][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [graylog] uncaught exception in thread [elasticsearch[graylog][search][T#5]]
org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
        at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction.registerChildNode(TransportAction.java:56) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:545) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction.executeSearch(TransportMultiSearchAction.java:149) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.handleResponse(TransportMultiSearchAction.java:172) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onFailure(TransportMultiSearchAction.java:157) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:98) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$5.onFailure(ActionListener.java:258) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:594) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:568) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:100) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

rrmike · December 29, 2021, 7:45pm

sudo tail -f /var/log/elasticsearch/gc.log
[2021-12-29T19:15:26.362+0000][17977][gc,heap     ] GC(56) Eden regions: 579->0(588)
[2021-12-29T19:15:26.362+0000][17977][gc,heap     ] GC(56) Survivor regions: 19->12(75)
[2021-12-29T19:15:26.362+0000][17977][gc,heap     ] GC(56) Old regions: 101->101
[2021-12-29T19:15:26.362+0000][17977][gc,heap     ] GC(56) Archive regions: 2->2
[2021-12-29T19:15:26.362+0000][17977][gc,heap     ] GC(56) Humongous regions: 28->24
[2021-12-29T19:15:26.362+0000][17977][gc,metaspace] GC(56) Metaspace: 77258K(79692K)->77258K(79692K) NonClass: 68064K(69888K)->68064K(69888K) Class: 9193K(9804K)->9193K(9804K)
[2021-12-29T19:15:26.362+0000][17977][gc          ] GC(56) Pause Young (Normal) (G1 Evacuation Pause) 726M->137M(1024M) 22.292ms
[2021-12-29T19:15:26.362+0000][17977][gc,cpu      ] GC(56) User=0.05s Sys=0.00s Real=0.02s
[2021-12-29T19:15:26.362+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 445330498 ns, Reaching safepoint: 369603 ns, At safepoint: 22449412 ns, Total: 22819015 ns
[2021-12-29T19:15:27.375+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1012688948 ns, Reaching safepoint: 233102 ns, At safepoint: 12001 ns, Total: 245103 ns
[2021-12-29T19:15:29.376+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 2000547772 ns, Reaching safepoint: 188902 ns, At safepoint: 22700 ns, Total: 211602 ns
[2021-12-29T19:15:31.377+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 2001382994 ns, Reaching safepoint: 177102 ns, At safepoint: 7300 ns, Total: 184402 ns
[2021-12-29T19:15:33.379+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 2001373507 ns, Reaching safepoint: 210502 ns, At safepoint: 7600 ns, Total: 218102 ns
[2021-12-29T19:15:33.797+0000][17977][gc,start    ] GC(57) Pause Young (Normal) (G1 Evacuation Pause)
[2021-12-29T19:15:33.797+0000][17977][gc,task     ] GC(57) Using 2 workers of 2 for evacuation
[2021-12-29T19:15:33.797+0000][17977][gc,age      ] GC(57) Desired survivor size 39321600 bytes, new threshold 15 (max threshold 15)
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) Age table with threshold 15 (max threshold 15)
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   1:   14434984 bytes,   14434984 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   2:     386448 bytes,   14821432 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   3:      61472 bytes,   14882904 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   4:     474488 bytes,   15357392 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   5:     346200 bytes,   15703592 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   6:     478896 bytes,   16182488 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   7:     195384 bytes,   16377872 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   8:     307536 bytes,   16685408 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age   9:     302680 bytes,   16988088 total
[2021-12-29T19:15:33.815+0000][17977][gc,age      ] GC(57) - age  10:       3664 bytes,   16991752 total
[2021-12-29T19:15:33.816+0000][17977][gc,age      ] GC(57) - age  11:       3080 bytes,   16994832 total
[2021-12-29T19:15:33.816+0000][17977][gc,age      ] GC(57) - age  12:       9904 bytes,   17004736 total
[2021-12-29T19:15:33.816+0000][17977][gc,age      ] GC(57) - age  13:      47496 bytes,   17052232 total
[2021-12-29T19:15:33.816+0000][17977][gc,age      ] GC(57) - age  14:     109616 bytes,   17161848 total
[2021-12-29T19:15:33.816+0000][17977][gc,age      ] GC(57) - age  15:      44776 bytes,   17206624 total
[2021-12-29T19:15:33.816+0000][17977][gc,phases   ] GC(57)   Pre Evacuate Collection Set: 0.2ms
[2021-12-29T19:15:33.816+0000][17977][gc,phases   ] GC(57)   Merge Heap Roots: 0.1ms
[2021-12-29T19:15:33.816+0000][17977][gc,phases   ] GC(57)   Evacuate Collection Set: 15.8ms
[2021-12-29T19:15:33.816+0000][17977][gc,phases   ] GC(57)   Post Evacuate Collection Set: 2.0ms
[2021-12-29T19:15:33.816+0000][17977][gc,phases   ] GC(57)   Other: 1.2ms
[2021-12-29T19:15:33.816+0000][17977][gc,heap     ] GC(57) Eden regions: 588->0(580)
[2021-12-29T19:15:33.816+0000][17977][gc,heap     ] GC(57) Survivor regions: 12->18(75)
[2021-12-29T19:15:33.816+0000][17977][gc,heap     ] GC(57) Old regions: 101->101
[2021-12-29T19:15:33.817+0000][17977][gc,heap     ] GC(57) Archive regions: 2->2
[2021-12-29T19:15:33.817+0000][17977][gc,heap     ] GC(57) Humongous regions: 33->23
[2021-12-29T19:15:33.817+0000][17977][gc,metaspace] GC(57) Metaspace: 77262K(79692K)->77262K(79692K) NonClass: 68068K(69888K)->68068K(69888K) Class: 9193K(9804K)->9193K(9804K)
[2021-12-29T19:15:33.817+0000][17977][gc          ] GC(57) Pause Young (Normal) (G1 Evacuation Pause) 734M->142M(1024M) 20.305ms
[2021-12-29T19:15:33.817+0000][17977][gc,cpu      ] GC(57) User=0.03s Sys=0.01s Real=0.02s
[2021-12-29T19:15:33.817+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 417439946 ns, Reaching safepoint: 206101 ns, At safepoint: 20496794 ns, Total: 20702895 ns
[2021-12-29T19:15:34.817+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000173956 ns, Reaching safepoint: 172301 ns, At safepoint: 8200 ns, Total: 180501 ns
[2021-12-29T19:15:35.818+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000493562 ns, Reaching safepoint: 173702 ns, At safepoint: 24300 ns, Total: 198002 ns
[2021-12-29T19:15:38.819+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 3000379995 ns, Reaching safepoint: 343804 ns, At safepoint: 11700 ns, Total: 355504 ns
[2021-12-29T19:15:39.826+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1006712234 ns, Reaching safepoint: 287903 ns, At safepoint: 11800 ns, Total: 299703 ns
[2021-12-29T19:15:40.834+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000323477 ns, Reaching safepoint: 7373970 ns, At safepoint: 57600 ns, Total: 7431570 ns
[2021-12-29T19:15:41.868+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1031019371 ns, Reaching safepoint: 2760127 ns, At safepoint: 38800 ns, Total: 2798927 ns
[2021-12-29T19:15:42.868+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000348684 ns, Reaching safepoint: 323303 ns, At safepoint: 34800 ns, Total: 358103 ns
[2021-12-29T19:15:43.873+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1004471027 ns, Reaching safepoint: 371503 ns, At safepoint: 30600 ns, Total: 402103 ns
[2021-12-29T19:15:44.415+0000][17977][gc,start    ] GC(58) Pause Young (Normal) (G1 Evacuation Pause)
[2021-12-29T19:15:44.415+0000][17977][gc,task     ] GC(58) Using 2 workers of 2 for evacuation
[2021-12-29T19:15:44.415+0000][17977][gc,age      ] GC(58) Desired survivor size 39321600 bytes, new threshold 15 (max threshold 15)
[2021-12-29T19:15:44.461+0000][17977][gc,age      ] GC(58) Age table with threshold 15 (max threshold 15)
[2021-12-29T19:15:44.461+0000][17977][gc,age      ] GC(58) - age   1:   11764824 bytes,   11764824 total
[2021-12-29T19:15:44.461+0000][17977][gc,age      ] GC(58) - age   2:    1862416 bytes,   13627240 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   3:     384120 bytes,   14011360 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   4:      60408 bytes,   14071768 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   5:     474328 bytes,   14546096 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   6:     344744 bytes,   14890840 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   7:     478736 bytes,   15369576 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   8:     194264 bytes,   15563840 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age   9:     307536 bytes,   15871376 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age  10:     302680 bytes,   16174056 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age  11:       3664 bytes,   16177720 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age  12:       3080 bytes,   16180800 total
[2021-12-29T19:15:44.462+0000][17977][gc,age      ] GC(58) - age  13:       9904 bytes,   16190704 total
[2021-12-29T19:15:44.463+0000][17977][gc,age      ] GC(58) - age  14:      47496 bytes,   16238200 total
[2021-12-29T19:15:44.463+0000][17977][gc,age      ] GC(58) - age  15:     109616 bytes,   16347816 total
[2021-12-29T19:15:44.463+0000][17977][gc,phases   ] GC(58)   Pre Evacuate Collection Set: 0.2ms
[2021-12-29T19:15:44.463+0000][17977][gc,phases   ] GC(58)   Merge Heap Roots: 0.2ms
[2021-12-29T19:15:44.463+0000][17977][gc,phases   ] GC(58)   Evacuate Collection Set: 43.9ms
[2021-12-29T19:15:44.463+0000][17977][gc,phases   ] GC(58)   Post Evacuate Collection Set: 1.5ms
[2021-12-29T19:15:44.463+0000][17977][gc,phases   ] GC(58)   Other: 2.4ms
[2021-12-29T19:15:44.463+0000][17977][gc,heap     ] GC(58) Eden regions: 580->0(576)
[2021-12-29T19:15:44.463+0000][17977][gc,heap     ] GC(58) Survivor regions: 18->17(75)
[2021-12-29T19:15:44.464+0000][17977][gc,heap     ] GC(58) Old regions: 101->101
[2021-12-29T19:15:44.464+0000][17977][gc,heap     ] GC(58) Archive regions: 2->2
[2021-12-29T19:15:44.464+0000][17977][gc,heap     ] GC(58) Humongous regions: 65->27
[2021-12-29T19:15:44.464+0000][17977][gc,metaspace] GC(58) Metaspace: 77302K(79692K)->77302K(79692K) NonClass: 68106K(69888K)->68106K(69888K) Class: 9195K(9804K)->9195K(9804K)
[2021-12-29T19:15:44.464+0000][17977][gc          ] GC(58) Pause Young (Normal) (G1 Evacuation Pause) 764M->145M(1024M) 49.129ms
[2021-12-29T19:15:44.464+0000][17977][gc,cpu      ] GC(58) User=0.04s Sys=0.00s Real=0.04s
[2021-12-29T19:15:44.464+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 537184397 ns, Reaching safepoint: 4513843 ns, At safepoint: 49416368 ns, Total: 53930211 ns
[2021-12-29T19:15:45.506+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1036972740 ns, Reaching safepoint: 4863347 ns, At safepoint: 16100 ns, Total: 4879447 ns
[2021-12-29T19:15:46.531+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1015862743 ns, Reaching safepoint: 9119887 ns, At safepoint: 60100 ns, Total: 9179987 ns
[2021-12-29T19:15:47.560+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1028263364 ns, Reaching safepoint: 323103 ns, At safepoint: 28901 ns, Total: 352004 ns
[2021-12-29T19:15:48.561+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1001003809 ns, Reaching safepoint: 365203 ns, At safepoint: 12900 ns, Total: 378103 ns
[2021-12-29T19:15:48.649+0000][17977][gc,start    ] GC(59) Pause Young (Normal) (G1 Evacuation Pause)
[2021-12-29T19:15:48.649+0000][17977][gc,task     ] GC(59) Using 2 workers of 2 for evacuation
[2021-12-29T19:15:48.649+0000][17977][gc,age      ] GC(59) Desired survivor size 39321600 bytes, new threshold 15 (max threshold 15)
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) Age table with threshold 15 (max threshold 15)
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   1:   16107056 bytes,   16107056 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   2:     290928 bytes,   16397984 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   3:     458856 bytes,   16856840 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   4:      26248 bytes,   16883088 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   5:      31528 bytes,   16914616 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   6:     474328 bytes,   17388944 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   7:     119472 bytes,   17508416 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   8:     478736 bytes,   17987152 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age   9:     189848 bytes,   18177000 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age  10:     136864 bytes,   18313864 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age  11:      16056 bytes,   18329920 total
[2021-12-29T19:15:48.670+0000][17977][gc,age      ] GC(59) - age  12:       2512 bytes,   18332432 total
[2021-12-29T19:15:48.671+0000][17977][gc,age      ] GC(59) - age  13:       3080 bytes,   18335512 total
[2021-12-29T19:15:48.671+0000][17977][gc,age      ] GC(59) - age  14:       6928 bytes,   18342440 total
[2021-12-29T19:15:48.671+0000][17977][gc,age      ] GC(59) - age  15:      47176 bytes,   18389616 total
[2021-12-29T19:15:48.671+0000][17977][gc,phases   ] GC(59)   Pre Evacuate Collection Set: 0.3ms
[2021-12-29T19:15:48.671+0000][17977][gc,phases   ] GC(59)   Merge Heap Roots: 0.2ms
[2021-12-29T19:15:48.671+0000][17977][gc,phases   ] GC(59)   Evacuate Collection Set: 18.7ms
[2021-12-29T19:15:48.671+0000][17977][gc,phases   ] GC(59)   Post Evacuate Collection Set: 1.5ms
[2021-12-29T19:15:48.671+0000][17977][gc,phases   ] GC(59)   Other: 0.6ms
[2021-12-29T19:15:48.671+0000][17977][gc,heap     ] GC(59) Eden regions: 576->0(573)
[2021-12-29T19:15:48.671+0000][17977][gc,heap     ] GC(59) Survivor regions: 17->19(75)
[2021-12-29T19:15:48.671+0000][17977][gc,heap     ] GC(59) Old regions: 101->101
[2021-12-29T19:15:48.671+0000][17977][gc,heap     ] GC(59) Archive regions: 2->2
[2021-12-29T19:15:48.671+0000][17977][gc,heap     ] GC(59) Humongous regions: 47->26
[2021-12-29T19:15:48.671+0000][17977][gc,metaspace] GC(59) Metaspace: 77330K(79692K)->77330K(79692K) NonClass: 68135K(69888K)->68135K(69888K) Class: 9195K(9804K)->9195K(9804K)
[2021-12-29T19:15:48.671+0000][17977][gc          ] GC(59) Pause Young (Normal) (G1 Evacuation Pause) 741M->146M(1024M) 21.547ms
[2021-12-29T19:15:48.671+0000][17977][gc,cpu      ] GC(59) User=0.02s Sys=0.00s Real=0.02s
[2021-12-29T19:15:48.671+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 87455031 ns, Reaching safepoint: 721007 ns, At safepoint: 21687506 ns, Total: 22408513 ns
[2021-12-29T19:15:49.674+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000131604 ns, Reaching safepoint: 3371032 ns, At safepoint: 19900 ns, Total: 3390932 ns
[2021-12-29T19:15:50.679+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000195309 ns, Reaching safepoint: 4004438 ns, At safepoint: 59500 ns, Total: 4063938 ns
[2021-12-29T19:15:51.682+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000235912 ns, Reaching safepoint: 2896527 ns, At safepoint: 15701 ns, Total: 2912228 ns
[2021-12-29T19:15:53.694+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 2011771140 ns, Reaching safepoint: 483405 ns, At safepoint: 16800 ns, Total: 500205 ns
[2021-12-29T19:15:54.702+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000168121 ns, Reaching safepoint: 7619873 ns, At safepoint: 38300 ns, Total: 7658173 ns
[2021-12-29T19:15:55.718+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1013892655 ns, Reaching safepoint: 2032620 ns, At safepoint: 15300 ns, Total: 2047920 ns
[2021-12-29T19:15:56.151+0000][17977][gc,start    ] GC(60) Pause Young (Normal) (G1 Evacuation Pause)
[2021-12-29T19:15:56.151+0000][17977][gc,task     ] GC(60) Using 2 workers of 2 for evacuation
[2021-12-29T19:15:56.151+0000][17977][gc,age      ] GC(60) Desired survivor size 38797312 bytes, new threshold 15 (max threshold 15)
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) Age table with threshold 15 (max threshold 15)
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   1:    8759416 bytes,    8759416 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   2:    1541760 bytes,   10301176 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   3:     220632 bytes,   10521808 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   4:     458424 bytes,   10980232 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   5:      21328 bytes,   11001560 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   6:      27536 bytes,   11029096 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   7:     474328 bytes,   11503424 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   8:     119472 bytes,   11622896 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age   9:     478736 bytes,   12101632 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  10:     189848 bytes,   12291480 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  11:     136120 bytes,   12427600 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  12:      16056 bytes,   12443656 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  13:       2512 bytes,   12446168 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  14:       2992 bytes,   12449160 total
[2021-12-29T19:15:56.170+0000][17977][gc,age      ] GC(60) - age  15:       6768 bytes,   12455928 total
[2021-12-29T19:15:56.170+0000][17977][gc,phases   ] GC(60)   Pre Evacuate Collection Set: 0.4ms
[2021-12-29T19:15:56.170+0000][17977][gc,phases   ] GC(60)   Merge Heap Roots: 0.2ms
[2021-12-29T19:15:56.170+0000][17977][gc,phases   ] GC(60)   Evacuate Collection Set: 16.6ms
[2021-12-29T19:15:56.170+0000][17977][gc,phases   ] GC(60)   Post Evacuate Collection Set: 1.8ms
[2021-12-29T19:15:56.170+0000][17977][gc,phases   ] GC(60)   Other: 0.6ms
[2021-12-29T19:15:56.170+0000][17977][gc,heap     ] GC(60) Eden regions: 573->0(581)
[2021-12-29T19:15:56.170+0000][17977][gc,heap     ] GC(60) Survivor regions: 19->13(74)
[2021-12-29T19:15:56.170+0000][17977][gc,heap     ] GC(60) Old regions: 101->101
[2021-12-29T19:15:56.170+0000][17977][gc,heap     ] GC(60) Archive regions: 2->2
[2021-12-29T19:15:56.170+0000][17977][gc,heap     ] GC(60) Humongous regions: 44->26
[2021-12-29T19:15:56.171+0000][17977][gc,metaspace] GC(60) Metaspace: 77402K(79692K)->77402K(79692K) NonClass: 68202K(69888K)->68202K(69888K) Class: 9199K(9804K)->9199K(9804K)
[2021-12-29T19:15:56.171+0000][17977][gc          ] GC(60) Pause Young (Normal) (G1 Evacuation Pause) 737M->140M(1024M) 19.812ms
[2021-12-29T19:15:56.171+0000][17977][gc,cpu      ] GC(60) User=0.02s Sys=0.01s Real=0.02s
[2021-12-29T19:15:56.171+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 430456300 ns, Reaching safepoint: 2319122 ns, At safepoint: 19946190 ns, Total: 22265312 ns
[2021-12-29T19:15:57.198+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1027197686 ns, Reaching safepoint: 294703 ns, At safepoint: 26800 ns, Total: 321503 ns
[2021-12-29T19:15:58.205+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000281333 ns, Reaching safepoint: 6659864 ns, At safepoint: 17400 ns, Total: 6677264 ns
[2021-12-29T19:15:59.210+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1002231755 ns, Reaching safepoint: 2490724 ns, At safepoint: 13100 ns, Total: 2503824 ns
[2021-12-29T19:16:00.242+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1031771040 ns, Reaching safepoint: 493705 ns, At safepoint: 14300 ns, Total: 508005 ns
[2021-12-29T19:16:01.258+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1015095385 ns, Reaching safepoint: 306003 ns, At safepoint: 32500 ns, Total: 338503 ns
[2021-12-29T19:16:01.998+0000][17977][gc,start    ] GC(61) Pause Young (Normal) (G1 Evacuation Pause)
[2021-12-29T19:16:01.998+0000][17977][gc,task     ] GC(61) Using 2 workers of 2 for evacuation
[2021-12-29T19:16:01.998+0000][17977][gc,age      ] GC(61) Desired survivor size 39321600 bytes, new threshold 15 (max threshold 15)
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) Age table with threshold 15 (max threshold 15)
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   1:    5369064 bytes,    5369064 total
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   2:    1884808 bytes,    7253872 total
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   3:    1093720 bytes,    8347592 total
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   4:     217440 bytes,    8565032 total
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   5:     454104 bytes,    9019136 total
[2021-12-29T19:16:02.039+0000][17977][gc,age      ] GC(61) - age   6:      20944 bytes,    9040080 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age   7:      22736 bytes,    9062816 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age   8:     426928 bytes,    9489744 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age   9:     119472 bytes,    9609216 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  10:     478736 bytes,   10087952 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  11:     189848 bytes,   10277800 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  12:     136120 bytes,   10413920 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  13:      16056 bytes,   10429976 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  14:       2512 bytes,   10432488 total
[2021-12-29T19:16:02.040+0000][17977][gc,age      ] GC(61) - age  15:       2992 bytes,   10435480 total
[2021-12-29T19:16:02.040+0000][17977][gc,phases   ] GC(61)   Pre Evacuate Collection Set: 0.3ms
[2021-12-29T19:16:02.041+0000][17977][gc,phases   ] GC(61)   Merge Heap Roots: 0.2ms
[2021-12-29T19:16:02.041+0000][17977][gc,phases   ] GC(61)   Evacuate Collection Set: 38.1ms
[2021-12-29T19:16:02.041+0000][17977][gc,phases   ] GC(61)   Post Evacuate Collection Set: 1.7ms
[2021-12-29T19:16:02.041+0000][17977][gc,phases   ] GC(61)   Other: 2.2ms
[2021-12-29T19:16:02.041+0000][17977][gc,heap     ] GC(61) Eden regions: 581->0(582)
[2021-12-29T19:16:02.041+0000][17977][gc,heap     ] GC(61) Survivor regions: 13->11(75)
[2021-12-29T19:16:02.041+0000][17977][gc,heap     ] GC(61) Old regions: 101->101
[2021-12-29T19:16:02.041+0000][17977][gc,heap     ] GC(61) Archive regions: 2->2
[2021-12-29T19:16:02.041+0000][17977][gc,heap     ] GC(61) Humongous regions: 88->33
[2021-12-29T19:16:02.041+0000][17977][gc,metaspace] GC(61) Metaspace: 77452K(79692K)->77452K(79692K) NonClass: 68252K(69888K)->68252K(69888K) Class: 9199K(9804K)->9199K(9804K)
[2021-12-29T19:16:02.042+0000][17977][gc          ] GC(61) Pause Young (Normal) (G1 Evacuation Pause) 783M->145M(1024M) 43.566ms
[2021-12-29T19:16:02.042+0000][17977][gc,cpu      ] GC(61) User=0.06s Sys=0.00s Real=0.04s
[2021-12-29T19:16:02.042+0000][17977][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 737973842 ns, Reaching safepoint: 2375223 ns, At safepoint: 43884419 ns, Total: 46259642 ns
[2021-12-29T19:16:03.058+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1016231001 ns, Reaching safepoint: 163802 ns, At safepoint: 21500 ns, Total: 185302 ns
[2021-12-29T19:16:04.059+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000162351 ns, Reaching safepoint: 185001 ns, At safepoint: 7501 ns, Total: 192502 ns
[2021-12-29T19:16:05.059+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000446056 ns, Reaching safepoint: 255903 ns, At safepoint: 45400 ns, Total: 301303 ns
[2021-12-29T19:16:07.060+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 2000359418 ns, Reaching safepoint: 349404 ns, At safepoint: 16900 ns, Total: 366304 ns
[2021-12-29T19:16:11.061+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 4001029776 ns, Reaching safepoint: 265903 ns, At safepoint: 23500 ns, Total: 289403 ns
[2021-12-29T19:16:18.063+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 7001085497 ns, Reaching safepoint: 387803 ns, At safepoint: 47301 ns, Total: 435104 ns
[2021-12-29T19:16:19.063+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 1000254999 ns, Reaching safepoint: 174701 ns, At safepoint: 7600 ns, Total: 182301 ns
[2021-12-29T19:16:22.066+0000][17977][safepoint   ] Safepoint "Cleanup", Time since last: 3000738015 ns, Reaching safepoint: 2331922 ns, At safepoint: 40301 ns, Total: 2372223 ns
[2021-12-29T19:16:22.487+0000][17977][gc,heap,exit] Heap
[2021-12-29T19:16:22.487+0000][17977][gc,heap,exit]  garbage-first heap   total 1048576K, used 395556K [0x00000000c0000000, 0x0000000100000000)
[2021-12-29T19:16:22.487+0000][17977][gc,heap,exit]   region size 1024K, 252 young (258048K), 11 survivors (11264K)
[2021-12-29T19:16:22.487+0000][17977][gc,heap,exit]  Metaspace       used 77641K, capacity 79361K, committed 79948K, reserved 1120256K
[2021-12-29T19:16:22.487+0000][17977][gc,heap,exit]   class space    used 9229K, capacity 9771K, committed 9804K, reserved 1048576K

Do you notice the time stamps? Why does /var/log/elasticsearch/gc.log think it’s 2021-12-29T19:15:33.816? That does not seem right. I’m on Eastern Time, it’s 14:43 right now.

Topic		Replies	Views
Some messages not showing Graylog Central (peer support)	3	180	September 6, 2022
Graylog port change not held at reboot Graylog Central (peer support)	3	646	September 1, 2021
Graylog not found any syslog Graylog Central (peer support) basic-configuration , elastic	7	572	May 12, 2022
Messages getting to server but not to Graylog Graylog Central (peer support)	5	2005	April 9, 2019
Graylog stopps processing messages. Only coming in but no out Graylog Central (peer support)	6	10430	November 12, 2018