We’ve got our Graylog Enterprise instance set up with AWS ElasticSearch service as the index and an S3 bucket for archive storage (indices rotated once a day). This unfortunately means that when an archive is done, data has to be downloaded from AWS, exported and compressed, and then uploaded to AWS. This works relatively well except for the fact that the bandwidth used exceeds our contracted rate for a couple of hours, which results in significant overage charges.
To remedy that, I’ve throttled the bandwidth on the log server with wondershaper/tc which succeeds in limiting the bandwidth. However, every archive that I run since the change fails to complete. It only writes one segment and then quits. I see this in the server log:
2018-02-05T13:48:14.673-06:00 INFO [RollingFileSegmentOutputStream] Creating new segment: /opt/s3/graylog-archives/graylog_39-20180205-163344-372/archive-segment-1.gz
2018-02-05T13:48:25.581-06:00 ERROR [ArchiveCreateJob] Archived only 4421000 out of 8508322 documents, not deleting/closing index graylog_39
2018-02-05T13:48:25.593-06:00 INFO [SystemJobManager] SystemJob <55148030-0a92-11e8-91ec-fee5de21aa98> [org.graylog.plugins.archive.job.ArchiveCreateSystemJob] finished in 11681221ms.
Is there anything I can do to fix this?