@gsmith thanks for the responses.
I have digged into it a little further and found a way to reproduce the problem using the OVA appliances.
https://packages.graylog2.org/appliances/ova
The behaviour seems to change between versions graylog-3.3.9-1.ova (updated via apt to 3.3.11) and graylog-4.0.0-5.rc.1.ova. I cannot seem to find differences in the GL/ES config files between these two that would affect the journaling/buffering behavoiur… The only difference i have spotted is that 3.3.11 dosent have the ES Support plugins installed, while 4.0.0 does have them, i have tried removing them from 4.0.0 but then graylog dosent want to start… so no luck there. The changelog dosent provide any info regarding changes that could have anything to do with this either. Could there maybe have been implemented something by accident between the two versions that breaks the buffering?
This might also just be a problem on ubuntu/debian, as you @gsmith dont have the problem on centos.
Also i can understand from reading around that the message processing goes something like this:
input → input buffer → journal → process buffer → output buffer → ES
following this, the behaviour from version 3.3.11 and earlier makes sense, where when ES is stopped, the journal and output buffer fills up, then the journal and process buffer and then just the journal, as the messages now have nowhere else to go, keeping in mind that they are all still present in the journal as the messages are not yet written to any other persistent memory.
Not sure if im completely right about this though…
With 4.0.0 and higher the messages do start counting up in the journal, but no matter what amount of messages are there they all vanish at the same time, when ES is started again and dont seem to be processed as is clear from the message counter that sits at 0 in that timeframe.