UDP buffer size problem despite system configuration

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Despite system configuration Graylog is unable to set a UDP buffer size bigger than 65536.

The system is configured for a UDP buffer size of 1048576 bytes.

Checked with a silly C program that does a socket(), setsockopt(), getsockopt().

% sysctl net.inet.udp.recvspace
net.inet.udp.recvspace: 1048576
./checkbuf 262144
SO_RCVBUF: requested: 262144, actual: 262144
 ./checkbuf 1048576
SO_RCVBUF: requested: 1048576, actual: 1048576

Fails as expected then exceeding UDP buffer size.
.

/checkbuf 2048576
setsockopt(): No buffer space available

I am not sure whether this is a limitation on OpenJDK 17, OpenJDK 17 on FreeBSD, netty or Graylog.

2. Describe your environment:

  • OS Information: FreeBSD 14.1 (amd64)

  • Package Version: Graylog 6.0.1

  • Service logs, configurations, and environment variables:
    2024-06-14 10:50:21,533 WARN o.g.i.t.UdpTransport [netty-transport-0] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=DNS syslog, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=XXXXXXX} (channel [id: 0x1bc943a9, L:/10.0.0.1:1312]) should be >= 1048576 but is 65536.

3. What steps have you already taken to try and solve the problem?
I have made sure the system is properly configured.

4. How can the community help?

If you are seeing these warnings on the logfile, check that your buffer size is properly configured using the attached program.

---- Check program, checkbuf.c

/*
 * checkbuf.c
 * Verifies UDP buffer size
 */

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <sys/types.h>
#include <sys/socket.h>


int main(int argc, char **argv)
{
    int s, r;
    unsigned int bufsiz, actual_bufsiz, len;

    if ( argc < 2 ) {
        fprintf(stderr, "Uso: %s buffer\n", argv[0]);
        exit(1);
    }
    bufsiz = atoi(argv[1]);
    
    s = socket(PF_INET, SOCK_DGRAM, 0);
    if ( s < 0 ) {
        perror("socket()");
        exit(1);
    }
    r = setsockopt(s, SOL_SOCKET, SO_RCVBUF, &bufsiz, sizeof(bufsiz));
    if ( r < 0 ) {
        perror("setsockopt()");
        exit(1);
    }
    /* Now verify */
    len = sizeof(actual_bufsiz);
    r = getsockopt(s, SOL_SOCKET, SO_RCVBUF, &actual_bufsiz, &len);
    if ( r < 0 ) {
        perror("getsockopt()");
        exit(1);
    }
    printf("SO_RCVBUF: requested: %d, actual: %d\n", bufsiz, actual_bufsiz);
    exit(0);
}

There is a workaround to prevent this issue:

When setting the buffer size to 0 on graylog.conf and not setting a buffer size for the input configuration, the setsockopt() system call fails and the socket keeps the default buffer size.

1048576 is the buffer size configured on my system.

Proto Recv-Q Send-Q Local Address          Foreign Address       R-HIWA S-HIWA R-LOWA S-LOWA R-BCNT S-BCNT R-BMAX S-BMAX   rexmt persist    keep    2msl  delack rcvtime
udp4       0      0 10.0.0.1.515      *.*                    **1048576**   9216      1   2048      0      0 2097152  73728
1 Like

I believe this behavior has always existed. I’m out of my depth understanding the specifics but does seem to be related to how the operating system is configured.

I did find receiveBufferSize (SO_RCVBUF) should be 262144 but is 212992. · Issue #1350 · Graylog2/graylog2-server · GitHub which says that the following command resolves the error/warning:

sysctl -w net.core.rmem_max=1048576

I just tested this in my lab where i was receiving:

[AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input PaloAlto9xInput{title=Log Replay - Palo 9+, type=org.graylog.integrations.inputs.paloalto9.PaloAlto9xInput, nodeId=null} (channel [id: 0x88f847b7, L:/[0:0:0:0:0:0:0:0%0]:2516]) should be >= 1048576 but is 425984.
[AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Log Replay - Syslog TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=null} (channel [id: 0x760a0c8a, L:/[0:0:0:0:0:0:0:0%0]:2515]) should be >= 1048576 but is 425984.

And after running the above command and restarting graylog-server i no longer receive the warnings.

1 Like

There seems to be some inconsistency/limitation on how buffer size is treated.

Maybe it depends on the operating system. You are mentioning Linux, I am mentioning FreeBSD.

On FreeBSD I have the right UDP buffer size configuration. However, if I specify a buffer size (either on graylog.conf or an UDP input definition) it is capped to 65536.

However, if I specify 0 for both (configuration file and input definition) the default buffer size (as configured on the OS) is used because Graylog tries a setsockopt() with a buffer size of 0, which fails but does not invalidate the socket :slight_smile:

I think I will open a Github issue.

If you want to check whether your problem is due to Graylog or the operating system try compiling and running the small program I have pasted here.

Just for the record in case someone lands here, there is a workaround on FreeBSD.

FIrst, you need to set up the default UDP buffer size with:

(Example for 1 MB, 1048576 bytes)
# sysctl net.inet.udp.recvsize=1048576

Second, make sure to both:

  • Set udp_recvbuffer_sizes = 0 on graylog.conf
  • Leave the input buffer sizes empty for the UDP inputs on Graylog

You can verify that the desired buffer size is used with:

% sysctl net.inet.udp.recvspace
net.inet.udp.recvspace: 524288

% netstat -p udp -x -n -a
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address       R-HIWA S-HIWA R-LOWA S-LOWA R-BCNT S-BCNT R-BMAX S-BMAX   rexmt persist    keep    2msl  delack rcvtime
udp4       0      0 *.53917                *.*                    524288   9216      1   2048      0      0 2097152  73728
udp4       0      0 *.17230                *.*                    524288   9216      1   2048      0      0 2097152  73728
udp4       0      0 *.18359                *.*                    524288   9216      1   2048      0      0 2097152  73728
udp4       0      0 *.50582                *.*                    524288   9216      1   2048      0      0 2097152  73728
udp4       0      0 *.50040                *.*                    524288   9216      1   2048      0      0 2097152  73728
udp4       0      0 *.27975                *.*                    524288   9216      1   2048      0      0 2097152  73728

The R-HIWA shows the actual receiver buffer size for the UDP sockets. The value should match net.inet.udp.recvspace.

Editing graylog.conf requires a Graylog restart. When you set that bogus value on graylog.conf the setsockopt(SO_RCVBUF) call will fail but Graylog will issue a warning and move forward, effectively leaving the default buffer size intact.

After graylog.conf is properly configured you can adjust the UDP buffer size by changing net.inet.udp.recvspace and as long as the input buffer size field is empty restarting the input will adjust the buffer size.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.