I am running into an issue I can’t solve on a single node install of Graylog 2.5 on Ubuntu 16.04 server with the following specs: 32 gb ram and 8 CPUs
Graylog is installed and running successfully receiving logs from syslog and http inputs on the node described above.
Issue:
- Add the Palo Alto input:
http://docs.graylog.org/en/2.5/pages/integrations/inputs/palo_alto_networks_input.html - The input is added in the UI successfully and is reported in the UI as:
Running
and netstat shows that the port in question is listening. - Update the PA NGFW to start sending logs to the ip/port of the graylog server. As SOON as the config on the FW is updated to send logs to the graylog server, the graylog dies in browser and can no longer be loaded in a browser, the error in browser is:
This site can’t be reached - graylog-sec.local took too long to respond.
The graylog-server server.log does not update at all when this condition occurs (the last log message is that the input is running). If I restart the graylog-server service it shows that the http server and rest api start up and the inputs are running (see the end of this post for the log snippet).
Initially I noticed that there were messages like this in the logs:
WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input Palo Alto Networks Input{title=pa-tcp, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=105bdb6c-01d4-4621-bc7b-81643b0414bd} should be 1048576 but is 212992
I updated the receive buffer defaults from 212992 to a number higher than 1048576, and confirmed the changed took effect (the logs no longer report this error about the PA input).
Restarting the graylog-server service, everything starts up successfully according to the logs, but the UI is still unreachable w/ the error I noticed above (site can’t be reached).
Server log snippet:
2019-01-29T16:58:08.106Z INFO [NetworkListener] Started listener bound to [10.xxx.xxx.xxx:9000]
2019-01-29T16:58:08.108Z INFO [HttpServer] [HttpServer] Started.
2019-01-29T16:58:08.108Z INFO [JerseyService] Started REST API at <https://10.xxx.xxx.xxx:9000/api/>
2019-01-29T16:58:08.109Z INFO [JerseyService] Started Web Interface at <https://10.xxx.xxx.xxx:9000/>
2019-01-29T16:58:08.109Z INFO [ServiceManagerListener] Services are healthy
2019-01-29T16:58:08.110Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2019-01-29T16:58:08.110Z INFO [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=2, JournalReader [RUNNING]=18, InputSetupService [RUNNING]=20, KafkaJournal [RUNNING]=60, ConfigurationEtagService [RUNNING]=119, OutputSetupService [RUNNING]=143, StreamCacheService [RUNNING]=152, PeriodicalsService [RUNNING]=365, LookupTableService [RUNNING]=439, JerseyService [RUNNING]=18915}
2019-01-29T16:58:08.120Z INFO [ServerBootstrap] Graylog server up and running.
2019-01-29T16:58:08.190Z INFO [InputStateListener] Input [Syslog TCP/5c4b5a0c878cc5594a1e22a7] is now STARTING
2019-01-29T16:58:08.192Z INFO [InputStateListener] Input [Palo Alto Networks Input (TCP)/5c5073cc878cc510adf1eff5] is now STARTING
2019-01-29T16:58:08.283Z INFO [InputStateListener] Input [Palo Alto Networks Input (TCP)/5c5073cc878cc510adf1eff5] is now RUNNING
2019-01-29T16:58:08.285Z INFO [InputStateListener] Input [Syslog TCP/5c4b5a0c878cc5594a1e22a7] is now RUNNING