Graylog for plesk nginx log_access

Plesk is one of the most using hosting panel and we like to add logs of all shared hosting server they use plesk to get logs centralized! Only the transfer of the nginx logs crashed why something is not so nice as on other linux servers, a problem of nginx log sending via UDP to the graylog server!

Can been one of them a workarround?

a) Find a solution to filter the first 50 bytes of the input to trash - the rest is then the nice input uncompressed and I thing so graylog can handle?
b) Find a solution, that nginx in plesk create the correct file for UDP with compression in gzip? I dont know that possible with compression the nginx produce in the beginning the 50 bytes, we and I thing so graylog not need?

More about the problem:

Nginx works for many domains in conjunction with the apache daemon. The log files for apache comes via gelf udp perfect, but the client-ips are every time the ip of the server. I thing so the mod_log_gelf.so use the wrong files (in conjuction with nginx).

About this I set up a 2nd stream for the nginx logs to an other UDP-input! I set in the nginx.conf a log_format with escape=json and then I send via access_log to the server and I have check with tcpdump the transfered datas:

2.1 In first the nginx in plesk will not compress the datas. A setting on “gzip on” in ngnix.conf brings errors!
2.2 In second the nginx in plesk produce something in the first 50 bytes, when the access_log send to the graylog server. When the access_log send the same to a local file, then the 50 bytes are not their - i show here:

0x0000:  9ad6 b45e df83 2899 3a99 a2f5 0800 4500  ...^..(.:.....E.
0x0010:  02c5 744c 4000 3a11 8bf7 2e69 5322 b01f  ..tL@.:....iS"..
0x0020:  0c3a 88b7 3071 02b1 a0c7 3c31 3930 3e41  .:..0q....<190>A
0x0030:  7567 2032 3120 3038 3a31 393a 3138 2070  ug.21.08:19:18.p
0x0040:  6c31 7365 7276 6572 206e 6769 6e78 3a20  l1server.nginx:.
0x0050:  7b20 2274 696d 6573 7461 6d70 223a 2022  {."timestamp":."
0x0060:  3230 3230 2d30 382d 3231 5430 383a 3139  2020-08-21T08:19
0x0070:  3a31 382b 3032 3a30 3022 2c20 2272 656d  :18+02:00",."rem
0x0080:  6f74 655f 6164 6472 223a 2022 3135 372e  ote_addr":."157.
0x0090:  3535 2e33 392e 3437 222c 2022 636f 6e6e  55.39.47",."conn
0x00a0:  6563 7469 6f6e 223a 2022 3238 3022 2c20  ection":."280",.
0x00b0:  2263 6f6e 6e65 6374 696f 6e5f 7265 7175  "connection_requ
0x00c0:  6573 7473 223a 2031 2c20 2270 6970 6522  ests":.1,."pipe"

It looks like a header in the file and the problem is the byte with the char “<” for graylog! Graylog stop with this warning:

com.fasterxml.jackson.core.JsonParseException: Unexpected character (’<’ (code 60)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’)
at [Source: (String)"<190>Aug 21 08:45:16 abcserver nginx: { “timestamp”: “2020-08-21T08:45:16+02:00”, “remote_addr”: “256.256.256.256”, “connection”: “643”, “connection_requests”: 1, “pipe”: “.”, “body_bytes_sent”: 1326, “request_length”: 198, “request_time”: 0.000, “response_status”: 200, “request”: “GET /robots.txt HTTP/1.1”, “request_method”: “GET”, “host”: “www.******.com”, “upstream_cache_status”: “”, “upstream_addr”: “”, “http_x_forwarded_for”: “”, “http_referrer”: “”, “http_user_agent”: “Mozilla/5.0 (compatible”[truncated 187 chars]; line: 1, column: 2]

bumping - is nobody here they can help?

Possible problems:

  1. You send one type of logs to wrong input. For example gelf to udp input. Please check that you use right input.
  2. You use json parser (extractor or pipeline rule) for another messages, which is sole json, and not syslog format? Check all your extractors and pipeline rule.
  3. Try to use Raw UDP Input for testing, if it work

Dear shoothub,

The 1st input - only apache with gelf udp - port 12201:
for the apache_gelf - wich is running ok, but the IP is not fine about the ngnix proxy
the transfer use only UDP and this is running good!

The 2nd input - only syslogs (other then apache) - port 1514:
The syslog transfer running in moment eqaul via UDP to syslog fine

The 3rd input - only ngnix with gelf udp - port 12401:

The Graylog-Server only accept UDP-connections only from machines of the internal network and
only for the specified ports !!!

Your answers:

“1. You send one type of logs to wrong input. For example gelf to udp input. Please check that you use right input.”

Nginx log is configured to a separate destination port, other the apache files they use their own port for graylog! 
Nginx syslog cant send to TCP - a cut from a website of serverfault dot com:

"There is no way to send via TCP in this way, which is quite a common way to send logs on a network, since you want the message delivery to be reliable (look at the RELP module for evidence of that need).

But also, only being able to send logs via UDP is quite common, a number of network devices (switches or routers for example) are only capable of sending by UDP, so sending to a syslog gateway ...."

End of cut !!!

“2. You use json parser (extractor or pipeline rule) for another messages, which is sole json, and not syslog format? Check all your extractors and pipeline rule.”

This is the default ngnix log syslog option as in ngnix is described on the logged server!
The only thing is, that I dont know why, plesk has a sepecial configuration why compression is NOT working! 
So, when you like, I can install for you an plesk-server for playing, so you can test itself!

“3. Yes UDP is working”

The files loads to the server, I can see this via tcpdump. I dont know where the file will been saved, in the graylog-server structure!
I havent found this informations about this input folder in the graylog documentation! I am sorry about this!

Back again to the problem, as I have 1st written:

The server with ngnix and plesk installation writes this typical 20 bytes at first, that the graylog server cant understand! I dont know wyh?!
I cant see, that this bullshit creates the linux syslog daemon or the ngnix daemon self! But this 20 bytes are not created, when they write on local disc!
So I have searched in google: never use Graylog for Plesk-Servers with ngnix !!! But the most servers with admin-interface runs under cpanel and plesk worldwide!

Please send you configuration on nginx side, how do you send logs to graylog.

Dear shoothub,

thanks for help of analyzing. Here the config-file for nginx - nginx.conf:

#user nginx;
worker_processes 1;

#error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;

#pid /var/run/nginx.pid;

include /etc/nginx/modules.conf.d/*.conf;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';
#

log_format gelf_json escape=json '{ "timestamp": "$time_iso8601", '
     '"remote_addr": "$remote_addr", '
     '"connection": "$connection", '
     '"connection_requests": $connection_requests, '
     '"pipe": "$pipe", '
     '"body_bytes_sent": $body_bytes_sent, '
     '"request_length": $request_length, '
     '"request_time": $request_time, '
     '"response_status": $status, '
     '"request": "$request", '
     '"request_method": "$request_method", '
     '"host": "$host", '
     '"upstream_cache_status": "$upstream_cache_status", '
     '"upstream_addr": "$upstream_addr", '
     '"http_x_forwarded_for": "$http_x_forwarded_for", '
     '"http_referrer": "$http_referer", '
     '"http_user_agent": "$http_user_agent", '
     '"http_version": "$server_protocol", '
     '"remote_user": "$remote_user", '
     '"http_x_forwarded_proto": "$http_x_forwarded_proto", '
     '"upstream_response_time": "$upstream_response_time", '
     '"nginx_access": true }';

#access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;
#tcp_nodelay        on;

#gzip  on;
#gzip_disable "MSIE [1-6]\.(?!.*SV1)";

server_tokens off;

include /etc/nginx/conf.d/*.conf;

}

override global parameters e.g. worker_rlimit_nofile

include /etc/nginx/*global_params;

Ok, and here the vhosts part:

access_log syslog:server=:12401 gelf_json;

Gzip cant set to on in the nginx.conf for plesk, otherwise the nginx daemon will not start - A problem of plesk-speciality, But I thing so, without gzip in raw-mode must been function same?!

Thanks

Detlef

I found your problem, and it was clear from beginning, I’ve already asked you if you use wrong Input for nginx data… you said no, but it’s not true…

Your problem:
You use wrong Graylog Input for nginx data. Nginx send message as json using custom template using SYSLOG protocol, and you try to send it to GELF Input in Graylog. GELF is completely different protocol from Syslog.
Nginx send syslog message e.g:
<190>Sep 29 15:28:05 graylog nginx: { "timestamp": "2020-09

But GELF formated message requires special syntax (also mandatory fields) in json format e.g.:
{ "version": "1.1", "host": "example.org", "short_message": "A short message", "level": 5, "_custom_field": "foo", "timestamp": 1385053862.3072}

Solution:
Change your port in nginx configuration access_log syslog:server=:PORT gelf_json; and point PORT to Syslog Input defined in Graylog. If you want to next parse (extract) fields defined in json format, use pipeline rule or extractor to do so.

Hi shoothub,

yes many thanks, thats look nice logical! Ok, I will change as you have wriiten and I thing so, then this will work. Many thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.