Using curl (and also browser api), I cannot recover the messages related to a server, which indeed (I verified it by doing a strace in the graylog process), I sent at this moment. Use this call: curl -s --user $username:$password -k -X GET -H “Content-Type: application/json” -H “X-Requested-By: SomeUniqueValue2” “https://$host: 9000/api/search/universal/relative/export?query=source&range=36000000&batch_size=5000&fields=source&pretty=true”
Can anybody help me?
What issue are you having? What is your expected outcome vs what is actually happening?
You run that command and then what happens? Does it complete? Does it return an error?
Are you trying to use the export api endpoint to export ALL messages in your cluster?
Hi, sir.
Well, I have 2 problems:
- Some devices that send logs are seen in the web interface, and are not seen using the API. That was my initial problem, but, I need to solve the following problem first.
- What worries me most is that certain devices are not seen in either interface. And clearly, the messages are sent from those computers (I did a strace and I see that the message is sent). How can I do some kind of debug of what graylog receives and how it is being processed by the application?
Here is the output of the strace about the rsyslogd process on the source computer:
[pid 14846] sendto(1, “<38>Dec 30 20:35:27 dalila sshd(pam_unix)[15258]: session opened for user root by (uid=0)”, 89, 0, {sa_family=AF_INET, sin_port=htons(1514), sin_addr=inet_addr(“10.214.3.140”)}, 16) = 89
Clearly, the 89 at the end indicates that the 89 bytes were correctly written to the socket connected to the graylog (whose ipe is 10.214.3.140, listening on port 1514).
From the server side, I only find this:
root@graylog:~# netstat -anp | grep 1514 | grep LISTEN
tcp6 0 0 :::1514 :::* LISTEN 1047/java
root@graylog:~# ps -fp 1047
UID PID PPID C STIME TTY TIME CMD
graylog 1047 1008 43 Dec26 ? 1-19:57:02 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+R
root@graylog:~# ls -l /proc/1047/fd | grep “.log”
lrwx------ 1 graylog graylog 64 Dec 30 20:36 280 → /var/lib/graylog-server/journal/messagejournal-0/00000000009108216836.log
l-wx------ 1 graylog graylog 64 Dec 28 14:44 5 → /var/log/graylog-server/server.log
root@graylog:~# file /var/lib/graylog-server/journal/messagejournal-0/00000000009108216836.log
/var/lib/graylog-server/journal/messagejournal-0/00000000009108216836.log: data
root@graylog:~# tail -1 /var/log/graylog-server/server.log
2023-12-30T20:39:19.428-03:00 WARN [RestClient] request [GET http://elastic2.litoral-gas.com.ar:9200/_cluster/settings?master_timeout=30s&include_defaults=true] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 “[discovery.zen.minimum_master_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.”]
I understand that I have to review some type of log within the application, but I still do not have the knowledge to do so. Might you help me?
I’m curious what you see on the graylog server using tcpdump (something like)
(replace the interface name with the one applicable to your system)
sudo tcpdump -i eth0 -nA port 1514
Can you confirm that your graylog syslog input is configured with the timezone that matches the timezone of the systems that send syslog to graylog?
I’m also curious what happens if you send a sample/test syslog message such as:
(replace the IP with the ip of your graylog server)
echo 'This is a test syslog message' | nc -w 1 172.31.31.250 1514
Hi, Miranda.
I did execute the command:
tcpdump -i ens192 -nA port 1514 | grep myip
in the graylog machine
If in the original machine, I try:
logger “hello from dalila”
I did not receive messages in the tcpdump.
But, If I send message with your nc command (echo “hello from dalila” | nc -w 1 graylog 1514", I did see messages in the tcpdump window:
root@graylog:~# tcpdump -i ens192 -nA port 1514 | grep -E “dalila|192.168.1.65”
tcpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:44:56.108313 IP 192.168.1.65.64019 > 10.214.3.140.1514: Flags [S], seq 216797085, win 5840, options [mss 139
8,sackOK,TS val 1366643849 ecr 0,nop,wscale 2], length 0
08:44:56.108362 IP 10.214.3.140.1514 > 192.168.1.65.64019: Flags [S.], seq 1533931055, ack 216797086, win 65535
, options [mss 1460,sackOK,TS val 268166444 ecr 1366643849,nop,wscale 2], length 0
08:44:56.114500 IP 192.168.1.65.64019 > 10.214.3.140.1514: Flags [.], ack 1, win 1460, options [nop,nop,TS val
1366643855 ecr 268166444], length 0
08:44:56.115140 IP 192.168.1.65.64019 > 10.214.3.140.1514: Flags [P.], seq 1:19, ack 1, win 1460, options [nop,
nop,TS val 1366643856 ecr 268166444], length 18
QuT…,hello from dalila
08:44:56.115159 IP 10.214.3.140.1514 > 192.168.1.65.64019: Flags [.], ack 19, win 16384, options [nop,nop,TS va
l 268166451 ecr 1366643856], length 0
08:44:56.115436 IP 192.168.1.65.64019 > 10.214.3.140.1514: Flags [F.], seq 19, ack 1, win 1460, options [nop,no
p,TS val 1366643856 ecr 268166444], length 0
08:44:56.119490 IP 10.214.3.140.1514 > 192.168.1.65.64019: Flags [F.], seq 1, ack 20, win 16384, options [nop,n
op,TS val 268166455 ecr 1366643856], length 0
08:44:56.126332 IP 192.168.1.65.64019 > 10.214.3.140.1514: Flags [.], ack 2, win 1460, options [nop,nop,TS val
1366643867 ecr 268166455], length 0
…&…?8!<189>date=2024-01-03 time=08:45:06 devname=“FG-BuenosAires1” devid=“FG3H0E5819902267” eventtime=1704
282306758124955 tz=“-0300” logid=“0000000013” type=“traffic” subtype=“forward” level=“notice” vd=“root” srcip=1
92.168.1.65 srcport=64019 srcintf=“port2” srcintfrole=“lan” dstip=10.214.3.140 dstport=1514 dstintf=“RI-IPLAN-R
ibera” dstintfrole=“undefined” srccountry=“Reserved” dstcountry=“Reserved” sessionid=292743794 proto=6 action="
timeout" policyid=2 policytype=“policy” poluuid=“3e494e76-83fc-51eb-2582-528a7d67e16b” policyname=“WAN In " ser
vice=“tcp/1514” trandisp=“noop” appcat=“unknown” applist=“g-default” duration=10 sentbyte=572 rcvdbyte=0 sentpk
t=10 rcvdpkt=0 vpn=“RI-IPLAN-Ribera” vpntype=“ipsec-static” srchwvendor=“Cisco” osname=“Windows” mastersrcmac=”
00:23:34:8f:39:c6" srcmac=“00:23:34:8f:39:c6” srcserver=0
…&…`C<189>date=2024-01-03 time=08:45:06 devname=“FG-Ribera2” devid=“FG3H0E5819901950” eventtime=170428230
6396664382 tz=“-0300” logid=“0000000013” type=“traffic” subtype=“forward” level=“notice” vd=“root” srcip=192.16
8.1.65 srcport=64019 srcintf=“RB-IPLAN-Ringo” srcintfrole=“undefined” dstip=10.214.3.140 dstport=1514 dstintf="
port13" dstintfrole=“undefined” srccountry=“Reserved” dstcountry=“Reserved” sessionid=662346212 proto=6 action=
“timeout” policyid=1 policytype=“policy” poluuid=“660d889c-45bf-51ed-96a6-673a6371aab9” policyname=“De WAN a Ri
b” service=“tcp/1514” trandisp=“noop” appcat=“unknown” applist=“g-default” duration=10 sentbyte=286 rcvdbyte=0
sentpkt=5 rcvdpkt=0 vpn=“RB-IPLAN-Ringo” vpntype=“ipsec-static”
I did install rsyslogd from sources, in the machine (is a old machine), because syslog is not working.
The rsyslogd is running (I’m transcribing because your site is not accepting more than one screenshot):
netstat -anp | grep syslog
udp 0 0 0.0.0.0:10839 0.0.0.0:* 6543/rsyslogd
unix 3 DGRAM 5792865 6543/rsyslogd /dev/log
ps -fp 6543
root 6543 1 0 2023 ? 00:00:03 rsyslogd
About timezone, in my machine, I have:
[root@dalila] # ls -l /etc/localtime
lrwxrwxrwx 1 root root 50 Jan 14 2014 /etc/localtime → /usr/share/zoneinfo/America/Argentina/Buenos_Aires
In graylog, I have:
I see the configured time zone, is gmt-3.
Thanks for your patient!
Greetings