Newbie install on Fedora, can log via curl but not PHP

[neekfenwick (Nick Fenwick)]

Newbie here, trying to make Graylog work on my simple local dev machine. I find that I can access the Graylog web UI and see a single message sent via command line Curl, but messages sent from my PHP code don’t seem to appear.

For example, this command line succeeds and I see the message in Graylog:
curl -XPOST http://127.0.0.1:12201/graylog -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

However, this code also completes (I have stepped through the code in VSCode and seen an ‘ok’ response from the curl request made by the GELF PHP code), yet I don’t see the message in Graylog:

$gelfPublisher = new Gelf\Publisher( new Gelf\Transport\TcpTransport('127.0.0.1',12201) );
$message = new Gelf\Message();
$message->setShortMessage("It works in TCP!")
        ->setLevel(\Psr\Log\LogLevel::ALERT)
        ->setFullMessage("There was a foo in bar")
;
$gelfPublisher->publish($message);

Still the graylog web UI shows a single log message ‘Hello there’ and none of the ‘It works in TCP!’ ones.

I’ve been through the basic Installation guide for CentOS (close enough to my Fedora system), so I now have the following packages installed:
graylog-server-4.2.8-1.noarch
elasticsearch-oss-7.10.2-1.x86_64
mongodb-org-server-4.4.4-1.el8.x86_64

I’ve followed the required installation steps such as defining password_secret and root_password_sha2 in /etc/graylog/server/server.conf and cluster.name and action.auto_create_index in /etc/elasticsearch/elasticsearch.yml.

The logs in /var/log/graylog-server/server.log and /var/log/elasticsearch/graylog.log show up to date log lines and no errors. Restarting the services with systemctl restart graylog-server.service and systemctl restart elasticsearch.service doesn’t seem to help.

I’ve read that the log data is stored in elasticsearch, but I don’t have personal knowledge of this software. I know mongod and can connect to it and query it but it just holds config. What’s a simple way I can connect to my elasticsearch service and query all logs (it’s very minimal at this point, just a “select *” kind of query would help), or other debugging steps I can take?

[gsmith]

Hello @neekfenwick

In layman’s terms what is the issue of this Graylog server? Can I ask what do you want to achieve?

From what was posted, it seams to work. I see you using GELF input, is this correct?

EDIT:

@neekfenwick

I’ve read that the log data is stored in elasticsearch, but I don’t have personal knowledge of this software.

You are correct, Graylog is the frontend, MongoDb keeps all the metadata and the heavy lifter is Elasticsearch.

Elasticsearch Guide [7.10] | Elastic

[neekfenwick (Nick Fenwick)]

We run a medium size online shop based on PHP, currently all logging is very simple stream to file, just a single source of logging. I’m looking at switching to a more structured logging approach using something like Graylog.

I’m using GELF input because this seems the most sensible thing to use from PHP, as in the example code I posted.

I’ve had another look at elasticsearch from the docs you linked to, it seems you can list indices with:

$ curl -XGET 'http://127.0.0.1:9200/_cat/indices'
green open gl-failures_1      J5KW3AOJQvi639-VGXdv4g 2 0 0 0  416b  416b
green open gl-failures_0      SHDB79mZS-u_n-lNgrmb3A 2 0 0 0  416b  416b
green open gl-events_0        fo7LN-Z1SqKu2QlL-9JlAA 4 0 0 0  832b  832b
green open graylog_0          sIPT4N-_RW-2rdLqyQTs8Q 4 0 1 0 8.5kb 8.5kb
green open gl-system-events_0 9-32g86CTz-Y5GhQFLGZcg 4 0 0 0  832b  832b

Then query all documents in the graylog_0 index:

$ curl -XGET 'http://127.0.0.1:9200/graylog_0/_search'
{"took":9,"timed_out":false,"_shards":{"total":4,"successful":4,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":"graylog_0","_type":"_doc","_id":"52d88070-c080-11ec-a59c-2cf05d743625","_score":1.0,"_source":{"gl2_accounted_message_size":118,"gl2_remote_ip":"127.0.0.1","foo":"bar","gl2_remote_port":52140,"streams":["000000000000000000000001"],"gl2_message_id":"01G12ZYAFTNSNTPBN2PAP9KYQ2","source":"example.org","message":"Hello there","gl2_source_input":"625fb43902042964a525386b","gl2_source_node":"c7808c29-a5d8-4e90-9dd7-591a5c1a954e","facility":"test","timestamp":"2022-04-20 08:03:11.957"}}]}}

Looks like only my ‘Hello there’ message created via Curl exists, the ones being created via the GELF PHP publisher are not, despite the PHP code not receiving any errors, it uses fwrite to write the message to a socket just fine.

There aren’t any other indexes in elasticsearch that look like they accidentally received my messages, all the gl-failures_0 and gl-system-events_0 look pretty normal and not something I should mess with.

One thing I don’t get is why on the Curl command line I specify the ‘graylog’ name in “http://127.0.0.1:12201/graylog” yet in the PHP Transport code only the IP and Port are specified, which makes me think perhaps the GELF messages are being sent to the wrong destination either in graylog or elasticsearch. I will keep googling how to use that library, maybe find some parameters I should be using.

[gsmith]

Hello,

Oh I see now. Thanks for the extra info.

Need to ask a couple questions.
These logs local? Meaning are on the same server? Or is this a remote device send logs?

What kind of log shipper are you using , Nxlog, FileBeat, Graylog Sidecar (is a wrapper for Winlogbeat, FileBeat , nxlog, etc…), Rsyslog, etc…?

I’m not sure about PHP but I do know is the logs have to be compatible with GELF input. For example, some of my Switch logs can not be ingest through GELF input. Just doesn’t happen, But Raw/plaintext input works great.

This link might help is you haven’t seen it before.

• Sending in log data

[neekfenwick (Nick Fenwick)]

@gsmith sorry, forgot to say, I’m using Monolog, it provides a Monolog\Handler\GelfHandler class, which you use with the GitHub - bzikarsky/gelf-php: A php implementation to send log-files to a gelf compatible backend like graylog library (known as graylog2/gelf-php), that’s the Gelf\Message stuff you saw in my code. Monolog has worked fine straight to file, and as I say I can step through the code when using the Gelf\Publisher stuff all the way down to the fwrite to a socket.

The graylog2/gelf-php library has a stable release and seems up to date (it replaces a deprecated older version) so it all looks like it should ‘just work’.

For this test install, I’ve just installed everything on my dev machine and everything’s using 127.0.0.1 (even in production it’s probably all going to be on the same machine as the web server).

In my graylog Inputs page, I see no Global inputs, and 1 Local input that I’ve called dev.premierrange (my local hostname for dev work) of type GELF HTTP. Maybe I need to specify this dev.premierrange identifier when setting up the Gelf\Publisher PHP object, but it doesn’t seem to take any such parameter.

[neekfenwick (Nick Fenwick)]

Taking Monolog out of the equation, the gelf-php library has a simple example gelf-php/simple.php at master · bzikarsky/gelf-php · GitHub … this seems to behave exactly like my earlier attempt, it goes through Gelf\Transport\UdpTransport into Gelf\Transport\StreamSocketClient and writes to a socket it has opened to graylog, yet no logs actually appear in graylog (or elasticsearch via direct query).

[gsmith]

@neekfenwick

I’m using Monolog, it provides a Monolog\Handler\GelfHandler class,

Ok, I haven’t used that before, to be honest this is the first time I’ve heard of that software.

@neekfenwick

In my graylog Inputs page, I see no Global inputs,

You need to create that as shown below for a global Input. There are two GELF Inputs TCP and UDP that can be used.

Here is an example of my Lab environment using GELF TCP/TLS. This us used by All my Windows servers, I Create one for Linux, Switches, firewall etc… It helps in the long run.

@neekfenwick

through Gelf\Transport\UdpTransport into Gelf\Transport\StreamSocketClient and writes to a socket it has opened to graylog, yet no logs actually appear in graylog (or elasticsearch via direct query).

You created a Input GELF UDP? I assume you used a tcpdump on this server?

[neekfenwick (Nick Fenwick)]

Aha! You are right, it seems the ‘GELF HTTP’ input I had created was not the right thing to do. Thank you for the help @gsmith !

I created a new ‘GELF UDP’ input on port 12201 and the simple Gelf\Logger message did then appear. Then I created a ‘GELF TCP’ input on port 12203 and updated my TcpTransport config to use 12203 and that, also, now appears in graylog.

In hindsight, I should have realised that a ‘GELF HTTP’ input wasn’t going to work, but I assumed that it would listen on the port 12201 that I’d seen in the docs, and since the socket connection did appear to succeed I was left rather confused as to what was going on.

So now I’ve proven TCP and UDP connections work, which is the right one when doing simple text logging from a web application backend (PHP in apache)? I’m guessing UDP because my limited knowledge of networking tells me that it’s unidirectional and so presumably has less overhead than TCP.

[gsmith]

@neekfenwick

Aha! You are right, it seems the ‘GELF HTTP’ input I had created was not the right thing to do. Thank you for the help @gsmith !

[tmacgbay]

@neekfenwick

I’m guessing UDP because my limited knowledge of networking tells me that it’s unidirectional and so presumably has less overhead than TCP

UDP sends the packet and doesn’t care if the transmission fails, TCP cares but uses slightly more overhead to care about it.