Securing inputs

[jimbrew (James)]

Hello I am using Graylog v4.2, which is running via docker on an Azure VM. I have setup an Nginx reverse proxy for HTTPS.

However how do I go about securing the inputs because anyone could log to this instance of Graylog all they need is the URL/IP address of the server and a port number.

What I want to achieve is only allow my web application to log to Graylog. I tired following this, but I am bit unclear as I don’t think I need to follow the first part. I thought I could just use my web application server certificate, store it in the Graylog trusted certs and would validate the web app certificate against what is in that folder. I keep getting the error NotSslRecordException: not an SSL/TLS record.

I have tried a .pem file and .crt file type

Here is my TCP input.

bind_address: 0.0.0.0
decompress_size_limit: 8388608
max_message_size: 2097152
number_worker_threads: 8
port: 12201
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: required
tls_client_auth_cert_file: /etc/graylog-trusted/
tls_enable: true
tls_key_file: <empty>
tls_key_password:********
use_null_delimiter: true

[jimbrew (James)]

So I wasn’t using the fullchain.pem. But now I am having another issue.

30 11:19:40,504 WARN : io.netty.channel.ChannelInitializer - Failed to initialize a channel. Closing: [id: 0xda67eb83, L:/172.24.0.4:12201 - R:/172.24.0.1:43656]

java.security.cert.CertificateParsingException: signed fields invalid

I updated my docker compose file to copy the certificate and add it to the trust store.

FROM graylog/graylog:4.2

COPY inputs-secure.json /usr/share/graylog/data/contentpacks/inputs.json
ENV GRAYLOG_CONTENT_PACKS_AUTO_INSTALL inputs.json
ENV GRAYLOG_CONTENT_PACKS_LOADER_ENABLED true
ENV GRAYLOG_CONTENT_PACKS_DIR data/contentpacks

USER 0
RUN mkdir /etc/graylog-trusted/
RUN chown -R graylog:graylog /etc/graylog-trusted/
COPY alto-cert.pem /etc/graylog-trusted/alto-cert.pem

RUN openssl x509 -in /etc/graylog-trusted/alto-cert.pem -out /etc/graylog-trusted/alto-cert.der -outform DER

# add the CA certificate to truststore
RUN cp /etc/graylog-trusted/alto-cert.pem /usr/local/share/ca-certificates/alto-cert.pem && update-ca-certificates
# test & import the .der file
RUN keytool -v -printcert -file /etc/graylog-trusted/alto-cert.der
RUN keytool -import -trustcacerts -file /etc/graylog-trusted/alto-cert.der -alias company.com -keystore /usr/local/openjdk-8/lib/security/cacerts -storepass changeit -noprompt

RUN sed -i "s/export GRAYLOG_SERVER_JAVA_OPTS='-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow'/export GRAYLOG_SERVER_JAVA_OPTS='-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=\/usr\/local\/openjdk-8\/lib\/security\/cacerts'/" "/etc/profile.d/graylog.sh"

USER graylog

It is not clear whether this issue is to do with the a certificate mismatch between the web application and the certificate I have trusted. Perhaps I have missed a step could anyone provide support?

[gsmith]

Hello @jimbrew

I’ll be honest when I first did this it was difficult to understand, perhaps I could shed some light on this.
I believe this is a combo of configuration and perhaps the certificates.
If your trying to use HTTPS on the Web UI then you would need to follow something like this.

• Using HTTPS

If you get that to work you can also use the two certificate made from that documentation for your INPUTS.

The working directory should now contain the PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) to be used with Graylog.

If your unfamiliar with how certificates and keystore work, I would suggest placing those two certs in your Graylog Directory since Graylog owns its directory, as for Docker I believe Graylogs PID is 1100.

Here is an example of HTTPS config

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Input used.

So to sum it up…
I created certificates for my Graylog instance using HTTPS , then I used those two certs for my input as shown above. This is only a demonstration of what could be done. In my Prod I would use different certificates for my inputs. This is definitely a learning experience.

Here is some add info, not sure if you have seen these posts.

Graylog3 with https (easy tutorial)

My ambient test: -Oracle Linux 7.6 (VMware 15); -Network: NAT; -Graylog3, MongoDB and Elastic: running together; Documentation base for RPM distro. First of all, read Docs » Configuring Graylog » Using HTTPS 0- mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/ 1- openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem 2- openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem 3- openssl pkcs8 -in pk…

Graylog 4.2.7 + docker compose + https = semi fail

1. Describe your incident: I am trying to migrate a working non-https graylog solution to an https one. I am able to connect to the web interface, but as soon as I do, the docker logs are populated in loop with the following lines: some-graylog | 2022-03-24 10:29:56,447 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://some-graylog:9000/api/system/metrics/multiple on node <795270ca-9ef2-43de-a800-063115f3a018>: PKIX path building failed: sun.security.pro…

Hope that helps

[jimbrew (James)]

@gsmith thanks for your feedback and suggestions.

I have tried most of that and got that HTTPs working, but what I am struggling with is thew concept of securing the inputs. With TLS enabled does this limit the hosts that can send messages to my graylog web server?

From what I gather anyone who has the IP address of the server and port number could easily send messages to the graylog server.

The beats stuff seems a bit overkill and overly complicated for my needs. My goal is to create a secure TCP input that authenticates itself against the client who is sending the messages.

[gsmith]

Hello

@jimbrew

but what I am struggling with is thew concept of securing the inputs.With TLS enabled does this limit the hosts that can send messages to my graylog web server?

No it does not, I have 200 nodes on one INPUT using TCP/TLS with the same configuration as shown above.

@jimbrew

From what I gather anyone who has the IP address of the server and port number could easily send messages to the graylog server.

Security Considerations

1. SELinux will be enforcing security policies
2. IPtables/Firewalld will provide firewall functionality
3. Inputs using TCP/TLS will be configured for added security
4. SSL certificates be used and HTTP will redirect to HTTPS
5. FTP is not allowed, and all file transfers must be encrypted during transport (SSL / SSH file transfer)
6. SFTP users chrooted (jailed) to their own directory.
7. SSH key based authentication (disable root access via SSH)
8. Using Active Directory
9. Create GPO’s for authentication failure/access.

Example_IPtables

[root@graylog server]# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN state NEW
DROP       tcp  --  anywhere             anywhere             tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG
           tcp  --  anywhere             anywhere             tcp dpt:9922 state NEW recent: SET name: SSH side: source mask: 255.255.255.255
DROP       tcp  --  anywhere             anywhere             tcp dpt:9922 state NEW recent: UPDATE seconds: 300 hit_count: 4 TTL-Match name: SSH side: source mask: 255.255.255.255
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:9922 state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:ntp state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:http state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:https state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-agent state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:zabbix-trapper state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51430 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51412 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51420 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51466 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51440 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:51411 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:27017 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:27018 state NEW,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:smtp state ESTABLISHED
LOGINPUT   all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:cslistener state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:51430 state NEW,ESTABLISHED
ACCEPT     tcp  --  graylog.domain.com  anywhere             tcp dpts:vrace:sec-t4net-srv
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:cslistener
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:50440
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:50440 state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:iop state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:iop state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:wbem-rmi state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:iop state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:51415 state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:lxi-evntsvc state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:lxi-evntsvc state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:51420 state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:ipfix state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:lxi-evntsvc state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:lxi-evntsvc state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:51415 state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:lxi-evntsvc state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:ipfix state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:hbci state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:xmltec-xmlmail state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:9833 state NEW,ESTABLISHEDA

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:9922 state ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:ntp state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:http state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:https state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http state NEW,RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https state NEW,RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:zabbix-agent state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:zabbix-trapper state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51430 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51412 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51420 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51466 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51440 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:51411 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:smtp state NEW,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:27017 state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:27018 state ESTABLISHED
LOGOUTPUT  all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:cslistener state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ldap state NEW,RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:ldap state NEW,RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply
ACCEPT     all  --  10.200.6.111         anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:50440 state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp dpt:iop state NEW,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             udp spt:http state ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:wbem-rmi state NEW,ESTABLISHED

Chain LOGINPUT (1 references)
target     prot opt source               destination
LOG        all  --  anywhere             anywhere             limit: avg 4/min burst 5 LOG level warning prefix "DROP INPUT: "

Chain LOGOUTPUT (1 references)
target     prot opt source               destination
LOG        all  --  anywhere             anywhere             limit: avg 4/min burst 5 LOG level warning prefix "DROP OUTPUT: "
[root@graylog server]#

@jimbrew

My goal is to create a secure TCP input that authenticates itself against the client who is sending the messages.

As shown earlier, even if someone has IP address or the port number they can send messages to this Address and port BUTTTTT it will not go through, They would need the certificates created on your Graylog server to get access , hence TCP/TLS,SSL

Hope that helps

[jimbrew (James)]

@gsmith Yes this helps. Thank so much for your suggestions and detailed explanations.

Would be nice if an input just had some API key against that has to be passed in along with the URL and port number. Make things a bit simpler.

Anyway I digress, your help is much appreciated. Will give that a bash!

[gsmith]

Hello @jimbrew

I’m glad it worked out for ya

You might want to look into Graylog Sidecar also.

• Introduction to Graylog Sidecar | Graylog

I grew to like this option specially when you have a large environment.