WARN [ProxiedResource] Unable to call and unable to start inputs

[pguillermet (Pablo Guillermet)]

Hi All

I installed a graylog server 4 on ubuntu 18 with elasticsearch-oss and nginx. The server is configured with https and a self signed certificate.

I’m having problem with the server inputs, I can create the input with the port 1514 but the service didn’t start, in the server.log file I have the following messages:

WARN  [UdpTransport] Failed to start channel for input SyslogUDPInput{title=WindowsEventsUDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null}
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
[InputLauncher] The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <602d760bb807c94ae169ab53> misfired. Reason: bind(..) failed: Permission denied.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
	at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
	at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
	at org.graylog2.inputs.transports.UdpTransport.launch(UdpTransport.java:135) ~[graylog.jar:?]
	at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
	... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Permission denied
2021-02-17T15:01:15.642-05:00 INFO  [InputStateListener] Input [Syslog UDP/602d760bb807c94ae169ab53] is now FAILED

and the other error is the message below

 WARN  [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:
    certificate: sha256//uj3j7KGQ9g0CLpT+ICOYNSi8lgYD505sBkquLTuAGw=

Thanks

[gsmith]

@pguillermet

Looks like permission issue accessing certs.
Did you allow Graylog to access the keystore and set the permiisionon the certs for Graylog to access those also?

How did you create your certificates?

@pguillermet

io.netty.channel.unix.Errors$NativeIoException: bind(…) failed: Permission denied
[InputLauncher] The [org.graylog2.inputs.syslog.udp.SyslogUDPInput] input with ID <602d760bb807c94ae169ab53> misfired. Reason: bind(…) failed: Permission denied.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(…) failed: Permission denied

[pguillermet (Pablo Guillermet)]

Hi, thanks for your reply.

I created the certificates with the next command
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/certs/graylog.key -out /etc/ssl/certs/graylog.crt

and I changed to 777 with chmod command, then I add the certificate to java keystore.

sudo cp -a “/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts” /etc/graylog/server/cacerts.jks

sudo keytool -importcert -keystore /etc/graylog/server/cacerts.jks -storepass changeit -alias graylog-self-signed -file /etc/ssl/certs/graylog.crt

sudo keytool -keystore /etc/graylog/server/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1

I have no idea which permission could be, the port that I used to create the input is 1514, is out of the private range.

[dickinsonzach]

I think they are suggesting the graylog user should have permissions the cert files. That is the user that runs the Graylog “stuff”.

[pguillermet (Pablo Guillermet)]

The certificates have the following permissions:

lrwxrwxrwx 1 root root 27 Jan 19 22:59 cacerts.jks → /etc/ssl/certs/java/cacerts
-rwxrwxrwx 1 root root 1541 Feb 10 13:14 graylog.crt*
-rwxrwxrwx 1 root root 1704 Feb 10 13:13 graylog.key*

the user must be owner of a directory or specific files that I missing?

[dickinsonzach]

I believe permissions on cert files themselves is the only thing suggested. I haven’t done this personally, just relaying a suggestion I have seen on the forums. FYI

[pguillermet (Pablo Guillermet)]

do you mean change the owner to graylog ubuntu user?

[pguillermet (Pablo Guillermet)]

I created a new input and didn’t start, here is the log error

2021-02-19T10:12:06.060-05:00 INFO [InputStateListener] Input [Syslog TCP/602fd545a4bcb73e26ebd797] is now STARTING
2021-02-19T10:12:06.129-05:00 INFO [InputStateListener] Input [Syslog TCP/602fd545a4bcb73e26ebd797] is now RUNNING
2021-02-19T10:12:06.137-05:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=InputTCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=0f4d5681-400c-44b8-b890-5fcac066222f} (channel [id: 0xdd930caf, L:/0:0:0:0:0:0:0:0%0:1514]) should be 1048576 but is 425984.
2021-02-19T10:12:07.075-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:07.282-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:07.285-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates/602fd545a4bcb73e26ebd797 on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:07.547-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:08.474-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:09.139-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:10.471-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:11.139-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:13.206-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:13.207-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

2021-02-19T10:12:14.050-05:00 INFO [InputStateListener] Input [Syslog TCP/602fd545a4bcb73e26ebd797] is now STOPPING
2021-02-19T10:12:14.056-05:00 INFO [InputStateListener] Input [Syslog TCP/602fd545a4bcb73e26ebd797] is now STOPPED
2021-02-19T10:12:14.057-05:00 INFO [InputStateListener] Input [Syslog TCP/602fd545a4bcb73e26ebd797] is now TERMINATED
2021-02-19T10:12:14.821-05:00 WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:

[dickinsonzach]

I really wish there was a definitive guide on how to get TLS/SSL going.

I am still struggling to get it up and running myself.

Is 192.168.0.2 the IP of your Graylog Server? I’m assuming you have the cluster all on the same server.

The “not verified” part made me think of “Adding a self-signed certificate to the JVM trust store” from here:

https://docs.graylog.org/en/4.0/pages/configuration/https.html

[pguillermet (Pablo Guillermet)]

Yes I did it, the only step that I miss is the next, beacuse I didn’t found were to do it:

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks . If you’ve been using another password to encrypt the JVM trust store than the default changeit , you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret .

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore and (optionally) javax.net.ssl.trustStorePassword system properties.

[pguillermet (Pablo Guillermet)]

I just added the line in the script /etc/init.d/graylog-server on line 47

DAEMON_ARGS="$GRAYLOG_SERVER_JAVA_OPTS DAEMON_LOG_OPTION -Dgraylog2.installation_source={GRAYLOG_INSTALLATION_SOURCE:=unknown} -Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks -Djavax.net.ssl.trustStoreType=jks -Djava.library.path=/usr/share/graylog-server/lib/sigar -jar $JAR_FILE server -p $PIDFILE -f /etc/graylog/server/server.conf $GRAYLOG_SERVER_ARGS"

But still having the message “WARN [ProxiedResource] Unable to call”

[gsmith]

@pguillermet
I’m sorry to hear you still having problems. This did take some time for me to understand how certificates work with graylog-server. One thing was clear, the certs need to be in the right format. Graylog needs to be able to access certs for the input (i.e. permissions). Graylog needs to be able to access the keystore. I can show you what i did but I also know everyone environment is a little bit different, so you might need configure these steps for your needs. The following steps is for CentOS 7 which has Elasticsearch, MongDb, and Graylog all on one Virtual machine. This helped me understand how and what to configure before I went into production envirment. An older problem I had with this was my DNS entry for my server. I had to make sure the Reverse lookup was configured.

1. Navigate to the following directory as follow;
   NOTE: I looked for my JAVA keystore called CACERTS.

• cd /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64/jre/lib/security

2. change the default password for Java CAcert store. Default password is changeit

• keytool -storepasswd -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64/jre/lib/security/cacerts

• Type “changeit”

• New-PASS Type “secret” /// this will be configured in graylog configuration file

3. Generate Certs for Graylog.

• keytool -genkey -alias dns.name.of.server -keyalg RSA -validity 365 -keystore keystore.jks
• openssl req -x509 -days 365 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem
• openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
• openssl pkcs8 -in pkcs5-plain.pem -topk8 -v2 des3 -out pkcs8-encrypted.pem -passout pass: secret
• keytool -list -v -keystore keystore.jks -alias dns.name.of.server
• keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12
• openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem
• openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem
• openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

4. Openjdk use this line. The password it requests is the password you changed in step 2.

• keytool -import -trustcacerts -file graylog-certificate.pem -alias dns.name.of.server -keystore /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64/jre/lib/security/cacerts

5. Move certificates to graylog folder

• NOTE: When I first started I found out by moving the files to Graylog Directory was easy for me to get the Inputs started and HTTPS enable for Graylog. Since Graylog owns this directory it was made simple. I have grown from this into something better but I found it to be a good learning experience to start with.

• move all *.pem, *.p12 and *.jks files to /etc/graylog/
  and chown graylog:graylog -R /etc/graylog
  so that the graylog user has access to them
  
  

6. Edit /etc/graylog/server/server.conf

• http_enable_tls = true
• http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
• http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
• http_tls_key_password = secret

7. Restart graylog and tail -f server.log

• systemctl restart graylog-server
• tail -f /var/log/graylog/server/server.log

My input configuration.

I hope this helps.

The following link/s help me understand how to make this happen.

GitHub - john-babio/graylog

Graylog3 with https (easy tutorial)

My ambient test: -Oracle Linux 7.6 (VMware 15); -Network: NAT; -Graylog3, MongoDB and Elastic: running together; Documentation base for RPM distro. First of all, read Docs » Configuring Graylog » Using HTTPS 0- mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/ 1- openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem 2- openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem 3- openssl pkcs8 -in pk…

https://docs.graylog.org/en/4.0/pages/configuration/https.html

[dickinsonzach]

Do I understand that the steps you outlined enabled you to create a TLS input? Did you also enable SSL on the web interface? Thank you, Zach.

[pguillermet (Pablo Guillermet)]

Yes, I will try creating new certificates

[pguillermet (Pablo Guillermet)]

HI gsmith

I tried but I’m having the same error:

WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=Bla, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=0f4d5681-400c-44b8-b890-5fcac066222f} (channel [id: 0x8cf27397, L:/0:0:0:0:0:0:0:0%0:51412]) should be 1048576 but is 425984.
INFO [InputStateListener] Input [Syslog TCP/6034053c8ec95d280ba22fb5] is now RUNNING
WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/metrics/multiple on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:
certificate: sha256/fj02ozJJViJ2Qz7Zjeel+FmYfyLvI9ljY3w2R6OvJAk=
WARN [ProxiedResource] Unable to call https://192.168.0.2:9000/api/system/inputstates/6034053c8ec95d280ba22fb5 on node <0f4d5681-400c-44b8-b890-5fcac066222f>: Hostname 192.168.0.2 not verified:
certificate: sha256/fj02ozJJViJ2Qz7Zjeel+FmYfyLvI9ljY3w2R6OvJAk=

Thanks

[gsmith]

Yes and yes.
You can see the input I use from above.

The certs graylog-certificate.pem and graylog-key.pem I used for my Input, and graylog-certificate.pem is also used for HTTPS, I have changed things around since then, but for learning this was my first step.

I do have notes stating when I created my certs to use the Fully Quailified Domain Name (FDQN), Make sure my host name matched my certifcates, also my alias was the same name as my FDQN when I inserted my certificates into the keystore.

[gsmith]

@pguillermet

Can you share your Server configuration file?
Since your using IP Address I take it you dont have a domain Controller in your environment?
Have you tried configuring your /etc/hosts file? Just a thought.

Here is one of my post while back, hope this will be able to help you figure out your problem.
My first step was just getting HTTPS enabled , then I moved to a INPUT using TCP/SSL.
Seams like your machine cant find your cert with IP Address (192.168.0.2). Question when you executed the steps I showed you from above, did you reconfigure this?

@pguillermet

Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts.jks

Graylog2 Using https

Hello All, I am having a problem with https on graylog2.2.3, Currently using CentOS-7.3 all in one graylog2 server. This is a Virtual machine with 8 cores, 10 GB of ram and 1 Tb storage. I created an Input for Localhost and tested to received messages, completed. No problems. I attempted to use the following: http://docs.graylog.org/en/2.2/pages/configuration/https.html Unfortunately, toward the end of the page I was having problems with JVM to pick up the new trust store. So, I attempted t…

@pguillermet @dickinsonzach
Have you seen this.

SSL Certs - Lesson Learned

After 3 days of fighting with certs to get my instance back up and running I’d like to share my thoughts about the process and what major suggestion I would add to the documentation. First of all the suggestion. COMMON NAME: need to be more specific that the common name needs to be EXACTLY what you set in the rest_uri and web_uri otherwise you’ll get java errors like “javax.net.ssl.SSLPeerUnverifiedException: Hostname not verified:” or other similarly cryptic messages. I would have been done 3…

[pguillermet (Pablo Guillermet)]

@SSL Certs - Lesson Learned

COMMON NAM

Hi, where I have to set the COMMON NAME?

[gsmith]

@pguillermet
Example:

[root@graylog bin]# openssl req -x509 -days 365 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem
Generating a 2048 bit RSA private key

writing new private key to ‘pkcs5-plain.pem’

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.

Country Name (2 letter code) [XX]:us
State or Province Name (full name) :NewYork
Locality Name (eg, city) [Default City]:newyork
Organization Name (eg, company) [Default Company Ltd]:comp
Organizational Unit Name (eg, section) :admin
Common Name (eg, your name or your server’s hostname) []:server.domain.com

[pguillermet (Pablo Guillermet)]

@SSL Certs - Lesson Learned

COMMON NAME: need to

Thanks, yes the common name it’s ok