Graylog SSL Certificates

Graylog Central (peer support)
1

I am struggling to set up HTTPS for my graylog instance. Here are some basic infos about my setup and what I have done so far:

  • OS:Ubuntu 18.04.1 LTS
  • Version: 2.5.0+ed06ce7, codename Trippy Trampoline
  • JVM: PID 9950, Oracle Corporation 1.8.0_191 on Linux 4.15.0-42-generic

Graylog is running fine with the default configuration using http over port 9000.
I now want to secure the connection with our companys own CA.
I’ve followed the section Converting a PKCS #12 (PFX) file to private key and certificate pair from here:

PKCS #12 key stores (PFX files) are commonly used on Microsoft Windows.

In this example, the PKCS #12 (PFX) file is named keystore.pfx :

$ openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem $ openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem $ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

The resulting graylog-certificate.pem and graylog-key.pem can be used in the Graylog configuration file.

For demonstrating purposes I’ve generated a new CA and issued a certificate for graylog so that I can post these here. When asked for a password I’ve always used secret. These are the two certificates created by the above commands:

Key:

-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQIagKg9ARhyhwCAggA
MAwGCCqGSIb3DQIJBQAwHQYJYIZIAWUDBAEqBBD7zmDKWPuI7auCjriheK/RBIIE
0LB5HEivJLmCHmwy9pGafbzTBXPJ/BBhDwKRA/YaSHk7NCT1kED87HEa1LX8y/pX
cOcqRyrVfmO3qywHxQYik3/qDq4lvF1c26nrGOk29fKG5qL9D5/huZOOqpDFXX1Y
hFS9hA7QuWgZE9275Nhsb2y+7L9HSSRnnD3xBFU2c+WAFRX3DnP9emt12ZJCobGY
SdCVmaRk6WwOo3af586lvlL233jeyZMSQXZEu4bXa3xvZWWR0lhlVvq7VPTX2Ycb
LLveFhmRCN8mgtHXTGv9ljKr1Mx0oP4NqbnX6524RKH/6tnXGr8jDveRGBFfhJeH
00NIIO4g2L7odtFcF0Ote+JzaDDtqJLNFVB/aNoARrJ+t7AOP3OFptljfhsndKHi
W459UUQeEfjgTyRrOwyNEsxsGUJfCxtP7StCO/aM2kNSE4XEUsESXNZla8HndBwT
MkIsIomKZasbuLOQPNwPoBCtQlkVPH2LGyb8mSILPiHkhXxOw7mTQXlJUvX7Q4E7
HPtqJT4+C+6SW4DWRWIeb1UPjXDHaR3P70pME0dh2gvsaBjoBoVBNc0YeENXsqx9
QljB3u+xonuPX1vEauxxB79WD2RQbN2ezWyNleUNxqa7fx2KXdyyybEjcAI6Zpp/
GgUccvrF8NYmuH2I6C/2fC8EFx+JO0Oiw4ENzKAnZvZpcDsEH73LlQ7xrI4jRK83
ljD5m+PYcqZsgYnYJblr9U6ZUUK1/fCW/PLyVn+AQbIy5V+jOmJoddMzVYbM4N8/
A3Zscl6Soz1Gn6PGgGzefnGivqYMbDHmxKk9rIShJ8zMcHfn4R4TwTKkZfI6vth7
SfZgWezkYCN/dqntnfJdLs/zoEsKHPOyyLn+vsEVkY+ZyP3t/a5sNUk5GoTeNCK/
g4Ah60472Zt7HE3c8lrXrWvf0F6cDpTcXrSLb6N8xSbo81DMTVLI8S/kf0/RVKS/
Mnvh2Ng3uwpg4i9v5bBmMi02XDZ7WQKen/GR1xAWkAB+arcRk7gaPJpbtXpjnbq3
ccZ0vzEh+NA/Sz2jGROvoJHuXn/pgn93gy4Ed0e+uIYpjejj8datDkMHCtMdQcuG
cS1m2X60m+X636iPLTY7/zAdmAQSbOj7n1v1+jzjFGSqXnEWG+uoYcnlydhZLiIS
DCbPA31HSQEaMJtIYlhVoyxGkxzsEr6I+QOGl5h/cCQ0JT6DBl/qljtR/GtIC2Ym
F6wQ6csq8IyBiqVmU53C0u99FZs5VutcvtTVzvARFI7d92Z7ISXMYRceko3aXiha
sGeZYX3rKDNLOleKAkkIoR/FrcY/J+7u36rTUivLOD492s8TXc/MgWWTD2hxSer5
1iS8Bwtdw8+4j08VaYDEI+sw9+E83l5G2IByk3HGYviaRWMAKXF8twXIv0BhL7Fq
QHu6Wrhes3HvPdfE7AZz9MxpTEoF2GVqUpETfATsyCmZRYc4/M/wqYLyaaYAcNUJ
CWLQJegz7UnRMi4QZY9HwIWJxjAc1FVXKr5kQsSsOzHRc268FEv152q21p1KI/+1
klAanFyTOxuQwn/UbAVkf4sduPg54K4pqL/0tAbg2+8ljpqiZu9sGbJd7Hmo269M
z1vuS/lmMwcPfOnRkCauiWpZ8HVK/2LzXKwNYnjTVw/7
-----END ENCRYPTED PRIVATE KEY-----

Certificate:

Bag Attributes
localKeyID: CA E8 A6 0B 80 02 11 5C EC CB D9 95 5A BC EA 07 72 52 2F C7
friendlyName: graylog.domain.local
subject=/C=DE/ST=Baden-W\xC3\xBCrttemberg/L=DevTown/O=IT/OU=IT/CN=graylog.domain.local/emailAddress=no@response.me
issuer=/C=DE/ST=Baden-W\xC3\xBCrttemberg/L=DevTown/O=IT/OU=IT/CN=Development CA/emailAddress=no@response.me
-----BEGIN CERTIFICATE-----
MIIEVjCCAz6gAwIBAgIIWLiD/b9yv4gwDQYJKoZIhvcNAQELBQAwgY4xCzAJBgNV
BAYTAkRFMRswGQYDVQQIDBJCYWRlbi1Xw7xydHRlbWJlcmcxEDAOBgNVBAcTB0Rl
dlRvd24xCzAJBgNVBAoTAklUMQswCQYDVQQLEwJJVDEXMBUGA1UEAxMORGV2ZWxv
cG1lbnQgQ0ExHTAbBgkqhkiG9w0BCQEWDm5vQHJlc3BvbnNlLm1lMB4XDTE4MTIx
OTAwMDAwMFoXDTE5MTIxODIzNTk1OVowgZQxCzAJBgNVBAYTAkRFMRswGQYDVQQI
DBJCYWRlbi1Xw7xydHRlbWJlcmcxEDAOBgNVBAcTB0RldlRvd24xCzAJBgNVBAoT
AklUMQswCQYDVQQLEwJJVDEdMBsGA1UEAxMUZ3JheWxvZy5kb21haW4ubG9jYWwx
HTAbBgkqhkiG9w0BCQEWDm5vQHJlc3BvbnNlLm1lMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAmeMCUIKROK/mcW8nZLTDHCSBVpeLFwY6kXkMbrGYvaC9
ZNS0IE4GjegLM8WUur9qtC9fA/P0lfTMtISb6QrRmKHvPJSUbCj0qh2R7a69/gOS
za5Vty1BQ+bOzVevNPq6e6QZ3Ig6yDwTidvhSRytdMobuTifIplcIqi8SczmqhIx
QtXAh1vloZZj3Aei8lwmlgDffS9oWWgMhPecSwpYhrOHHlCNquOHvQ/n3fkiNMS3
hwz3o5k2RKQF39T7FI8yHD7lAzEzG4nIkooJ5K+zFnG6i+3RSl4KC9SupsBVOgLi
bKAVs9xP9l1kX15791TJ7x0vtidszI8AlyZG+iSCPQIDAQABo4GvMIGsMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFGkKjM3OIIFc22TEwE8O8iwgzIa2MAsGA1UdDwQE
AwIF4DATBgNVHSUEDDAKBggrBgEFBQcDATAoBgNVHREEITAfghRncmF5bG9nLmRv
bWFpbi5sb2NhbIIHZ3JheWxvZzARBglghkgBhvhCAQEEBAMCBkAwHgYJYIZIAYb4
QgENBBEWD3hjYSBjZXJ0aWZpY2F0ZTANBgkqhkiG9w0BAQsFAAOCAQEAHECmquLo
b7wb2IrFhfb3M4AFlfbeUt/LXZrn3ESwiFpa1hAxTGBzeQVfAQnzHGCV/3iLQ3lp
P4paSEM2R8D+V67427RsCRPbeJKsAC3r0qFpdeX4RRzuUAHKhrxbGnDQ1ICVgT7V
2r785wisi4BHnOqHw7inmR/CDe7gfCHXop6PgXzuENC78dpVfh1feNXdxs3u6exC
LOSKSYCHoi/vYkd+yLslbRCYaeKBp4lIqmyF0wrToWr3foxw63TpLbX0CzEPRJj8
JVCdk5ApHu9f/pfiiyOd+1Cza5YUGNfAUqDC5g3H2MKwZQ9gGYCWHfcj4aF3z3T4
F4dFeVSwzIOxSA==
-----END CERTIFICATE-----
Bag Attributes:
subject=/C=DE/ST=Baden-W\xC3\xBCrttemberg/L=DevTown/O=IT/OU=IT/CN=Development CA/emailAddress=no@response.me
issuer=/C=DE/ST=Baden-W\xC3\xBCrttemberg/L=DevTown/O=IT/OU=IT/CN=Development CA/emailAddress=no@response.me
-----BEGIN CERTIFICATE-----
MIIEFDCCAvygAwIBAgIIOORznMskPbcwDQYJKoZIhvcNAQELBQAwgY4xCzAJBgNV
BAYTAkRFMRswGQYDVQQIDBJCYWRlbi1Xw7xydHRlbWJlcmcxEDAOBgNVBAcTB0Rl
dlRvd24xCzAJBgNVBAoTAklUMQswCQYDVQQLEwJJVDEXMBUGA1UEAxMORGV2ZWxv
cG1lbnQgQ0ExHTAbBgkqhkiG9w0BCQEWDm5vQHJlc3BvbnNlLm1lMCAXDTE4MTIx
OTAwMDAwMFoYDzIxMTgxMjE4MjM1OTU5WjCBjjELMAkGA1UEBhMCREUxGzAZBgNV
BAgMEkJhZGVuLVfDvHJ0dGVtYmVyZzEQMA4GA1UEBxMHRGV2VG93bjELMAkGA1UE
ChMCSVQxCzAJBgNVBAsTAklUMRcwFQYDVQQDEw5EZXZlbG9wbWVudCBDQTEdMBsG
CSqGSIb3DQEJARYObm9AcmVzcG9uc2UubWUwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDU641kGOtQZouasjWBLHs3tUJ0GGQesxI5LaoskPCgXBzpE2Uz
W4wmtp9l/XvH+63TXYxdWmDmqXnQqM2eQPHNiD0R51eBybWo6NCTC141utdOoiPO
XIG8zRFjuGsAyAMn8MBM+4aWoShSg0DUS8lpAD4Q7/qqDW5B1lYEHM85RyLCmYMS
Gq0YQ7qT0sJdjVbL7pytlUOAP6mc91098OCB0g25l6+crG2XiCYgQzu6DUdyHJnq
wLSg4vTdXuMJmJBG8Jf+a8TbSMdqCxBSc963TXuGwFBoAhzWuxURuNy8cUQI8nHg
RVvd5Z0PdH5ySruiV5UEa4quaGA7M95jAofNAgMBAAGjcjBwMA8GA1UdEwEB/wQF
MAMBAf8wHQYDVR0OBBYEFJZPXDApC/n5nLnR31QgmplYk5LYMAsGA1UdDwQEAwIB
BjARBglghkgBhvhCAQEEBAMCAAcwHgYJYIZIAYb4QgENBBEWD3hjYSBjZXJ0aWZp
Y2F0ZTANBgkqhkiG9w0BAQsFAAOCAQEAOHIJDPL0SJHJcv88hbBcWPE7k1wYUrYS
prLHxyPRHBza5ZznsoRJkt9l6O0MDMiYb/nN1sxFPLSQXgNRXbvV4slkPI9RXAFt
2I4EmoosZ5f6+ZCPDYYHfjlycx+M/xnR55xh9h+ihgAkJZQuNlcbmPG/BdaMJujM
p8n8GiudEOGAWmmvkPIzgvPqHJ9QVy5haZyPxF195UvS13W2YoFiOTm0z+wwDJPW
9izmx/sppP/C1s4KzwfBQC/VGJtCJJdXJu43Jauh5h5JjfhEO5bsAP+ctDxY1EeK
BXZjaN4L0FPAKon3ofonuw0N6gzXPdDe/g6JT//ZFPmTnlzHewyZEw==
-----END CERTIFICATE-----

I’ve set up my server.conf accordingly:

rest_listen_uri = https://0.0.0.0:443/api/
rest_enable_tls = true
rest_tls_cert_file = /etc/ssl/graylog-certificate.pem
rest_tls_key_file = /etc/ssl/graylog-key.pem
rest_tls_key_password = network21
web_listen_uri = https://0.0.0.0:443/
web_enable_tls = true
web_tls_cert_file = /etc/ssl/graylog-certificate.pem
web_tls_key_file = /etc/ssl/graylog-key.pem
web_tls_key_password = network21

Unfortunately it seems like graylog is unable to read these certificate files properly:

2018-12-19T13:57:54.703+01:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.io.IOException: ObjectIdentifier() – data isn’t an object ID (tag = 48)
at sun.security.util.ObjectIdentifier.(ObjectIdentifier.java:257) ~[?:1.8.0_191]
at sun.security.util.DerInputStream.getOID(DerInputStream.java:314) ~[?:1.8.0_191]
at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_191]
at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_191]
at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_191]
at sun.security.x509.AlgorithmId.(AlgorithmId.java:114) ~[?:1.8.0_191]
at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_191]
at javax.crypto.EncryptedPrivateKeyInfo.(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_191]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:69) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:384) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:207) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:141) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
2018-12-19T13:57:54.709+01:00 INFO [Server] SIGNAL received. Shutting down.

What am I missing here?
Certificates are not really my strong suit so any help is greatly appreciated!

2

DISREGARD THIS (I misread your original message):
You’re pointing to the same .PEM file for everything. Unfortunately Graylog wants the certificate in a .CRT PEM format certificate and the private key in a PKCS8 format .PEM file. In my experience they need to be two separate files.

BUT PAY ATTENTION TO THIS:

Also, make absolutely sure there are no leading or trailing white spaces or lines around the key and the certificate.

Also, if you’re generating the files on Windows, don’t forget to dos2unix the files after copying them to the Graylog box.

3

you might find this part of the documentation helpful: http://docs.graylog.org/en/2.5/pages/secure/sec_graylog_beats.html

4

After hours of trying a bunch of different methods I just gave up and set up an nginx proxy instead. This is the nginx site configuration if anyone cares:

server {
    listen      443 ssl;
    server_name graylog.domain.local;
    ssl_certificate /etc/ssl/graylog.domain.local.crt;
    ssl_certificate_key /etc/ssl/graylog.domain.local.pem;

    location /
    {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Graylog-Server-URL https://$server_name/api;
      proxy_pass       http://127.0.0.1:9000;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name graylog.domain.local;

    location / {
        return 301 https://$host$request_uri;
    }
}

Thanks anyways for the help guys!

1 Like
5

I’ve grabbed my own documentation and here’s what we do to make things work:

We generate the keypair and certificate on our CA. Input to this is a certificate request file that includes all wanted aliases and which has the flag to mark the key as exportable. The template in question “ClientServerExportablePrivate” ensures that the keypair can be used for both ServerAuth and ClientAuth and that the private key is exportable.

[Version]
signature="$Windows NT$"
[NewRequest]
Subject="CN=server314.corp.broehaha.nl,O=Broehaha"
HashAlgorithm=SHA256
KeyAlgorithm=RSA
KeyLength=2048
Exportable=True
MachineKeySet=True
[RequestAttributes]
CertificateTemplate="ClientServerExportablePrivate"
[Extensions]
2.5.29.17="{text}"
_continue_="dns=graylog.corp.broehaha.nl&"
_continue_="dns=graylog&"
_continue_="dns=server314.corp.broehaha.nl&"
_continue_="dns=server314&"
_continue_="dns=192.168.3.14&"
_continue_="ipaddress=192.168.3.14&"

This request is used to generate a keypair, after which the CSR is handled and the cert is signed. Now, with ADCS (the PKI I’m using) it’s a bit messy, but in the end I end up with two files: server314.cer and server314.pfx. The latter is a PFX exported package of certificate+keys (the Powershell command to get this is Export-PfxCertificate).

Converting the PFX and cert is then a matter of three OpenSSL commands:

openssl x509 -in .\server314-elastic.cer -outform pem -out .\server314-elastic.crt
openssl pkcs12 -in .\server314-elastic.pfx -nocerts -out .\server314-elastic.key
openssl pkcs8 -in .\server314-elastic.key -topk8 -out .\server314-elastic.pem

The resulting .PEM file needs to be edited with something like Notepad to take off any extraneous bits. There’s a bunch of metadata that gets added on, which must NOT be in the file. You must only have the BEGINKEY and ENDKEY and everything in-between.

The .PEM and .CRT files are then uploaded to the Linux box that runs Graylog. Over there, we do one more trick, which is dos2unix server314.crt; dos2unix server314.pem to make sure all the line endings are fine.

One last important step: we need to make sure that the Linux box trusts our CA’s certificate! Actually, you’ll need the whole chain, so both the root and any intermediaries… The process of making these trusted differs per distribution, on RHEL-derivatives it’s along these lines:

sudo cp /tmp/issuing-ca.crt /etc/pki/ca-trust/source/anchors/
sudo cp /tmp/root-ca.crt /etc/pki/ca-trust/source/anchors/
sudo cp /tmp/chain-ca.pem /etc/pki/ca-trust

cd /etc/pki/ca-trust
sudo chmod 644 /etc/pki/ca-trust/chain-ca.pem
sudo restorecon ./*

sudo update-ca-trust extract

The two individual certs are snagged by update-ca-trust to fix the system-wide trust. The “chain-ca.pem” file is a certificate chain of the two individual .CRTs, which can be used by Graylog as the trust store.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.