Cant get https up and running

Marco · February 13, 2019, 9:46am

After following the documentation for the setup of https to the letter,
and checking back with all of these:

https://community.graylog.org/t/graylog-ssl-certificates/8100
https://community.graylog.org/t/https-isnt-working-for-me/8154
https://community.graylog.org/t/jerseyservice-wont-start-after-enabling-https/2836
https://community.graylog.org/t/cant-get-https-working-on-graylog/1215
https://github.com/Graylog2/graylog2-server/issues/2752
https://github.com/Graylog2/graylog2-server/issues/2215

(sorry, too many links for a new user…)

I still can’t figure out how the get graylog to use https.

No matter what I do, I always end up with a failed JerseyService and graylog starting in a loop,
apparently due to:

2019-02-13T09:16:46.576+01:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
	Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
	Caused by: java.io.IOException: ObjectIdentifier() -- data isn't an object ID (tag = 48)
		at sun.security.util.ObjectIdentifier.(ObjectIdentifier.java:257) ~[?:1.8.0_171]
		at sun.security.util.DerInputStream.getOID(DerInputStream.java:314) ~[?:1.8.0_171]
		at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_171]
		at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_171]
		at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_171]
		at sun.security.x509.AlgorithmId.(AlgorithmId.java:114) ~[?:1.8.0_171]
		at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_171]
		at javax.crypto.EncryptedPrivateKeyInfo.(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_171]
		at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:69) ~[graylog.jar:?]
		at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:384) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:207) ~[graylog.jar:?]
		at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:141) ~[graylog.jar:?]
		at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
		at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
		at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171]

Without tls enabled, everything is running just fine.

The other posts seem to indicate a problem with the certificate, though I cannot figure out what could possibly the issue. I get my certificate from a WindowsCA.

Then I do…

openssl pkcs12 -in npgulmlog02.intern.swp.de.pfx -nokeys -out graylog-certificate.pem
openssl pkcs12 -in npgulmlog02.intern.swp.de.pfx -nocerts -out graylog.pkcs5.priv.key
openssl pkcs8 -in graylog.pkcs5.priv.key -topk8 -out graylog-private.pem

Now, there are multiple certs in the cert-file. A certificate chain.

As that didn’t work I tried

openssl x509 -in graylog-certificate.pem -outform pem -out graylog-x509.pem

as well. That didn’t work either.

I imported the certs into the java-keystore…

$keytool -importcert -keystore /etc/ssl/certs/java/cacerts -storepass changeit -alias graylog -file /etc/graylog/certs/graylog-certificate.pem

And reconfigured the hosts ca-certificates:

$ cp /etc/graylog/certs/graylog-certificate.pem /etc/ssl/certs/
$ cp /etc/graylog/certs/graylog-private.pem /etc/ssl/private/
$ dpkg-reconfigure -f noninteractive ca-certificates

But that didn’t change anything either.

Is there anywhere ELSE but the certificates I need to look at? Wouldn’t be the first time an error-message and google-searches are misleading

graylog was installed from .deb-packages.

Key	Value
OS	Debian 8
graylog-server	2.5.1-1
elasticsearch	5.6.14
mongodb	1:2.4.10-5+deb8u1
java	openjdk version “1.8.0_171”

Totally_Not_A_Robot · February 13, 2019, 10:52am

Hey

I don’t know off the top of my head where Debian stores the systemwide trust store. In CentOS/RHEL it’s /etc/ssl/ca-trust/anchors, after which you need to trigger an update of the actual keystores with a separate command.

I did not have to update any Java keystores with certs or keys; you won’t have to use keytool.

Now, I did not actually see you import the Root CA and Issuing CA certificate files. Those need to go into the systemwide trust store. It’s not the Graylog cert that needs to be trusted, only the issuers.

Aside from that, most everything you’ve done looks similar to my own setup. I assume that you’ve setup the file locations in your server configuration correctly and that you’ve also added the right passphrase

EDIT:
My full instructions can be found here →

… I say “full instructions”, but I’ve also skipped over the graylog server config file

Marco · February 13, 2019, 1:24pm

Thanks Tess!

At least I can scratch the Java Keystore from the list

The root-CA get’s deployed by puppet, so that should not be an issue.
But I’ve copied them manually to /usr/local/share/ca-certificates/ and ran dpkg-reconfigure again just to be sure.

I’ve cleaned the pem-files as you described in your full instructions as well. But no luck.

I have the whole certificate-chain in my graylog-certificate.pem, though. Could that be an issue?
And there are aliases in the cert, as well, should have mentionned that before.

Certificates are here:

$ ls -lvhF /etc/graylog/certs/
insgesamt 8,0K
-r--r--r-- 1 graylog graylog 2,3K Feb 13 09:46 graylog-certificate.pem
-r--r----- 1 graylog graylog 1,9K Feb 12 17:16 graylog-private.pem

And config looks like this (sorted here, for a quicker overview)
Everything is one host.

allow_highlighting = false
allow_leading_wildcard_searches = false
content_packs_auto_load = grok-patterns.json
elasticsearch_analyzer = standard
elasticsearch_hosts = https://127.0.0.1:9200
elasticsearch_index_prefix = graylog
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 30
elasticsearch_replicas = 0
elasticsearch_shards = 1
http_non_proxy_hosts = localhost,127.0.0.1,10.54.4.133
inputbuffer_processors = 2
inputbuffer_ring_size = 65536
inputbuffer_wait_strategy = blocking
is_master = true
lb_recognition_period_seconds = 3
message_journal_dir = /elasticsearch/graylog-journal/journal
message_journal_enabled = true
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
mongodb_uri = mongodb://localhost/graylog
node_id_file = /etc/graylog/server/node-id
output_batch_size = 500
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
output_flush_interval = 1
outputbuffer_processors = 3
password_secret = SuperSecretExtraLongPassword
plugin_dir = /usr/share/graylog-server/plugin/
processbuffer_processors = 5
processor_wait_strategy = blocking
proxied_requests_thread_pool_size = 32
rest_enable_tls = true
rest_listen_uri = https://10.54.4.133:9000/api/
rest_tls_cert_file = /etc/graylog/certs/graylog-certificate.pem
rest_tls_key_file = /etc/graylog/certs/graylog-private.pem
rest_tls_key_password =  redactedNoQuotesOrSpecialCharacters
rest_transport_uri = https://10.54.4.133:9000/api/
retention_strategy = delete
ring_size = 65536
root_password_sha2 = SomeSHA2PasswordHash
root_timezone = CET
root_username = graylog
rotation_strategy = time
web_enable_tls = true
web_listen_uri = https://10.54.4.133:9000/
web_tls_cert_file = /etc/graylog/certs/graylog-certificate.pem
web_tls_key_file = /etc/graylog/certs/graylog-private.pem
web_tls_key_password =  redactedNoQuotesOrSpecialCharacters

Note that I’m using https in the URI. Is that correct? Found that at https://mintopsblog.com/2018/03/04/graylog-basic-installation-with-https-ssl-configuration/ and it looked like the right thing to do.

I (temporarily) tried to route any requests to port 80 and later port 443 to port 9000 using iptables like this:

iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 9000

but that didn’t change the behaviour either. (Flushed the tables after)

Is this really is about certificates?
I’d expect some sort of “CertificateError” and not “ObjectIdentifier() – data isn’t an object ID (tag = 48)”

Probably some typo causing havoc will jump at me right after posting
It must be something obvious…

Thanks again!
And if you find the aforementionned typo, please let me know

Totally_Not_A_Robot · February 13, 2019, 1:30pm

I assume you mean the Root CA cert gets deployed Then that’s pretty darn cool and you guys are on top of things!

I’ve cleaned the pem-files as you described in your full instructions as well. But no luck.

Darn

I have the whole certificate-chain in my graylog-certificate.pem, though. Could that be an issue?

Yeah, it could. Let’s pare the PEM file down to nothing but the server cert for Graylog.

And there are aliases in the cert, as well, should have mentionned that before.

Aliases are fine. Our certs have plenty of them! Long name, short name, long alias, short alias, IP address, etc etc.

rest_enable_tls = true
rest_listen_uri = https://10.54.4.133:9000/api/
rest_tls_cert_file = /etc/graylog/certs/graylog-certificate.pem
rest_tls_key_file = /etc/graylog/certs/graylog-private.pem
rest_tls_key_password =  redactedNoQuotesOrSpecialCharacters
rest_transport_uri = https://10.54.4.133:9000/api/

All of that looks great too.

This basically should bring up Graylog on 9000, with HTTPS and certs.

EDIT:
Hold the horses! Spaces! And blank chars! Those were my bane in the early hours of setting up TLS for Graylog. Your certificate file should contain only ONE cert and ZERO whitespace before or after. Ditto for the private key file.

Marco · February 13, 2019, 2:08pm

One cert and zero whitespace (I highlight those *3"!§$1 spaces in vim…)
No trailing whitespace in server.conf, or else I would see passwordish issues when running without https.

But still, no luck

I need to buy a chicken to shake around the rack

OK. More digging in the server-logs…

Marco · February 13, 2019, 3:42pm

Ok, finally found a difference in the logs that could have significance.

When running http:

2019-02-13T15:02:50.464+01:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2019-02-13T15:03:03.504+01:00 INFO  [NetworkListener] Started listener bound to [10.54.4.133:9000]
2019-02-13T15:03:03.506+01:00 INFO  [HttpServer] [HttpServer] Started.
2019-02-13T15:03:03.506+01:00 INFO  [JerseyService] Started REST API at 
2019-02-13T15:03:03.506+01:00 INFO  [JerseyService] Started Web Interface at 
2019-02-13T15:03:03.507+01:00 INFO  [ServiceManagerListener] Services are healthy
2019-02-13T15:03:03.508+01:00 INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=26, KafkaJournal [RUNNING]=78, BufferSynchronizerService [RUNNING]=108, InputSetupService [RUNNING]=165, StreamCacheService [RUNNING]=223, JournalReader [RUNNING]=225, ConfigurationEtagService [RUNNING]=269, PeriodicalsService [RUNNING]=297, LookupTableService [RUNNING]=406, JerseyService [RUNNING]=13727}
2019-02-13T15:03:03.512+01:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2019-02-13T15:03:03.521+01:00 INFO  [ServerBootstrap] Graylog server up and running.

When running https, the line reading

INFO  [JerseyService] Enabling CORS for HTTP endpoint

never appears.

and instead of

INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]

I get

2019-02-13T15:56:40.825+01:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2019-02-13T15:56:40.830+01:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]

The question that remains is: How do I enable CORS for HTTPS?

Marco · February 14, 2019, 4:55pm

Great! I just wasted a couple of days because the Windows-CA decided to be helpful and “correct” my certificates and silently ignored that I need data-encryption.

Self-signed cert made on a linux-cli works without issue. Head->Desk

Well at least I know what the issue is now…

Totally_Not_A_Robot · February 14, 2019, 7:53pm

Great! I just wasted a couple of days because the Windows-CA decided to be helpful and “correct” my certificates and silently ignored that I need data-encryption.

Ahh! There you go then, yeah that would do it.

Mind you, there’s a reason why the CA “decided to be helpful”: security rules. AD CS works with “templates” which define the allowed settings for requested certificates. When submitting a CSR you actually include details on the desired template, to make sure you get the right kind of certificate.

In our case we actually had to create a completely new template to allow for the combination of factors that we needed for the whole Graylog stack (Graylog + MongoDB + ElasticSearch). Our template was created by:

Duplicating the standard “WebServer” template, renaming it “GraylogStack”.
In this new template we added “Client Authentication” as an allowed usage, so Graylog can auth to MongoDB and ElasticSearch.
In this new template we also checked the box that would allow the export of the private key.

Number 3 was needed because we were generating the keypairs on the CA, instead of on the Graylog boxen. This allowed us to prepare all certificates+keypairs beforehand, so the installation of the whole stack could immediately proceed with the right certs. In total, for our cluster environment, we ended up with ten keypairs and certificates: 3x Graylog data-inputs, 1x Graylog GUI, 3x ElasticSearch, 3x MongoDB.

If you would do a stranger a favor: never ever use self-signed certificates in a live environment. If your company has their own PKI, take the time to work with the PKI team to setup the correct certs. It takes a bit to learn, but in all honesty it’s A) not that complicated, B) helpful in assuring your network’s security and C) actually pretty darn cool. Could you guess that I’m my environment’s PKI admin?

Marco · February 15, 2019, 8:04am

Yes, I’m aware, but I did add the needed extras and checked the extra checkbox telling it “this is important”. So until I did the explicit check with CA-provided cert and self-signed cert I wasn’t aware of the error and was looking for the error in all the wrong places.

Thanks for the HowTo!
You are describing my exact use-case there and using those windows-tools can be quite frustrating for the command-line-loving Linux-Guy from time to time sigh
(especially with german translations in the UI)

Of course! Otherwise I would just have given up and would have stuck a reverse proxy in front of an unencrypted stack.

I thought I felt a tiny hint in that direction…

You are awesome!
Thanks again!

Totally_Not_A_Robot · February 15, 2019, 9:08am

Mann, auah! Du musst wirklich’ne Maso wesen!

You are describing my exact use-case there and using those windows-tools can be quite frustrating for the command-line-loving Linux-Guy from time to time sigh

But wait, there’s more! Now I just have to tell you about the CLI to ADCS! Let’s assume you’ve uploaded the CSR to D:\ on the CA host. Replace ${CANAME} with the name of your CA…

$Result=certutil -submit -attrib "CertificateTemplate:GraylogStack" -config localhost\${CANAME} D:\graylog.csr
$RequestID=(($Result | Select-String "RequestID: """).Split('"')[1])
certutil -resubmit ${RequestID} 
certreq -config localhost\${CANAME} -retrieve ${RequestID} D:\graylog.cer

Of course, if you’re doing proper PKI, you would not approve your own request with the certutil -resubmit. You would ask a PKI-admin, or a colleague, to validate and approve the request.

You are awesome!

https://memegenerator.net/img/instances/74038472.jpg

Marco · February 15, 2019, 2:15pm

Could you --pretty please-- post a couple of screenshots on how exactly I need to modify the default webserver-template?

My PKI-Admin and I have been trying to find the right combination of carefully hidden checkboxes to tick with not much success :-/

When i do…

openssl x509 -in cert.pem -purpose -noout -text

…my (working) self-signed cert looks like this:

8< - - - Cut here - - -
        X509v3 extensions:
            X509v3 Key Usage: 
                Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
8< - - - Cut here - - -

while the (not working) cert from the AD shows me this beauty:

8< - - - Cut here - - -
        X509v3 extensions:
            1.3.6.1.4.1.311.21.7: 
                0..&+.....7.....3...=...=...W.....F......A..d...
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment, Data Encipherment
            1.3.6.1.4.1.311.21.10: 
                0.0
..+.......0
..+.......
8< - - - Cut here - - -

…after adding “Client Authentication” as you mentionned before and checking “encryption” in some user property in order to get “Data Encipherment”.

And of course, the cert from AD will cause graylog to be unable to start.

Or have you imported CSR via your PKI Webinterface?
That’s what we are trying at the moment. (update: with no success either, as that must use an existing template…)

Totally_Not_A_Robot · February 15, 2019, 4:18pm

In my case, I have generated the whole keypair on the CA. I then did all the conversions I described earlier on the CA as well. I edited the files to remove all extraneous whitespaces and chain information. Then I migrated the PEM formated cert and the PKCS8 keyfile to the Linux boxen that run the Graylog stack.

OOOOOHHHHhhhh!!!

One damn important step that I should not forget: after transferring the files to the Linux box I pushed both files through dos2unix, converting the line endings from CRLF to CR (or is it LF?). Otherwise Graylog will still puke on it.

Marco · February 15, 2019, 4:53pm

Well, I…

exported cert and key into a pfx-file
copied that to the linux box and then
ran the openssl-commands from the docs over it to get cert and encrypted key out of it.

And that gives me whitespace-less base64-encoded files that graylog chokes on.
Extracting the different parts on a linux box should make the dos2unix part unneccessary, and take care of any character-encoding-weirdness as well.
My head is going to explode sometime soon

You were using mmc.exe on Windows, right?

Totally_Not_A_Robot · February 15, 2019, 5:32pm

Not me

You mean for the submission of the key req and CSR right? That’d be certlm.msc which is in fact part of MMC. But no, I usually handcraft the certificate request in a TXT file, which I use to keygen with certreq -new. #nerd

The request file would look like this:

[Version]
Signature = $Windows NT$    
 
[NewRequest]
Subject = "CN=myserver.mydomain.local"
KeyLength = 2048
Exportable = True
MachineKeySet = True
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"

[RequestAttributes]
CertificateTemplate=GraylogStack

[Extensions]
2.5.29.17 = "{text}"
_continue_ "dns=myserver&"
_continue_ "dns=graylog&"
_continue_ "dns=graylog.mydomain.local&"
_continue_ "ip=192.168.1.100&"

More details here:

I run that file through certreq -new D:\graylog.inf D:\graylog.csr, which generates the keypair based on the config and the template settings and outputs the CSR into file indicated. From there on, I proceed as described earlier.

Marco · February 15, 2019, 6:26pm

OK, that doesn’t look that much but just enough different from openssl.
I’ll try that back at work on tuesday.

It’s just the “CertificateTemplate=GraylogStack”-bit that troubles me.
If I can’t get the template setup right in the first place, any manual intervention on my behalf will fail.

So all you changed from the default webserver template was “Client Authentication” and nothing else?

And on a side-note: can Graylog handle sha512-certificates? Or do I need to use sha256, like in the server.conf?

This will haunt me the entire weekend where I don’t have access to the servers shudder

Totally_Not_A_Robot · February 15, 2019, 6:37pm

I will check when I’m at the office on Monday. Like you, I don’t have access right now.

Well, do you have a homelab? We can try things out! Mine is, as of yet, missing the Graylog stack. I’m still setting up a bunch of other things first, but Graylog will follow very soon. And since I’ve built a PKI already, I’ll definitely set it up with my own certs.

Unfortunately I’m studying for an exam that’s due in two weeks, so I don’t have much time to waste on my lab ;_;

Marco · February 19, 2019, 1:04pm

I don’t have the necessary Windows-stuff at home, so I can’t really test all that at home. (Licensing and such…)
Studying for your exam takes priority, of course. I can work with self-signed cert and a reverse-proxy for the time being and fake security. Though I’m curious how a working setup looks on the windows-pki side. mutters insults about the german gui

Totally_Not_A_Robot · February 19, 2019, 7:02pm

Boy am I gonna make your day!

https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2012-r2

Did you know that Microsoft offer eval versions of all their server and desktop OSes? And while an eval is valid for 3 months, you can actually renew the eval pretty simply. That makes it very simple to run a bunch of Windows servers in your homelab.

Want to run things a little more professionally, but cheaply? Have your employer spring for an “Action Pack” MS Partnership.

https://www.sherweb.com/blog/become-microsoft-certified/

$475 gets you ten licenses of lotsa Microsoft products.

I’ll be back at the office tomorrow; I’ll take a few quick screenshots for ya

Totally_Not_A_Robot · February 20, 2019, 7:10am

Basically, this is it. Really nothing exciting.

The instructions I have listed to make this template are:

Duplicate the default ADCS WebServer template, rename it to your liking.
Cryptography tab:
- Provider is the Key Storage Provider
- Requests can use any provider available on the subject’s computer
- Algorithm is RSA 2048
- Request hash is sha256
- Use alternate signature hash must be set to NO.
Request handling tab:
- Allow the private key to be exported is set to Yes.
Extensions tab:
- Application policies to both server auth as well as client auth.

I’ve chosen to make the private key exportable, because I’m generating the keypairs and converting them into the desired format on the CA itself. If you’re generating the keys and certs on the Graylog stack boxen, you will not have to tick that box.

Marco · February 20, 2019, 2:07pm

This is slowly driving me mad…
I tried this exact configuration (and various variations of it) and still graylog will show the exact same error as before

Something is seriously worng over here. I’ll post it when I find it…

Topic		Replies	Views
Graylog HTTPS issues Graylog Central (peer support)	11	2177	February 7, 2018
Graylog crashes on startup if HTTPS is enabled Graylog Central (peer support)	20	1147	May 8, 2020
Graylog startup failed Graylog Central (peer support)	4	3320	January 12, 2018
Understand Https Issues Graylog Central (peer support)	2	336	April 18, 2018
Trying to enable SSL Graylog Central (peer support)	7	663	March 4, 2020

Cant get https up and running

Related topics