I have deployed a Graylog v2.4.3 OVA/OVF based appliance on VMware for evaluation and are having some issues getting HTTPS web access working. I had some issues getting the OVF to initially load on our version of vCenter but I have been able to complete this with other info I found here and HTTP access seems to be working correctly.
Since this is the appliance I finally worked out that you must use only the graylog-ctl script based method to deploy system config. Under the “Install custom SSL certificates” section it describes replacing the
crt and key files and then restarting web server.
Using the instructions I found here Using HTTPS I generated the required certificates and replaced them. (making sure I changed the IP and DNS in the SAN.(subject alternative names)
Unfortunately I have been unable to achieve access to interface on https://{ip}:9000/ like I can still do via HTTP. I get the following message on IE11 and a similar thing on other browsers.(firefox / chrome)
Now I noticed on the “Using HTTPS” page there was the following note:
Not sure exactly what relevance that has the appliance and if the instructions would event work.
Can I have some further HTTPS appliance based info on how it should be configured… I suspect the problem may be the certificates may not be in the correct format based on the instructions I used to create them but there are not any which are specific for the appliance that I could find.
Further to this I have checked out the default certificate shipped with the appliance and it has CN=graylog
So as a test I re-deployed the appliance from OVF package and set the LAN DNS entry (on our LAN infrastructure DNS not in the graylog VM) for graylog={assigned DHCP IP} and I still get the above behaviour. (calling up http://graylog = works and https://graylog:9000 = not working)
I suspect there may be something wrong or missing from the appliance package as shipped (or I have missed something obvious)
Hmm actually you are right I have not added the graylog.crt to trusted CA on the test client.
Normally on this PC if a cert is untrusted we can make a connection with warning, download the cert and then trust via the browser.
I did this (manually by dumping graylog.crt to VM console and cut&paste to PC) and it is now trusted,
But unfortunately that has not fixed the problem… (same response as per initial post)
Or do you mean that the trust issue is server side… (note we are still using VM exactly as supplied without any modification)
To be honest - without knowing what steps exactly you have done and what files you put where and what commands you had run and what the content of your Graylog server.log is - everything I can do is guessing.
Will these changes work on the appliance that seems to be pretty much controlled by graylog-ctl
To be honest - without knowing what steps exactly you have done and what files you put where and what commands you had run and what the content of your Graylog server.log is - everything I can do is guessing.
Originally I tried to setup my own certificates but when I ran into issues I have redeployed Graylog appliance from scratch using OVF again. Since the certificate supplied inside the appliance ‘out of box’ and the hostname have been configured as ‘graylog’, I setup LAN DNS external to VM to reflect this. (purposely done so no changes have been made to deployed appliance)
what the content of your Graylog server.log is
Can you direct me specifically to where this log file is located in the appliance and I will check it out.
Customizing the OVA is a little more tricky and is not what the applicance is made for. If you need that customization you should think of installing your own Graylog following one of the step-by-step guides.
the default file locations are written down in the docs:
The log file /var/log/graylog/server/current has nothing in it and nothing is being collected/added based on a failure to open https (I think it may have been archived since the initial deployment)
The log file /var/log/graylog/nginx/current has nothing related to https in it only healthy messgaes related to http activity. (which works)
Customizing the OVA is a little more tricky and is not what the applicance is made for
Customisation is not what I intended, I just needed to get up and running quickly and eval the application. I needed to get HTTPS working and I would have thought this would be a very simple 5 minute task or work out of the box with no config. I feel I have got sidetracked on what is a very simple task (that being getting https working before even starting to look at the actual product functionality)
Here is a question.:
The appliance as released in OVF format (that can be downloaded from your website) has a configured default hostname of graylog. The certificate supplied with the release has CN=graylog.(common name)
Should this be operational with the hostname assigned as graylog without any setup??
So I can connect to server with supplied configuration on http://{IP Address} so the deployment is mostly working
Since the appliance is set to DHCP address mode by default the certificate will not have the IP address in it.
If we set the external DNS so the local LAN DNS name graylog points to the assigned IP (by DHCP) will https://graylog:9000 work without regenerating certificates.
If it will not work (which I am starting to suspect) can you provide the exact process to create self signed certificates for the appliance specifically…
I found instructions here but these only mention a manual server install not the appliance. I found instructions for the appliance to assign the certificates here but there is no mention on how to actually create/generate these files and what formats are required. Is the process the same as the server version or is there some differences?? Does the graylog-ctl script handle the import of the certificate (looks like it might need to be in different format) to java keystore etc…
To be honest I would have thought that graylog-ctl script would manage the cert for you automatically. On change of hostname operation regen the cert with the new hostname, on change of static IP regen the cert with the IP as a subject. For DHCP mode (as per default) leave hostname as graylog as its deployed. Just a suggestion to make it easy for new users trying to get up and running quickly with the appliance. Fair enough on app installation make them do it all manually.
Also related to the above if I eventually want to use a proper certificate chain (either a publically issued certificate or a private CA) where do the root certificates get installed to… I assume there would be one spot for nginx and a similar parallel spot for java. Haven’t found any doco on that yet and if I do put graylog into production this is something I will need to do…
Also another interesting update, I have logged in via HTTP to have a peak around and come across the following message, not sure if its related to my above problem…
I clicked on the link and it looks like syslog is the issue - not sure how to fix it…
Clicking the green start button on the right hand side of window didn’t seem to fix it…
The log file /var/log/graylog/server/current has nothing in it and nothing is being collected/added based on a failure to open https (I think it may have been archived since the initial deployment)
That nothing is written to the logfile indicates that Graylog might not run - or you have something mixed up in the Setup.
How to create those needed certificates can be found in this community or by just using the search engine of your trust - as we are speaking of webserver certificates. Our documentation covers that, some other ressources are stackOverflow the digitalocean blog and I think thousand of others.
To create a Self-Signed certificate for a server, I have created this script or you can create a custom ca for development. Both are mentioned several times in this community.
Both of the above scripts do only help, but not take off the pain of getting knowledge how tls (https) is working in modern web infrastructures and what is needed on both ends - the client and the server.
The certificate provided with the OVA can be seen as a placeholder - nothing more.
the enforce of SSL in the OVA is done via the used NGINX proxy and not Graylog itself. No need to import the Key into the JVM Keystore
without the error from the logfile I can only wild guess why one input is not working.
How to create those needed certificates can be found in this community or by just using the search engine of your trust - as we are speaking of webserver certificates. Our documentation covers that
Ok found all those resources in the beginning, looks like I was initially on the right track…
•the enforce of SSL in the OVA is done via the used NGINX proxy and not Graylog itself. No need to import the Key into the JVM Keystore
Ok so the certs are used by a reverse proxy to provide HTTPS - i.e. the reverse proxy is already implemented in the appliance build.
•without the error from the logfile I can only wild guess why one input is not working.
I see your point here its a bit like “the chicken and the egg”. Hopefully starting again from scratch I will get it right.
I will check out your scripts for cert generation against what I am doing. (kept a list of everything,)
I already have an internal CA for all my other management devices, but I think I will try to get self signed working first…
I have re-done the certificate configuration completely again both manually and with your self signed script and it still doesn’t work.
Interestingly enough however I have discovered that http://graylog:9000 works but https still does not.
I had a look at some external documentation for nginx SSL proxy config to see how it compared with what was already in the appliance. I could only find one configuration file at /opt/graylog/conf/nginx/nginx.conf and it didn’t seem to have any SSL config in it so I do not know if there is another location or its actually missing.
At this stage I am going to assume its missing and I might have a look at a previous version of the appliance to see if its configured differently. Can you check the https reverse proxy config for v2.4.3 and see if its complete and works?
Update: I have checked a number of the v2.4 releases and they seem to all have the same(no https) config.
Not sure exactly what relevance that has the appliance and if the instructions would event work.
I can now confirm that this is the problem.
Initially I did a trawl through other community posts, websites and the Nginx proxy doco and built the config myself as I could not seem to find the correct process in the documentation.
After doing all this I discovered that most of this config is build into the graylog-ctl tool already but you have to do a number of things to get it working which in my opinion is not clear under the most obvious “Install custom SSL certificates” section of the graylog_ctl script page as its spread across a number of sections.
You need to enable the SSL config in the nginx reverse proxy with both of these commands
graylog-ctl enforce-ssl
graylog-ctl reconfigure
This does more than just ‘enforce SSL’ it actually builds the SSL reverse proxy config as well so it should be given a name more like ‘enable SSL’ in my opinion
Then you need to update the certificates in the “Install custom SSL certificates” section of the page.
The doco should be updated to list this in a more complete process in my opinion.
After this it should work. The config is almost identical to what I had manually discovered but I had only TLS1.2 and a few less ciphers enabled.(which is probably better from a SSL security perspective) If you do it manually you will lose all the config after the next reconfigure so using graylog-ctl is the better option. (but if you do update the default config to tighten security keep a copy of the ‘/opt/graylog/conf/nginx/nginx.conf’ as this gets overwritten.
I would also suggest adding an option to the graylog-ctl script to generate a self signed certificate based on current hostname and IP would be a good idea to make it easy for new users and will reduce support queries. Also maybe having an option to generate a CSR and install a supplied certificate may not be a bad idea either.(so you don’t have to play with OpenSSL to do this stuff which can be time consuming if you haven’t done it before)
All this greatly simplify people starting off with the appliance in the initial stages.
With browser operating standards being upgraded in the near future I think you will find more people will be wanting SSL config as standard.
