Issues accessing Graylog login behind ssl proxy in apache


(Stephen) #1

Hi there,

Currently my server is running behind an apache proxy, this seems to work but when the login screen should load, i get the following error

We are experiencing problems connecting to the Graylog server running on https://127.0.0.1:9000/api/. Please verify that the server is healthy and working correctly.

You will be automatically redirected to the previous page once we can connect to the server.

Do you need a hand? We can help you.
Less details

This is the last response we received from the server:

Error message
    Bad request
Original Request
    GET https://127.0.0.1:9000/api/system/sessions
Status code
    undefined
Full error message
    Error: Request has been terminated
    Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

I’ve been going through suggestions and setting regarding this same error from other users none of the suggestion seem to work for me or they are either for proxy or ssl, not both. Any help would be appreciated

As far as i can tell, there are no errors on the graylog-server logs nor in apache. Below are settings (with obvious redactions) and logs

Graylog server 2.2.3+7adc951
JRE: Oracle Corporation 1.8.0_121 on Linux 4.4.0-31-generic
OS: Ubuntu 14.04.5 LTS
Apache: Apache/2.4.7 (Ubuntu)
host firewall off while we finish the setup

grep . /etc/graylog/server/server.conf | grep -v "\#"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = NOPE
root_username = NOPE
root_password_sha2 = NOPE
root_email = "NOPE"
root_timezone = America/New_York
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://domain.name.here/api
rest_enable_gzip = false
rest_enable_tls = true
rest_tls_cert_file = /etc/apache2/certs/all.crt
rest_tls_key_file = /etc/apache2/certs/all.pem
web_listen_uri = http://127.0.0.1:9000/
web_endpoint_uri = https://domain.name.here:443/api/
web_enable_gzip = false
web_enable_tls = true
web_tls_cert_file = /etc/apache2/certs/all.crt
web_tls_key_file = /etc/apache2/certs/all.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_network_host = 127.0.0.1
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32 
grep . /etc/apache2/sites-enabled/default-ssl.conf | grep -v "\#"
<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin webmaster@localhost
                ServerName domain.name.here
                ProxyRequests Off
                DocumentRoot /var/www/html
                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined
                SSLEngine on
                SSLProxyEngine on
                SSLProxyVerify none
                SSLProxyCheckPeerCN off
                SSLProxyCheckPeerName off
                SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4
                SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4
                SSLProtocol all -SSLv3 -SSLv2
                SSLProxyProtocol all -SSLv3 -SSLv2
                SSLHonorCipherOrder On
                SSLCertificateFile /etc/apache2/certs/all.crt
                SSLCertificateKeyFile /etc/apache2/certs/all.key
                SSLCACertificateFile /etc/apache2/certs/combined.crt
                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>
                BrowserMatch "MSIE [2-6]" \
                                nokeepalive ssl-unclean-shutdown \
                                downgrade-1.0 force-response-1.0
                BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
                <Proxy *>
                        Order deny,allow
                        Allow from all
                </Proxy>
                <Location />
                        RequestHeader set X-Graylog-Server-URL "https://127.0.0.1:9000/api/"
                        ProxyPass https://127.0.0.1:9000/
                        ProxyPassReverse https://127.0.0.1:9000/
                </Location>
        </VirtualHost>
</IfModule>       
curl -XGET 127.0.0.1:9200
{
  "name" : "Blackwing",
  "cluster_name" : "graylog",
  "cluster_uuid" : "GcR_wGwvQ8yfAcSd6tM5EQ",
  "version" : {
    "number" : "2.4.4",
    "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
    "build_timestamp" : "2017-01-03T11:33:16Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
} 
curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
curl -k https://127.0.0.1:9000/api/system/sessions
{"session_id":null,"username":null,"is_valid":false}
tail -f /var/log/apache2/access.log
some.ip.here.XX - - [19/Apr/2017:10:47:16 -0400] "GET / HTTP/1.1" 200 7143 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"
some.ip.here.XX - - [19/Apr/2017:10:47:20 -0400] "GET /config.js HTTP/1.1" 200 531 "https://domain.name.here/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"

tail -f /var/log/graylog-server/server.log
2017-04-19T10:46:35.870-04:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-04-19T10:46:35.874-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-04-19T10:46:35.875-04:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-04-19T10:46:35.876-04:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-04-19T10:46:35.890-04:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-04-19T10:46:35.891-04:00 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-04-19T10:46:36.306-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-04-19T10:46:36.656-04:00 INFO  [Version] HV000001: Hibernate Validator null
2017-04-19T10:46:40.165-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2017-04-19T10:46:40.204-04:00 INFO  [NodeId] Node ID: 3ac10362-abad-4897-af01-48a34e519755
2017-04-19T10:46:40.558-04:00 INFO  [LogManager] Loading logs.
2017-04-19T10:46:40.777-04:00 INFO  [LogManager] Logs loading complete.
2017-04-19T10:46:40.778-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-04-19T10:46:40.817-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-04-19T10:46:40.867-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-04-19T10:46:40.951-04:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-04-19T10:46:41.019-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:139}] to localhost:27017
2017-04-19T10:46:41.031-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 9]}, minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216, roundTripTimeNanos=1187160}
2017-04-19T10:46:41.052-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:140}] to localhost:27017
2017-04-19T10:46:41.546-04:00 INFO  [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] version[2.4.4], pid[30352], build[fcbb46d/2017-01-03T11:33:16Z]
2017-04-19T10:46:41.547-04:00 INFO  [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] initializing ...
2017-04-19T10:46:41.558-04:00 INFO  [plugins] [graylog-3ac10362-abad-4897-af01-48a34e519755] modules [], plugins [graylog-monitor], sites []
2017-04-19T10:46:43.972-04:00 INFO  [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] initialized
2017-04-19T10:46:44.100-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-04-19T10:46:46.352-04:00 INFO  [RulesEngineProvider] No static rules file loaded.
2017-04-19T10:46:46.819-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-04-19T10:46:48.392-04:00 INFO  [ServerBootstrap] Graylog server 2.2.3+7adc951 starting up
2017-04-19T10:46:48.394-04:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_121 on Linux 4.4.0-31-generic
2017-04-19T10:46:48.394-04:00 INFO  [ServerBootstrap] Deployment: deb
2017-04-19T10:46:48.395-04:00 INFO  [ServerBootstrap] OS: Ubuntu 14.04.5 LTS (trusty)
2017-04-19T10:46:48.395-04:00 INFO  [ServerBootstrap] Arch: amd64
2017-04-19T10:46:48.407-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-04-19T10:46:48.465-04:00 INFO  [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] starting ...
2017-04-19T10:46:48.465-04:00 INFO  [PeriodicalsService] Starting 26 periodicals ...
2017-04-19T10:46:48.469-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.472-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-04-19T10:46:48.481-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.484-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-04-19T10:46:48.488-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-04-19T10:46:48.489-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-04-19T10:46:48.493-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-04-19T10:46:48.499-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-04-19T10:46:48.503-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-04-19T10:46:48.503-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-04-19T10:46:48.504-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.507-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-04-19T10:46:48.508-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-04-19T10:46:48.510-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.515-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-04-19T10:46:48.518-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-04-19T10:46:48.520-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-04-19T10:46:48.521-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-04-19T10:46:48.800-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:141}] to localhost:27017
2017-04-19T10:46:48.800-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:142}] to localhost:27017
2017-04-19T10:46:48.805-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:143}] to localhost:27017
2017-04-19T10:46:48.807-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:144}] to localhost:27017
2017-04-19T10:46:48.807-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:145}] to localhost:27017
2017-04-19T10:46:48.811-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:146}] to localhost:27017
2017-04-19T10:46:48.819-04:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:148}] to localhost:27017
2017-04-19T10:46:48.819-04:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:147}] to localhost:27017
2017-04-19T10:46:48.821-04:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:149}] to localhost:27017
2017-04-19T10:46:48.905-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-04-19T10:46:48.952-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-04-19T10:46:48.952-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-04-19T10:46:48.953-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-04-19T10:46:48.965-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-04-19T10:46:48.975-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-04-19T10:46:48.981-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-04-19T10:46:48.983-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-04-19T10:46:48.996-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-04-19T10:46:48.997-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-04-19T10:46:49.013-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-04-19T10:46:49.203-04:00 INFO  [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2017-04-19T10:46:49.329-04:00 INFO  [transport] [graylog-3ac10362-abad-4897-af01-48a34e519755] publish_address {127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2017-04-19T10:46:49.345-04:00 INFO  [discovery] [graylog-3ac10362-abad-4897-af01-48a34e519755] graylog/MK1ePtvlRZWhCSCp5Msoiw
2017-04-19T10:46:49.929-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2017-04-19T10:46:52.350-04:00 WARN  [discovery] [graylog-3ac10362-abad-4897-af01-48a34e519755] waited for 3s and no initial state was set by the discovery
2017-04-19T10:46:52.351-04:00 INFO  [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] started
2017-04-19T10:46:52.490-04:00 INFO  [service] [graylog-3ac10362-abad-4897-af01-48a34e519755] detected_master {Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300}, added {{Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300}])
2017-04-19T10:47:08.301-04:00 INFO  [NetworkListener] Started listener bound to [127.0.0.1:9000]
2017-04-19T10:47:08.304-04:00 INFO  [HttpServer] [HttpServer] Started.
2017-04-19T10:47:08.305-04:00 INFO  [JerseyService] Started REST API at <https://127.0.0.1:9000/api/>
2017-04-19T10:47:08.305-04:00 INFO  [JerseyService] Started Web Interface at <https://127.0.0.1:9000/>
2017-04-19T10:47:08.308-04:00 INFO  [ServiceManagerListener] Services are healthy
2017-04-19T10:47:08.310-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=31, KafkaJournal [RUNNING]=32, OutputSetupService [RUNNING]=36, InputSetupService [RUNNING]=368, ConfigurationEtagService [RUNNING]=384, JournalReader [RUNNING]=387, StreamCacheService [RUNNING]=396, PeriodicalsService [RUNNING]=553, IndexerSetupService [RUNNING]=4062, JerseyService [RUNNING]=19856}
2017-04-19T10:47:08.312-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD] to Running?[LB:ALIVE]
2017-04-19T10:47:08.322-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2017-04-19T10:47:20.368-04:00 INFO  [jvm] [graylog-3ac10362-abad-4897-af01-48a34e519755] [gc][young][31][4] duration [929ms], collections [1]/[1.4s], total [929ms]/[1.5s], memory [1.3gb]->[206.4mb]/[3.8gb], all_pools {[young] [1.1gb]->[34.1mb]/[1.6gb]}{[survivor] [164.6mb]->[128.2mb]/[204.7mb]}{[old] [0b]->[51.9mb]/[2gb]}

If theres anything else that would help troubleshoot, do let me know. We curently have a PoC running with no issues but it is not behind a proxy and is unsecured. For production we need to put up walls

Thanks,
Stephen


(Jochen) #2

Is your web browser with which you access the Graylog web interface, running on the same machine that’s hosting Graylog?

You might want to check the web_endpoint_uri setting and the X-Graylog-Server-URL HTTP request header…


(Stephen) #3

I guess is it the little things that get missed. :scream: Going crazy looking for the issue on graylog settings, the issue was indeed the apache conf (browser is not on the local graylog machine)

RequestHeader set X-Graylog-Server-URL "https://some.domain.name/api/"

solved the issue. Thanks! @jochen

(edit)I’ll add then, anyone looking for HTTPS/SSL behind an apache reverse proxy with these versions, The above configurations with the above correction work :smile:
Graylog server 2.2.3+7adc951
JRE: Oracle Corporation 1.8.0_121 on Linux 4.4.0-31-generic
OS: Ubuntu 14.04.5 LTS
Apache: Apache/2.4.7 (Ubuntu)


(Jochen) #4

You don’t have to set both, web_endpoint_uri and the X-Graylog-Server-URI HTTP request header. One is enough, see http://docs.graylog.org/en/2.2/pages/configuration/web_interface.html#how-does-the-web-interface-connect-to-the-graylog-server for details.