Hi there,
Currently my server is running behind an apache proxy, this seems to work but when the login screen should load, i get the following error
We are experiencing problems connecting to the Graylog server running on https://127.0.0.1:9000/api/. Please verify that the server is healthy and working correctly.
You will be automatically redirected to the previous page once we can connect to the server.
Do you need a hand? We can help you.
Less details
This is the last response we received from the server:
Error message
Bad request
Original Request
GET https://127.0.0.1:9000/api/system/sessions
Status code
undefined
Full error message
Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.
I’ve been going through suggestions and setting regarding this same error from other users none of the suggestion seem to work for me or they are either for proxy or ssl, not both. Any help would be appreciated
As far as i can tell, there are no errors on the graylog-server logs nor in apache. Below are settings (with obvious redactions) and logs
Graylog server 2.2.3+7adc951
JRE: Oracle Corporation 1.8.0_121 on Linux 4.4.0-31-generic
OS: Ubuntu 14.04.5 LTS
Apache: Apache/2.4.7 (Ubuntu)
host firewall off while we finish the setup
grep . /etc/graylog/server/server.conf | grep -v "\#"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = NOPE
root_username = NOPE
root_password_sha2 = NOPE
root_email = "NOPE"
root_timezone = America/New_York
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://127.0.0.1:9000/api/
rest_transport_uri = https://domain.name.here/api
rest_enable_gzip = false
rest_enable_tls = true
rest_tls_cert_file = /etc/apache2/certs/all.crt
rest_tls_key_file = /etc/apache2/certs/all.pem
web_listen_uri = http://127.0.0.1:9000/
web_endpoint_uri = https://domain.name.here:443/api/
web_enable_gzip = false
web_enable_tls = true
web_tls_cert_file = /etc/apache2/certs/all.crt
web_tls_key_file = /etc/apache2/certs/all.pem
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300
elasticsearch_network_host = 127.0.0.1
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
grep . /etc/apache2/sites-enabled/default-ssl.conf | grep -v "\#"
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin webmaster@localhost
ServerName domain.name.here
ProxyRequests Off
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLProtocol all -SSLv3 -SSLv2
SSLProxyProtocol all -SSLv3 -SSLv2
SSLHonorCipherOrder On
SSLCertificateFile /etc/apache2/certs/all.crt
SSLCertificateKeyFile /etc/apache2/certs/all.key
SSLCACertificateFile /etc/apache2/certs/combined.crt
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-6]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
<Location />
RequestHeader set X-Graylog-Server-URL "https://127.0.0.1:9000/api/"
ProxyPass https://127.0.0.1:9000/
ProxyPassReverse https://127.0.0.1:9000/
</Location>
</VirtualHost>
</IfModule>
curl -XGET 127.0.0.1:9200
{
"name" : "Blackwing",
"cluster_name" : "graylog",
"cluster_uuid" : "GcR_wGwvQ8yfAcSd6tM5EQ",
"version" : {
"number" : "2.4.4",
"build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
"build_timestamp" : "2017-01-03T11:33:16Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}
curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "graylog",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4,
"active_shards" : 4,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
curl -k https://127.0.0.1:9000/api/system/sessions
{"session_id":null,"username":null,"is_valid":false}
tail -f /var/log/apache2/access.log
some.ip.here.XX - - [19/Apr/2017:10:47:16 -0400] "GET / HTTP/1.1" 200 7143 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"
some.ip.here.XX - - [19/Apr/2017:10:47:20 -0400] "GET /config.js HTTP/1.1" 200 531 "https://domain.name.here/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"
tail -f /var/log/graylog-server/server.log
2017-04-19T10:46:35.870-04:00 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-04-19T10:46:35.874-04:00 INFO [CmdLineTool] Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-04-19T10:46:35.875-04:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-04-19T10:46:35.876-04:00 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-04-19T10:46:35.890-04:00 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-04-19T10:46:35.891-04:00 INFO [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-04-19T10:46:36.306-04:00 INFO [CmdLineTool] Running with JVM arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-04-19T10:46:36.656-04:00 INFO [Version] HV000001: Hibernate Validator null
2017-04-19T10:46:40.165-04:00 INFO [InputBufferImpl] Message journal is enabled.
2017-04-19T10:46:40.204-04:00 INFO [NodeId] Node ID: 3ac10362-abad-4897-af01-48a34e519755
2017-04-19T10:46:40.558-04:00 INFO [LogManager] Loading logs.
2017-04-19T10:46:40.777-04:00 INFO [LogManager] Logs loading complete.
2017-04-19T10:46:40.778-04:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-04-19T10:46:40.817-04:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-04-19T10:46:40.867-04:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-04-19T10:46:40.951-04:00 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-04-19T10:46:41.019-04:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:139}] to localhost:27017
2017-04-19T10:46:41.031-04:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[2, 4, 9]}, minWireVersion=0, maxWireVersion=0, maxDocumentSize=16777216, roundTripTimeNanos=1187160}
2017-04-19T10:46:41.052-04:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:140}] to localhost:27017
2017-04-19T10:46:41.546-04:00 INFO [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] version[2.4.4], pid[30352], build[fcbb46d/2017-01-03T11:33:16Z]
2017-04-19T10:46:41.547-04:00 INFO [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] initializing ...
2017-04-19T10:46:41.558-04:00 INFO [plugins] [graylog-3ac10362-abad-4897-af01-48a34e519755] modules [], plugins [graylog-monitor], sites []
2017-04-19T10:46:43.972-04:00 INFO [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] initialized
2017-04-19T10:46:44.100-04:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-04-19T10:46:46.352-04:00 INFO [RulesEngineProvider] No static rules file loaded.
2017-04-19T10:46:46.819-04:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-04-19T10:46:48.392-04:00 INFO [ServerBootstrap] Graylog server 2.2.3+7adc951 starting up
2017-04-19T10:46:48.394-04:00 INFO [ServerBootstrap] JRE: Oracle Corporation 1.8.0_121 on Linux 4.4.0-31-generic
2017-04-19T10:46:48.394-04:00 INFO [ServerBootstrap] Deployment: deb
2017-04-19T10:46:48.395-04:00 INFO [ServerBootstrap] OS: Ubuntu 14.04.5 LTS (trusty)
2017-04-19T10:46:48.395-04:00 INFO [ServerBootstrap] Arch: amd64
2017-04-19T10:46:48.407-04:00 WARN [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-04-19T10:46:48.465-04:00 INFO [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] starting ...
2017-04-19T10:46:48.465-04:00 INFO [PeriodicalsService] Starting 26 periodicals ...
2017-04-19T10:46:48.469-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.472-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-04-19T10:46:48.481-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.484-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-04-19T10:46:48.488-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-04-19T10:46:48.489-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-04-19T10:46:48.493-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-04-19T10:46:48.499-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-04-19T10:46:48.503-04:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-04-19T10:46:48.503-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-04-19T10:46:48.504-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.507-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-04-19T10:46:48.508-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-04-19T10:46:48.510-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-04-19T10:46:48.515-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-04-19T10:46:48.518-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-04-19T10:46:48.520-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-04-19T10:46:48.521-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-04-19T10:46:48.800-04:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:141}] to localhost:27017
2017-04-19T10:46:48.800-04:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:142}] to localhost:27017
2017-04-19T10:46:48.805-04:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:143}] to localhost:27017
2017-04-19T10:46:48.807-04:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:144}] to localhost:27017
2017-04-19T10:46:48.807-04:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:145}] to localhost:27017
2017-04-19T10:46:48.811-04:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:146}] to localhost:27017
2017-04-19T10:46:48.819-04:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:148}] to localhost:27017
2017-04-19T10:46:48.819-04:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:147}] to localhost:27017
2017-04-19T10:46:48.821-04:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:149}] to localhost:27017
2017-04-19T10:46:48.905-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2017-04-19T10:46:48.952-04:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-04-19T10:46:48.952-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-04-19T10:46:48.953-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-04-19T10:46:48.965-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-04-19T10:46:48.975-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-04-19T10:46:48.981-04:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-04-19T10:46:48.983-04:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-04-19T10:46:48.996-04:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-04-19T10:46:48.997-04:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-04-19T10:46:49.013-04:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-04-19T10:46:49.203-04:00 INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2017-04-19T10:46:49.329-04:00 INFO [transport] [graylog-3ac10362-abad-4897-af01-48a34e519755] publish_address {127.0.0.1:9350}, bound_addresses {127.0.0.1:9350}
2017-04-19T10:46:49.345-04:00 INFO [discovery] [graylog-3ac10362-abad-4897-af01-48a34e519755] graylog/MK1ePtvlRZWhCSCp5Msoiw
2017-04-19T10:46:49.929-04:00 INFO [JerseyService] Enabling CORS for HTTP endpoint
2017-04-19T10:46:52.350-04:00 WARN [discovery] [graylog-3ac10362-abad-4897-af01-48a34e519755] waited for 3s and no initial state was set by the discovery
2017-04-19T10:46:52.351-04:00 INFO [node] [graylog-3ac10362-abad-4897-af01-48a34e519755] started
2017-04-19T10:46:52.490-04:00 INFO [service] [graylog-3ac10362-abad-4897-af01-48a34e519755] detected_master {Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300}, added {{Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Blackwing}{8wLNnVN6R3WV8yhp2XDcGw}{127.0.0.1}{127.0.0.1:9300}])
2017-04-19T10:47:08.301-04:00 INFO [NetworkListener] Started listener bound to [127.0.0.1:9000]
2017-04-19T10:47:08.304-04:00 INFO [HttpServer] [HttpServer] Started.
2017-04-19T10:47:08.305-04:00 INFO [JerseyService] Started REST API at <https://127.0.0.1:9000/api/>
2017-04-19T10:47:08.305-04:00 INFO [JerseyService] Started Web Interface at <https://127.0.0.1:9000/>
2017-04-19T10:47:08.308-04:00 INFO [ServiceManagerListener] Services are healthy
2017-04-19T10:47:08.310-04:00 INFO [ServerBootstrap] Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=31, KafkaJournal [RUNNING]=32, OutputSetupService [RUNNING]=36, InputSetupService [RUNNING]=368, ConfigurationEtagService [RUNNING]=384, JournalReader [RUNNING]=387, StreamCacheService [RUNNING]=396, PeriodicalsService [RUNNING]=553, IndexerSetupService [RUNNING]=4062, JerseyService [RUNNING]=19856}
2017-04-19T10:47:08.312-04:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD] to Running?[LB:ALIVE]
2017-04-19T10:47:08.322-04:00 INFO [ServerBootstrap] Graylog server up and running.
2017-04-19T10:47:20.368-04:00 INFO [jvm] [graylog-3ac10362-abad-4897-af01-48a34e519755] [gc][young][31][4] duration [929ms], collections [1]/[1.4s], total [929ms]/[1.5s], memory [1.3gb]->[206.4mb]/[3.8gb], all_pools {[young] [1.1gb]->[34.1mb]/[1.6gb]}{[survivor] [164.6mb]->[128.2mb]/[204.7mb]}{[old] [0b]->[51.9mb]/[2gb]}
If theres anything else that would help troubleshoot, do let me know. We curently have a PoC running with no issues but it is not behind a proxy and is unsecured. For production we need to put up walls
Thanks,
Stephen
Going crazy looking for the issue on graylog settings, the issue was indeed the apache conf (browser is not on the local graylog machine)