Graylog on centos 7 with apache


(JEHANZAIB YOUNIS) #1

Hi folks,

I am having difficulty to configure graylog with apache. I made it work without any issue on debian with ngnix but cant make it work with apache on centos. but this time i have a google cloud machine. The local IP of the machine is 10.152.x.x and there is one external public IP. I have problem to configure the graylog server i think.
elasticsearch seems working fine. also the mongodb and graylog server. cant find anything in the graylog log file.

[root@graylog]# curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'

{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 4,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[root@graylog]# curl -k https:// 127.0.0.1:9000/api/system/sessions
curl: (7) Failed connect to 127.0.0.1:9000; Connection refused

here is my config for apache

<VirtualHost *:80>
    ServerName log.mycompany.net
    ProxyRequests Off
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "http://log.mycompany.net/api/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

</VirtualHost>

(Jochen) #2

Is Graylog running on https://127.0.0.1:9000?

What’s the output of the following command?

# lsof -i :9000

(JEHANZAIB YOUNIS) #3

no result.
[root@log ]# lsof -i :9000
[root@log ]#


(Jochen) #4

Maybe you should start Graylog and check the logs of your Graylog node.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(JEHANZAIB YOUNIS) #5

I checked the logs but cant find any clue yet. please note this is the google compute so the IP is 10.152.0.2.
my external public ip is 35.189.17.xxx

Does the settings look good?


rest_listen_uri = http:// log.mycompany.net:9000/api/
rest_transport_uri = http:// 10.152.0.2:9000/api/

web_enable = true
web_listen_uri = http:// 10.152.0.2:9000/
web_endpoint_uri = http:// 10.152.0.2:9000/


here are the server log if you can see anything suspecious .

2018-03-29T21:42:03.650Z WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-03-29T21:42:03.694Z INFO  [PeriodicalsService] Starting 26 periodicals ...
2018-03-29T21:42:03.694Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-03-29T21:42:03.885Z INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2018-03-29T21:42:03.896Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-03-29T21:42:03.907Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2018-03-29T21:42:03.914Z INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2018-03-29T21:42:03.915Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-03-29T21:42:04.002Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2018-03-29T21:42:04.003Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2018-03-29T21:42:04.003Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2018-03-29T21:42:04.083Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-03-29T21:42:04.114Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2018-03-29T21:42:04.136Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-03-29T21:42:04.192Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-03-29T21:42:04.222Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2018-03-29T21:42:04.265Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2018-03-29T21:42:04.287Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2018-03-29T21:42:04.314Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2018-03-29T21:42:04.344Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2018-03-29T21:42:04.344Z INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2018-03-29T21:42:04.491Z INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-03-29T21:42:04.522Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2018-03-29T21:42:04.552Z INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2018-03-29T21:42:04.629Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-03-29T21:42:04.672Z INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2018-03-29T21:42:04.699Z INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2018-03-29T21:42:04.741Z INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2018-03-29T21:42:04.807Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2018-03-29T21:42:04.810Z INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2018-03-29T21:42:22.629Z ERROR [ServiceManager] Service JerseyService [FAILED] has failed in the STARTING state.
java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_161]
        at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_161]
        at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_161]
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_161]
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_161]
        at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:131) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:88) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:238) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:218) ~[graylog.jar:?]
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:209) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:723) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:277) ~[graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:236) ~[graylog.jar:?]
        at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:141) ~[graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-03-29T21:42:22.640Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2018-03-29T21:42:22.641Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2018-03-29T21:42:22.642Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2018-03-29T21:42:22.643Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2018-03-29T21:42:22.646Z INFO  [JournalReader] Stopping.
2018-03-29T21:42:22.647Z INFO  [LogManager] Shutting down.
2018-03-29T21:42:22.671Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2018-03-29T21:42:22.674Z ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2018-03-29T21:42:22.681Z INFO  [Buffers] Waiting until all buffers are empty.
2018-03-29T21:42:22.682Z INFO  [Buffers] All buffers are empty. Continuing.
2018-03-29T21:42:22.683Z INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2018-03-29T21:42:22.685Z INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2018-03-29T21:42:22.694Z INFO  [LogManager] Shutdown complete.
2018-03-29T21:42:22.695Z INFO  [ServiceManagerListener] Services are now stopped.
2018-03-29T21:42:22.695Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2018-03-29T21:42:22.705Z INFO  [Server] SIGNAL received. Shutting down.
2018-03-29T21:42:22.710Z INFO  [GracefulShutdown] Graceful shutdown initiated.
2018-03-29T21:42:22.711Z INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2018-03-29T21:42:22.712Z WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-03-29T21:42:26.713Z INFO  [GracefulShutdown] Goodbye.

(JEHANZAIB YOUNIS) #6

I think i found this now. do you know the best way to give permission ? i think graylog is not allowed to open the port ? or you think its something else?

2018-03-29T22:45:47.720Z INFO  [JerseyService] Started REST API at <http://10.152.0.2:9000/api/>
2018-03-29T22:45:50.205Z ERROR [ServiceManager] Service JerseyService [FAILED] has failed in the STARTING state.
java.net.SocketException: Permission denied

(JEHANZAIB YOUNIS) #7

Finally i made the server work but now another issue. When i enter the username and password. i see

Error - the server returned: 404 - cannot POST http://log.mycompany.net:9000/system/sessions (404)

Now i have these settings in server.conf

rest_listen_uri = http://0.0.0.0:9000/api/
rest_transport_uri = http://10.152.0.2:9000/api/

web_listen_uri = http://0.0.0.0:9000/
web_endpoint_uri = http://log.mycompany.net:9000

(Jochen) #8

web_endpoint_uri has to be set to the public URI of the Graylog REST API, not the URI of the web interface.


(system) #9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.