Graylog 2.5.1 ElacticSearch X-Pack 6.1 issue


(Paweł Karoluk) #1

Hi, is anyone using ElasticSearch 6.1.4 + X-Pack. I have an Access Denied with Graylog 2.5.1 bootstap. It seems that graylog tried to get _all/_aliases which was dropped in elastic 6.1.

graylog user role:

client_service:
  cluster:
  - monitor
  - manage_index_templates
  indices:
  - names: "graylog_*"
    privileges:
    - monitor
    - manage
    - view_index_metadata
    - read_cross_cluster
    - read
    - index
    - create_index
    - delete_index
    - create
    - write
    - delete

|[2019-01-14T00:44:24,776] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[cluster:monitor/health], indices=[_all], request=[ClusterHealthRequest]|
|---|---|
|[2019-01-14T00:44:24,927] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[indices:monitor/settings/get], indices=[graylog_deflector], request=[GetSettingsRequest]|
|[2019-01-14T00:44:24,930] [transport] [access_denied]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[indices:admin/aliases/get], indices=[*,-*], request=[GetAliasesRequest]|
|[2019-01-14T00:44:26,778] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[cluster:monitor/health], indices=[_all], request=[ClusterHealthRequest]|

#2

I’m not sure, but the “_all” indices match with your “graylog_*” role?


(Paweł Karoluk) #3

I tried names: “*” but the result was the same.


(Paweł Karoluk) #4

I had dumped all traffic and it looks like Graylog doesn’t perform PUT /_template/graylog-internal during bootstrapping. When I tried to perform it manually as graylog user it works so it’s not a wrong X-Pack configuration.

I checked also

elasticsearch_version=6
elasticsearch_disable_version_check= true

settings but it didn’t help


(Jan Doberstein) #5

just something that comes to my mind - Why did you not use the latest stable 6.x release?


(Paweł Karoluk) #6

Elastic 6.1.4 is the laster version supported by my puppet module but is not a cause.

My current Server config:

allow_highlighting = true
allow_leading_wildcard_searches = true
content_packs_auto_load = grok-patterns.json
content_packs_dir = /usr/share/graylog-server/contentpacks
elasticsearch_disable_version_check = true
elasticsearch_hosts = httX://graylog:<SECRET>@10.254.0.10:9200
elasticsearch_index_prefix = graylog
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 30
elasticsearch_replicas = 1
elasticsearch_shards = 4
elasticsearch_template_name = graylog-internal
gc_warning_threshold = 1s
http_connect_timeout = 5s
http_read_timeout = 10s
http_write_timeout = 10s
inputbuffer_processors = 2
inputbuffer_ring_size = 131072
inputbuffer_wait_strategy = blocking
is_master = true
ldap_connection_timeout = 2000
message_journal_dir = /var/lib/graylog-server/journal
message_journal_enabled = true
message_journal_max_size = 10gb
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
mongodb_uri = mongodb://mongo:mongo@10.10.0.1:27017/graylog
mongodb_useauth = true
node_id_file = /etc/graylog/server/node-id
output_batch_size = 500
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
output_flush_interval = 1
outputbuffer_processors = 4
password_secret = <SECRET>
plugin_dir = /usr/share/graylog-server/plugin
processbuffer_processors = 4
processor_wait_strategy = blocking
proxied_requests_thread_pool_size = 32
rest_enable_gzip = true
rest_enable_tls = false
rest_listen_uri = httX://10.254.0.13:9000/api/
rest_transport_uri = httX://10.254.0.13:9000/api/
retention_strategy = delete
ring_size = 131072
root_password_sha2 = <SECRET>
root_timezone = Europe/Warsaw
root_username = admin
rotation_strategy = count
transport_email_enabled = true
transport_email_from_email = graylog@example.com
transport_email_hostname = smtp.example.com
transport_email_port = 25
transport_email_subject_prefix = [graylog]
transport_email_use_auth = false
transport_email_use_ssl = false
transport_email_use_tls = false
transport_email_web_interface_url = httX://graylog.example.com
trusted_proxies = 127.0.0.1/32,10.254.0.13/32
web_enable = true
web_enable_cors = false
web_enable_gzip = true
web_enable_tls = false
web_endpoint_uri = httX://graylog.example.com
web_listen_uri = httX://10.254.0.13:9000/

(due forum limitation http was replaced by httX)


(Jan Doberstein) #7

let me clarify - you use elasticsearch plugin to restrict access to elasticsearch. That is not working.

without knowing that is actually in your logs of Graylog, Elasticsearch and may an additional audit log of the xpack nobody will be able to help.

Did you disable xpack security and checked if everything is working? Just to make sure the issue is only with the the authentication.


(Paweł Karoluk) #8

With xpack.security.enabled=false during graylog bootstrap (from tcpdump):

1. GET /_cluster/health/_all?local=true&timeout=60s -> 200 OK
2. GET /_cluster/health/graylog_*?timeout=60s -> 200 OK
3. GET /_cat/nodes?h=name%2Chost%2Cip%2CfileDescriptorMax&full_id=true&format=json -> 200 OK
4. PUT /_template/graylog-internal -> 200 OK
5. PUT /graylog_0 -> 200 OK
6. GET /_cluster/health/graylog_0?wait_for_status=yellow -> 200 OK
7. POST /_aliases -> 200 OK
8. GET /graylog_*/_alias?expand_wildcards=open -> 200 OK
9. POST /_bulk -> 200 OK

Than I turned on X-Pack and dump looks like:

1. GET /_cluster/health/_all?local=true&timeout=60s -> 200 OK
2. GET /_cluster/health/graylog_*?timeout=60s -> 200 OK
3. GET /graylog_deflector/_settings -> 404 Not Found
4. GET /graylog_*/_alias?expand_wildcards=open -> 404 Not Found
5. GET /_cat/nodes?h=name%2Chost%2Cip%2CfileDescriptorMax&full_id=true&format=json -> 200 OK
6. GET /_all/_alias -> 404 Not Found
7. POST /_bulk -> 200 OK

As I said It’s not permission issue, all requests I can perform manually as graylog user.
I wonder why graylog doesn’t even try to PUT /_template/graylog-internal or /graylog_0


(Jan Doberstein) #9

did you checked your Graylog server.log?


(Paweł Karoluk) #11

Server.log:

2019-01-14T14:44:55.368+01:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-01-14T14:44:55.370+01:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
org.graylog2.indexer.IndexNotFoundException: Couldn't collect aliases for index pattern graylog_*

Index not found for query: . Try recalculating your index ranges.
        at org.graylog2.indexer.cluster.jest.JestUtils.buildIndexNotFoundException(JestUtils.java:118) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:84) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.getIndexNamesAndAliases(Indices.java:316) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getNewestIndexNumber(MongoIndexSet.java:151) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getNewestIndex(MongoIndexSet.java:146) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:252) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_191]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Than I set the template, index and alias manually and rotate the index from the Graylog.

2019-01-14T15:29:24.141+01:00 INFO [DeflectorResource] Cycling deflector for index set &lt;5c165a96ab04d33ff4a69aa6&gt;. Reason: REST request.
2019-01-14T15:29:24.183+01:00 INFO [MongoIndexSet] Cycling from &lt;graylog_0&gt; to &lt;graylog_1&gt;.
2019-01-14T15:29:24.183+01:00 INFO [MongoIndexSet] Creating target index &lt;graylog_1&gt;.
2019-01-14T15:29:24.234+01:00 INFO [Indices] Successfully created index template graylog-internal
2019-01-14T15:29:24.441+01:00 INFO [MongoIndexSet] Waiting for allocation of index &lt;graylog_1&gt;.
2019-01-14T15:29:24.527+01:00 INFO [MongoIndexSet] Index &lt;graylog_1&gt; has been successfully allocated.
2019-01-14T15:29:24.527+01:00 INFO [MongoIndexSet] Pointing index alias &lt;graylog_deflector&gt; to new index &lt;graylog_1&gt;.
2019-01-14T15:29:24.589+01:00 INFO [SystemJobManager] Submitted SystemJob &lt;ca640bb0-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2019-01-14T15:29:24.589+01:00 INFO [MongoIndexSet] Successfully pointed index alias &lt;graylog_deflector&gt; to index &lt;graylog_1&gt;.
2019-01-14T15:29:54.826+01:00 INFO [SetIndexReadOnlyJob] Flushing old index &lt;graylog_0&gt;.
2019-01-14T15:29:55.002+01:00 INFO [SetIndexReadOnlyJob] Setting old index &lt;graylog_0&gt; to read-only.
2019-01-14T15:29:55.155+01:00 INFO [SystemJobManager] Submitted SystemJob &lt;dc9c3320-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2019-01-14T15:29:55.222+01:00 INFO [OptimizeIndexJob] Optimizing index &lt;graylog_0&gt;.
2019-01-14T15:29:55.255+01:00 INFO [SystemJobManager] SystemJob &lt;dc9c3320-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 99ms.
2019-01-14T15:29:55.284+01:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_0.
2019-01-14T15:29:55.551+01:00 INFO [MongoIndexRangeService] Calculated range of [graylog_0] in [260ms].
2019-01-14T15:29:55.626+01:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog_0.
2019-01-14T15:29:55.626+01:00 INFO [SystemJobManager] SystemJob &lt;ca640bb0-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 1034ms.

That’s why I’m sure the problem lies the bootstrap


(Jan Doberstein) #12

As you are the first person that ever came up in this community with this issue, I guess that this is more a local problem.

Maybe someone who has the same configuration can provide his role configuration to you.