Graylog 2.5.1 ElacticSearch X-Pack 6.1 issue

pkaroluk · January 13, 2019, 11:55pm

Hi, is anyone using ElasticSearch 6.1.4 + X-Pack. I have an Access Denied with Graylog 2.5.1 bootstap. It seems that graylog tried to get _all/_aliases which was dropped in elastic 6.1.

graylog user role:

client_service:
  cluster:
  - monitor
  - manage_index_templates
  indices:
  - names: "graylog_*"
    privileges:
    - monitor
    - manage
    - view_index_metadata
    - read_cross_cluster
    - read
    - index
    - create_index
    - delete_index
    - create
    - write
    - delete

|[2019-01-14T00:44:24,776] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[cluster:monitor/health], indices=[_all], request=[ClusterHealthRequest]|
|---|---|
|[2019-01-14T00:44:24,927] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[indices:monitor/settings/get], indices=[graylog_deflector], request=[GetSettingsRequest]|
|[2019-01-14T00:44:24,930] [transport] [access_denied]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[indices:admin/aliases/get], indices=[*,-*], request=[GetAliasesRequest]|
|[2019-01-14T00:44:26,778] [transport] [access_granted]|origin_type=[rest], origin_address=[10.254.0.13], principal=[graylog], action=[cluster:monitor/health], indices=[_all], request=[ClusterHealthRequest]|

macko003 · January 14, 2019, 7:49am

I’m not sure, but the “_all” indices match with your “graylog_*” role?

pkaroluk · January 14, 2019, 9:07am

I tried names: “*” but the result was the same.

pkaroluk · January 14, 2019, 12:16pm

I had dumped all traffic and it looks like Graylog doesn’t perform PUT /_template/graylog-internal during bootstrapping. When I tried to perform it manually as graylog user it works so it’s not a wrong X-Pack configuration.

I checked also

elasticsearch_version=6
elasticsearch_disable_version_check= true

settings but it didn’t help

jan · January 14, 2019, 12:18pm

just something that comes to my mind - Why did you not use the latest stable 6.x release?

pkaroluk · January 14, 2019, 1:36pm

Elastic 6.1.4 is the laster version supported by my puppet module but is not a cause.

My current Server config:

allow_highlighting = true
allow_leading_wildcard_searches = true
content_packs_auto_load = grok-patterns.json
content_packs_dir = /usr/share/graylog-server/contentpacks
elasticsearch_disable_version_check = true
elasticsearch_hosts = httX://graylog:<SECRET>@10.254.0.10:9200
elasticsearch_index_prefix = graylog
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 30
elasticsearch_replicas = 1
elasticsearch_shards = 4
elasticsearch_template_name = graylog-internal
gc_warning_threshold = 1s
http_connect_timeout = 5s
http_read_timeout = 10s
http_write_timeout = 10s
inputbuffer_processors = 2
inputbuffer_ring_size = 131072
inputbuffer_wait_strategy = blocking
is_master = true
ldap_connection_timeout = 2000
message_journal_dir = /var/lib/graylog-server/journal
message_journal_enabled = true
message_journal_max_size = 10gb
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
mongodb_uri = mongodb://mongo:mongo@10.10.0.1:27017/graylog
mongodb_useauth = true
node_id_file = /etc/graylog/server/node-id
output_batch_size = 500
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
output_flush_interval = 1
outputbuffer_processors = 4
password_secret = <SECRET>
plugin_dir = /usr/share/graylog-server/plugin
processbuffer_processors = 4
processor_wait_strategy = blocking
proxied_requests_thread_pool_size = 32
rest_enable_gzip = true
rest_enable_tls = false
rest_listen_uri = httX://10.254.0.13:9000/api/
rest_transport_uri = httX://10.254.0.13:9000/api/
retention_strategy = delete
ring_size = 131072
root_password_sha2 = <SECRET>
root_timezone = Europe/Warsaw
root_username = admin
rotation_strategy = count
transport_email_enabled = true
transport_email_from_email = graylog@example.com
transport_email_hostname = smtp.example.com
transport_email_port = 25
transport_email_subject_prefix = [graylog]
transport_email_use_auth = false
transport_email_use_ssl = false
transport_email_use_tls = false
transport_email_web_interface_url = httX://graylog.example.com
trusted_proxies = 127.0.0.1/32,10.254.0.13/32
web_enable = true
web_enable_cors = false
web_enable_gzip = true
web_enable_tls = false
web_endpoint_uri = httX://graylog.example.com
web_listen_uri = httX://10.254.0.13:9000/

(due forum limitation http was replaced by httX)

jan · January 14, 2019, 2:00pm

let me clarify - you use elasticsearch plugin to restrict access to elasticsearch. That is not working.

without knowing that is actually in your logs of Graylog, Elasticsearch and may an additional audit log of the xpack nobody will be able to help.

Did you disable xpack security and checked if everything is working? Just to make sure the issue is only with the the authentication.

pkaroluk · January 14, 2019, 3:18pm

With xpack.security.enabled=false during graylog bootstrap (from tcpdump):

1. GET /_cluster/health/_all?local=true&timeout=60s -> 200 OK
2. GET /_cluster/health/graylog_*?timeout=60s -> 200 OK
3. GET /_cat/nodes?h=name%2Chost%2Cip%2CfileDescriptorMax&full_id=true&format=json -> 200 OK
4. PUT /_template/graylog-internal -> 200 OK
5. PUT /graylog_0 -> 200 OK
6. GET /_cluster/health/graylog_0?wait_for_status=yellow -> 200 OK
7. POST /_aliases -> 200 OK
8. GET /graylog_*/_alias?expand_wildcards=open -> 200 OK
9. POST /_bulk -> 200 OK

Than I turned on X-Pack and dump looks like:

1. GET /_cluster/health/_all?local=true&timeout=60s -> 200 OK
2. GET /_cluster/health/graylog_*?timeout=60s -> 200 OK
3. GET /graylog_deflector/_settings -> 404 Not Found
4. GET /graylog_*/_alias?expand_wildcards=open -> 404 Not Found
5. GET /_cat/nodes?h=name%2Chost%2Cip%2CfileDescriptorMax&full_id=true&format=json -> 200 OK
6. GET /_all/_alias -> 404 Not Found
7. POST /_bulk -> 200 OK

As I said It’s not permission issue, all requests I can perform manually as graylog user.
I wonder why graylog doesn’t even try to PUT /_template/graylog-internal or /graylog_0

jan · January 14, 2019, 3:47pm

did you checked your Graylog server.log?

pkaroluk · January 14, 2019, 4:24pm

Server.log:

2019-01-14T14:44:55.368+01:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-01-14T14:44:55.370+01:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
org.graylog2.indexer.IndexNotFoundException: Couldn't collect aliases for index pattern graylog_*

Index not found for query: . Try recalculating your index ranges.
        at org.graylog2.indexer.cluster.jest.JestUtils.buildIndexNotFoundException(JestUtils.java:118) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:84) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:57) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.getIndexNamesAndAliases(Indices.java:316) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getNewestIndexNumber(MongoIndexSet.java:151) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.getNewestIndex(MongoIndexSet.java:146) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:252) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_191]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Than I set the template, index and alias manually and rotate the index from the Graylog.

2019-01-14T15:29:24.141+01:00 INFO [DeflectorResource] Cycling deflector for index set &lt;5c165a96ab04d33ff4a69aa6&gt;. Reason: REST request.
2019-01-14T15:29:24.183+01:00 INFO [MongoIndexSet] Cycling from &lt;graylog_0&gt; to &lt;graylog_1&gt;.
2019-01-14T15:29:24.183+01:00 INFO [MongoIndexSet] Creating target index &lt;graylog_1&gt;.
2019-01-14T15:29:24.234+01:00 INFO [Indices] Successfully created index template graylog-internal
2019-01-14T15:29:24.441+01:00 INFO [MongoIndexSet] Waiting for allocation of index &lt;graylog_1&gt;.
2019-01-14T15:29:24.527+01:00 INFO [MongoIndexSet] Index &lt;graylog_1&gt; has been successfully allocated.
2019-01-14T15:29:24.527+01:00 INFO [MongoIndexSet] Pointing index alias &lt;graylog_deflector&gt; to new index &lt;graylog_1&gt;.
2019-01-14T15:29:24.589+01:00 INFO [SystemJobManager] Submitted SystemJob &lt;ca640bb0-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2019-01-14T15:29:24.589+01:00 INFO [MongoIndexSet] Successfully pointed index alias &lt;graylog_deflector&gt; to index &lt;graylog_1&gt;.
2019-01-14T15:29:54.826+01:00 INFO [SetIndexReadOnlyJob] Flushing old index &lt;graylog_0&gt;.
2019-01-14T15:29:55.002+01:00 INFO [SetIndexReadOnlyJob] Setting old index &lt;graylog_0&gt; to read-only.
2019-01-14T15:29:55.155+01:00 INFO [SystemJobManager] Submitted SystemJob &lt;dc9c3320-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2019-01-14T15:29:55.222+01:00 INFO [OptimizeIndexJob] Optimizing index &lt;graylog_0&gt;.
2019-01-14T15:29:55.255+01:00 INFO [SystemJobManager] SystemJob &lt;dc9c3320-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 99ms.
2019-01-14T15:29:55.284+01:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_0.
2019-01-14T15:29:55.551+01:00 INFO [MongoIndexRangeService] Calculated range of [graylog_0] in [260ms].
2019-01-14T15:29:55.626+01:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog_0.
2019-01-14T15:29:55.626+01:00 INFO [SystemJobManager] SystemJob &lt;ca640bb0-1808-11e9-96d8-0800270c2c16&gt; [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 1034ms.

That’s why I’m sure the problem lies the bootstrap

jan · January 14, 2019, 6:55pm

As you are the first person that ever came up in this community with this issue, I guess that this is more a local problem.

Maybe someone who has the same configuration can provide his role configuration to you.

system · January 28, 2019, 6:55pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog 4.0.1 / 7.10.1 Error only on Graylog Master Graylog Central (peer support)	4	1083	January 28, 2021
Updating Elasticsearch 5.x to 6.8 Graylog Central (peer support)	7	368	January 4, 2023
Graylog not able to communicate secured elasticsearch(Searchguard) Graylog Central (peer support)	5	5843	February 20, 2018
Graylog does not search! Graylog Central (peer support)	15	3499	July 4, 2019
Unusual behaviour while using elasticsearch non super user Graylog Central (peer support)	5	1010	August 12, 2019

Graylog 2.5.1 ElacticSearch X-Pack 6.1 issue

Related topics