Graylog docker reports blocked by: forbidden

[i4admin (Casper Jensen)]

Hi,
I think this is a easy solution im just stumped as to how.

So i noticed i was running a old version of elasticsearch in my docker-compose file. so i decided to change it to the newst version mentioned in the graylog documentation. When i did that my index got blocked with the following error:

Blockquote
blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
graylog_1 | at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:110) ~[graylog.jar:?]
graylog_1 | at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:60) ~[graylog.jar:?]
graylog_1 | at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
graylog_1 | at org.graylog2.indexer.indices.Indices.cycleAlias(Indices.java:655) ~[graylog.jar:?]
graylog_1 | at org.graylog2.indexer.MongoIndexSet.pointTo(MongoIndexSet.java:357) ~[graylog.jar:?]
graylog_1 | at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:166) ~[graylog.jar:?]
graylog_1 | at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
graylog_1 | at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_252]
graylog_1 | at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
graylog_1 | at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
graylog_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
graylog_1 | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_252]
graylog_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_252]
graylog_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_252]
graylog_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
graylog_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
graylog_1 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

I think it just need to set something somewhere so my index is not blocked anymore. but i can’t find how to do it?

Can anyone help?

[i4admin (Casper Jensen)]

the wierd thing is though that i can search data from 2020-06-22 so apparently not the entire index is forbidden??

here is my docker compose file:

Blockquote
version: ‘3’
services:
MongoDB: Docker
mongo:
image: mongo:3
volumes:
- ./graylog/database/mongo_data:/data/db
networks:
- graylog
Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/d ocker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.5
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1g
networks:
- graylog
Graylog: Docker
graylog:
user: $USER
image: graylog/graylog:3.3
volumes:
Mount local configuration directory into Docker container
- ./graylog/config:/usr/share/graylog/data/config
Mount local plugin file into Docker container
- ./graylog/plugin/:/usr/share/graylog/plugin/
Logs
- ./graylog/Logs/:/var/log/graylog-server/
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=somepasswordpepper
Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4 bb8a81f6f2ab448a918
- GRAYLOG_WEB_ENDPOINT_URI=http://syslog.i4.dk:9000/api
- GRAYLOG_HTTP_EXTERNAL_URI=http://syslog.i4.dk:9000/
networks:
- graylog
dns:
- 8.8.8.8
- 8.8.4.4
depends_on:
- mongo
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
Syslog TCP
- 514:514
Syslog UDP
- 514:514/udp
GELF TCP
- 12201:12201
GELF UDP
- 12201:12201/udp
networks:
graylog:
driver: bridge
Volumes for persisting data, see https://docs.docker.com/storage/ volumes/
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_journal:
driver: local

[i4admin (Casper Jensen)]

Oh and also.
I found this:

How to undo setting Elasticsearch Index to readonly?

Where they suggest to do this:

PUT your_index/_settings
{
  "index": {
    "blocks.read_only": true
  }
}

But i can’t find how to implement this with a graylog solution?
Like with Kibana i can run script code from the web interface, but how am i suppose to do it with graylog. The solution escapes me

[i4admin (Casper Jensen)]

and its not a diskspace issue

[i4admin (Casper Jensen)]

Uh i think i know what i have to do. just need help now getting the syntax right

curl -xpost “http://localhost:9000/_put” -d’
PUT graylog_0/_settings
{
“index”: {
“blocks.read_only”: true
}
}’
curl: (5) Could not resolve proxy: post

am i right?

[i4admin (Casper Jensen)]

ok so i got a little further now… people are proberly thinking by now “if we just let this guy hanging he will figure it out on his own” but i really would like to have this solved quickly

I found the documention for the API browser:
https://docs.graylog.org/en/3.0/pages/configuration/rest_api.html

but when i click the api browser button it redirects me to wrong IP - i will proberly need to check the config

[i4admin (Casper Jensen)]

ok so ip 172.18.0.4 is proberly some docker bridge network ip cause the docker server has a interface with ip 172.18.0.1

not sure how to solve this yet

[i4admin (Casper Jensen)]

ok i’m a little stumped…

so i found these in the graylog.conf

Default: http://$http_bind_address/
http_publish_uri = http://XXX.YYY.ZZ:9000/

and

Default: $http_publish_uri
http_external_uri = http://XXX.YYY.ZZ:9000/

I tried changing them to 0.0.0.0 and the local ip of the docker server and even dns name. all failed.

I think its the right settings im tampering with i just don’t understand what to put in them in order to get my api adress to not use docker brodcast ip but the “right” one.

It won’t load my API in web atleast.

[jan (Jan Doberstein)]

[FORBIDDEN/12/index read-only / allow delete

This message indicate that your Elasticsearch does not allow to write to it.

the Curl command to enable write again needs to be fired against the Elasticsearch API, not Graylog.

[i4admin (Casper Jensen)]

@jan
Thanks for answering. I how been scovering the net since this was also my conclution, i thought the API i found was for everything… apparently only graylog.

I have foudn the put command i need to fire against the elasticsearch but i can’t find how to do it.

can you perhaps help me with this? i know its not a graylog issue but instead of me having to create a account at elaticsearch forum and wait for response i was hopeing you would get going faster?

[jan (Jan Doberstein)]

you should first find the reason for the read only status of elasticsearch. most likely you have not enough disk space available so elasticsearch switched into the high watermark state. But the elasticsearch log should tell you that.

After the reason is solved you can fire the curl command agains the elasticsearch api - in your case you need to open the port via docker compose or jump into a shell inside the container OR add a cerebro container and perform the action with cerebro.

[i4admin (Casper Jensen)]

thanks @jan i will give that a go