Unable to receive logs from cisco devices

Graylog Central (peer support)
1

I have configured graylog. It worked fine for 3 days and from past 3 days it is not receiving any logs from cisco devices. I have even tried adding new devices but still i am unable to get any logs. I am using UDP txt method to collect the data. Please support asap.

2019-04-09T04:58:11.160Z WARN [Messages] Failed to index message: index=<graylog_2> id=<12cd0601-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:11.195Z WARN [Messages] Failed to index message: index=<graylog_2> id=<12cd0602-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:11.195Z ERROR [Messages] Failed to index [3] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:13.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<1363eed0-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:13.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:47.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<2849d120-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:47.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:56.300Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-04-09T04:59:37.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<46173620-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:59:37.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:00:27.159Z WARN [Messages] Failed to index message: index=<graylog_2> id=<63e49b20-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:00:27.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:01:17.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<81b1d911-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:01:17.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:02:07.159Z WARN [Messages] Failed to index message: index=<graylog_2> id=<9f800160-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:02:07.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:02:57.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:02:57.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:03:47.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:03:47.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:04:37.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:04:37.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
@@@

i am receiving these errors please support to resolve.

2

Have you tried it?
Login to graylog and check system overview.
Eg. notifications, index failures, elasticsearch cluster status.
First tip, you fill up your disk…
Also the full log (with end of the lines) contains more information.

3

Either your ES disk is full or your index is read-only. Hard to tell since you didn’t copy the complete error message…

4

sir there are not alerts on web portal. Moreover, the disk space is as follow: ubuntu@graylog:~ df -h Filesystem Size Used Avail Use% Mounted on udev 22G 0 22G 0% /dev tmpfs 4.4G 764K 4.4G 1% /run /dev/mapper/graylog--vg-root 19G 9.8G 7.5G 57% / tmpfs 22G 0 22G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 22G 0 22G 0% /sys/fs/cgroup tmpfs 4.4G 0 4.4G 0% /run/user/1000 ubuntu@graylog:~

the log file only contains the logs i posted nothing else could be found :frowning:

5

2019-04-09T04:58:11.160Z WARN [Messages] Failed to index message: index=<graylog_2> id=<12cd0601-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:11.195Z WARN [Messages] Failed to index message: index=<graylog_2> id=<12cd0602-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:11.195Z ERROR [Messages] Failed to index [3] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:13.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<1363eed0-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:13.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:47.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<2849d120-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:58:47.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T04:58:56.300Z ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2019-04-09T04:59:37.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<46173620-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T04:59:37.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:00:27.159Z WARN [Messages] Failed to index message: index=<graylog_2> id=<63e49b20-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:00:27.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:01:17.158Z WARN [Messages] Failed to index message: index=<graylog_2> id=<81b1d911-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:01:17.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:02:07.159Z WARN [Messages] Failed to index message: index=<graylog_2> id=<9f800160-5a84-11e9-9ca8-00505696f558> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:02:07.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:02:57.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:02:57.159Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:03:47.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:03:47.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2019-04-09T05:04:37.158Z WARN [Messages] Failed to index message: index=<graylog_2> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-04-09T05:04:37.158Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
@@@

6
Timestamp Index Letter ID Error message
11 minutes ago graylog_2 962e4f50-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
11 minutes ago graylog_2 8c517020-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
11 minutes ago graylog_2 8a2875a1-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
11 minutes ago graylog_2 8a0d9aa0-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
11 minutes ago graylog_2 8974b600-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
12 minutes ago graylog_2 78604e10-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
12 minutes ago graylog_2 662ba910-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
12 minutes ago graylog_2 62b43590-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
12 minutes ago graylog_2 62982210-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
12 minutes ago graylog_2 61ff8b90-5b41-11e9-8086-00505696f558 {“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}
7

That should give you an idea.

8

use proper markdown to format your posting will make it far more readable for all!

I would advice to check your ELASTICSEARCH logs now to check WHY the index is read-only.

Sherlock you need to combine yourself and check what will be the next step in the line to get the answer …

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.