Error Massage - limit fields problem

Graylog Central (peer support)
,
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Error Massage

OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]

2. Describe your environment:

  • OS Information: AlmaLinux

  • Package Version: Graylog 6.3.5+ & Datanode

  • Service logs, configurations, and environment variables:

    image
    image1573×353 33.1 KB

3. What steps have you already taken to try and solve the problem?

I try to run below command to extend fields, but it is no any respone.

curl -X PUT “localhost:9200/graylog_14/_settings” -H ‘Content-Type: application/json’ -d ‘{ “index.mapping.total_fields.limit”: 5000 }’

curl: (52) Empty reply from server

4. How can the community help?

Can anyone help me?
On the other hand, I’m wondering if the problem is with my Datanode installation causing my curl 9200 to fail?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hello,

Does a simple GET work ? curl “localhost:9200/graylog_14/_settings”

Maybe add “-v” to curl to get more details.

I handle this settings with a template to apply it to all indices (only the new ones).

Finally I recommend to split your logs in separate indices, for example one for windows, one for firewalls… And maybe you can rotate your indices more often (for example every 8 hours). It will reduce the number of fields per index.

3

below it is connect detail.

[gladmin@nitcgl ~]$ curl “localhost:9200/graylog_14/_settings” -v

  • Host localhost:9200 was resolved.
  • IPv6: ::1
  • IPv4: 127.0.0.1
  • Trying [::1]:9200…
  • Connected to localhost (::1) port 9200

GET /graylog_14/_settings HTTP/1.1
Host: localhost:9200
User-Agent: curl/8.9.1
Accept: /

  • Request completely sent off
  • Empty reply from server
  • shutting down connection #0
    curl: (52) Empty reply from server
    [gladmin@nitcgl ~]$ curl “https://localhost:9200/graylog_14/_settings” -v
  • Host localhost:9200 was resolved.
  • IPv6: ::1
  • IPv4: 127.0.0.1
  • Trying [::1]:9200…
  • Connected to localhost (::1) port 9200
  • ALPN: curl offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
  • CApath: none
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Request CERT (13):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (OUT), TLS alert, unknown CA (560):
  • SSL certificate problem: unable to get local issuer certificate
  • closing connection #0
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: curl - SSL CA Certificates

This is my first time using Graylog, and I’d like to ask how to route traffic to other indices. I usually find the relevant information in the detfault stream and then route it to other streams using conditions (such as IP). Currently, whenever I syslog to Graylog through the firewall, I immediately get an error message saying “over 1000 filed entries”.

4

Ok so your OpenSearch requires TLS.

Add “-k” to your curl command to skip TLS checks if you don’t have the CA.

To route logs into indices you need to use Streams. When you configure a Stream you can choose to route filtered logs into a specific index and to delete them from the default index.

5

it will be pop up the URL rejected. I didn’t know which problem with datanode….

[root@nitcgl gladmin]# curl -X PUT -k -u admin:password \ “https://localhost:9200/graylog_15/_settings” \ -H ‘Content-Type: application/json’ \ -d ‘{“index.mapping.total_fields.limit”: 5000}’
curl: (3) URL rejected: Malformed input to a URL function
curl: (3) URL rejected: Malformed input to a URL function
curl: (3) URL rejected: Malformed input to a URL function
curl: (3) URL rejected: Malformed input to a URL function
curl: (3) URL rejected: Malformed input to a URL function

and the other hand, the error happen on deault stream, so I can’t route to another index….

6

If you have special characters in the password, you need to use simple quotes:

-u ‘admin:password’

Regarding the default stream you can remove logs from it, when you create a Stream choose an index (other than “Default index set”) and check “Removes matches from ‘Default Stream’.

However if a log matches multiple Streams linked with multiple indices, it will be duplicated in all indices.