OpenSearchException

Graylog Central (peer support)
1

Hi everyone,
In the System - Overview - Indexer failures section I get the following message:

OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]

there are over 4000 records for about a month… Do you have any idea how to fix it? I find similar errors for elastich but nothing related to opensearch.

S.o. Ubuntu 22.04 - Graylog 5.0.5 - OpenSearch 2.5.0 - Mongodb 6.0
I’m not very experienced…
Thank you all for the fantastic support

2

There is a maximum of 1000 fields per index. If you have a ton of different sources, or if your sources have a ton of different fields, you will hit this limit.

The limit can be increased, but the better way to handle it is to split your traffic into different indices. The easiest way is to use streams. Set up a stream rule to separate different types of logs, (eg. FW, Windows, Apache, etc) and route those streams each into their own index. Use index routing in the stream page to do that,

image
image1679×492 108 KB

Details avaialble here: Streams

1 Like
3

You will also need to manually rotate your index to fix the problem on the original default index. Index model. All the way at the bottom of that page.

4

the problem continues.

I found that these errors come from Windows Event Log (GELF UDP) input. I see the logs correctly, but evidently some logs arrive longer than expected.
There is only one windows machine under log.

I use nxlog to send to graylog. Can you suggest me any other alternative?
alternatively can you show me some guides to increase the limit of the fields?

5

Hey @blake86

Like @joe.gross suggested your best bet is to create a different INPUT’s and attach them to a different INDICES, problem solved. There is a very good reason why you get that error

[1000] has been exceeded].

6

Thank you all, I divided the indexes as indicated, but in the index dedicated to Windows log (only 1 pc) the problem persists… Do you have any solutions or suggestions?

7

Unless you are collecting from Windows 11. Try using winlogbeats. I have seen this problem in the past with NXLog.

Something just occurred to me. What config are you using? Have you seen the sample configuration we provide in the Sidecar section of the UI?

Begin Sample Configuration:

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>


<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Input eventlog>
        Module im_msvistalog
        PollInterval 1
        SavePos True
        ReadFromLast True
        
        #Channel System
        #<QueryXML>
        #  <QueryList>
        #   <Query Id='1'>
        #    <Select Path='Security'>*[System/Level=4]</Select>
        #    </Query>
        #  </QueryList>
        #</QueryXML>
</Input>


<Input file>
	Module im_file
	File 'C:\Windows\MyLogDir\\*.log'
	PollInterval 1
	SavePos	True
	ReadFromLast True
	Recursive False
	RenameCheck False
	Exec $FileName = file_name(); # Send file name with each message
</Input>


<Output gelf>
	Module om_tcp
	Host 192.168.1.1
	Port 12201
	OutputType  GELF_TCP
	<Exec>
	  # These fields are needed for Graylog
	  $gl2_source_collector = '${sidecar.nodeId}';
	  $collector_node_id = '${sidecar.nodeName}';
	</Exec>
</Output>


<Route route-1>
  Path eventlog => gelf
</Route>
<Route route-2>
  Path file => gelf
</Route>
8

Here’s one for Winlogbeat. Comment out the first three lines if you’re not using Sidecar or it will cause problems.

Begin Sample Configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["192.168.1.1:5044"]
path:
  data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security
9

Hey @blake86

Unless you created a custum index, Opensearch uses dynamic mapping by defualt. It would help to show your work or any setting you have done.

@joe.gross

Hey good call on the…

I need that, you solved my issue without know it :laughing:

1 Like
10

Thanks for the reply, I had already noticed this configuration, but even with this I get an error OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]. The operating system used for the test is Win11 and Win10. With Win10 Winlogbeat works great with Sidecar.

11

I attach the nxlog configuration.
nxlog.pdf (117.0 KB)

to install graylog i used the official guide: Ubuntu installation

to compile /etc/graylog/server/server.conf and /etc/opensearch/opensearch.yml I used this guide: How To Install Graylog V5 On Ubuntu - YouTube (min 4:35 and 10:03)

Input:
gelf tcp

12

@blake86,

Are you manually rotating your indices each time you make a change? If it’s hit the 1000 field limit, it must be rotated. If the new configuration works as intended, you won’t see that error again.

13

How do you manually rotate an index finger?
Is there a reference guide? Thanks for your precious time!

14

@joe.gross,

Thanks, the solution is sidecar with winlogbeat.
But windows 11 22H2 have problems with sidecar.
Luckily everyone in the company is Windows 10 except my notebook.

Thanks to you and everyone for the fantastic support.

15

Sorry for not replying earlier, I missed this before. To manually rotate an index, go to System/Indices > Index Name. There will be a Maintenance button on the top right. Click that and Manually Rotate Index is one of the options.

Glad you solved it.

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.