Data lake errors?

Graylog Central (peer support)
1

Nice, spent 30 minutes writing a post and the forum destroyed it.

So, short.

What does this mean? Failed to read required dictionary page for id: 2

Data lake created on a local directory: /var/db/dataLake1, owned by graylog:graylog

Graylog directory (journals, etc): /var/db/graylog

Data is being written to /var/db/graylog/data-lake-journal

If I try to preview data lake data, I get “Failed to read required dictionary page for id: 2’

And on the log,

2025-10-29 11:14:09,634 ERROR o.g.p.d.i.q.DataLakeNativeIcebergTableScanQueryService [query-engine-data-lake-jobs-1] Error reading data from Data Lake
java.lang.IllegalStateException: Failed to read required dictionary page for id: 2
at org.apache.iceberg.parquet.ParquetDictionaryRowGroupFilter$EvalVisitor.dict(ParquetDictionaryRowGroupFilter.java:422)
at org.apache.iceberg.parquet.ParquetDictionaryRowGroupFilter$EvalVisitor.ltEq(ParquetDictionaryRowGroupFilter.java:228)
at org.apache.iceberg.parquet.ParquetDictionaryRowGroupFilter$EvalVisitor.ltEq(ParquetDictionaryRowGroupFilter.java:81)
at org.apache.iceberg.expressions.ExpressionVisitors$BoundExpressionVisitor.predicate(ExpressionVisitors.java:156)
at org.apache.iceberg.expressions.ExpressionVisitors.visitEvaluator(ExpressionVisitors.java:390)
at org.apache.iceberg.parquet.ParquetDictionaryRowGroupFilter$EvalVisitor.eval(ParquetDictionaryRowGroupFilter.java:119)
at org.apache.iceberg.parquet.ParquetDictionaryRowGroupFilter.shouldRead(ParquetDictionaryRowGroupFilter.java:75)
at org.apache.iceberg.parquet.ReadConf.(ReadConf.java:109)
at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:74)
at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:94)
at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:40)
at java.base/java.lang.Iterable.spliterator(Iterable.java:101)
at com.google.common.collect.Streams.stream(Streams.java:69)
at org.graylog.plugins.datalake.iceberg.query.DataLakeNativeIcebergTableScanQueryService.icebergIterableToRows(DataLakeNativeIcebergTableScanQueryService.java:220)
at org.graylog.plugins.datalake.iceberg.query.DataLakeNativeIcebergTableScanQueryService.query(DataLakeNativeIcebergTableScanQueryService.java:174)
at org.graylog.plugins.datalake.iceberg.query.DataLakeNativeIcebergTableScanQueryService.query(DataLakeNativeIcebergTableScanQueryService.java:100)
at org.graylog.plugins.datalake.iceberg.search.DataLakeQueryBackend.lambda$executeQuery$0(DataLakeQueryBackend.java:76)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at org.graylog.plugins.datalake.iceberg.search.DataLakeQueryBackend.executeQuery(DataLakeQueryBackend.java:85)
at org.graylog.plugins.datalake.iceberg.search.DataLakeQueryBackend.doRun(DataLakeQueryBackend.java:56)
at org.graylog.plugins.datalake.iceberg.search.DataLakeQueryBackend.doRun(DataLakeQueryBackend.java:31)
at org.graylog.plugins.views.search.engine.QueryBackend.run(QueryBackend.java:94)
at org.graylog.plugins.views.search.engine.QueryEngine.prepareAndRun(QueryEngine.java:162)
at org.graylog.plugins.views.search.engine.QueryEngine.lambda$execute$5(QueryEngine.java:131)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2025-10-29 11:14:10,790 WARN o.g.p.d.i.i.l.MetricsSeekableInput [Finalizer] MetricsLocalSeekableInput was not closed properly!

Graylog 7.0.0.rc.2, FreeBSD 14.3, OpenJDK 21.

As a matter or fact, both /var/db/graylog and /var/db/dataLake1 are different filesystems (ZFS datasets).

Any idea? Thanks.

2

More errors.



Data Lake optimization warning

(triggered an hour ago)

Iceberg optimization failed with message: Can't read value in column [timestamp] required int64 timestamp (TIMESTAMP(MICROS,true)) = 2 at value 0 out of 0 in current page. repetition level: -1, definition level: -1
3

Hello @borjam

It seems both errors point to the same issue, that column 2 (timestamp) appears to have no entries but as why I’m not sure.

What is the storage backend being used here and is this a single node setup?

4

Hi Borjam

Any chance you could export a “Cluster Support Bundle” of your logs from the System > Logging page and attach it here, or email it to us? (tully.elliston@graylog.com)

Cheers

5

Of course, on its way by email.

Thanks!

6

It is a single node set up (with three Opensearch nodes).

The OS is FreeBSD and I created a filesystem data lake. The filesystem is ZFS.

7

I have updated the test instance to 7.0.0 and now I find this which I guess is related.:

Unsupported OS/arch, cannot find /freebsd/amd64/libzstd-jni-1.5.7-5.so or load zstd-jni-1.5.7-5 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.7-5 in your system.

Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/db/graylog/data/libnative/libzstd-jni-1.5.7-51597341120340121630.so: /var/db/graylog/data/libnative/libzstd-jni-1.5.7-51597341120340121630.so: mmap of data failed: Permission denied

no zstd-jni-1.5.7-5 in java.library.path: /usr/java/packages/lib/amd64:/usr/lib:/usr/local/lib

Unsupported OS/arch, cannot find /freebsd/amd64/libzstd-jni-1.5.7-5.so or load zstd-jni-1.5.7-5 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.7-5 in your system. [in thread "DataLakeJournalReader"]

Does it mean that FreeBSD is no longer supported? I am running OpenJDK 21 but I don’t jave JNI bindings for the zstd libraries.

I guess I could try to compile it, is it a prerequisite now?

8

Hey @borjam,

Are you using the instance of Java that ships with Graylog or are you defining a different instance within the Graylog java options?

9

As I am running it on FreeBSD and the supplied Java VM is compiled on Linux, I am using a Java VM compiled for FreeBSD.

Right now it is OpenJDK 21.

I hadn’t added any additional Java libraries because they are not mentioned in the installation requirements.

After the error message I saw on the log message I have added now the zstd-jni binary version for FreeBSD-amd64 from Maven Central and now I don’t see complains about the missing library.

I have downloaded the JVM bundled version in order to check for any additional libraries it might need.

Thank you, I wasn’t aware that it might need others.