Cannot search sub string in message on v6.1.1

Graylog Central (peer support)
1

1. Describe your incident:

Noticed this issue when started getting false positives for an alert.

For example I have these logs:

2024-10-31_11-22
2024-10-31_11-221019×522 54.7 KB

If I search for an exact message I have no problem:
message:"ports: port F16 is now off-line"

2024-10-31_11-25
2024-10-31_11-25250×315 20 KB

Now if I attempt to search a sub-string of the whole message I get zero results. I’ve tried all words inside the message ports:, port F16, is now, off-line… nothing works…

2. Describe your environment:

  • OS Information: Rocky linux 9
  • all apps under docker
  • using opensearch
{
  "name" : "node2",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "lyOdd7lWS3CCpcugthCmbQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.17.1",
    "build_type" : "tar",
    "build_hash" : "1893d20797e30110e5877170e44d42275ce5951e",
    "build_date" : "2024-09-26T21:59:32.078798875Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
{
  "cluster_name" : "opensearch-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 51,
  "active_shards" : 152,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

  • mongo v5.0.29

  • Service logs, configurations, and environment variables:

Env variables:

      GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/config/node-id"
      GRAYLOG_ROOT_USERNAME: "administrator"
      GRAYLOG_ROOT_EMAIL: "inbox@domain.net"
      GRAYLOG_ROOT_TIMEZONE: "UTC"
      GRAYLOG_ALLOW_HIGHLIGHTING: true
      GRAYLOG_ROTATION_STRATEGY: "count"
      GRAYLOG_RETENTION_STRATEGY: "delete"
      GRAYLOG_ELASTICSEARCH_MAX_DOCS_PER_INDEX: 20000000
      GRAYLOG_ELASTICSEARCH_MAX_NUMBER_OF_INDICES: 100
      GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: mail.domain.net
      GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL: "https://${REGION_SHORTNAME}.domain.net"
      GRAYLOG_TRANSPORT_EMAIL_ENABLED: true
      GRAYLOG_TRANSPORT_EMAIL_PORT: 25
      GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: false
      GRAYLOG_TRANSPORT_EMAIL_USE_TLS: false
      GRAYLOG_TRANSPORT_EMAIL_USE_SSL: false
      GRAYLOG_TRANSPORT_EMAIL_FROM_EMAIL: "no-reply@domain.net"
      GRAYLOG_TRANSPORT_SUBJECT_PREFIX: "[graylog]"
      GRAYLOG_TRUSTED_PROXIES: "x.x.x.x/32,x.x.x.y/32"
      GRAYLOG_CONTENT_PACKS_LOADER_ENABLED: true
      GRAYLOG_CONTENT_PACKS_DIR: "/usr/share/graylog/data/contentpacks"
      GRAYLOG_CONTENT_PACKS_AUTO_INSTALL: "dcom-content-pack.json"
      GRAYLOG_SERVER_JAVA_OPTS: "-Djavax.net.ssl.trustStore=/usr/local/openjdk-8/lib/security/cacerts -Xms10g -Xmx10g"

Here is my graylog.conf file

node_id_file = /usr/share/graylog/data/config/node-id
password_secret = blabla
root_password_sha2 = blabla
bin_dir = /usr/share/graylog/bin
data_dir = /usr/share/graylog/data
plugin_dir = /usr/share/graylog/plugin
http_bind_address = 0.0.0.0:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = true
elasticsearch_analyzer = standard
output_batch_size = 5000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 15
outputbuffer_processors = 15
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 5
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
prometheus_exporter_enabled = true
prometheus_exporter_bind_address = 0.0.0.0:9833

3. What steps have you already taken to try and solve the problem?

Tried using a new index set since I noticed that the indices were set to mappings.properties.message.type = keyword I would have thought that maybe it should have been type=text. But graylog auto sets it to keyword so maybe it’s right.

4. How can the community help?

Why can I not search a sub-string of a message?

Any input is appreciated

Thank you,
Dave

2

Issue fixed.

It was the fact that the message type field was wrong. Not sure how it was keyword.

I stopped graylog and opensearch nodes
Purged the data directory of opensearch
Restarted my opensearch cluster
Restarted graylog nodes

Now all my indices have the correct mappings.properties.message.type

{
  "aruba_ap_index_0": {
    "mappings": {
      "message": {
        "full_name": "message",
        "mapping": {
          "message": {
            "type": "text",
            "analyzer": "standard"
          }
        }
      }
    }
  }
}

Which then allows me to do searches of sub-string of a message

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.