Replace timestamp from another indexed field

Graylog Central (peer support)
1

Description of your problem

I am a newbie to Graylog. I have graylog/mongodb/es running in docker using docker-compose.yaml (listed). I have WebSphere Liberty writing logs in JSON format. I used filebeat to point the liberty log file to the Beats input that I created in Graylog:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /wlp-javaee7-19.0.0.12/usr/servers/browsi-git-maven/logs/messages.log

output.logstash:
 hosts: ["192.168.1.165:5044"]

Description of steps you’ve taken to attempt to solve the issue

In my Beats input, I created a JSON extractor to take the message field (which was already JSON) and extract the keys/values from it. Works great.

Now, I would like to take one of those extracted fields, namely:

  • ibm_datetime

and override the timestamp field in Graylog.

What is the best way of overriding the timestamp field with one of the JSON indexed fields?

ibm_datetime fields has the following format:

2021-09-08T19:32:22.707-0400

Operating system information

  • macOS Big Sur
  • Docker

Package versions

version: '3'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:4.2
    networks:
      - graylog
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 1g
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.1
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    networks:
      - graylog
    restart: always
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # Beats TCP
      - 5044:5044
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
networks:
  graylog:
    driver: bridge
2

Looks like I should use pipeline rule with set_field for timestamp field.

3

Hello @corporate_gadfly

I would have to agree with you on the pipeline. Maybe something like this might help.

rule “ibm_datetime”
when
    true
then
    let new_date = parse_date(to_string($message.http_time), “yyyy-MM-dd’T’HH:mm:ss”, "CET,UTC,etc...");
    set_field(“timestamp”, new_date);
end

EDIT: I found this here.

Hope that helps.

4

@gsmith is correct, but use this correct format for your field format 2021-09-08T19:32:22.707-0400

let new_date = parse_date(to_string($message.ibm_datetime), "yyyy-MM-dd'T'HH:mm:ss.SSSZZ");
set_field("timestamp", new_date);
1 Like
5

Thx @shoothub , @gsmith .

Love it when a plan comes together. Will try it out this morning.

1 Like
6

TY, both.

For completeness, I did the following:

  • Since I’m using JSON extractor, I made sure to move Message Filter Chain before Pipeline Processor in System → Configurations
    image
    image697×241 11.5 KB
  • To match the ibm_datetime field, my rule looks like following:
rule "replace timestamp"
when
  has_field("ibm_datetime")
then
  let timestamp = parse_date(to_string($message.ibm_datetime), "yyyy-MM-dd'T'HH:mm:ss.SSSZ");
  set_field("timestamp", timestamp);
end
1 Like
7

@shoothub

Thanks for catching my mistake. I should have known better.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.