Kubernetes metadata sent by Filebeats but not shown in Graylog

Graylog Central (peer support)
,
1

Hi all,

I have an Azure AKS (Kubernetes 1.11.2), and a filebeat 6.3.2 on each node.

filebeat.yml :

  filebeat.prospectors:
    - type: docker
      containers.ids:
      - "*"
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
                    
  output.file.enabled: false
  output.elasticsearch.enabled: false
  output.logstash:
    hosts: 
    - graylog.monitoring.svc.cluster.local:5044

The filebeat is sending the kubernetes metadata but I don’t see them in Graylog :

{
  "@timestamp": "2018-09-26T12:41:48.373Z",
  "@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.3.2"
  },
  "message": "{2018-09-26T12:41:48+00:00} [DEBUG] [LOG] is4_ip_da_presentation \u0026$ Technical \u0026$ end consuming event 92ae0ac3-0be0-4d23-9977-9
2b1fc713a2e - no subscription corresponding (msg/expected) : GloubiDeclared/TechnicalReferentialCreated \u0026$ null \u0026$ To be defined \u0026$ nul
l \u0026$ E:\\agent\\_work\\57\\s\\src\\Component.Bus\\Subscribe\\AzureSubscriptions.cs 91 \u0026$ null \u0026$ null",
  "source": "/var/lib/docker/containers/c248ed867de4382b8957984da62fc81547a725856d91bd6559de971ef2fc2f63/c248ed867de4382b8957984da62fc81547a725856d91b
d6559de971ef2fc2f63-json.log",
  "kubernetes": {
"pod": {
  "name": "ip-da-7749d86858-crbk2"
},
"node": {
  "name": "aks-agentpool-xxx"
},
"namespace": "ci",
"labels": {
  "pod-template-hash": "3305842414",
  "app": "ip-da"
},
"container": {
  "name": "ip-da"
}
  },
  "meta": {
"cloud": {
  "instance_id": "xxx",
  "instance_name": "aks-agentpool-xxx",
  "machine_type": "Standard_D2s_v3",
  "region": "northeurope",
  "provider": "az"
}
  },
  "offset": 111659,
  "stream": "stdout",
  "input": {
"type": "docker"
  },
  "prospector": {
"type": "docker"
  },
  "beat": {
"name": "filebeat-nd5l4",
"hostname": "filebeat-nd5l4",
"version": "6.3.2"
  },
  "host": {
"name": "filebeat-nd5l4"
  }

I configured a beats input in Graylog with this Grok pattern :

{%{TIMESTAMP_ISO8601:time}} [%{DATA:logLevel}] [%{DATA:code}] %{DATA:caller} &$ %{DATA:logMessageType} &$ %{DATA:message} &$ %{DATA:correlationId} &$ %{DATA:userId} &$ %{DATA:data} &$ %{DATA:operation} &$ %{DATA:error} &$ (?.*)

It splits the fields just fine, I just miss the kubernetes metadata fields.

Graylog : graylog/graylog:2.4.6-1
ES : docker.elastic.co/elasticsearch/elasticsearch:5.6.3
MongoDB : mongo:3

Each deployed as a deployment + ClusterIP service, so each one as its pod.

Graylog yml of the deployment (graylog.conf let as is) :

spec:
  containers:
  - image: "graylog/graylog:2.4.6-1"
    name: graylog
    ports:
      - containerPort: 9000
        name: http
      - containerPort: 5044
        name: tcp
      - containerPort: 12201
        name: gelf
    env:
    - name: GRAYLOG_PASSWORD_SECRET
      value: "redacted"
    - name: GRAYLOG_ROOT_PASSWORD_SHA2
      value: redacted
    - name: GRAYLOG_WEB_ENDPOINT_URI
      value: https :// my-company-dns-graylog-pointing-to-ingress/api
    - name: "GRAYLOG_ELASTICSEARCH_HOSTS"
      value: "h t t p : // elasticsearch-discovery.monitoring.svc.cluster.local : 9200/"
    - name: "GRAYLOT_ELASTICSEARCH_DISCOVERY_ENABLED"
      value: "true"
    - name: "GRAYLOG_MONGODB_URI"
      value: "mongodb :// mongo.monitoring.svc.cluster.local : 27017/graylog"

I saw in a github issue that the beats plugin in Graylog should handle the kubernetes metadata. What is wrong with my conf ?

Thanks.

2

Anyone has an idea ?

3

what if you remove the grok pattern on the input and just let it run? I have never seen issues with the Beats input to create the single fields out of the given information.

4

@jan Thank you for your answer.

If I remove the grok pattern, I only have those fields in Graylog :

  • facility = filebeat
  • file = /var/lib/docker/containers/xxx/xxx-json.log
  • message = {2018-10-02T12:16:33+00:00} [INFO] [LOG] xxx
  • name = filebeat-4ntqb (name of the filebeat pod sending the log)
  • offset = 0
  • source = filebeat-4ntqb
  • timestamp
  • type = null

But the data send by filebeat :

2018-10-02T12:16:33Z	DEBUG	[publish]	pipeline/processor.go:291	Publish event: {
  "@timestamp": "2018-10-02T12:16:33Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.2"
  },
  "stream": "stdout",
  "meta": {
    "cloud": {
      "region": "northeurope",
      "instance_id": "xxx",
      "instance_name": "aks-nodepool1-xxx-0",
      "machine_type": "Standard_D2s_v3",
      "provider": "az"
    }
  },
  "offset": 0,
  "source": "/var/lib/docker/containers/xxx/xxx-json.log",
  "prospector": {
    "type": "docker"
  },
  "input": {
    "type": "docker"
  },
  "kubernetes": {
    "pod": {
      "name": "ip-da-xxx-xxx"
    },
    "node": {
      "name": "aks-nodepool1-xxx-0"
    },
    "namespace": "ci",
    "labels": {
      "pod-template-hash": "xxx",
      "app": "ip-da"
    },
    "container": {
      "name": "ip-da"
    }
  },
  "beat": {
    "version": "6.3.2",
    "name": "filebeat-4ntqb",
    "hostname": "filebeat-4ntqb"
  },
  "host": {
    "name": "filebeat-4ntqb"
  },
  "message": "{2018-10-02T12:16:33+0000} [DEBUG] [LOG] xxx"
}
5

not sure, but that might be an issue with the input on Graylog. Beat input is improved in 3.0 ( https://github.com/Graylog2/graylog-plugin-beats/pull/29 ) what might be the solution for your described behaviour.

If you are able, can you spin up an alpha instance and test it? Docker images are given with the 3.0 tag. Or did you need OS packages?

6

@jan trying to deploy DOcker image 3.0 I got this javascript exception in the UI :

Blocage du chargement du contenu mixte actif (mixed active content) « http://pod.private.ip:9000/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.ee58472ed9a318366aad.js

Where does it search for this IP ? I set the following env in the kubernetes deployment yaml :

env:
- name: GRAYLOG_REST_LISTEN_URI
value: https ://my.public.dns.com/api
- name: GRAYLOG_REST_TRANSPORT_URI
value: https ://my.public.dns.com/api
- name: GRAYLOG_WEB_ENDPOINT_URI
value: https ://my.public.dns.com/api

7

sorry never used kubernetes - the only I can offer is http://docs.graylog.org/en/2.4/pages/installation/docker.html#settings

8

@jan the problem is the config.js behing populated with :

gl2ServerUrl: ‘http://10.1.0.129:9000/api/’

instead of gl2ServerUrl: ‘https://my.public.dns.com/api/’

window.appConfig = {
  gl2ServerUrl: '${serverUri}',
  gl2AppPathPrefix: '${appPathPrefix}',
  rootTimeZone: '${rootTimeZone}',
};
9

@jan Setting

env:

fixed the problem.

10

@jan with Docker image 3.0 (3.0.0-alpha.2+49bdce2) it works as expected.

I got fields kubernetes_container_name, etc… from filebeat.

Thanks a lot !

11

thank you for letting me know - so upcoming versions will have that fixes!

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.