Help required to ingest logs contains Java stack traces

Graylog Central (peer support)
1

Hi All,

logs have Stack traces are not ingested into graylog. Below is the example stack trace which didn’t ingested to our graylog system.

2020-12-09T16:50:58.421-08:00 [sm9265vcrmnzkblgxqnufqmhusa4fk8j-4-887194 appperf@example.com]   INFO common.logging.StdOutStdErrLogger - Unable to create /opt/b/logs/heapdumps: No such file or directory
2020-12-09T16:50:58.421-08:00 [sm9265vcrmnzkblgxqnufqmhusa4fk8j-4-887194 appperf@example.com] 2c.example.test 2744354 ERROR dagorlad.common.AppWorker - Get exception from job and persist it by jobRecorder with job id 2885517269286428272
java.lang.OutOfMemoryError: Java heap space
	at com.example.tools.Misc$LongStack.ensureIndex(Misc.java:678) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.tools.Misc$LongStack.push(Misc.java:685) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.tools.Misc$LongStack.add(Misc.java:702) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.graph.SignatureGraph.signaturesToEdges(SignatureGraph.java:318) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.graph.SignatureGraph.signatureGraphData(SignatureGraph.java:352) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.graph.SignatureGraph.signatureGraph(SignatureGraph.java:393) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.dagorlad.grapher.GrapherWorkerService.buildAndUploadTopoModelLandmark(GrapherWorkerService.java:353) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.dagorlad.grapher.GrapherWorkerService.buildGraphHelper(GrapherWorkerService.java:206) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.dagorlad.grapher.GrapherWorker.buildGraph(GrapherWorker.java:484) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_92]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_92]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_92]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_92]
	at com.example.dagorlad.common.DagCallRouter.tryRoutingCall(DagCallRouter.java:59) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.dagorlad.common.BaseWorker.callFunctionInternal(BaseWorker.java:101) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
	at com.example.dagorlad.common.BaseWorker.callFunction(BaseWorker.java:119) ~[example-platform-all-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]

P.S : 1) we are using below GROK pattern for parsing the LongStack

      CUSTOM_LOG %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:issuer}\] %{IPORHOST:remote_addr} %{DATA:unknownid} %{LOGLEVEL:loglevel} (?m)%{GREEDYDATA:message}
  2) Graylog version 4.0.1 is deployed on server

It would be very great, if somebody help me to ingest the above mentioned logs into graylog.

Thanks,
Karthikeyan

2

OutOfMemoryError, means you have low memory. Check RAM of graylog box, and also JVM heap size. Increase RAM or decrease JVM heap size.

3

I don’t think @karthikdevops was referring to the message as originating from Graylog. As I understand the problem, the overall message didn’t get ingested, and so he requires help ingesting the stack trace.

@karthikdevops can you please ensure your code/log messages are surrounded by ``` please?

1 Like
4

@aaronsachs , Thanks for understanding the problem exactly and your solution.
How do I add ``` ? I am using tail plugin of telegraf to ingest the application logs into graylog.

This is my configuration to push the logs using telegraf to graylog.

[[inputs.tail]]
  files = ["/opt/test/logs/**.log"]
  from_beginning = false
  name_override = "2a.example.test"
  grok_custom_pattern_files = []
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_patterns = '''
  CUSTOM_LOG %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:issuer}\] %{IPORHOST:remote_addr} %{DATA:unknownid} %{LOGLEVEL:loglevel} (?m)%{GREEDYDATA:message}
   '''
  grok_timezone = "UTC"
  data_format = "grok"
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.