Creating a new pipeline for a new field

Graylog Central (peer support)
1

I’m still in the process of learning how to craft pipeline rules, and I could really use some assistance. Currently, I’m ingesting Android logs into Graylog for analysis purposes. My goal is to enrich these logs by creating additional fields. This way, when I’m using a dashboard, I can easily sort and filter by these fields.

To clarify, when I say “creating a field,” I mean adding new fields to each log entry. For example:

  • Android_ID

This is necessary because within the logs, there are various code IDs like:

  • E/TrafficController(166)
  • D/AppLovinSdk(20295)
  • E/TrafficController(166)

Below, I’ve provided some examples of the logs I’m working with:

03-26 10:41:03.099 D/AppLovinSdk(20295): [SessionTracker] Application Paused
03-26 10:41:03.101 D/AppLovinSdk(20295): [AppLovinEventService] Tracking event: "paused" with parameters: {}
03-26 10:41:03.118 D/AppLovinSdk(20295): [PersistentPostbackManager] Wrote updated postback queue to disk.
03-26 10:41:03.118 D/AppLovinSdk(20295): [PersistentPostbackManager] Enqueued postback: PostbackRequest{uniqueId='c68694e4-6a1e-4bd9-a7aa-22897abba387', communicatorRequestId='null', httpMethod='null', targetUrl='https://rt.applovin.com/4.0/pix', backupUrl='https://rt.applvn.com/4.0/pix', attemptNumber=0, isEncodingEnabled=true, isGzipBodyEncoding=false}
03-26 10:41:03.118 D/AppLovinSdk(20295): [PersistentPostbackManager] Preparing to submit postback...PostbackRequest{uniqueId='c68694e4-6a1e-4bd9-a7aa-22897abba387', communicatorRequestId='null', httpMethod='null', targetUrl='https://rt.applovin.com/4.0/pix', backupUrl='https://rt.applvn.com/4.0/pix', attemptNumber=0, isEncodingEnabled=true, isGzipBodyEncoding=false}
03-26 10:41:03.126 D/AppLovinSdk(20295): [PersistentPostbackManager] Wrote updated postback queue to disk.
03-26 10:41:03.127 I/AppLovinSdk(20295): [ConnectionManager] Sending POST request to id=#1086116902 "rt.applovin.com/4.0/pix"...
03-26 10:41:03.127 D/AppLovinSdk(20295): [ConnectionManager] Request to #1086116902 "rt.applovin.com/4.0/pix" is 1:96943fdea087a1d759e08734fce0068af23ef96e:m9PHRQa9f46yMTIcLTDNuyRVKGm3E0Qn77ef89dwtC1i-Dr2C_GoGE:2_08v1wKLRNL6QIrlhB5IRwsAazK5ELg8PbUCK22jDuouxks-XFUZ03jVplprvM7
03-26 10:41:03.237 I/AppLovinSdk(20295): [ConnectionManager] Successful POST returned 200 in 0.11 s over wifi to #1086116902 "rt.applovin.com/4.0/pix"
03-26 10:41:03.237 D/AppLovinSdk(20295): [ConnectionManager] 1:96943fdea087a1d759e08734fce0068af23ef96e:m9PHRQa9f46yMTIcLTDNuyRVKGm3E0Qn77ef89dwtC1i-Dr2C_GoGE:BP08v1wKLROO1Xs_I_ccFKaCohEdQYr6guSQIVKpVC_xubulrTEIO7DUMPX_YEkhpRrZ5eVsC_xp9nerZKtYCVcY0BT-3TRDNxm9ewnL23GGsn070MYC5g**
03-26 10:41:03.245 D/AppLovinSdk(20295): [PersistentPostbackManager] Wrote updated postback queue to disk.
03-26 10:41:04.089 D/AppLovinSdk(20295): [TaskCollectSignals] Running signal collection for SignalProviderSpec{adObject={"name":"LINKEDIN_DSP","server_parameters":{"sdk_key":"15550566edbcae560bac1ccdaa33392eeb80cd9b1d4797a52fdccfc3ea114d15"},"class":"com.applovin.dsp.adapters.LinkedInDspAdapter","adapter_timeout_ms":1000,"max_signal_length":32768,"scode":"2!v3!380931.1711420862196!EMjzK8FNmGjVh7IxnjwCTLCmaiaLcmpekfPBaZEufto*","run_on_ui_thread":false}} on the background thread
03-26 10:41:04.089 D/AppLovinSdk(20295): [MediationAdapterManager] Not attempting to load LINKEDIN_DSP due to prior errors
03-26 10:41:04.089 D/AppLovinSdk(20295): [TaskCollectSignals] Running signal collection for SignalProviderSpec{adObject={"name":"FACEBOOK_NETWORK","server_parameters":{"placement_ids":["381912867904016_381922467903056"]},"class":"com.applovin.mediation.adapters.FacebookMediationAdapter","adapter_timeout_ms":30000,"max_signal_length":32768,"scode":"2!v3!380931.1711420862196!29DYTqII9Jld2XisdZyAbtzNrPuc1fbt9viiqcOg2o8*","run_on_ui_thread":false}} on the background thread
03-26 10:41:04.089 D/AppLovinSdk(20295): [MediationAdapterManager] Not attempting to load FACEBOOK_NETWORK due to prior errors
03-26 10:41:04.089 D/AppLovinSdk(20295): [TaskCollectSignals] Running signal collection for SignalProviderSpec{adObject={"name":"MINTEGRAL_NATIVE_BIDDING","server_parameters":{"app_id":"134229","app_key":"e9f5a420e488ee97938c67164b1b35af","credentials":{"98bae13c884c9524":{"ad_unit_id":"328548","placement_id":"231395"}},"placement_id":"231395"},"class":"com.applovin.mediation.adapters.MintegralMediationAdapter","adapter_timeout_ms":3000,"max_signal_length":32768,"scode":"2!v3!380931.1711420862196!biXKpbKJi_9Z8YtYI5hSgN6HDSOvEz172MQciTxZ8nzM0PhBFDopcKFkIwfm5jBp","run_on_ui_thread":false}} on the background thread
03-26 10:41:04.089 D/AppLovinSdk(20295): [MediationAdapterManager] Loaded MINTEGRAL_NATIVE_BIDDING
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationAdapterManager] Loaded MINTEGRAL_NATIVE_BIDDING
03-26 10:41:04.090 I/AppLovinSdk(20295): [MediationAdapterInitializationManager] Initializing adapter SignalProviderSpec{adObject={"name":"MINTEGRAL_NATIVE_BIDDING","server_parameters":{"app_id":"134229","app_key":"e9f5a420e488ee97938c67164b1b35af","credentials":{"98bae13c884c9524":{"ad_unit_id":"328548","placement_id":"231395"}},"placement_id":"231395"},"class":"com.applovin.mediation.adapters.MintegralMediationAdapter","adapter_timeout_ms":3000,"max_signal_length":32768,"scode":"2!v3!380931.1711420862196!biXKpbKJi_9Z8YtYI5hSgN6HDSOvEz172MQciTxZ8nzM0PhBFDopcKFkIwfm5jBp","run_on_ui_thread":false}}
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationAdapterWrapper] MintegralMediationAdapter: running initialize...
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationAdapterWrapper] Initializing MintegralMediationAdapter on thread: Thread[AppLovinSdk:auxiliary_operations:GoGE,10,main] with 'run_on_ui_thread' value: false
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationAdapterWrapper] MintegralMediationAdapter: finished initialize
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationService] Collecting signal for adapter: MINTEGRAL_NATIVE_BIDDING
03-26 10:41:04.090 D/AppLovinSdk(20295): [MediationAdapterWrapper] MintegralMediationAdapter: running collect_signal...
03-26 10:41:04.090 I/AppLovinSdk(20295): [MintegralMediationAdapter] Collecting signal...
03-26 10:41:03.390 D/My Ads  (20557): My Ads set network applovin
03-26 10:41:03.390 D/My Ads  (20557): My Ads Success : ca-app-pub-3940256099942544/9214589741 | true | 3 | 2 | 1 | 
03-26 10:41:03.391 D/My Ads  (20557): My Ads Success : ca-app-pub-3940256099942544/1033173712 | true | 2 | 2 | 1 | 
03-26 10:41:03.392 D/My Ads  (20557): My Ads Success : ca-app-pub-8147305324909118/8119441288 | true | 2 | 2 | 1 | 
03-26 10:41:03.394 D/My Ads  (20557): My Ads Success : ca-app-pub-8147305324909118/4599832826 | true | 2 | 2 | 1 | 
03-26 10:41:03.395 D/My Ads  (20557): My Ads Success : 58f4deb582dae58d | true | 2 | 2 | 1 | 
03-26 10:41:03.395 D/My Ads  (20557): My Ads Success : 835c16467a123a94 | true | 2 | 2 | 1 | 
03-26 10:41:03.491 I/FA-Ads  (20295): Application backgrounded at: timestamp_millis: 1711420861483
03-26 10:41:03.073 E/TrafficController(166): Failed to delete the counterSet: Function not implemented
03-26 10:41:03.073 E/TrafficController(166): Failed to delete the counterSet: Function not implemented

The below is my pipeline code:

rule "android_new_field"
when
  has_field("message")
then
  let message_field = to_string($message.message);
  let pattern = "^%{TIMESTAMP_ISO8601} %{GREEDYDATA:New_field_A}:";
  let matches = grok(pattern, message_field);
  set_fields(matches);
end
2

Hey @wilsonshow

I assume that GROK pipeline doesn’t work? What do you see?

3

Hey @gsmith

There is no new field appear.

I try a new pipeline to check on a message with this pipeline rule :

rule "D/AppLovinSdk"
when
  contains(to_string($message.message), "D/AppLovinSdk")
then
  let sdk_name = regex("D/AppLovinSdk\\(\\d+\\)", to_string($message.message));
  set_field("Android_ID", "D/AppLovinSdk(20295)");
end

It working. But is some how cheating.
I have created a grok pattern on my graylog name “Android_Log”:

\w\/\w+\s*\(\d+\)

but I am unsure on how to put it into the pipeline.

Hopfully you or someone will have some answer to it.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.