1 logfile for 2 usergroups

[Stefan73 (Stefan Unger)]

I have a logfile with several fields which I parse with GROK.
I have two usergroups, one should see all fields parsed with GROK, the other usergroup should see less fields.
I have created two Streams and connected each to a different pipeline.
I have given usergroup A View-rights on A-stream
and
usergroup B View-rights on B-stream.

Problem is that as soon as I start the pipeline that GROKs the additional fields, they also become visilbe in the other Stream, so all users can see all data.

What is the best way to seperate the two usergroups for this logfile, so each group can only see the data which it should see?

[gsmith]

Hey @Stefan73

Sorry for that later responce, can you show your piplines?

[Stefan73 (Stefan Unger)]

Pipeline for default users that see less fields:

rule "custom cls BIS 6.7 pipeline"
when
   contains("BIS_server_lgw",to_string($message.filebeat_fields_kind)) OR 
   contains("pl-jdbc_lgw",to_string($message.filebeat_fields_kind)) OR
   contains("BIS_audit_lgw",to_string($message.filebeat_fields_kind))
then
  let nocomma = to_string($message.message);
  //GROK message & overwrite ingest timestamp with logfile timestamp
  let groked_message = grok(pattern: "%{TIMESTAMP_ISO8601:logdate}\t ?%{DATA:loglevel}\t ?%{DATA:class}\t ?%{DATA:thread}\t ?(\\[%{DATA:bundle}\\])?\t ?(%{DATA:module})?\t ?(%{DATA:instance})?\t ?(%{DATA:UNWANTED})?\t ?(%{DATA:contextid}?( \\| %{DATA:app} \\| %{DATA:session} \\| %{DATA:requestheaderinfo} \\|)?)\t ?(\\[%{DATA:account}\\] %{BASE10NUM:exectime:float} )?%{GREEDYMULTILINET:message}", value: nocomma, only_named_captures: true);
  set_fields(groked_message);
  set_field("timestamp", parse_date(value: to_string(groked_message.logdate), pattern: "yyyy-MM-dd'T'HH:mm:ss.SSSZ"));

Pipeline for Users that should see all fields:

rule "custom cls BIS 6.7 pipeline with user"
when
   contains("BIS_server_lgw",to_string($message.filebeat_fields_kind)) OR 
   contains("pl-jdbc_lgw",to_string($message.filebeat_fields_kind)) OR
   contains("BIS_audit_lgw",to_string($message.filebeat_fields_kind))
then
  let nocomma = to_string($message.message);
  //GROK message & overwrite ingest timestamp with logfile timestamp
  let groked_message = grok(pattern: "%{TIMESTAMP_ISO8601:logdate}\t ?%{DATA:loglevel}\t ?%{DATA:class}\t ?%{DATA:thread}\t ?(\\[%{DATA:bundle}\\])?\t ?(%{DATA:module})?\t ?(%{DATA:instance})?\t ?(%{DATA:user})?\t ?(%{DATA:contextid}?( \\| %{DATA:app} \\| %{DATA:session} \\| %{DATA:requestheaderinfo} \\|)?)\t ?(\\[%{DATA:account}\\] %{BASE10NUM:exectime:float} )?%{GREEDYMULTILINET:message}", value: nocomma, only_named_captures: true);
  set_field("user", to_string(groked_message.user));
  route_to_stream(name: "BIS with user", remove_from_default: true);