Google workspace plugin and configuration

[gianluca-valentini]

Hi @gsmith, @tmacgbay
I finally installed and started graylog 5 enterprice with free license (small business).
Configuring Google Workspace input now I can ingest the audit log.

Now I need to read the Gmail log too. I follow the documentation here, but no messages are incoming…

I see some differences from input field name from documentation and Graylog input
For example in documentation I see:

Project ID Alpha-numeric project ID for the Google Cloud project

but in the input there is
Gmail Account User Email

Considering the wrong references to P12 I think that there is something that I wrong in the input configuration.
Can you help me to understand what I shoud set in the input Gmail Account User Mail?
Thanks
Gianluca

[gsmith]

Hey @gianluca-valentini

I havent played with that yet, but do you see any warn/errors in the Log files by chance?

@gianluca-valentini

Considering the wrong references to P12 I think that there is something that I wrong in the input configuration.

In your Google Cloud did you create P12 key? If so then copy that over to Graylog, for testing place it in Graylog Directory /etc/graylog/my_cert.p12

Then use that path in the input see if that works.

Adding on:

@gianluca-valentini

Can you help me to understand what I shoud set in the input Gmail Account User Mail?

This would be the Team account for the company or user account. This is also where you make the Key for the input on Graylog.

https://go2docs.graylog.org/5-0/getting_in_log_data/google_input.html

[gianluca-valentini]

Hi @gsmith
thanks a lot for your answer. I followed the instructions linked in the documentation.
I will test the p12 too.
The problem is that the Gmail input configuration that we can read in the documentation talks about service account json.
Let me test it. May be I will follow the descriptions present in the graylog input instead.

Thanks
Gianluca

[gianluca-valentini]

Hi @gsmith

if i use the p12 file the Gmail input does not start while the Workspace one (that use the same p12) is working fine

I’m confused now

[gsmith]

Hey,

Can graylog access it? Check permissions on the certs, also what does it show in the log/s?

[gianluca-valentini]

Hi,
Graylog can access it. The other input with the same p12 works fine (Google Workspace input)
If I set the p12 file, as described in the input documentation I have the follow error:

2023-03-10T16:16:29.481+01:00 ERROR [GmailTransport] Gmail Client count not be acquired. []
2023-03-10T16:16:29.481+01:00 ERROR [InputLauncher] The [org.graylog.enterprise.integrations.gmail.GmailInput] input with ID <63ff3996760dcc79b767fbbc> misfired. Reason: expected primitive class, but got: class com.google.api.client.json.GenericJson.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: Unable to create Gmail client
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:161) ~[graylog.jar:?]
        at org.graylog.enterprise.integrations.gmail.GmailInput.launch(GmailInput.java:90) ~[?:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:91) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog2.plugin.inputs.MisfireException: Unable to create Gmail client
        at org.graylog.enterprise.integrations.gmail.GmailTransport.doLaunch(GmailTransport.java:78) ~[?:?]
        at org.graylog2.plugin.inputs.transports.ThrottleableTransport.launch(ThrottleableTransport.java:76) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        ... 8 more
Caused by: java.lang.IllegalArgumentException:
        at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[?:?]
        at com.google.api.client.json.JsonParser.parse(JsonParser.java:360) ~[?:?]
        at com.google.api.client.json.JsonParser.parse(JsonParser.java:335) ~[?:?]
        at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79) ~[?:?]
        at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73) ~[?:?]
        at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:548) ~[?:?]
        at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:527) ~[?:?]
        at org.graylog.enterprise.integrations.gmail.external.GmailClientFactory.getClient(GmailClientFactory.java:41) ~[?:?]
        at org.graylog.enterprise.integrations.gmail.GmailTransport.doLaunch(GmailTransport.java:74) ~[?:?]
        at org.graylog2.plugin.inputs.transports.ThrottleableTransport.launch(ThrottleableTransport.java:76) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        ... 8 more

This means tha it required a valid json file (not a p12 one).

Adding the service-account json file I read this error in the log:

INFO  [InputStateListener] Input [Gmail Log Events/63ff3996760dcc79b767fbbc] is now RUNNING
ERROR [GmailClient] Exception Occurred while listing Available Tables in BigQuery Invalid resource name projects/s-audit-logs@xxx-audit.iam.gserviceaccount.com; Project id: s-audit-logs@xxx-audit.iam.gserviceaccount.com
ERROR [BigQueryService] Exception Occurred while deleting tables older than checkpoint timeInvalid resource name projects/s-audit-logs@xxx-audit.iam.gserviceaccount.com; Project id: s-audit-logs@xxx-audit.iam.gserviceaccount.com

What is missing? Is big query something that I have to pay to use it?
Thanks
Gianluca

[gsmith]

Hey @gianluca-valentini

@gianluca-valentini

Is big query something that I have to pay to use it?

I think thats on the Google side but im not 100% sure. Also this is a good read if you have time

Graylog Labs - Ingesting Google Cloud Audit Logs with Graylog

Hey Folks, As this is my first actual post, just to introduce myself, I’m Kyle Pearson and I am a solutions engineer here at Graylog, I’ve been a consultant in the log management and SIEM space for a few years and have seen enough log management solutions in production to probably last me a lifetime, but yet, I get excited every-time I talk to anyone about it. Cloud log sources come up more and more these days, so I recently wrote a blog based on ingesting Google Cloud Audit Logs into Graylog. …

[gianluca-valentini]

Hi @gsmith
thanks a lot. That guide is very good for Google Cloud log ingestion but it can be used to configure Gmail too?

[kpearson (Kyle Pearson)]

So short answer is that it can somewhat be used, as all the Google input stuff works in a similar fashion using log sinks and BigQuery.

Can I just ask, are you seeing events come in from the workspace input or is it just running and receiving no events (if so check the section in the blog about the writer identity)?

Note that the project Id and client Id are the same as those I listed in the blog so your input maybe has the incorrect values currently unless your project Id in GCP is also you email address.