Cloudtrail input failure after 4.2.9 update

Graylog Central (peer support)
1

I noticed this after updating to 4.2.9 today (unsure if this was a prior issue, but unlikely). I have 3 nodes in a cluster with 1 being a voting-only. Master node works fine and all inputs start with no issue

2022-05-24T10:57:43.923-07:00 INFO  [CloudTrailTransport] Starting cloud trail subscriber
2022-05-24T10:57:43.924-07:00 INFO  [InputStateListener] Input [AWS CloudTrail/61955be4763c9f3c3d36f5f4] is now STARTING
2022-05-24T10:57:44.064-07:00 INFO  [InputStateListener] Input [AWS CloudTrail/61955be4763c9f3c3d36f5f4] is now RUNNING

Second node is having issues that seem to be java related. All components in the cluster have identical versions including java

org.graylog2.shared.inputs.NoSuchInputTypeException: There is no input of type <org.graylog.aws.inputs.cloudtrail.CloudTrailInput> registered.
	at org.graylog2.shared.inputs.MessageInputFactory.create(MessageInputFactory.java:41) ~[graylog.jar:?]
	at org.graylog2.inputs.InputServiceImpl.getMessageInput(InputServiceImpl.java:381) ~[graylog.jar:?]
	at org.graylog2.inputs.InputEventListener.startInput(InputEventListener.java:107) [graylog.jar:?]
	at org.graylog2.inputs.InputEventListener.inputCreated(InputEventListener.java:74) [graylog.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87) [graylog.jar:?]
	at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144) [graylog.jar:?]
	at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Input dashboard shows the same

Unknown Input (org.graylog.aws.inputs.cloudtrail.CloudTrailInput)

Everything was updated via dnf and md5sums for all jar files match. Not really sure how to go about troubleshooting the root cause.

2

Hello && Welcome

AWS input working? if so are you able to see this INPUT configured on this second node?
It seams that there is something missing or the node with the issue didn’t get updated correctly.
By chance is this only one node having issue?
Have you tried to recreate AWS input?

3

it is the only node having the issue. the last commit to this class is here: Load AWS plugin on all nodes by thll · Pull Request #652 · Graylog2/graylog-plugin-aws · GitHub
it kind of sheds some light on it i guess. once i downgraded back to 4.0.16 the issue went away, but 4.2.5+ produce the problem, though i havent tried all patches in 4.2.
All other AWS inputs for Kinesis/Cloudwatch work fine in 4.2.9

I did try to recreate it but unless its on the Master node if fails, however, a temporary workaround seems to be only having it run on the Master node, but that doesnt seem ideal.

I upgraded to 4.3 since that pull request mentioned it would be released then, but that ended up causing all kinds of MongoDB issues which resulted in me having to rollback from snapshots…once i finally got it back up and running i called it a day.

4

Thanks for the added feed back.

I completed understand, this is starting to sound like a bug but if the other node (#3) is not having this issue I’m wondering why only the #2 node is…

Just an idea, maybe upgrade in order → 4.0 → 4.1 then to 4.2.

5

I completed understand, this is starting to sound like a bug but if the other node (#3) is not having this issue I’m wondering why only the #2 node is…

to clarify i have 3 nodes. 1 master, 1 slave, and 1 voting-only (ES and Mongo only)

Just an idea, maybe upgrade in order → 4.0 → 4.1 then to 4.2.

just discovered going from 4.0.16 to 4.1.0-4 produces the same issue…now im really confused at whats happening.

6

Hello,
I was going to ask about some spec’s about this environment but I assume its been working correct. Perhaps ask this question here.

To be honest I have not had this happen. One of my large setups is 6 node cluster

  • Node#1 Graylog /MongoDb (master)
  • Node#2 Graylog /MongoDb
  • Node#3 Graylog /MongoDb
  • Nodes# 4-6 Elasticsearch cluster

Sorry I cant be more help, but maybe someone else has had this happen.

7

no worries. i appreciate the replies anyway. i caved and just have that one input running on a single node for now until i can dig deeper.

1 Like
8

Hello,
Just an FYI, I found this in the doc’s here .
This part:

is_leader = true

If you are running more than one instance of Graylog server you have to select only one graylog-servernode as the leader.
This node will perform periodical and maintenance actions that replica nodes won’t.
Replica nodes will accept messages just the same as leader nodes.
Nodes will fall back to replica mode if there already is a leader in the cluster.

Looks like they changed is_master to is_leader

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.