AWS CloudWatch Event Sink Implementation #1965

adnanhemani · 2025-06-27T02:21:21Z

As per discussion at this morning's Polaris Community Sync, I am introducing this event listener that will allow users to sink events to AWS CloudWatch Logs - where they can search through events for auditing purposes, if they wish. I do intend to introduce similar changes for Azure and GCP once this is merged so that we have parity across all CSPs - but this will help nail down the exact pattern to make this work.

Below is a screenshot of a sample event sent to CloudWatch:

snazy · 2025-06-27T08:38:21Z

Didn't look into the whole PR, but two things I noticed:

The PR's doing more than mentioned in the summary & description
There's no (integration) testing for the feature (IIRC there are solutions to test cloudwatch locally)

adnanhemani · 2025-06-27T09:43:09Z

Hi @snazy,

The PR's doing more than mentioned in the summary & description

Yes, I'm assuming you're referring to the creation of one new event (onAfterCatalogCreated). It is much faster to use that to rapidly test new event listeners. If it is a sticking point, I'm happy to break that into a separate, few-line PR. WDYT?

There's no (integration) testing for the feature (IIRC there are solutions to test cloudwatch locally)

I didn't want to sink (no pun intended) more time trying to setup LocalStack for this repository if this approach was wildly off-the-mark. If you think the general approach is reasonable and the main things that will need change are implementation details, I will be glad to add that in the next revision. IIRC, this is also something that we only get with AWS - both Azure and GCP equivalents of CloudWatch do not have equivalent testing solutions (this could be outdated information but I can't find anything in a quick Google search). If having parity across CSPs on testing is important, maybe we settle for unit tests instead? Let me know your thoughts.

…ent_listener

adnanhemani · 2025-07-12T12:57:02Z

I've added a test using LocalStack, which should help emulate CloudWatch. This PR is ready for a full review.

runtime/defaults/src/main/resources/application.properties

service/common/src/main/java/org/apache/polaris/service/events/EventListenerConfiguration.java

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java

...e/common/src/test/java/org/apache/polaris/service/events/AwsCloudWatchEventListenerTest.java

...java/org/apache/polaris/service/quarkus/events/QuarkusPolarisEventListenerConfiguration.java

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

snazy

There are some thread-safety/concurrency issues and memory leaks in this change, which have to be addressed.

The implementation can also be simplified.

I'd also recommend to move this to its own module to streamline CI and testing in general.

snazy · 2025-07-17T09:26:28Z

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

+  private static final String DEFAULT_LOG_STREAM_NAME = "polaris-cloudwatch-default-stream";
+  private static final String DEFAULT_LOG_GROUP_NAME = "polaris-cloudwatch-default-group";
+  private static final String DEFAULT_REGION = "us-east-1";


Those hard coded defaults shouldn't be there. Defaulting to an AWS region is not great and will cause user confusion. I do not mind having defaults of the log stream/group name in the Quarkus configuration, but region is awkward IMHO.

I had added the defaults based on your previous advice that things should work "out of the box". Without a default region, this listener will fail for users.

Shifted the defaults to QuarkusPolarisEventListenerConfiguration.

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java

snazy · 2025-07-17T09:43:39Z

...e/common/src/test/java/org/apache/polaris/service/events/AwsCloudWatchEventListenerTest.java

+      LoggerFactory.getLogger(AwsCloudWatchEventListenerTest.class);
+
+  private static final DockerImageName LOCALSTACK_IMAGE =
+      DockerImageName.parse("localstack/localstack:3.4");


Hard coding dependency versions in source isn't good practice. This should be replaced with the container-spec-helper.

Done. I'm not sure if my changes are what you meant, but please let me know if not.

eric-maynard · 2025-07-21T21:08:04Z

...c/main/java/org/apache/polaris/service/events/aws/cloudwatch/AwsCloudWatchEventListener.java

+
+  // Event overrides below
+  @Override
+  public void onAfterTableRefreshed(AfterTableRefreshedEvent event) {


I still think we need to refactor these out so that the listener itself with all its Future-wrangling logic does not become exceedingly long

I've done this on the new revision!

eric-maynard · 2025-07-21T23:16:30Z

...c/main/java/org/apache/polaris/service/events/aws/cloudwatch/AwsCloudWatchEventListener.java

+      return;
+    }
+    InputLogEvent inputLogEvent = createLogEvent(eventAsJson, getCurrentTimestamp(callContext));
+    Future<?> future = executorService.submit(() -> sendAndHandleCloudWatchCall(inputLogEvent));


Two thoughts:

For this listener, I think we should offer a simple synchronous submission mode

As we add more listeners, seems like some of this asynchronous handling logic will be duplicated so it might be worth refactoring it out. This can wait until we add more listeners though.

For this listener, I think we should offer a simple synchronous submission mode

I still disagree with this, to be honest. It will likely cause a sizable slowdown to fast operations. But I can add a mode that does this in-line instead. Adding to the next revision.

As we add more listeners, seems like some of this asynchronous handling logic will be duplicated so it might be worth refactoring it out. This can wait until we add more listeners though.

Sounds good, will do on the next PR.

eric-maynard

I think this is getting really close. Left a couple of comments on the listener structure.

eric-maynard · 2025-07-22T16:21:47Z

...mon/src/main/java/org/apache/polaris/service/events/jsonEventListener/JsonEventListener.java

+
+import java.util.HashMap;
+
+public abstract class JsonEventListener extends PolarisEventListener {


nit: it's more like a MapBasedEventListener right? There's no JSON here

The map will transform into a JSON string when it sinks, which is why I made it this name. But open to change if you think MapBasedEventListener is better?

eric-maynard · 2025-07-22T16:21:55Z

...mon/src/main/java/org/apache/polaris/service/events/jsonEventListener/JsonEventListener.java

+        properties.put("table_identifier", event.tableIdentifier().toString());
+        transformAndSendEvent(properties);
+    }
+


super nit: extra whitespace

eric-maynard

LGTM, thanks!

singhpk234

Thank you @adnanhemani for the change, it mostly LGTM !

singhpk234 · 2025-07-26T00:51:38Z

...rvice/quarkus/events/jsonEventListener/aws/cloudwatch/QuarkusAwsCloudWatchConfiguration.java

+@StaticInitSafe
+@ConfigMapping(prefix = "polaris.event-listener.aws-cloudwatch")
+@ApplicationScoped
+public interface QuarkusAwsCloudWatchConfiguration extends AwsCloudWatchConfiguration {


since its a public interface would be nice to have a doc

singhpk234 · 2025-07-26T00:54:29Z

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

+    try {
+      client.createLogGroup(CreateLogGroupRequest.builder().logGroupName(logGroup).build());
+    } catch (ResourceAlreadyExistsException ignored) {
+      LOGGER.debug("Log group {} already exists", logGroup);
+    }
+
+    try {
+      client.createLogStream(
+          CreateLogStreamRequest.builder().logGroupName(logGroup).logStreamName(logStream).build());
+    } catch (ResourceAlreadyExistsException ignored) {
+      LOGGER.debug("Log stream {} already exists", logStream);
+    }


do we only on sdk retries for this creation ?
what if we create a log group but creating log stream fails, are we ok overall ?

singhpk234 · 2025-07-26T00:55:39Z

...apache/polaris/service/events/jsonEventListener/aws/cloudwatch/Dockerfile-localstack-version

+# Dockerfile to provide the image name and tag to a test.
+# Version is managed by Renovate - do not edit.
+FROM localstack/localstack:3.4
+


nit: extra line ?

singhpk234 · 2025-07-26T00:57:52Z

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

+  @Override
+  protected void transformAndSendEvent(HashMap<String, Object> properties) {
+    properties.put("realm", callContext.getRealmContext().getRealmIdentifier());
+    properties.put("principal", securityContext.getUserPrincipal().getName());


how about also adding the activated prinicpal roles too ?

singhpk234 · 2025-07-26T00:59:52Z

...ache/polaris/service/events/jsonEventListener/aws/cloudwatch/AwsCloudWatchEventListener.java

+        EventWithRetry currValue = futures.remove(future);
+        if (currFutureState.equals(Future.State.FAILED)
+            || currFutureState.equals(Future.State.CANCELLED)) {
+          if (currValue.retryCount >= 3) {


[good to have] is it possible to make this retry count configurable ?

adnanhemani added 2 commits June 26, 2025 19:14

Add AWS CloudWatch integration through Event Listener

2e07dde

cleanup

06eaca4

github-project-automation bot added this to Basic Kanban Board Jun 27, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jun 27, 2025

spotlessapply

1515fbb

adnanhemani added 5 commits July 12, 2025 04:55

Added unit test with LocalStack

854501b

typo

c1c94b2

spotlessapply

ab3c5f9

Merge remote-tracking branch 'origin/main' into ahemani/cloudwatch_ev…

ab9ccbe

…ent_listener

recompile from main

a641136

eric-maynard reviewed Jul 15, 2025

View reviewed changes

adnanhemani added 4 commits July 15, 2025 21:57

first revision change, based on review from @eric-maynard

5a355d1

merge from origin/main

bab4439

spotlessapply

04c310a

Merge branch 'main' into ahemani/cloudwatch_event_listener

d4b44ff

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

eric-maynard reviewed Jul 16, 2025

View reviewed changes

service/common/src/main/java/org/apache/polaris/service/events/AwsCloudWatchEventListener.java Outdated Show resolved Hide resolved

snazy requested changes Jul 17, 2025

View reviewed changes

adnanhemani added 7 commits July 21, 2025 00:18

injected securitycontext and callcontext

4d0554a

todo

cc715ad

modify test

518aaaa

first draft of revision

8758255

resolve comments from @eric-maynard and @snazy

f3f62a0

refactor into separate package

d21dabc

typo

9054511

adnanhemani requested a review from snazy July 21, 2025 07:39

eric-maynard reviewed Jul 21, 2025

View reviewed changes

adnanhemani added 3 commits July 22, 2025 03:01

revising comments from @eric-maynard

828760a

Merge branch 'main' into ahemani/cloudwatch_event_listener

ae79600

spotlessapply

9d47684

eric-maynard reviewed Jul 22, 2025

View reviewed changes

eric-maynard approved these changes Jul 22, 2025

View reviewed changes

singhpk234 reviewed Jul 26, 2025

View reviewed changes


		import java.util.HashMap;

		public abstract class JsonEventListener extends PolarisEventListener {

AWS CloudWatch Event Sink Implementation #1965

Are you sure you want to change the base?

AWS CloudWatch Event Sink Implementation #1965

Uh oh!

Conversation

adnanhemani commented Jun 27, 2025

Uh oh!

snazy commented Jun 27, 2025

Uh oh!

adnanhemani commented Jun 27, 2025

Uh oh!

adnanhemani commented Jul 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eric-maynard left a comment

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.