Skip to content

Add compatibility with GCP Schema Registries #1153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

magnusdriver
Copy link

What changes did you make? (Give an overview)

Added new cluster property "gcpSchemaRegistry" to enable compatibility with new GCP Schema Registries.
This property can be used in config.yaml file like this:

kafka:
  clusters:
    - name: local  # Unique name identifier for the Kafka cluster
      bootstrap-servers: kafka1:9092,kafka2:9092  # List of Kafka broker addresses

      schemaRegistry: https://managedkafka.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/schemaRegistries/REGISTRY_ID
      gcpSchemaRegistry: true # false by default

Or can be enabled with the env variable: KAFKA_CLUSTERS_0_GCPSCHAMAREGISTRY=true

Important changes in code are:

  • Add bearer token generation for auth with GCP Schema Registry in WebClientConfigurator
  • Add custom bearer token provider in SchemaRegistrySerde
  • Add the new gcpSchemaRegistry config property in kafbat-ui-api
  • Add new schema fields kafka-sr-api for compatibility level as GCP Schema Registry return different format.
  • Add condition to use compatibility or compatibliityLevel field based on gcpSchemaRegistry enabled or not in
    SchemaRegistryService

It's needed to use gcloud auth application-default login to get the credentials to connect to GCP Schema
Registries or run kafka-ui in a compute engine instance or GKE with a service account with the needed permissions.

Note: This functionality only works now with Avro schema types as these are the ones I work with.
It could work with Protobuf schemas, but not with JSON ones as GCP Schema Registries don't
support them.

How Has This Been Tested? (put an "x" (case-sensitive!) next to an item)

  • Manually (please, describe, if necessary)

    Changes tested connecting to GCP and Confluent Schema Registries to check:

    • All Schemas registered are shown in Schema Registry dashboard
    • Topic events are deserialized correctly in the topics dashboards.

    It was impossible to pass unit tests even before adding any change :'(

Checklist (put an "x" (case-sensitive!) next to all the items, otherwise the build will fail)

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (e.g. ENVIRONMENT VARIABLES)
  • My changes generate no new warnings (e.g. Sonar is happy)
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Check out Contributing and Code of Conduct

@magnusdriver magnusdriver requested a review from a team as a code owner June 26, 2025 15:52
@kapybro kapybro bot added status/triage Issues pending maintainers triage status/triage/manual Manual triage in progress status/triage/completed Automatic triage completed and removed status/triage Issues pending maintainers triage labels Jun 26, 2025
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi magnusdriver! 👋

Welcome, and thank you for opening your first PR in the repo!

Please wait for triaging by our maintainers.

Please take a look at our contributing guide.

@magnusdriver magnusdriver changed the title Add all changes to connect with GCP Schema Registries Add compatibility with GCP Schema Registries Jun 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/triage/completed Automatic triage completed status/triage/manual Manual triage in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy