Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avro schema name in source connector is randomly recomputed with every query execution #145

Open
pantaoran opened this issue Mar 17, 2023 · 1 comment

Comments

@pantaoran
Copy link
Contributor

pantaoran commented Mar 17, 2023

When using this as a source connector with the queryMode set to query, then the name of the avro schema which gets registered in Schema Registry contains a random integer.
This happens here in the code: https://github.com/SAP/kafka-connect-sap/blob/master/src/main/scala/com/sap/kafka/connect/source/querier/TableQuerier.scala#L95

This makes the connector unusable, because the name changes with every execution (apparently with every batch as specified by batch.max.rows, making it incompatible with previously registered versions of the schema. This then crashes the connector after the first batch was fetched, because it tries to set the same schema with a new name, which is incompatible with schema registry (unless using compatibility NONE, which I cannot in good conscience use in production).

This seems broken to me @elakito
Can you please find a way to re-use the same schema name with every query? Maybe use a hash of the query or something?

@pantaoran
Copy link
Contributor Author

More insights: this only happens when the queryMode is set to query.
When I set it to table then it uses the table.name as the name for the avro schema, which is stable enough.

@pantaoran pantaoran changed the title Set avro schema name (source connector)? Avro schema name in source connector is randomly recomputed with every query execution Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant