-
Notifications
You must be signed in to change notification settings - Fork 133
[FEATURE] cursor replication with kop #1778
Comments
KoP does not use cursor to store the committed offsets. Like Kafka, KoP just stores the offsets in the offset topic, which is I think the reason is that KoP reads the committed offset from the memory cache directly. KoP only loads the cache from the offset topic when starting. |
are there any configs to setup for the consumer offsets to be replicated or the default pulsar things will work ? How to know the cache of KoP and flush if needed for a topic/consumer group ? |
TL; DR, current KoP does not work well with geo-replication, we might need to improve it later.
You only needs to replicate the
All Kafka admin operations just query the cache of KoP. To understand this issue, you might need to know some implementation details:
Assuming you replicated the offset topic from cluster-A to cluster-B, if a Kafka consumer connected to KoP when the offset topic is not fully replicated, KoP might not load the latest metadata, e.g.
Then the consumer to cluster-B will start consuming from offset 201, not 1001. What's worse is, if you triggered the metadata loading, the metadata won't update. e.g.
|
Is your feature request related to a problem? Please describe.
One of the good things solved by pulsar is the subscription cursor replication for native pulsar applications. When kop is used there is same understanding, But in practice things are a bit different with kafka's consumer groups
Describe the solution you'd like
kafka consumer group offset replication should work across clusters similar to the native subscription replication for consumers
Describe alternatives you've considered
None
Additional context
We have 2 clusters running kop with namespace level replication setup.
cluster-scus
ontopic-1
for 10900 messagescluster-scus
ontopic-1
, stop at 10900 offset, lag 0cluster-scus
cluster-wus
ontopic-1
, consumer stats at 500 offset, lag 10400cluster-wus
cluster-scus
ontopic-1
, consumer resumes from 10900 offset, lag 0The text was updated successfully, but these errors were encountered: