- Environment: flink 1.13.6 on yarn
- Application: flink api stream application, consume one topic named a with group.id g , with checkpoint, auto commit offset is closed, offset committment is triggered by checkpoint. checkpoint every minutes at 24 second, such as 12:00:24, 12:01:24...
- Problem Description:
The offset commited to kafka is suddenly increase from a small value (we had a big lag) to a value close to the partioin log offset (infer to nearly no lag at all), and vice versa.
For better illustration, the following is two pictures of offset monitoring to describe the wierd error.
For more detail, I seek to the offset topic to get the commit detail, find that the topic is operation nomally and the offset commited is wierd. (so it s not the problem of kafka).
As I mentioned, I check the status of the topic and seek to offset topic find the commited messages, make sure is the problem of the client side. Is it a bug of flink kafka source or checkpoint?
Why do I get a ValueError in my Kafka Consumer if I seek to another position?
I m using python 3.9.16 and kafka-python version 2.0.2. I m running on my Macbook Pro IOS 11.6.5. I m still getting my feet wet with Kafka so it s entirely possible I m doing things the wrong way. ...