Flink 检查点失败 - 10分钟后检查点暂停
原标题：Flink Checkpoint Failure - Checkpoints time out after 10 mins
We got one or two CheckPoint Failure during processing data every day. The data volume is low, like under 10k, and our interval setting is 2 minutes . (The reason for processing very slow is we need to sink the data to another API endpoint which take some time to process at the end of flink job, so the time is Streaming data + Sink to external API endpoint).
The root issue is: Checkpoints time out after 10 mins, this caused by the data processing time longer than 10 mins, so the checkpoint time out. We might increase the parallelism to fast the processing, but if the data become bigger, we have to increase the parallelism again, so don t want to use this way.
Suggested solution: I saw someone suggest to set the pause between old and new checkpoint, but I have some question here is, if I set the pause time there, will the new checkpoint missing the state in the pause time?
Aim: How to avoid this issue and record the correct state that doesn t miss any data?
Failed checkpoint:
Completed checkpoint:
subtask didn t respond
Thanks
问题回答
There are several related configuration variables you can set -- such as the checkpoint interval, the pause between checkpoints, and the number of concurrent checkpoints. No combination of these settings will result in data being skipped for checkpointing.
Setting an interval between checkpoints means that Flink won t initiate a new checkpoint until some time has passed since the completion (or failure) of the previous checkpoint -- but this has no effect on the timeout.
Sounds like you should extend the timeout, which you can do like this:
env.getCheckpointConfig().setCheckpointTimeout(n);
where n is measured in milliseconds. See the section of the Flink docs on enabling and configuring checkpointing for more details.
相关问题
How do I modify PDF without a library using C# and stream it back to client in ASP.NET?
I m having an issue where I m corrupted a PDF and not sure of a proper solution. I ve seen several posts on people trying to just do a basic stream or trying to modify the file with a third party ...
OutOfMemoryException on MemoryStream writing
I have a little sample application I was working on trying to get some of the new .Net 4.0 Parallel Extensions going (they are very nice). I m running into a (probably really stupid) problem with an ...
How can I compose output streams, so output goes multiple places at once?
I d like to compose two (or more) streams into one. My goal is that any output directed to cout, cerr, and clog also be outputted into a file, along with the original stream. (For when things are ...
flush in java.io.FileWriter
I have a question in my mind that, while writing into the file, before closing is done, should we include flush()??. If so what it will do exactly? dont streams auto flush??
EDIT:
So flush what it ...
Creating a FILE * stream that results in a string
I m looking for a way to pass in a FILE * to some function so that the function can write to it with fprintf. This is easy if I want the output to turn up in an actual file on disk, say. But what I ...
Decoding Manchester biphase-mark in software (representing a SMPTE timecode audio stream)
I ve never had to deal with signal/stream encoding or decoding before, at least not beyond fairly basic protocols like HTTP, so forgive me if I m making this harder than I should be.
Several of the ...
Most performant way to write binary data to a file in C#
I am trying to optimize a class that serializes objects in binary format and writes them in a file. I am currently using a FileStream (in sync mode because of the size of my objects) and a ...
how do you stream a sound clip on a web page so it is not easily downloadable?
This isn t my area of expertise at all, so I was wondering if anyone could point me in the right direction for streaming a sound clip from a web page.
The main restriction is that it can t be easily ...
