Suddenly, the Stream Analytics Job stopped writing to ADLS Storage
I have an Event Hub Namespace wich connects to a Stream Analytics Job, wich finally writes to an ADLS Storage. Then, the data is queried in SQL Synapse (Serverless)
Everything was fine in the dev & test:
I sent messages to Event Hub using Generate Data (Preview)
I run my queries on Synapse
The output files on ADLS are JSON lines, means that every message is stored in a blob, which make bigger every time since new message are appened to the same file.
Suddenly, I was sending message to Evet Hub via Azure Portal and a second after launching my query to see "my real time data". At some moment, sending new message would t traslate on files stored at Blob. This was rare and jump into the Stream Analytics Job to debug, an every thing seems to be on place.
Watching the activity log of the Stream Analitycs Job, I realized there was something rare.
First Occurred: 5/28/2023 12:49:48 AM UTC | Resource Name: sporadic-contribution-adls-output-json | Message: Unable to upload blobs to storage because of invalid blob block ids. Please ensure that blob file event=sporadic_contribution/date=2023-05-28/hour=00/0_62b04999def54281be2fb6de7f556b63_1.json is not being modified by another process at the same time. Blob storage error code: InvalidBlockList
I wonder if this error is related to the fact that I launch my SQL queries while the blob is geting updated on new entries comming from Event Hub ¿Is there any race condition? It should not. Besides, the SQL queries just read the Blob, not write While new entries are comming from Event Hub, and pushing to Blob (appended to the same file) via Stream Analytics, ¿The blob enters to and inconsistent state that conflicts with my SQL queries?
Best regards