Airflow DAG failing silently in prod, nothing in the logs β€” anyone else hit this?

Siddharth 3 months ago

Our daily ETL DAG just stopped running for 3 days and nobody noticed because the Slack alert webhook had quietly broken too. Turned out to be a stale DAG file cache issue but took half a day to find.

1 Reply

Tanvi 3 months ago

Yes! We had something similar β€” our worker had run out of disk space so new tasks weren't even being picked up, failed silent instead of erroring loudly.