Dagster is an open-source orchestrator for data pipelines that decides what runs, in what order, and when. The tool tracks how tables, datasets, machine learning models, and reports connect to the source data they were built from.
Tracking broken data
When a source table contains errors, the downstream tables built from it inherit those problems. Dagster’s lineage graph identifies which assets sit downstream, providing a clear starting point for fixing broken dashboards or models.
Developers describe data assets as Python functions using the @dg.asset decorator. The function parameters name the assets they depend on, allowing the system to build a dependency graph displayed in a web interface. The project includes an example that chains country population figures, a continent-change model, and a set of derived continent statistics.
Installation requires a single command: uv add dagster dagster-webserver dagster-dg-cli. The project supports Python versions 3.9 through 3.14 and maintains a library of integrations for common data tools, including dbt.
Read Also: Obsidian introduces AI agent security controls to block risky actions
Asset checks function as data quality tests written in code next to the assets they cover. Checks can verify that a column contains no null values. Teams already using dbt can load those tests into Dagster as asset checks. The UI displays the status of every check—passed, failed, or unrun.
Marking a check as blocking ensures that a failed validation stops the materialization of downstream assets, preventing a broken table from being used to build the next one. Schedules and sensors can automate the execution of assets and their checks.
Partitions split an asset into slices, such as one per day. When slices are missing or need rebuilding, Dagster runs a backfill. By default, a backfill launches one run per partition, so a request for 100 days of history results in 100 runs. A backfill policy can consolidate these into a single run over the entire range, such as one Snowflake query, though the system still records each partition as filled. This policy only applies to backfills launched from the asset graph or an asset page; backfills started from a job page ignore the setting.
Dagster is available for free on GitHub.
Leave a Reply