Testing data quality
Tests can be applied to either sources or models, allowing you to verify your data at every build step against specified rules or assertions.
Y42 provides three types of tests:
- Table tests: assertions applicable at the table level, e.g. if the table row count is within a certain range or if the table includes specific columns.
- Column tests: assertions applicable at the column level, e.g. unique, not null, or containing certain values.
- Anomaly detection tests: assertions designed to identify data quality issues, including volume, freshness, or anomalies within individual columns.
To define your own customized tests, add them to the tests folder.
You can customize the location of this folder by modifying the test-paths configuration in the dbt_project.yml file.
- assert_purchasing_total_is_positive.sql
- assert_vendors_valid_names.sql
- dbt_project.yml
Test execution
All tests are converted into SQL select queries before being executed.
Each query returns rows that fail to meet the specified test criteria.
Hence, if the test's query returns at least one row, the test will fail.
Conversely, if the test query does not return any nows, then the test will pass.