site_name: DQOps Data Quality Operations Center Documentation

site_url: https://dqops.com/docs/

repo_url: https://github.com/dqops/dqo

site_description: DQOps is an an open-source data quality platform for the whole data platform lifecycle from profiling new data sources to automating data quality monitoring.

# Docs navigation

nav:

- What is DQOps?: "index.md"

- Documentation guide: "documentation-guide.md"

- Getting started:

- "getting-started/index.md"

- Installation: "getting-started/installation.md"

- Connect to a data source: "getting-started/add-data-source-connection.md"

- Review initial results and run monitoring checks: "getting-started/review-results-and-run-monitoring-checks.md"

- Review results on the dashboards: "getting-started/review-results-on-dashboards.md"

- DQOps concepts:

- "dqo-concepts/index.md"

- Data quality process: "dqo-concepts/data-quality-process.md"

- Data quality rule mining: "dqo-concepts/data-quality-rule-mining.md"

- What is a data quality check:

- "dqo-concepts/definition-of-data-quality-checks/index.md"

- Data profiling checks: "dqo-concepts/definition-of-data-quality-checks/data-profiling-checks.md"

- Data monitoring checks: "dqo-concepts/definition-of-data-quality-checks/data-observability-monitoring-checks.md"

- Partition checks: "dqo-concepts/definition-of-data-quality-checks/partition-checks.md"

- User interface overview: "dqo-concepts/dqops-user-interface-overview.md"

- Configuring data sources: "dqo-concepts/configuring-data-sources.md"

- Configuring table metadata: "dqo-concepts/configuring-table-metadata.md"

- Configuring data quality checks: "dqo-concepts/configuring-data-quality-checks-and-rules.md"

- Running data quality checks: "dqo-concepts/running-data-quality-checks.md"

- Data observability: "dqo-concepts/data-observability.md"

- DQOps user home: "dqo-concepts/dqops-user-home-folder.md"

- Data quality sensors: "dqo-concepts/definition-of-data-quality-sensors.md"

- Data quality rules: "dqo-concepts/definition-of-data-quality-rules.md"

- Data quality dashboards: "dqo-concepts/types-of-data-quality-dashboards.md"

- Data quality dimensions: "dqo-concepts/data-quality-dimensions.md"

- Auditing time periods: "dqo-concepts/auditing-data-quality-with-time-periods.md"

- Data grouping: "dqo-concepts/measuring-data-quality-with-data-grouping.md"

- Data quality KPIs: "dqo-concepts/definition-of-data-quality-kpis.md"

- Incremental data quality monitoring: "dqo-concepts/incremental-data-quality-monitoring.md"

- Incident management: "dqo-concepts/grouping-data-quality-issues-to-incidents.md"

- Command-line interface: "dqo-concepts/command-line-interface.md"

- Data quality error sampling: "dqo-concepts/data-quality-error-sampling.md"

- Storing data quality results: "dqo-concepts/data-storage-of-data-quality-results.md"

- Deployment architecture: "dqo-concepts/architecture/dqops-architecture.md"

- Quality check execution flow: "dqo-concepts/architecture/data-quality-check-execution-flow.md"

- Installation guide:

- "dqops-installation/index.md"

- Download DQOps using PIP: "dqops-installation/install-dqops-using-pip.md"

- Run DQOps in Docker: "dqops-installation/run-dqops-as-docker-container.md"

- Install DQOps from GitHub: "dqops-installation/install-dqops-from-github.md"

- Install DQOps from release package: "dqops-installation/install-dqops-from-release-package.md"

- Data sources:

- "data-sources/index.md"

- Amazon Athena: "data-sources/athena.md"

- Amazon Redshift: "data-sources/redshift.md"

- Amazon S3: "data-sources/aws.md"

- Apache Spark: "data-sources/spark.md"

- Avro files: "data-sources/avro.md"

- ClickHouse: "data-sources/clickhouse.md"

- CSV files: "data-sources/csv.md"

- Databricks: "data-sources/databricks.md"

- Delta Lake: "data-sources/deltalake.md"

- DuckDB: "data-sources/duckdb.md"

- Google BigQuery: "data-sources/bigquery.md"

- Goggle Cloud Storage: "data-sources/gcp.md"

- IBM DB2: "data-sources/db2.md"

- Iceberg: "data-sources/iceberg.md"

- JSON files: "data-sources/json.md"

- MariaDB: "data-sources/mariadb.md"

- Microsoft Azure Blob Storage: "data-sources/azure.md"

- Microsoft SQL Server: "data-sources/sql-server.md"

- MySQL: "data-sources/mysql.md"

- Oracle: "data-sources/oracle.md"

- Parquet files: "data-sources/parquet.md"

- PostgreSQL: "data-sources/postgresql.md"

- Presto: "data-sources/presto.md"

- QuestDB: "data-sources/questdb.md"

- SAP HANA: "data-sources/hana.md"

- SingleStoreDB: "data-sources/single-store.md"

- Snowflake: "data-sources/snowflake.md"

- Teradata: "data-sources/teradata.md"

- Timescale: "data-sources/timescale.md"

- Trino: "data-sources/trino.md"

########## INCLUDE TYPES OF CHECKS REFERENCE - DO NOT MODIFY MANUALLY

- Categories of checks:

- 'categories-of-data-quality-checks/index.md'

- Accepted values: 'categories-of-data-quality-checks/how-to-validate-accepted-values-in-columns.md'

- Anomaly detection: 'categories-of-data-quality-checks/how-to-detect-anomaly-data-quality-issues.md'

- Blanks and whitespaces: 'categories-of-data-quality-checks/how-to-detect-blank-and-whitespace-values.md'

- Boolean values: 'categories-of-data-quality-checks/how-to-detect-data-quality-issues-in-bool-fields.md'

- Comparing tables: 'categories-of-data-quality-checks/how-to-reconcile-data-and-detect-differences.md'

- Custom SQL and multi column checks: 'categories-of-data-quality-checks/how-to-detect-data-quality-issues-with-custom-sql.md'

- Data accuracy: 'categories-of-data-quality-checks/how-to-detect-accuracy-data-quality-issues.md'

- Data type conversions: 'categories-of-data-quality-checks/how-to-verify-text-values-are-parsable.md'

- Data type detection: 'categories-of-data-quality-checks/how-to-detect-data-type-changes.md'

- Data volume: 'categories-of-data-quality-checks/how-to-detect-data-volume-issues-and-changes.md'

- Empty and incomplete columns: 'categories-of-data-quality-checks/how-to-detect-empty-or-incomplete-columns-with-nulls.md'

- Invalid dates: 'categories-of-data-quality-checks/how-to-detect-invalid-dates.md'

- Numeric statistics: 'categories-of-data-quality-checks/how-to-detect-data-quality-issues-in-numeric-fields.md'

- PII values: 'categories-of-data-quality-checks/how-to-detect-pii-values-and-sensitive-data.md'

- Referential integrity: 'categories-of-data-quality-checks/how-to-detect-data-referential-integrity-issues.md'

- Table availability: 'categories-of-data-quality-checks/how-to-table-availability-issues-and-downtimes.md'

- Table schema changes: 'categories-of-data-quality-checks/how-to-detect-table-schema-changes.md'

- Text patterns: 'categories-of-data-quality-checks/how-to-detect-bad-values-not-matching-patterns.md'

- Text statistics: 'categories-of-data-quality-checks/how-to-detect-data-quality-issues-in-text-fields.md'

- Timeliness and freshness: 'categories-of-data-quality-checks/how-to-detect-timeliness-and-freshness-issues.md'

- Uniqueness and duplicates: 'categories-of-data-quality-checks/how-to-detect-data-uniqueness-issues-and-duplicates.md'

########## END INCLUDE TYPES OF CHECKS REFERENCE

- Use cases and examples:

- "examples/index.md"

- Data accuracy:

- Integrity check between columns in different tables: "examples/data-accuracy/integrity-check-between-columns-in-different-tables.md"

- Data availability:

- Detect table availability issues: "examples/data-availability/detect-table-availability-issues.md"

- Data completeness:

- Detect incomplete columns: "examples/data-completeness/detect-null-values.md"

- Detect empty or incomplete tables: "examples/data-completeness/detect-empty-or-incomplete-tables.md"

- Data consistency:

- Percentage of rows having only accepted values: "examples/data-consistency/percentage-of-rows-with-a-text-found-in-set.md"

- Detect data type changes in the landing zone tables: "examples/data-consistency/detect-data-type-changes.md"

- Data reasonability:

- Percentage of false boolean values: "examples/data-reasonability/percentage-of-false-values.md"

- Percentage of values in range: "examples/data-reasonability/percentage-of-integer-values-in-range.md"

- A text not exceeding a maximum length: "examples/data-reasonability/text-not-exceeding-a-maximum-length.md"

- Data uniqueness:

- Percentage of duplicates: "examples/data-uniqueness/percentage-of-duplicates.md"

- Data validity:

- Detect invalid emails: "examples/data-validity/detect-invalid-emails.md"

- Detect invalid IP4 address: "examples/data-validity/detect-invalid-ip4-addresses.md"

- Percentage of negative values: "examples/data-validity/percentage-of-negative-values.md"

- Percentage of rows passing SQL condition: "examples/data-validity/percentage-of-rows-passing-sql-condition.md"

- Percentage of texts matching date regex: "examples/data-validity/percentage-of-texts-matching-date-regex.md"

- Percentage of valid currency codes: "examples/data-validity/percentage-of-valid-currency-codes.md"

- Percentage of valid latitude and longitude: "examples/data-validity/percentage-of-valid-latitude-and-longitude.md"

- Percentage of valid UUID: "examples/data-validity/percentage-of-valid-uuid.md"

- Percentage of rows containing USA zipcodes: "examples/data-validity/percentage-of-values-that-contains-usa-zipcode.md"

- Schema:

- Detect table schema changes: "examples/schema/detect-table-schema-changes.md"

- Data quality monitoring:

- Detect empty tables: "examples/data-quality-monitoring/detect-empty-tables.md"

- Running checks with a scheduler: "examples/data-quality-monitoring/running-checks-with-a-scheduler.md"

- Moving configured checks between environments: "examples/data-quality-monitoring/moving-configured-checks-between-environments.md"

- Working with DQOps:

- "working-with-dqo/index.md"

- Daily monitoring of data quality: "working-with-dqo/daily-monitoring-of-data-quality.md"

- Basic data statistics: "working-with-dqo/collecting-basic-data-statistics.md"

- Running data quality checks: "working-with-dqo/run-data-quality-checks.md"

- Review the data quality results on dashboards: "working-with-dqo/review-the-data-quality-results-on-dashboards.md"

- Delete data quality results: "working-with-dqo/delete-data-quality-results.md"

- Schedules:

- "working-with-dqo/configure-scheduling-of-data-quality-checks/index.md"

- Configuring the schedules by modifying the YAML files: "working-with-dqo/configure-scheduling-of-data-quality-checks/configuring-schedules-by-modifying-yaml-file.md"

- Time zone configuration: "working-with-dqo/configure-scheduling-of-data-quality-checks/timezone-configuration.md"

- Cron formatting: "working-with-dqo/configure-scheduling-of-data-quality-checks/cron-formatting.md"

- Incidents and notifications: "working-with-dqo/managing-data-quality-incidents-with-dqops.md"

- Activate and deactivate multiple checks: "working-with-dqo/activate-and-deactivate-multiple-checks.md"

- Set up data grouping: "working-with-dqo/set-up-data-grouping-for-data-quality-checks.md"

- Compare tables: "working-with-dqo/compare-tables-between-data-sources.md"

- User and access management: "working-with-dqo/access-management.md"

- Managing errors: "working-with-dqo/managing-errors.md"

- Creating custom data quality checks: "working-with-dqo/creating-custom-data-quality-checks.md"

- Working with DQOps Shell: "working-with-dqo/working-with-dqo-shell.md"

- Working with files: "working-with-dqo/working-with-files.md"

- Integrations:

- "integrations/index.md"

- Airflow:

- Setup: "integrations/airflow/index.md"

- Run checks operator: "integrations/airflow/run-checks-operator.md"

- Collect statistics operator: "integrations/airflow/collect-statistics-operator.md"

- Table import operator: "integrations/airflow/table-import-operator.md"

- Table status operator: "integrations/airflow/table-status-operator.md"

- Wait for job operator: "integrations/airflow/wait-for-job-operator.md"

- Run checks use case: "integrations/airflow/run-checks-use-case.md"

- Azure Data Factory: "integrations/azure-data-factory/index.md"

- Data lineage:

- "integrations/data-lineage/index.md"

- Marquez: "integrations/data-lineage/marquez/index.md"

- Dbt:

- "integrations/dbt/index.md"

- Dbt cloud with run checks use case: "integrations/dbt/dbt-cloud-run-checks-use-case.md"

- Dbt cloud with table status use case: "integrations/dbt/dbt-cloud-table-status-use-case.md"

- Dbt core with run checks case: "integrations/dbt/dbt-core-use-case.md"

- Looker Studio: "integrations/looker-studio/creating-custom-data-quality-dashboards.md"

- Slack: "integrations/slack/configuring-slack-notifications.md"

- Visual Studio Code: "integrations/visual-studio-code/index.md"

- Webhooks: "integrations/webhooks/index.md"

########## INCLUDE CLI COMMANDS - DO NOT MODIFY MANUALLY

- Command-line interface:

- 'command-line-interface/index.md'

- dqo: 'command-line-interface/dqo.md'

- cls: 'command-line-interface/cls.md'

- connection: 'command-line-interface/connection.md'

- table: 'command-line-interface/table.md'

- check: 'command-line-interface/check.md'

- collect: 'command-line-interface/collect.md'

- column: 'command-line-interface/column.md'

- settings: 'command-line-interface/settings.md'

- cloud: 'command-line-interface/cloud.md'

- sensor: 'command-line-interface/sensor.md'

- scheduler: 'command-line-interface/scheduler.md'

- data: 'command-line-interface/data.md'

- run: 'command-line-interface/run.md'

- rule: 'command-line-interface/rule.md'

- sso: 'command-line-interface/sso.md'

########## END INCLUDE CLI COMMANDS

########## INCLUDE PYTHON CLIENT REFERENCE - DO NOT MODIFY MANUALLY

- REST API Python client:

- 'client/index.md'

- models:

- 'client/models/index.md'

- check_results: 'client/models/check_results.md'

- check_results_overview: 'client/models/check_results_overview.md'

- checks: 'client/models/checks.md'

- column_quality_policies: 'client/models/column_quality_policies.md'

- columns: 'client/models/columns.md'

- common: 'client/models/common.md'

- connections: 'client/models/connections.md'

- dashboards: 'client/models/dashboards.md'

- data_domains: 'client/models/data_domains.md'

- data_grouping_configurations: 'client/models/data_grouping_configurations.md'

- data_lineage: 'client/models/data_lineage.md'

- data_sources: 'client/models/data_sources.md'

- default_column_check_patterns: 'client/models/default_column_check_patterns.md'

- default_table_check_patterns: 'client/models/default_table_check_patterns.md'

- dictionaries: 'client/models/dictionaries.md'

- environment: 'client/models/environment.md'

- error_samples: 'client/models/error_samples.md'

- errors: 'client/models/errors.md'

- filtered_notifications_configurations: 'client/models/filtered_notifications_configurations.md'

- incidents: 'client/models/incidents.md'

- jobs: 'client/models/jobs.md'

- labels: 'client/models/labels.md'

- log_shipping: 'client/models/log_shipping.md'

- rule_mining: 'client/models/rule_mining.md'

- rules: 'client/models/rules.md'

- schemas: 'client/models/schemas.md'

- search: 'client/models/search.md'

- sensor_readouts: 'client/models/sensor_readouts.md'

- sensors: 'client/models/sensors.md'

- shared_credentials: 'client/models/shared_credentials.md'

- table_comparison_results: 'client/models/table_comparison_results.md'

- table_comparisons: 'client/models/table_comparisons.md'

- table_quality_policies: 'client/models/table_quality_policies.md'

- tables: 'client/models/tables.md'

- users: 'client/models/users.md'

- operations:

- 'client/operations/index.md'

- check_results: 'client/operations/check_results.md'

- check_results_overview: 'client/operations/check_results_overview.md'

- checks: 'client/operations/checks.md'

- column_quality_policies: 'client/operations/column_quality_policies.md'

- columns: 'client/operations/columns.md'

- connections: 'client/operations/connections.md'

- dashboards: 'client/operations/dashboards.md'

- data_catalog_synchronization: 'client/operations/data_catalog_synchronization.md'

- data_domains: 'client/operations/data_domains.md'

- data_grouping_configurations: 'client/operations/data_grouping_configurations.md'

- data_lineage: 'client/operations/data_lineage.md'

- data_sources: 'client/operations/data_sources.md'

- default_column_check_patterns: 'client/operations/default_column_check_patterns.md'

- default_table_check_patterns: 'client/operations/default_table_check_patterns.md'

- defaults: 'client/operations/defaults.md'

- dictionaries: 'client/operations/dictionaries.md'

- environment: 'client/operations/environment.md'

- error_samples: 'client/operations/error_samples.md'

- errors: 'client/operations/errors.md'

- filtered_notifications_configurations: 'client/operations/filtered_notifications_configurations.md'

- healthcheck: 'client/operations/healthcheck.md'

- incidents: 'client/operations/incidents.md'

- jobs: 'client/operations/jobs.md'

- labels: 'client/operations/labels.md'

- log_shipping: 'client/operations/log_shipping.md'

- rule_mining: 'client/operations/rule_mining.md'

- rules: 'client/operations/rules.md'

- schemas: 'client/operations/schemas.md'

- search: 'client/operations/search.md'

- sensor_readouts: 'client/operations/sensor_readouts.md'

- sensors: 'client/operations/sensors.md'

- shared_credentials: 'client/operations/shared_credentials.md'

- table_comparison_results: 'client/operations/table_comparison_results.md'

- table_comparisons: 'client/operations/table_comparisons.md'

- table_quality_policies: 'client/operations/table_quality_policies.md'

- tables: 'client/operations/tables.md'

- timezones: 'client/operations/timezones.md'

- users: 'client/operations/users.md'

########## END INCLUDE PYTHON CLIENT REFERENCE

########## INCLUDE CHECK REFERENCE - DO NOT MODIFY MANUALLY

- Data quality checks:

- 'checks/index.md'

- table:

- 'checks/table/index.md'

- accuracy:

- 'checks/table/accuracy/index.md'

- total row count match percent: 'checks/table/accuracy/total-row-count-match-percent.md'

- availability:

- 'checks/table/availability/index.md'

- table availability: 'checks/table/availability/table-availability.md'

- comparisons:

- 'checks/table/comparisons/index.md'

- row count match: 'checks/table/comparisons/row-count-match.md'

- column count match: 'checks/table/comparisons/column-count-match.md'

- custom_sql:

- 'checks/table/custom_sql/index.md'

- sql condition failed on table: 'checks/table/custom_sql/sql-condition-failed-on-table.md'

- sql condition passed percent on table: 'checks/table/custom_sql/sql-condition-passed-percent-on-table.md'

- sql aggregate expression on table: 'checks/table/custom_sql/sql-aggregate-expression-on-table.md'

- sql invalid record count on table: 'checks/table/custom_sql/sql-invalid-record-count-on-table.md'

- import custom result on table: 'checks/table/custom_sql/import-custom-result-on-table.md'

- schema:

- 'checks/table/schema/index.md'

- column count: 'checks/table/schema/column-count.md'

- column count changed: 'checks/table/schema/column-count-changed.md'

- column list changed: 'checks/table/schema/column-list-changed.md'

- column list or order changed: 'checks/table/schema/column-list-or-order-changed.md'

- column types changed: 'checks/table/schema/column-types-changed.md'

- timeliness:

- 'checks/table/timeliness/index.md'

- data freshness: 'checks/table/timeliness/data-freshness.md'

- data freshness anomaly: 'checks/table/timeliness/data-freshness-anomaly.md'

- data staleness: 'checks/table/timeliness/data-staleness.md'

- data ingestion delay: 'checks/table/timeliness/data-ingestion-delay.md'

- reload lag: 'checks/table/timeliness/reload-lag.md'

- uniqueness:

- 'checks/table/uniqueness/index.md'

- duplicate record count: 'checks/table/uniqueness/duplicate-record-count.md'

- duplicate record percent: 'checks/table/uniqueness/duplicate-record-percent.md'

- volume:

- 'checks/table/volume/index.md'

- row count: 'checks/table/volume/row-count.md'

- row count anomaly: 'checks/table/volume/row-count-anomaly.md'

- row count change: 'checks/table/volume/row-count-change.md'

- row count change 1 day: 'checks/table/volume/row-count-change-1-day.md'

- row count change 7 days: 'checks/table/volume/row-count-change-7-days.md'

- row count change 30 days: 'checks/table/volume/row-count-change-30-days.md'

- column:

- 'checks/column/index.md'

- accepted_values:

- 'checks/column/accepted_values/index.md'

- text found in set percent: 'checks/column/accepted_values/text-found-in-set-percent.md'

- number found in set percent: 'checks/column/accepted_values/number-found-in-set-percent.md'

- expected text values in use count: 'checks/column/accepted_values/expected-text-values-in-use-count.md'

- expected texts in top values count: 'checks/column/accepted_values/expected-texts-in-top-values-count.md'

- expected numbers in use count: 'checks/column/accepted_values/expected-numbers-in-use-count.md'

- text valid country code percent: 'checks/column/accepted_values/text-valid-country-code-percent.md'

- text valid currency code percent: 'checks/column/accepted_values/text-valid-currency-code-percent.md'

- accuracy:

- 'checks/column/accuracy/index.md'

- total sum match percent: 'checks/column/accuracy/total-sum-match-percent.md'

- total min match percent: 'checks/column/accuracy/total-min-match-percent.md'

- total max match percent: 'checks/column/accuracy/total-max-match-percent.md'

- total average match percent: 'checks/column/accuracy/total-average-match-percent.md'

- total not null count match percent: 'checks/column/accuracy/total-not-null-count-match-percent.md'

- anomaly:

- 'checks/column/anomaly/index.md'

- sum anomaly: 'checks/column/anomaly/sum-anomaly.md'

- mean anomaly: 'checks/column/anomaly/mean-anomaly.md'

- median anomaly: 'checks/column/anomaly/median-anomaly.md'

- min anomaly: 'checks/column/anomaly/min-anomaly.md'

- max anomaly: 'checks/column/anomaly/max-anomaly.md'

- mean change: 'checks/column/anomaly/mean-change.md'

- mean change 1 day: 'checks/column/anomaly/mean-change-1-day.md'

- mean change 7 days: 'checks/column/anomaly/mean-change-7-days.md'

- mean change 30 days: 'checks/column/anomaly/mean-change-30-days.md'

- median change: 'checks/column/anomaly/median-change.md'

- median change 1 day: 'checks/column/anomaly/median-change-1-day.md'

- median change 7 days: 'checks/column/anomaly/median-change-7-days.md'

- median change 30 days: 'checks/column/anomaly/median-change-30-days.md'

- sum change: 'checks/column/anomaly/sum-change.md'

- sum change 1 day: 'checks/column/anomaly/sum-change-1-day.md'

- sum change 7 days: 'checks/column/anomaly/sum-change-7-days.md'

- sum change 30 days: 'checks/column/anomaly/sum-change-30-days.md'

- bool:

- 'checks/column/bool/index.md'

- true percent: 'checks/column/bool/true-percent.md'

- false percent: 'checks/column/bool/false-percent.md'

- comparisons:

- 'checks/column/comparisons/index.md'

- sum match: 'checks/column/comparisons/sum-match.md'

- min match: 'checks/column/comparisons/min-match.md'

- max match: 'checks/column/comparisons/max-match.md'

- mean match: 'checks/column/comparisons/mean-match.md'

- not null count match: 'checks/column/comparisons/not-null-count-match.md'

- null count match: 'checks/column/comparisons/null-count-match.md'

- conversions:

- 'checks/column/conversions/index.md'

- text parsable to boolean percent: 'checks/column/conversions/text-parsable-to-boolean-percent.md'

- text parsable to integer percent: 'checks/column/conversions/text-parsable-to-integer-percent.md'

- text parsable to float percent: 'checks/column/conversions/text-parsable-to-float-percent.md'

- text parsable to date percent: 'checks/column/conversions/text-parsable-to-date-percent.md'

- custom_sql:

- 'checks/column/custom_sql/index.md'

- sql condition failed on column: 'checks/column/custom_sql/sql-condition-failed-on-column.md'

- sql condition passed percent on column: 'checks/column/custom_sql/sql-condition-passed-percent-on-column.md'

- sql aggregate expression on column: 'checks/column/custom_sql/sql-aggregate-expression-on-column.md'

- sql invalid value count on column: 'checks/column/custom_sql/sql-invalid-value-count-on-column.md'

- import custom result on column: 'checks/column/custom_sql/import-custom-result-on-column.md'

- datatype:

- 'checks/column/datatype/index.md'

- detected datatype in text: 'checks/column/datatype/detected-datatype-in-text.md'

- detected datatype in text changed: 'checks/column/datatype/detected-datatype-in-text-changed.md'

- datetime:

- 'checks/column/datetime/index.md'

- date values in future percent: 'checks/column/datetime/date-values-in-future-percent.md'

- date in range percent: 'checks/column/datetime/date-in-range-percent.md'

- text match date format percent: 'checks/column/datetime/text-match-date-format-percent.md'

- integrity:

- 'checks/column/integrity/index.md'

- lookup key not found: 'checks/column/integrity/lookup-key-not-found.md'

- lookup key found percent: 'checks/column/integrity/lookup-key-found-percent.md'

- nulls:

- 'checks/column/nulls/index.md'

- nulls count: 'checks/column/nulls/nulls-count.md'

- nulls percent: 'checks/column/nulls/nulls-percent.md'

- nulls percent anomaly: 'checks/column/nulls/nulls-percent-anomaly.md'

- not nulls count: 'checks/column/nulls/not-nulls-count.md'

- not nulls percent: 'checks/column/nulls/not-nulls-percent.md'

- empty column found: 'checks/column/nulls/empty-column-found.md'

- nulls percent change: 'checks/column/nulls/nulls-percent-change.md'

- nulls percent change 1 day: 'checks/column/nulls/nulls-percent-change-1-day.md'

- nulls percent change 7 days: 'checks/column/nulls/nulls-percent-change-7-days.md'

- nulls percent change 30 days: 'checks/column/nulls/nulls-percent-change-30-days.md'

- numeric:

- 'checks/column/numeric/index.md'

- number below min value: 'checks/column/numeric/number-below-min-value.md'

- number above max value: 'checks/column/numeric/number-above-max-value.md'

- negative values: 'checks/column/numeric/negative-values.md'

- negative values percent: 'checks/column/numeric/negative-values-percent.md'

- number below min value percent: 'checks/column/numeric/number-below-min-value-percent.md'

- number above max value percent: 'checks/column/numeric/number-above-max-value-percent.md'

- number in range percent: 'checks/column/numeric/number-in-range-percent.md'

- integer in range percent: 'checks/column/numeric/integer-in-range-percent.md'

- min in range: 'checks/column/numeric/min-in-range.md'

- max in range: 'checks/column/numeric/max-in-range.md'

- sum in range: 'checks/column/numeric/sum-in-range.md'

- mean in range: 'checks/column/numeric/mean-in-range.md'

- median in range: 'checks/column/numeric/median-in-range.md'

- percentile in range: 'checks/column/numeric/percentile-in-range.md'

- percentile 10 in range: 'checks/column/numeric/percentile-10-in-range.md'

- percentile 25 in range: 'checks/column/numeric/percentile-25-in-range.md'

- percentile 75 in range: 'checks/column/numeric/percentile-75-in-range.md'

- percentile 90 in range: 'checks/column/numeric/percentile-90-in-range.md'

- sample stddev in range: 'checks/column/numeric/sample-stddev-in-range.md'

- population stddev in range: 'checks/column/numeric/population-stddev-in-range.md'

- sample variance in range: 'checks/column/numeric/sample-variance-in-range.md'

- population variance in range: 'checks/column/numeric/population-variance-in-range.md'

- invalid latitude: 'checks/column/numeric/invalid-latitude.md'

- valid latitude percent: 'checks/column/numeric/valid-latitude-percent.md'

- invalid longitude: 'checks/column/numeric/invalid-longitude.md'

- valid longitude percent: 'checks/column/numeric/valid-longitude-percent.md'

- non negative values: 'checks/column/numeric/non-negative-values.md'

- non negative values percent: 'checks/column/numeric/non-negative-values-percent.md'

- patterns:

- 'checks/column/patterns/index.md'

- text not matching regex found: 'checks/column/patterns/text-not-matching-regex-found.md'

- texts not matching regex percent: 'checks/column/patterns/texts-not-matching-regex-percent.md'

- invalid email format found: 'checks/column/patterns/invalid-email-format-found.md'

- invalid email format percent: 'checks/column/patterns/invalid-email-format-percent.md'

- text not matching date pattern found: 'checks/column/patterns/text-not-matching-date-pattern-found.md'

- text not matching date pattern percent: 'checks/column/patterns/text-not-matching-date-pattern-percent.md'

- text not matching name pattern percent: 'checks/column/patterns/text-not-matching-name-pattern-percent.md'

- invalid uuid format found: 'checks/column/patterns/invalid-uuid-format-found.md'

- invalid uuid format percent: 'checks/column/patterns/invalid-uuid-format-percent.md'

- invalid ip4 address format found: 'checks/column/patterns/invalid-ip4-address-format-found.md'

- invalid ip6 address format found: 'checks/column/patterns/invalid-ip6-address-format-found.md'

- invalid usa phone format found: 'checks/column/patterns/invalid-usa-phone-format-found.md'

- invalid usa zipcode format found: 'checks/column/patterns/invalid-usa-zipcode-format-found.md'

- invalid usa phone format percent: 'checks/column/patterns/invalid-usa-phone-format-percent.md'

- invalid usa zipcode format percent: 'checks/column/patterns/invalid-usa-zipcode-format-percent.md'

- pii:

- 'checks/column/pii/index.md'

- contains usa phone percent: 'checks/column/pii/contains-usa-phone-percent.md'

- contains email percent: 'checks/column/pii/contains-email-percent.md'

- contains usa zipcode percent: 'checks/column/pii/contains-usa-zipcode-percent.md'

- contains ip4 percent: 'checks/column/pii/contains-ip4-percent.md'

- contains ip6 percent: 'checks/column/pii/contains-ip6-percent.md'

- schema:

- 'checks/column/schema/index.md'

- column exists: 'checks/column/schema/column-exists.md'

- column type changed: 'checks/column/schema/column-type-changed.md'

- text:

- 'checks/column/text/index.md'

- text min length: 'checks/column/text/text-min-length.md'

- text max length: 'checks/column/text/text-max-length.md'

- text mean length: 'checks/column/text/text-mean-length.md'

- text length below min length: 'checks/column/text/text-length-below-min-length.md'

- text length below min length percent: 'checks/column/text/text-length-below-min-length-percent.md'

- text length above max length: 'checks/column/text/text-length-above-max-length.md'

- text length above max length percent: 'checks/column/text/text-length-above-max-length-percent.md'

- text length in range percent: 'checks/column/text/text-length-in-range-percent.md'

- min word count: 'checks/column/text/min-word-count.md'

- max word count: 'checks/column/text/max-word-count.md'

- uniqueness:

- 'checks/column/uniqueness/index.md'

- distinct count: 'checks/column/uniqueness/distinct-count.md'

- distinct percent: 'checks/column/uniqueness/distinct-percent.md'

- duplicate count: 'checks/column/uniqueness/duplicate-count.md'

- duplicate percent: 'checks/column/uniqueness/duplicate-percent.md'

- distinct count anomaly: 'checks/column/uniqueness/distinct-count-anomaly.md'

- distinct percent anomaly: 'checks/column/uniqueness/distinct-percent-anomaly.md'

- distinct count change: 'checks/column/uniqueness/distinct-count-change.md'

- distinct count change 1 day: 'checks/column/uniqueness/distinct-count-change-1-day.md'

- distinct count change 7 days: 'checks/column/uniqueness/distinct-count-change-7-days.md'

- distinct count change 30 days: 'checks/column/uniqueness/distinct-count-change-30-days.md'

- distinct percent change: 'checks/column/uniqueness/distinct-percent-change.md'

- distinct percent change 1 day: 'checks/column/uniqueness/distinct-percent-change-1-day.md'

- distinct percent change 7 days: 'checks/column/uniqueness/distinct-percent-change-7-days.md'

- distinct percent change 30 days: 'checks/column/uniqueness/distinct-percent-change-30-days.md'

- whitespace:

- 'checks/column/whitespace/index.md'

- empty text found: 'checks/column/whitespace/empty-text-found.md'

- whitespace text found: 'checks/column/whitespace/whitespace-text-found.md'

- null placeholder text found: 'checks/column/whitespace/null-placeholder-text-found.md'

- empty text percent: 'checks/column/whitespace/empty-text-percent.md'

- whitespace text percent: 'checks/column/whitespace/whitespace-text-percent.md'

- null placeholder text percent: 'checks/column/whitespace/null-placeholder-text-percent.md'

- text surrounded by whitespace found: 'checks/column/whitespace/text-surrounded-by-whitespace-found.md'

- text surrounded by whitespace percent: 'checks/column/whitespace/text-surrounded-by-whitespace-percent.md'

########## END INCLUDE CHECK REFERENCE

- Reference:

- 'reference/index.md'

########## INCLUDE SENSOR REFERENCE - DO NOT MODIFY MANUALLY

- Data quality sensors:

- 'reference/sensors/index.md'

- table:

- 'reference/sensors/table/index.md'

- accuracy table sensors: 'reference/sensors/table/accuracy-table-sensors.md'

- availability table sensors: 'reference/sensors/table/availability-table-sensors.md'

- custom_sql table sensors: 'reference/sensors/table/custom_sql-table-sensors.md'

- schema table sensors: 'reference/sensors/table/schema-table-sensors.md'

- timeliness table sensors: 'reference/sensors/table/timeliness-table-sensors.md'

- uniqueness table sensors: 'reference/sensors/table/uniqueness-table-sensors.md'

- volume table sensors: 'reference/sensors/table/volume-table-sensors.md'

- column:

- 'reference/sensors/column/index.md'

- accepted_values column sensors: 'reference/sensors/column/accepted_values-column-sensors.md'

- accuracy column sensors: 'reference/sensors/column/accuracy-column-sensors.md'

- bool column sensors: 'reference/sensors/column/bool-column-sensors.md'

- conversions column sensors: 'reference/sensors/column/conversions-column-sensors.md'

- custom_sql column sensors: 'reference/sensors/column/custom_sql-column-sensors.md'

- datatype column sensors: 'reference/sensors/column/datatype-column-sensors.md'

- datetime column sensors: 'reference/sensors/column/datetime-column-sensors.md'

- integrity column sensors: 'reference/sensors/column/integrity-column-sensors.md'

- nulls column sensors: 'reference/sensors/column/nulls-column-sensors.md'

- numeric column sensors: 'reference/sensors/column/numeric-column-sensors.md'

- patterns column sensors: 'reference/sensors/column/patterns-column-sensors.md'

- pii column sensors: 'reference/sensors/column/pii-column-sensors.md'

- range column sensors: 'reference/sensors/column/range-column-sensors.md'

- sampling column sensors: 'reference/sensors/column/sampling-column-sensors.md'

- schema column sensors: 'reference/sensors/column/schema-column-sensors.md'

- text column sensors: 'reference/sensors/column/text-column-sensors.md'

- uniqueness column sensors: 'reference/sensors/column/uniqueness-column-sensors.md'

- whitespace column sensors: 'reference/sensors/column/whitespace-column-sensors.md'

########## END INCLUDE SENSOR REFERENCE

########## INCLUDE RULE REFERENCE - DO NOT MODIFY MANUALLY

- Data quality rules:

- 'reference/rules/index.md'

- Averages: 'reference/rules/Averages.md'

- Change: 'reference/rules/Change.md'

- Comparison: 'reference/rules/Comparison.md'

- Percentile: 'reference/rules/Percentile.md'

- Stdev: 'reference/rules/Stdev.md'

########## END INCLUDE RULE REFERENCE

########## INCLUDE YAML REFERENCE - DO NOT MODIFY MANUALLY

- DQOps YAML files schema:

- 'reference/yaml/index.md'

- profiling:

- 'reference/yaml/profiling/index.md'

- table profiling checks: 'reference/yaml/profiling/table-profiling-checks.md'

- column profiling checks: 'reference/yaml/profiling/column-profiling-checks.md'

- monitoring:

- 'reference/yaml/monitoring/index.md'

- table daily monitoring checks: 'reference/yaml/monitoring/table-daily-monitoring-checks.md'

- table monthly monitoring checks: 'reference/yaml/monitoring/table-monthly-monitoring-checks.md'

- column daily monitoring checks: 'reference/yaml/monitoring/column-daily-monitoring-checks.md'

- column monthly monitoring checks: 'reference/yaml/monitoring/column-monthly-monitoring-checks.md'

- partitioned:

- 'reference/yaml/partitioned/index.md'

- table daily partitioned checks: 'reference/yaml/partitioned/table-daily-partitioned-checks.md'

- table monthly partitioned checks: 'reference/yaml/partitioned/table-monthly-partitioned-checks.md'

- column daily partitioned checks: 'reference/yaml/partitioned/column-daily-partitioned-checks.md'

- column monthly partitioned checks: 'reference/yaml/partitioned/column-monthly-partitioned-checks.md'

- ConnectionYaml: 'reference/yaml/ConnectionYaml.md'

- TableYaml: 'reference/yaml/TableYaml.md'

- DashboardYaml: 'reference/yaml/DashboardYaml.md'

- SensorDefinitionYaml: 'reference/yaml/SensorDefinitionYaml.md'

- ProviderSensorYaml: 'reference/yaml/ProviderSensorYaml.md'

- RuleDefinitionYaml: 'reference/yaml/RuleDefinitionYaml.md'

- CheckDefinitionYaml: 'reference/yaml/CheckDefinitionYaml.md'

- DefaultNotificationsYaml: 'reference/yaml/DefaultNotificationsYaml.md'

- TableLevelDataQualityPolicyYaml: 'reference/yaml/TableLevelDataQualityPolicyYaml.md'

- ColumnLevelDataQualityPolicyYaml: 'reference/yaml/ColumnLevelDataQualityPolicyYaml.md'

- DefaultSchedulesYaml: 'reference/yaml/DefaultSchedulesYaml.md'

- LocalSettingsYaml: 'reference/yaml/LocalSettingsYaml.md'

- IncidentNotificationMessage: 'reference/yaml/IncidentNotificationMessage.md'

########## END INCLUDE YAML REFERENCE

########## INCLUDE PARQUET FILES REFERENCE - DO NOT MODIFY MANUALLY

- Parquet data files:

- 'reference/parquetfiles/index.md'

- check_results: 'reference/parquetfiles/check_results.md'

- errors: 'reference/parquetfiles/errors.md'

- incidents: 'reference/parquetfiles/incidents.md'

- sensor_readouts: 'reference/parquetfiles/sensor_readouts.md'

- statistics: 'reference/parquetfiles/statistics.md'

- error_samples: 'reference/parquetfiles/error_samples.md'

########## END INCLUDE PARQUET FILES REFERENCE

extra_css:

- stylesheets/extra.css

extra_javascript:

- javascripts/lazyload.min.js

# Visual settings

theme:

name: material

custom_dir: docs

features:

- navigation.indexes

- navigation.footer

- navigation.instant

- navigation.tracking

- navigation.prune

- content.tabs.link

- content.code.copy

- content.code.annotate

logo: images/logo.svg

favicon: images/favicon.ico

language: en

icon:

repo: fontawesome/brands/github

palette:

- media: "(prefers-color-scheme: light)"

scheme: default

primary: white

accent: teal

toggle:

icon: material/weather-night

name: Switch to dark mode

- media: "(prefers-color-scheme: dark)"

scheme: slate

primary: white

accent: teal

toggle:

icon: material/weather-sunny

name: Switch to light mode

font: false

# Disable search, the index is too big

plugins:

- glightbox:

skip_classes:

- glightbox-ignored-image

- meta-descriptions:

export_csv: false

quiet: true

enable_checks: true

min_length: 50

max_length: 160

trim: true

- group:

enabled: !ENV ENABLE_SEARCH

plugins:

- search

# Markdown extensions

markdown_extensions:

- pymdownx.snippets

- pymdownx.superfences

- pymdownx.tabbed:

alternate_style: true

- pymdownx.highlight:

anchor_linenums: true

- pymdownx.inlinehilite

- admonition

- pymdownx.details

- attr_list

- md_in_html

- def_list

- pymdownx.superfences:

custom_fences:

- name: mermaid

class: mermaid

format: !!python/name:pymdownx.superfences.fence_code_format

- pymdownx.emoji:

emoji_index: !!python/name:material.extensions.emoji.twemoji

emoji_generator: !!python/name:material.extensions.emoji.to_svg

- meta

hooks:

- tools/mkdocs/hooks.py

extra:

analytics:

provider: google

property: G-9W3BYMXFK5

homepage: https://dqops.com/

generator: false

social:

- icon: fontawesome/brands/github

link: https://github.com/dqops/dqo

- icon: fontawesome/brands/twitter

link: https://twitter.com/DQOpsCenter

- icon: fontawesome/brands/linkedin

link: https://www.linkedin.com/company/dqops-com/

- icon: fontawesome/brands/docker

link: https://hub.docker.com/r/dqops/dqo

- icon: fontawesome/brands/facebook

link: https://www.facebook.com/dqops

copyright: Copyright © 2021 - 2024 DQOps by Documati