From db2ea1dc3abc007ae51d57aecca210d4092ce5f8 Mon Sep 17 00:00:00 2001 From: Dylan Hercher Date: Mon, 20 Jul 2020 23:07:30 +0000 Subject: [PATCH 1/4] adding filters to the readme --- README.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/README.md b/README.md index 593cab388..a8b58a58b 100644 --- a/README.md +++ b/README.md @@ -145,6 +145,26 @@ via `pip install teradatasql` if you have a license. } ``` +## Query Configurations + +It is possible to customize the configuration for a given validation. The CLI expects that you are trying to compare two identical tables; however, you can customize each query (sour or target) either by running the validation with a custom configuration in Python or editing a save YAML configuration file. + +### Filters + +Currently the only form of filter supported is a custom filter written by you in the syntax of the given source. In future we will also release pre-built filters to cover certain usecases (ie. `SELECT * FROM table WHERE created_at > 30 days ago;`). + +#### Custom Filters +``` +{ + "type": "custom", + "source": "created_at > '2020-01-01 00:00:00'::TIMESTAMP", + "target": "created_at > '2020-01-01 00:00:00'", +} +``` + +Note that you are writing the query to execute, which does not have to match between source and target as long as the reuslts can be expected to align. + + ## Validation Reports The data validation tool can write the results of a validation run to Google From 1947b14ea679c153d659375758c77846a2d75c20 Mon Sep 17 00:00:00 2001 From: Dylan Hercher Date: Mon, 20 Jul 2020 23:14:47 +0000 Subject: [PATCH 2/4] adding a section on how to add a new data source --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index a8b58a58b..3952bd87e 100644 --- a/README.md +++ b/README.md @@ -190,6 +190,20 @@ By default the handler will print to stdout. } ``` +## Add Support for an existing Ibis Data Source + +If you want to add an Ibis Data Source which exists, but was not yet supported in the Data Validation tool, it is a simple process. + +1. In data_validation/data_validation.py + - Import the extened Client for the given source (ie. from ibis.sql.mysql.client import MySQLClient). + - Add the "": Client to the global CLIENT_LOOKUP dictionary. + +2. In ibis_addon/operations.py + - Add the RawSQL operator to the data source registry (for custom filter support). + +3. You are done, you can reference the data source via the config. + - Config: {"source_type": "", ...KV Values required in Client...} + ## Deploy to Composer ``` #!/bin/bash From 75509f7e9cda38c0fbad424cc6116a8104f16aff Mon Sep 17 00:00:00 2001 From: Dylan Hercher Date: Mon, 20 Jul 2020 23:22:59 +0000 Subject: [PATCH 3/4] spelling --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3952bd87e..18f917258 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ Currently the only form of filter supported is a custom filter written by you in } ``` -Note that you are writing the query to execute, which does not have to match between source and target as long as the reuslts can be expected to align. +Note that you are writing the query to execute, which does not have to match between source and target as long as the results can be expected to align. ## Validation Reports From 4bd1cfafc0a21f8ebc49e501cdb0e466e1c28ff1 Mon Sep 17 00:00:00 2001 From: Tim Swast Date: Tue, 21 Jul 2020 08:33:49 -0500 Subject: [PATCH 4/4] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 18f917258..15cf1e867 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ via `pip install teradatasql` if you have a license. ## Query Configurations -It is possible to customize the configuration for a given validation. The CLI expects that you are trying to compare two identical tables; however, you can customize each query (sour or target) either by running the validation with a custom configuration in Python or editing a save YAML configuration file. +It is possible to customize the configuration for a given validation. The CLI expects that you are trying to compare two identical tables; however, you can customize each query (sour or target) either by running the validation with a custom configuration in Python or editing a saved YAML configuration file. ### Filters