Skip to content

Commit

Permalink
docs updates, typing fix
Browse files Browse the repository at this point in the history
  • Loading branch information
dmedora committed Nov 2, 2021
1 parent 8aad1bf commit f71beb0
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 4 deletions.
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,9 @@ case specific CLI arguments or editing the saved YAML configuration file.
For example, the following command creates a YAML file for the validation of the
`new_york_citibike` table: `data-validation validate column -sc my_bq_conn -tc
my_bq_conn -tbls bigquery-public-data.new_york_citibike.citibike_trips -c
citibike.yaml`
citibike.yaml`.

The vaildation config file is saved to the GCS path specified by the `PSO_DV_CONFIG_HOME` env variable if that has been set; otherwise, it is saved to wherever the tool is run.

Here is the generated YAML file named `citibike.yaml`:

Expand Down Expand Up @@ -219,6 +221,8 @@ data-validation run-config -c citibike.yaml
View the complete YAML file for a GroupedColumn validation on the
[examples](docs/examples.md#) page.

You can view a list of all saved validation YAML files using `data-validation run-config list`.

### Aggregated Fields

Aggregate fields contain the SQL fields that you want to produce an aggregate
Expand Down
4 changes: 2 additions & 2 deletions data_validation/state_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ def _get_connection_path(self, name: str) -> str:
return os.path.join(self._get_connections_directory(),
f"{name}.connection.json")

def create_validation_yaml(self, name: str, yaml_config: dict[str, str]):
def create_validation_yaml(self, name: str, yaml_config: Dict[str, str]):
"""Create a validation file and store the given config as YAML.
Args:
Expand All @@ -111,7 +111,7 @@ def create_validation_yaml(self, name: str, yaml_config: dict[str, str]):
self._write_file(validation_path, yaml_config_str)
print(yaml_config)

def get_validation_config(self, name: str) -> dict[str, str]:
def get_validation_config(self, name: str) -> Dict[str, str]:
"""Get a validation configuration from the expected file.
Args:
Expand Down
2 changes: 1 addition & 1 deletion docs/connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ a directory specified by the env variable `PSO_DV_CONFIG_HOME`.
## GCS Connection Management (recommended)

The connections can also be stored in GCS using `PSO_DV_CONFIG_HOME`.
To do so simply add the GCS path to the environment.
To do so simply add the GCS path to the environment. Note that if this path is set, query validation configs will also be saved here.

eg.
`export PSO_DV_CONFIG_HOME=gs://my-bucket/my/connections/path/`
Expand Down

0 comments on commit f71beb0

Please sign in to comment.