fix: More robust __exit__ on DataValidation() #1206
Conversation
data_validation/data_validation.py
Outdated
| self.config_manager.source_client.con.dispose() | ||
| self.config_manager.target_client.con.dispose() | ||
| if hasattr(self.config_manager.source_client, "con"): | ||
| self.config_manager.source_client.con.dispose() |
There was a problem hiding this comment.
We might want to check if dispose() is a supported method too. I know in Teradata I get the following now: PM-WARNING: Exception closing connections: 'TeradataCursor' object has no attribute 'dispose'
There was a problem hiding this comment.
@nj1973 - this might be a problem specific to Postgres. I did not see this issue on Oracle.
Please take a look at my PR regarding too many files in a config directory. There by changing the way we did validation, I was able to process about 150 files in sequence. I was impressed that the python oracle driver and/or the garbage collection were working well.
Sundar Mudupalli
There was a problem hiding this comment.
In my testing I could see the connection count climbing in both Oracle and PostgreSQL.
There was a problem hiding this comment.
We might want to check if dispose() is a supported method too. I know in Teradata I get the following now:
PM-WARNING: Exception closing connections: 'TeradataCursor' object has no attribute 'dispose'
Hmm, I think I should wrap the dispose() calls in a check for Oracle and PostgreSQL. They are the systems with a known problem. I made the assumption that there would be consistency across other engine's interfaces but that was clearly a bad assumption.
|
/gcbrun |
No description provided.