Adds support to read and write to Databricks delta tables

[pthalasta]

Adds support for querying data from Databricks delta tables and write the results to delta tables. Resolves #611

[ghost]

Pull requests from external contributors require approval from a nv-morpheus organization member with write or admin permissions before CI can begin.

[raykallen]

/ok to test

[raykallen]

/ok to test

[mdemoret-nv]

Suggestions are inline. Biggest issue is that there are no tests that exercise these new stages. To guarantee that they will continue to work, we need tests built that run these stages.

[pthalasta]

will take a look at the changes

[BartleyR]

@pthalasta Pinging here to resurface. This would be a good stage to add once issues are addressed.

[pthalasta]

@mdemoret-nv i can resolve the comments except the usage of JDK as it is a requirement and regardless of which source we install it from there will be increase in container size. As this stage reads data from pyspark via databricks-connect, i don't think it will work without JDK. There are early stage python libs to read from delta without any involvement of databricks/spark, if JDK alternative is a must, i can probably test with early phase python libs, please let me know.

[gurpreets-nvidia]

@cwharris @pthalasta @mdemoret-nv : what are next steps with this MR? Are any discussions happening offline to find a way to resolve the JDK issue and close/merge this MR?

[cwharris]

@gurpreets-nvidia we are indeed discussions options. It looks like adding the JRE, JDK, and Databricks Connect will introduce an additional ~300MB to the release container. We're currently in the process of determining what CVE's would be introduced. Once we understand the CVE implications we can determine whether including it is the right way to go or if we need to look for a workaround.

[pthalasta]

@cwharris i see two options

• Load delta table into memory using python based delta table libs and then filter data the data. This doesn’t require spark so no, JDK. However, there can be memory issues in certain cases if we try to load all the data without filtering
• Pre-filter data using spark or distributed compute that can read the delta tables and share the resulting df as done in the approach above and use the result df as needed in downstream jobs.

Let me know how to proceed in closing on this MR

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ajschmidt8]

/ok to test

[dagardner-nv]

/ok to test

[dagardner-nv]

/ok to test

[efajardo-nv]

/ok to test

[cwharris]

/ok to test

[efajardo-nv]

/ok to test

[cwharris]

/ok to test

[dagardner-nv]

/ok to test

[cwharris]

/ok to test

[pthalasta]

@cwharris i think we have all tests and checks passing. There was no change to documentation part, so not sure if that is needed?

[pthalasta]

@dagardner-nv i see that the documentation is failing due to import of stages by sphinx. Is this expected?

[dagardner-nv]

/ok to test

[dagardner-nv]

/ok to test

[pthalasta]

@mdemoret-nv @cwharris can we merge if there are no other changes required and all looks good?

[mdemoret-nv]

/merge