HyperAI integration: orchestrator and service connector

[christianversloot]

Describe changes

HyperAI (hyperai.ai) has recently become one of our suppliers of GPU instances. Unfortunately, unlike the major public cloud providers, HyperAI does not yet have an SDK like sagemaker. Still, it is critical for us to keep using ZenML, as many of our pipelines are built that way. This PR implements a HyperAI integration by means of an orchestrator and service connector.

Service connector:

• The service connector is effectively a SSH based service connector. Using paramiko, it provides an authenticated SSHClient given the configured ip_address (with an optional instance_name serving as a nickname to distinguish multiple IP addresses), username, base64_ssh_key and optionally ssh_passphrase.
• Unfortunately, the ZenML CLI does not support multiline entry via cli. That is why the key can be provided in Base64 encoded format; the service connector ensures that it is decoded.
• It has support for multiple key types: RSA, DSA (DSS), ECDSA and ED255519.

The orchestrator:

• Effectively uses Docker Compose and its service_completed_successfully depends_on condition to compose a Docker Compose file which guarantees the order of execution, including more complex pipelines by using step.spec.upstream_steps.
• Uses the provided SSH client to upload the file to the instance and then executes it (and hence assumes a very lightweight setup at the user side with Docker and Docker Compose being installed).
• Has support for scheduled pipelines (then also assumes Cron daemon is running).
• Has a built-in cleanup mechanism for non-scheduled pipeline runs, which are automatically deleted after 7 days upon starting a new pipeline run. Following the paradigm where users are responsible for cleaning up their schedule pipeline runs (per the ZenML docs), it does nothing with those.
• Optionally (via configuration, but set to False by default recognizing possible security implications) is able to authenticate the instance to the stack's configured container registry by logging in. Once again, this is entirely optional: if, from a security perspective, the user does not want this, they are free to leave it to False; then, they must ensure that the instance is logged in themselves.
• Allows for mounts to be made between folders on the instance and the container, hence allowing data stored on these GPU instances to be readily available to the pipeline run. Mounts that are made can be configured by the user in component configuration and can be changed at any time.

Logo:
The code assumes a HyperAI logo to be present in a seemingly public bucket on your end. I can ask the HyperAI team to provide a proper logo that can be put in this bucket so that it's visible within the ZenML dashboard.

This way:

• Users have full flexibility as to whether they want to create one service connector/orchestrator combination per instance; reuse service connectors (and thus keys) with multiple instances; even provide single users with different data access patterns by giving them different orchestrators with different mounts.
• Users have full freedom to deploy any Docker based pipeline using this orchestrator: as with SageMaker, the orchestrator fully respects the Docker configuration set by the user (in fact, I used the local Docker orchestrator to find inspiration).

Testing:

• I have added limited tests because I could not find many nor do I have an understanding about any testing infrastructure you may have.
• I did test thoroughly via a local ZenML / ZenML dashboard setup and it worked very well.

As discussed with @htahir1 , we're actively awaiting HyperAI usage (and so is the HyperAI team, as they've been in touch as far as I know) so we'd prefer to start using this integration as soon as possible. Do note that this does not mean that we should merge it recklessly, we're just very excited :)

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

Summary by CodeRabbit

• New Features
  • Introduced the HyperAI Service Connector for enhanced authentication options, supporting various key types.
  • Added comprehensive documentation for configuring HyperAI Connectors and orchestrating pipelines on HyperAI instances, including support for Docker Compose and scheduled runs.
  • Implemented the ZenML HyperAI orchestrator for deploying machine learning pipelines on HyperAI instances, with support for GPU-backed hardware via CUDA.
• Documentation
  • New guides and documentation for setting up and using the HyperAI Service Connector and orchestrator.
• Tests
  • Added tests for the HyperAI orchestrator to ensure correct attribute settings.

[schustmi]

LGTM 🦭

[socket-security]

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
pypi/[email protected]	Transitive: environment, eval, filesystem, network, shell, unsafe	+32	318 MB	typeshed_bot

View full report↗︎