Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Active Stack as part of configuration YAML #2370

Open
wants to merge 38 commits into
base: develop
Choose a base branch
from

Commits on Jan 29, 2024

  1. Configuration menu
    Copy the full SHA
    60af95f View commit details
    Browse the repository at this point in the history
  2. remove test generated file

    kabinja committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    ac820e5 View commit details
    Browse the repository at this point in the history
  3. fix typos

    kabinja committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    9f76559 View commit details
    Browse the repository at this point in the history
  4. fix typo

    kabinja committed Jan 29, 2024
    Configuration menu
    Copy the full SHA
    fd1f482 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Configuration menu
    Copy the full SHA
    63bc189 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. Update src/zenml/new/pipelines/pipeline.py

    fix typo in docstring
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    a98112c View commit details
    Browse the repository at this point in the history
  2. Update src/zenml/stack/utils.py

    add missing return type
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    c2d7774 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2024

  1. Configuration menu
    Copy the full SHA
    8242efb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f075c4a View commit details
    Browse the repository at this point in the history
  3. fix linting

    kabinja committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    fbe12e0 View commit details
    Browse the repository at this point in the history
  4. add documentation

    kabinja committed Feb 4, 2024
    Configuration menu
    Copy the full SHA
    80a578d View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2024

  1. Configuration menu
    Copy the full SHA
    120885a View commit details
    Browse the repository at this point in the history
  2. Update docs/book/user-guide/advanced-guide/pipelining-features/config…

    …ure-steps-pipelines.md
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    001a031 View commit details
    Browse the repository at this point in the history
  3. Update docs/book/user-guide/advanced-guide/pipelining-features/config…

    …ure-steps-pipelines.md
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    323d33c View commit details
    Browse the repository at this point in the history
  4. Update src/zenml/stack/utils.py

    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    8959f07 View commit details
    Browse the repository at this point in the history
  5. Update src/zenml/stack/utils.py

    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    kabinja and strickvl committed Feb 5, 2024
    Configuration menu
    Copy the full SHA
    dfc3115 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. update TOC (zenml-io#2406)

    strickvl committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    f141fed View commit details
    Browse the repository at this point in the history
  2. fix docstring indentation

    kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    2cdcfac View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    590b074 View commit details
    Browse the repository at this point in the history
  4. Update has_custom_name for legacy artifacts (zenml-io#2384)

    * update `has_custom_name` for legacy artifacts
    
    * add test
    
    * handle special characters in artifact names
    
    * update test signature
    
    * update test signature
    
    ---------
    
    Co-authored-by: Safoine El Khabich <[email protected]>
    2 people authored and kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    b64fae9 View commit details
    Browse the repository at this point in the history
  5. Use native VertexAI scheduler capability instead of old GCP official …

    …workaround (zenml-io#2310)
    
    * Use native VertexAI pipelines scheduling facility
    
    * deprecate VertexOrchestrator no longer used fields
    
    * updating docs
    
    * remove dead code
    
    * style: 🎨 linting
    
    * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    
    * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    
    * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    
    * Remove no longer used libraries
    
    * feat: add support for start_time and end_time in the schedule
    
    * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    
    * docs: Add details about schedule parameters
    
    * Apply suggestions from code review
    
    ---------
    
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    Co-authored-by: Hamza Tahir <[email protected]>
    Co-authored-by: Barış Can Durak <[email protected]>
    4 people authored and kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    ce9a34f View commit details
    Browse the repository at this point in the history
  6. HyperAI integration: orchestrator and service connector (zenml-io#2372)

    * Add init for HyperAI integration
    
    * WIP: HyperAI service connector
    
    * WIP
    
    * WIP: HyperAI Service Connector
    
    * WIP: HyperAI Orchestrator
    
    * Replace Docker compose write with temporary file and SCP
    
    * Variable assignment error
    
    * Set dependency
    
    * Set basic values of the HyperAI settings and config
    
    * Add config property
    
    * Allow mounts to be made
    
    * Remove newline
    
    * Finish (untested) orchestrator
    
    * Import HyperAI integration
    
    * Import HyperAI service connector in service connector registry
    
    * Rename resource type
    
    * Rename auth method
    
    * Force key to be base64
    
    * Fixes to service connector
    
    * Identify instance by name and IP address
    
    * Strip IP address Python
    
    * Strip IP address Python
    
    * Return paramiko client
    
    * WIP
    
    * Mimic sagemaker integration
    
    * Fixes to make HyperAI orchestrator visible
    
    * Fixes to make orchestrator work
    
    * Temp change default local ip for testing
    
    * Environment fix
    
    * Use upstream steps to determine dependencies
    
    * Add support for scheduled pipelines
    
    * Polish schedules
    
    * Add configuration support for multiple Paramiko key types
    
    * Add Base64 instructions
    
    * Rename various vars
    
    * Add instructions about possible cron
    
    * Some docstring edits
    
    * Add setting for CR autologin
    
    * Add rudimentary Docker login
    
    * Move value
    
    * Add docstring
    
    * Remove unused def
    
    * Extract Paramiko key type given service connector configuration
    
    * Add better warnings
    
    * Check for None differently
    
    * Automatic Docker login if configured
    
    * Add HyperAI orchestrator flavor to docs
    
    * Basic docs for HyperAI orchestrator
    
    * Add HyperAI service connector to auth management docs
    
    * Add HyperAI service connector to docs
    
    * Set autologin to False by default
    
    * Add test similar to Airflow orchestrator
    
    * Formatting
    
    * Revert changes needed to run successfully locally
    
    * Add mount path validation
    
    * Improve error handling and formatting
    
    * Format mount paths differently
    
    * Upgrade azureml-core to 1.54.0.post1
    
    * Fix docstring
    
    * Update src/zenml/integrations/hyperai/service_connectors/hyperai_service_connector.py
    
    Co-authored-by: Michael Schuster <[email protected]>
    
    * Rename def into _validate_mount_paht
    
    * Update config docstring to default to False
    
    * Move Settings, Config and Flavor to lavor folder
    
    * Remove type from docstring
    
    * Remove type from docstring
    
    * Remove type check convered by pydantic
    
    * Select container registry more efficiently
    
    * Remove redundant type conversion
    
    * Move Paramiko client creation into helper method
    
    * Reformatting
    
    * Fix imports
    
    * Temp changes for local testing
    
    * Fix imports
    
    * Revert "Temp changes for local testing"
    
    This reverts commit 76fdb29.
    
    * Rename HYPERAI_RESOURCE_TYPE into hyperai-instance
    
    * Rename ip_address into hostname
    
    * Update src/zenml/integrations/hyperai/service_connectors/hyperai_service_connector.py
    
    Co-authored-by: Stefan Nica <[email protected]>
    
    * Raise AuthorizationException if client cannot be created
    
    * Remove RuntimeError in two places because it will never arrive in that state anymore
    
    * Remove try/catch statement
    
    * Let exception fall through if applicable
    
    * Remove raises
    
    * Add warning hint about long-lived credentials
    
    * Renames in docs based on changes
    
    * Add missing io import
    
    * Formatting
    
    * Add automatic_cleanup_pipeline_files to HyperAIOrchestratorConfig
    
    * Remove redundant variable assignment
    
    * Clean only if users configure auto cleaning
    
    * Update docs
    
    * Work in progress: multi IP service connector
    
    * Resources
    
    * Append hostname instead
    
    * Omit assigning value
    
    * Rename config value
    
    * Ensure that hostname is passed to Paramiko client
    
    * Raise NotImplementedError instead of pass value
    
    * Formatting
    
    * Changes to _verify
    
    * Reflect changes in service connector docs
    
    * Fix connector value validation to allow arrays to be used with the CLI
    
    * Reflect changes in orchestrator docs
    
    * Fix connector verification to allow the multi-instance case
    
    * Ensure that pipelines can run when scheduled by setting run ID dynamically
    
    * Reformatting
    
    * Add information about scheduled pipelines to docs
    
    * Use service connector username to create Compose files on instance
    
    * Add GPU reservation if configured that way
    
    * Formatting
    
    * Add instruction
    
    * Add prerequisites for HyperAI instance
    
    * Formatting and docstrings
    
    * Fixed remaining linter errors
    
    * Applied review suggestions
    
    * Add paramiko to API docs mocks
    
    * HyperAI orchestrator config tests; make additional assertions available and fix is_remote
    
    * Remove GPU-based Dockerfile
    
    * Ensure that shell commands are escaped when used
    
    * Provide password to stdin differently
    
    * Escape case where file cannot be written to HyperAI instance
    
    * Escape inputs differently
    
    * Use network mode host to avoid non-overlapping IPv4 network pool error
    
    * Disable security check for paramiko auto-add-policy
    
    * Changes to escaping
    
    * Silenced remaining security issues and fixed remaining linter errors
    
    ---------
    
    Co-authored-by: Michael Schuster <[email protected]>
    Co-authored-by: Stefan Nica <[email protected]>
    Co-authored-by: Alex Strick van Linschoten <[email protected]>
    4 people authored and kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    c1abd87 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    a0a12e8 View commit details
    Browse the repository at this point in the history
  8. Prepare release 0.55.2 (zenml-io#2401)

    * Prepare release 0.55.2
    
    * Auto-update of Starter template
    
    ---------
    
    Co-authored-by: GitHub Actions <[email protected]>
    2 people authored and kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    479b292 View commit details
    Browse the repository at this point in the history
  9. update TOC (zenml-io#2406)

    strickvl authored and kabinja committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    e51297c View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    1022574 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Configuration menu
    Copy the full SHA
    855c96b View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. Configuration menu
    Copy the full SHA
    4c284fe View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9865043 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c34b389 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Configuration menu
    Copy the full SHA
    e27df0e View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Configuration menu
    Copy the full SHA
    876a9fb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d07e90c View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Configuration menu
    Copy the full SHA
    5126923 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. Configuration menu
    Copy the full SHA
    226791c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7e72e00 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2024

  1. temporary_active_stack

    kabinja committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    951ea5b View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. Configuration menu
    Copy the full SHA
    597ffbb View commit details
    Browse the repository at this point in the history