Use native VertexAI scheduler capability instead of old GCP official …

…workaround (zenml-io#2310) * Use native VertexAI pipelines scheduling facility * deprecate VertexOrchestrator no longer used fields * updating docs * remove dead code * style: 🎨 linting * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Co-authored-by: Alex Strick van Linschoten <[email protected]> * Update docs/book/stacks-and-components/component-guide/orchestrators/vertex.md Co-authored-by: Alex Strick van Linschoten <[email protected]> * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Co-authored-by: Alex Strick van Linschoten <[email protected]> * Remove no longer used libraries * feat: add support for start_time and end_time in the schedule * Update src/zenml/integrations/gcp/orchestrators/vertex_orchestrator.py Co-authored-by: Alex Strick van Linschoten <[email protected]> * docs: Add details about schedule parameters * Apply suggestions from code review --------- Co-authored-by: Alex Strick van Linschoten <[email protected]> Co-authored-by: Hamza Tahir <[email protected]> Co-authored-by: Barış Can Durak <[email protected]>
kabinja · Feb 6, 2024 · ce9a34f · ce9a34f
1 parent b64fae9
commit ce9a34f
Show file tree

Hide file tree

Showing 9 changed files with 85 additions and 633 deletions.
diff --git a/docs/book/stacks-and-components/component-guide/orchestrators/vertex.md b/docs/book/stacks-and-components/component-guide/orchestrators/vertex.md
diff --git a/src/zenml/integrations/gcp/__init__.py b/src/zenml/integrations/gcp/__init__.py
@@ -49,8 +49,6 @@ class GcpIntegration(Integration):
         "google-cloud-container>=2.21.0",
         "google-cloud-storage>=2.9.0",
         "google-cloud-aiplatform>=1.21.0",  # includes shapely pin fix
-        "google-cloud-scheduler>=2.7.3",
-        "google-cloud-functions>=1.8.3",
         "google-cloud-build>=3.11.0",
         "kubernetes",
     ]

diff --git a/src/zenml/integrations/gcp/flavors/vertex_orchestrator_flavor.py b/src/zenml/integrations/gcp/flavors/vertex_orchestrator_flavor.py
@@ -133,7 +133,11 @@ class VertexOrchestratorConfig(  # type: ignore[misc] # https://github.com/pydan
     gpu_limit: Optional[int] = None
 
     _resource_deprecation = deprecation_utils.deprecate_pydantic_attributes(
-        "cpu_limit", "memory_limit", "gpu_limit"
+        "cpu_limit",
+        "memory_limit",
+        "gpu_limit",
+        "function_service_account",
+        "scheduler_service_account",
     )
 
     @property

diff --git a/src/zenml/integrations/gcp/google_cloud_function.py b/src/zenml/integrations/gcp/google_cloud_function.py
diff --git a/src/zenml/integrations/gcp/google_cloud_scheduler.py b/src/zenml/integrations/gcp/google_cloud_scheduler.py