Optional override for Vertex custom job name #208

jeremy-thomas-roc · 2023-09-01T13:52:29Z

Add optional parameter to VertexAICustomJob to name jobs. This helps in monorepo situations where all Vertex jobs will just be {repo-name}-{uuid}, which is not very helpful when running lots of scheduled jobs.

Screenshots

Checklist

References any related issue by including "Closes #" or "Closes ".
- If no issue exists and your change is not a small fix, please create an issue first.
Includes tests or only affects documentation.
Passes pre-commit checks.
- Run pre-commit install && pre-commit run --all locally for formatting and linting.
Includes screenshots of documentation updates.
- Run mkdocs serve view documentation locally.
Summarizes PR's changes in CHANGELOG.md

…tHQ#196) Bumps [JamesIves/github-pages-deploy-action](https://github.com/jamesives/github-pages-deploy-action) from 4.4.2 to 4.4.3. - [Release notes](https://github.com/jamesives/github-pages-deploy-action/releases) - [Commits](JamesIves/github-pages-deploy-action@v4.4.2...v4.4.3) --- updated-dependencies: - dependency-name: JamesIves/github-pages-deploy-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander Streed <[email protected]>

* Persist Labels to Vertex AI Custom Job * ensure compatible with GCP requirements * small bug not fetching key-value pairs * Add test, changelog, and format * pre commit hook changes * Feedback for method name --------- Co-authored-by: Alexander Streed <[email protected]>

jeremy-thomas-roc · 2023-09-11T13:44:30Z

@desertaxle Can I get some feedback on this? I believe the failing tests happened when there was a dependency issue of the prefect side, but I am unable to re-run them to verify.

desertaxle · 2023-09-18T13:17:38Z

@jeremy-thomas-roc I think we could use the existing name attribute on the VertexAICustomJob job to create more unique names for created jobs. By default, name will be the flow run name, which has the added advantage of making it easier to map Vertex AI jobs to flow runs. Let me know if that would work for your use case or it you have any questions on implementation.

jeremy-thomas-roc · 2023-09-18T19:41:35Z

@desertaxle that works for me, implemented that change

It appears that name can be None when this property is retrieved, which I see from one of the failing tests. Is this an issue where the name is not created before the job_name?

desertaxle · 2023-09-18T21:03:05Z

@jeremy-thomas-roc That's true that name can be None if the block is used outside of a deployment. A simple check to see if name is set should work. We do something similar in prefect-azure. You can see PrefectHQ/prefect-azure#92 as an example.

jeremy-thomas-roc · 2023-09-19T13:54:03Z

@desertaxle I have made that change, but it appears an unrelated test is failing due to an API error, not sure why that is happening

desertaxle

Couple of small nits, but otherwise this is good to go!

desertaxle · 2023-09-19T14:19:33Z

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added

 ### Changed
+ - `job_name` in `VertexAICustomJob` is now based on the flow run name instead of the image repository.


Suggested change

- `job_name` in `VertexAICustomJob` is now based on the flow run name instead of the image repository.

- Use flow run name for name of created custom jobs - [#208](https://github.com/PrefectHQ/prefect-gcp/pull/208)

desertaxle · 2023-09-19T14:20:07Z

prefect_gcp/aiplatform.py

+            repo_name = self.name or self.image.split("/")[2]
+            return f"{repo_name}-{uuid4().hex}"


Suggested change

repo_name = self.name or self.image.split("/")[2]

return f"{repo_name}-{uuid4().hex}"

base_name = self.name or self.image.split("/")[2]

return f"{base_name}-{uuid4().hex}"

jeremy-thomas-roc · 2023-09-19T15:12:00Z

@desertaxle suggested changes implemented

jeremy-thomas-roc and others added 24 commits April 24, 2023 08:58

add accelerator count

de4f9d1

update changelog

b4856d6

added test

67c2791

fix jobs not canceling in vertex

fea1f1f

add accelerator count

6da509a

update changelog

28b2972

added test

f591110

fix jobs not canceling in vertex

ad00206

Merge branch 'main' of https://github.com/jeremy-thomas-roc/prefect-gcp

4115e3e

revert bad changes

177bfff

update changelog

8e35bb3

update changelog

ad9b02e

Merge branch 'main' into main

bcb94df

Merge branch 'main' into main

3df1e10

added test

9b87c5f

revert bad changes

ead7fbf

update changelog

b75e64d

update changelog

be4d994

Merge branch 'main' of https://github.com/jeremy-thomas-roc/prefect-gcp

257620f

revert

2eec164

add optional vertex job name

4a543fe

update changelog

8dc58b6

jeremy-thomas-roc requested a review from a team as a code owner September 1, 2023 13:52

revert bad merge

3c80609

change job name to reference flow run name

11c0296

jeremy-thomas-roc and others added 2 commits September 18, 2023 15:42

changelog

35b8f17

Merge branch 'main' into main

4e30e9f

jeremy-thomas-roc added 2 commits September 19, 2023 09:26

handle none values

71e1383

handle none values

5b06a03

desertaxle suggested changes Sep 19, 2023

View reviewed changes

code review changes

455e592

desertaxle approved these changes Sep 19, 2023

View reviewed changes

desertaxle merged commit dfa07ca into PrefectHQ:main Sep 19, 2023
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optional override for Vertex custom job name #208

Optional override for Vertex custom job name #208

jeremy-thomas-roc commented Sep 1, 2023 •

edited

Loading

jeremy-thomas-roc commented Sep 11, 2023

desertaxle commented Sep 18, 2023

jeremy-thomas-roc commented Sep 18, 2023 •

edited

Loading

desertaxle commented Sep 18, 2023

jeremy-thomas-roc commented Sep 19, 2023

desertaxle left a comment

desertaxle Sep 19, 2023

desertaxle Sep 19, 2023

jeremy-thomas-roc commented Sep 19, 2023

	- `job_name` in `VertexAICustomJob` is now based on the flow run name instead of the image repository.
	- Use flow run name for name of created custom jobs - [#208](https://github.com/PrefectHQ/prefect-gcp/pull/208)

		repo_name = self.name or self.image.split("/")[2]
		return f"{repo_name}-{uuid4().hex}"

Optional override for Vertex custom job name #208

Optional override for Vertex custom job name #208

Conversation

jeremy-thomas-roc commented Sep 1, 2023 • edited Loading

Screenshots

Checklist

jeremy-thomas-roc commented Sep 11, 2023

desertaxle commented Sep 18, 2023

jeremy-thomas-roc commented Sep 18, 2023 • edited Loading

desertaxle commented Sep 18, 2023

jeremy-thomas-roc commented Sep 19, 2023

desertaxle left a comment

Choose a reason for hiding this comment

desertaxle Sep 19, 2023

Choose a reason for hiding this comment

desertaxle Sep 19, 2023

Choose a reason for hiding this comment

jeremy-thomas-roc commented Sep 19, 2023

jeremy-thomas-roc commented Sep 1, 2023 •

edited

Loading

jeremy-thomas-roc commented Sep 18, 2023 •

edited

Loading