WorkerV2 - Cloud Run V2 API

[japerry911]

New: Add support for Cloud Run API v2 (Worker)

Still ToDo:

• Add in force parameter option for deleting jobs in future whenever that is fixed on GCPs end
  • Cloud Run Job v2 API does not include the force query parameter for Jobs.Delete googleapis/google-api-python-client#2250

Overview

• Add support for Cloud Run API v2 (Worker)
  • more features are available, for example longer timeouts
• Closes Upgrade Cloud Run to use API v2 #199

How to Test Locally:

Example

• Anonymous Block example:

CloudRunJobV2(
    credentials=gcp_credentials_block,  # GCPCredentials Block
    image=(
        f"us-central1-docker.pkg.dev/{variables.GCP_PROJECT_ID}/prefect-flows/flows:"
        f"{variables.IMAGE_TAG}"
    ),
    region="us-central1",
    cpu=1,
    memory=512,
    memory_unit="Mi",
    env=variables.GCP_BLOCK_ENV,  # environment variables object
    keep_job=False,
    timeout=86400,  # 24 hours
    launch_stage="BETA",
    max_retries=1,
)

Screenshots

• Starting Deployment/Flow
  

• Starting in Cloud Run Jobs via Cloud Run API V2
  

• Full 1 day timeout being used in example
  

• Cancelling Example (currently delete all executions by hand instead of using Force parameter until GCP fixes that error in above issue)
  
  

Thank you, let me know of any/all questions/comments/feedback, happy to have worked on this.

Checklist

• References any related issue by including "Closes #" or "Closes ".
  • If no issue exists and your change is not a small fix, please create an issue first.
• Includes tests or only affects documentation.
• Passes pre-commit checks.
  • Run pre-commit install && pre-commit run --all locally for formatting and linting.
• Includes screenshots of documentation updates.
  • Run mkdocs serve view documentation locally.
• Summarizes PR's changes in CHANGELOG.md

[zzstoatzz]

hey @japerry911 - thank you so much for the work here!

I've done a quick run through the v2 worker in particular and have a few comments / questions 🙂

• this is looking really good so far! we really appreciate thorough PRs like this - especially a more complex implementation like a worker 💙
• would you be willing to copy the infra block implementation to a separate PR? i think it would be good for future reference if they were separate - plus we'll have to think a little about the ambiguity of Cloud Run v2 vs Cloud Run v2 Worker in the docs / collection registry. a related nit, could we update this PR name to be "{verb} {feature}" like "Add v2 Cloud Run API worker" (if that sounds good to you)?
• i am wondering if there are some places we are using Optional where its simply a field having a default value, whose value should generally not be None (left comment on at least one of them)

I will review the tests and the rest as soon as I can, and take it for a spin :) ! feel free to reach out with any questions

[japerry911]

hey @japerry911 - thank you so much for the work here!

I've done a quick run through the v2 worker in particular and have a few comments / questions 🙂

• this is looking really good so far! we really appreciate thorough PRs like this - especially a more complex implementation like a worker 💙
• would you be willing to copy the infra block implementation to a separate PR? i think it would be good for future reference if they were separate - plus we'll have to think a little about the ambiguity of Cloud Run v2 vs Cloud Run v2 Worker in the docs / collection registry. a related nit, could we update this PR name to be "{verb} {feature}" like "Add v2 Cloud Run API worker" (if that sounds good to you)?
• i am wondering if there are some places we are using Optional where its simply a field having a default value, whose value should generally not be None (left comment on at least one of them)

I will review the tests and the rest as soon as I can, and take it for a spin :) ! feel free to reach out with any questions

@zzstoatzz All this sounds great, thank you for diving in!

Good idea, I'll break it out into two branches over the next few days (can probably get it done by Friday morning). Should I open PR for just Cloud Run v2 Block first, then the worker after that merges, or what's the best way to go about it?

I left some comments/questions on a few of the modifications. Lemme know what you think, and I will make all of those updates. And lastly, I'll take a deeper look at "i am wondering if there are some places we are using Optional where its simply a field having a default value, whose value should generally not be None (left comment on at least one of them)".

Thanks again for review @zzstoatzz ! Looking forward to getting these PRs into the repo.

[zzstoatzz]

Good idea, I'll break it out into two branches over the next few days (can probably get it done by Friday morning). Should I open PR for just Cloud Run v2 Block first, then the worker after that merges, or what's the best way to go about it?

i think the worker is a good place to start if that sounds alright, since the worker implementations should be a little more up to date with our best practices in core and could inspire some updates to the infra block implementation (plus just generally we are more focused on actively maintaining workers nowadays)

thanks for the quick updates :)

[japerry911]

Good idea, I'll break it out into two branches over the next few days (can probably get it done by Friday morning). Should I open PR for just Cloud Run v2 Block first, then the worker after that merges, or what's the best way to go about it?

i think the worker is a good place to start if that sounds alright, since the worker implementations should be a little more up to date with our best practices in core and could inspire some updates to the infra block implementation.

thanks for the quick updates :)

The worker uses these imports from the Block though:

from prefect_gcp.cloud_run_v2 import CloudRunJobV2Result, ExecutionV2, JobV2

That's why I went about merging them into 1 PR, since the block would only run with Agent 😿 .

[japerry911]

Oh I see, I probably should just build those classes that are imported and keep the others out for a separate PR?

[japerry911]

@zzstoatzz , hey Nate, I made the modifications we spoke about and tested them as well.

Let me know what you think!

Thank you!

[zzstoatzz]

hey @japerry911 - the code is looking great to me 🙂

I will QA this as soon as possible, then come back for a once over :)

[zzstoatzz]

hey @japerry911 - things are looking great!

I am going to discuss with the rest of the team on how exactly we should differentiate (if anything additional) this in the UI and then include this in a metadata scrape - I'll keep this open for the moment while I get that feedback.

I just want to thank you again for this super high quality contribution - thank you so much 💙. Back with you ASAP

[zzstoatzz]

such a great contribution @japerry911 💙 thanks again!!

[serinamarie]

@zzstoatzz It seems like there was a 404 in loading this that wasn't caught

[discdiver]

I can work on that @serinamarie - I see another thing in the API docs that needs fixed, too.

[serinamarie]

it was fixed/merged! thanks though