-
Notifications
You must be signed in to change notification settings - Fork 26
WorkerV2 - Cloud Run V2 API #220
WorkerV2 - Cloud Run V2 API #220
Conversation
…uilding out abstract cmds now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @japerry911 - thank you so much for the work here!
I've done a quick run through the v2 worker in particular and have a few comments / questions 🙂
- this is looking really good so far! we really appreciate thorough PRs like this - especially a more complex implementation like a worker 💙
- would you be willing to copy the infra block implementation to a separate PR? i think it would be good for future reference if they were separate - plus we'll have to think a little about the ambiguity of
Cloud Run v2
vsCloud Run v2 Worker
in the docs / collection registry. a related nit, could we update this PR name to be "{verb} {feature}" like "Add v2 Cloud Run API worker" (if that sounds good to you)? - i am wondering if there are some places we are using
Optional
where its simply a field having a default value, whose value should generally not beNone
(left comment on at least one of them)
I will review the tests and the rest as soon as I can, and take it for a spin :) ! feel free to reach out with any questions
@zzstoatzz All this sounds great, thank you for diving in! Good idea, I'll break it out into two branches over the next few days (can probably get it done by Friday morning). Should I open PR for just Cloud Run v2 Block first, then the worker after that merges, or what's the best way to go about it? I left some comments/questions on a few of the modifications. Lemme know what you think, and I will make all of those updates. And lastly, I'll take a deeper look at "i am wondering if there are some places we are using Thanks again for review @zzstoatzz ! Looking forward to getting these PRs into the repo. |
i think the worker is a good place to start if that sounds alright, since the worker implementations should be a little more up to date with our best practices in core and could inspire some updates to the infra block implementation (plus just generally we are more focused on actively maintaining workers nowadays) thanks for the quick updates :) |
The worker uses these imports from the Block though: from prefect_gcp.cloud_run_v2 import CloudRunJobV2Result, ExecutionV2, JobV2 That's why I went about merging them into 1 PR, since the block would only run with Agent 😿 . |
Oh I see, I probably should just build those classes that are imported and keep the others out for a separate PR? |
@zzstoatzz , hey Nate, I made the modifications we spoke about and tested them as well. Let me know what you think! Thank you! |
hey @japerry911 - the code is looking great to me 🙂 I will QA this as soon as possible, then come back for a once over :) |
hey @japerry911 - things are looking great! I am going to discuss with the rest of the team on how exactly we should differentiate (if anything additional) this in the UI and then include this in a metadata scrape - I'll keep this open for the moment while I get that feedback. I just want to thank you again for this super high quality contribution - thank you so much 💙. Back with you ASAP |
such a great contribution @japerry911 💙 thanks again!! |
- AI Platform: aiplatform.md | ||
- Deployment Steps: deployments/steps.md | ||
- Workers: | ||
- Cloud Run: cloud_run_worker.md | ||
- Cloud Run V2: cloud_run_worker_v2.md |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@zzstoatzz It seems like there was a 404 in loading this that wasn't caught
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can work on that @serinamarie - I see another thing in the API docs that needs fixed, too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it was fixed/merged! thanks though
New: Add support for Cloud Run API v2 (Worker)
Still ToDo:
force
parameter option for deleting jobs in future whenever that is fixed on GCPs endforce
query parameter for Jobs.Delete googleapis/google-api-python-client#2250Overview
How to Test Locally:
Example
Screenshots
Starting Deployment/Flow
![Screenshot 2023-10-26 at 11 00 38 PM](https://private-user-images.githubusercontent.com/21957238/278541399-a3ca7a84-a844-42eb-9faa-b372dae6994c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNjE3ODUsIm5iZiI6MTcyNTM2MTQ4NSwicGF0aCI6Ii8yMTk1NzIzOC8yNzg1NDEzOTktYTNjYTdhODQtYTg0NC00MmViLTlmYWEtYjM3MmRhZTY5OTRjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDExMDQ0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFmN2JlOGI3MDg4YjBhNTlmNjgyZDA4ZmU4NjNiYjE3OTgxOWNmYTRlZWZjZTc0YmU3ZmM1YTA2YTQwOGVhOTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.gunAaP9HLmomHB-AzmYJY-rxBbIcDlj2Js78Oo750Mo)
Starting in Cloud Run Jobs via Cloud Run API V2
![Screenshot 2023-10-26 at 11 00 53 PM](https://private-user-images.githubusercontent.com/21957238/278541396-48608bd5-0b11-4965-836e-2aba11577711.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNjE3ODUsIm5iZiI6MTcyNTM2MTQ4NSwicGF0aCI6Ii8yMTk1NzIzOC8yNzg1NDEzOTYtNDg2MDhiZDUtMGIxMS00OTY1LTgzNmUtMmFiYTExNTc3NzExLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDExMDQ0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI4Mjk1OWMxNTM4MmU5NDAzYmQ4YmI4YTFmYmFlMGRhYzI0NDRkNmViM2ZlZDEwYjBkOTk3YTMzMThlNDU5N2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.X7VQgkTT_vbkvQyjzqUuh47C69l5aMajzy-v9jiAsgA)
Full 1 day timeout being used in example
![Screenshot 2023-10-26 at 11 01 21 PM](https://private-user-images.githubusercontent.com/21957238/278541528-41c23e31-3bd2-440a-9e08-45d45b2047da.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNjE3ODUsIm5iZiI6MTcyNTM2MTQ4NSwicGF0aCI6Ii8yMTk1NzIzOC8yNzg1NDE1MjgtNDFjMjNlMzEtM2JkMi00NDBhLTllMDgtNDVkNDViMjA0N2RhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDExMDQ0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk3YTZhMmY4ZjlhMzg1NTU4ZWNlZjYyM2ZiODg2NjNjYmFhNTBjMWYxOWVkYWZjOTc5MjJmYWU4NTc2OTU1YTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.zCOEFnieWEB7RWW8X_r-6y1iHEOjJoo_q_KHjDcI5Vs)
Cancelling Example (currently delete all executions by hand instead of using Force parameter until GCP fixes that error in above issue)
![Screenshot 2023-10-26 at 11 02 11 PM](https://private-user-images.githubusercontent.com/21957238/278541593-45b49d50-e777-4d12-9680-f6a2d6ce1e3f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNjE3ODUsIm5iZiI6MTcyNTM2MTQ4NSwicGF0aCI6Ii8yMTk1NzIzOC8yNzg1NDE1OTMtNDViNDlkNTAtZTc3Ny00ZDEyLTk2ODAtZjZhMmQ2Y2UxZTNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDExMDQ0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdiNWI4MTY2MjM1MGYwMGM4ODczYTAyN2M4NjFlNzU2NDJjNzg4NDg1NWFiNjA2ZjkzYjUyNDdmOWU2N2Y5ZjAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.dni7rzaUzBo9UgU7eRYtMRLRdRQY2RGjyg4CYLDLv-8)
![Screenshot 2023-10-26 at 11 02 15 PM](https://private-user-images.githubusercontent.com/21957238/278541609-3220c9b7-60b1-49b8-8b4f-467ab30744b1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNjE3ODUsIm5iZiI6MTcyNTM2MTQ4NSwicGF0aCI6Ii8yMTk1NzIzOC8yNzg1NDE2MDktMzIyMGM5YjctNjBiMS00OWI4LThiNGYtNDY3YWIzMDc0NGIxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDExMDQ0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJmMjZjMjg3ZDA5NjY1ZDc2NWY4NzgyZjgyYjAxNzEzZjJkNmM3MmIzM2U5ODZjYjVjNTFmNTQ1MTY1MGI3NmYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-Mac_LO84LJINLO0OBJi7GE6HDgJFRVswMrktF3Bkxg)
Thank you, let me know of any/all questions/comments/feedback, happy to have worked on this.
Checklist
pre-commit
checks.pre-commit install && pre-commit run --all
locally for formatting and linting.mkdocs serve
view documentation locally.