New development environment + setup document #1507

hacktobeer · 2024-07-05T07:32:44Z

Description of the change

This PR contains

new development configuration with skaffold, minikube and vscode
vscode launch configurations for worker and server
vscode extensions configuration
changed worker and server Dockerfile to enable debugging and hot reloading
debug.py code to enable live debugging and hot reloading of code
skaffold configuration to manage the build/deploy development cycle

Applicable issues

None

Additional information

None

Checklist

All tests were successful.
Documentation updated.

…etails.

wajihyassine

Thanks for the PR, the work here will help standardize our development flows

Overall LGTM, just left a couple questions

docs/developer/develop-minikube.md

skaffold.yaml

hacktobeer · 2024-07-18T10:40:18Z

Thanks @wajihyassine. I have made the changes and added some extra explanation in the comment and document regarding hot reloading. Let me know if that makes more sense now (and resolve the comment).

jleaniz

LGTM - note that you can use the reload parameter with uvicorn to provide hot reloading for the API server too.

wajihyassine

LGTM

aarontp

Cool, looking forward to this. Just a few comments.

aarontp · 2024-07-19T00:05:35Z

docs/developer/develop-minikube.md

+### Configure VSCode
+Start VSCode and install the following extensions:
+* [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+* [Gemini Code Assist + Google Cloud Code](https://marketplace.visualstudio.com/items?itemName=GoogleCloudTools.cloudcode)


So is this only needed when doing development in GCE then? I can't seem to load the link here right now so not sure how these pieces fit together, but is the Gemini code assist part required?

No, the extension will manage all the dependencies (among others: minikube, kubectl and skaffold) for you independent if your machine is a local or a remote GCE. It will install and configure the dependecies in a way that do not interfere with your machine configuration.

The Gemini code assist is included as of lately (unfortunatly..?) and is not activated by default.

Interesting, so the only thing we're using it for is dependency management and don't actually use for anything cloud? What form are all those dependencies in, are they just vscode extensions or are there other packages it installs too? Is there maybe a lighter weight or better way to do that, or is that just easiest?

Could we maybe add a note clarifying exactly what it's used for (and not used for, e.g. actual cloud management) and that Gemini is not used or enabled by default?

The exension manages minikube, skaffold, kubectl and their setup and configuration with a single Install click of the extension. I can't think of an easier way. It will also install these dependencies in a seperate folder so it will not interfere with your system.

The extension will be using the Cloud component for the GKE development option to be added in a further PR.

Gemini code assist is not enabled by default, added a note.

The extension and its use is described in more details at the top of this document in the Components section.

aarontp · 2024-07-19T00:17:26Z

docs/developer/develop-minikube.md

+## Setup
+### Requirements
+Please install the following requirements on your system.
+* [Docker](https://docs.docker.com/engine/install/), you can install Docker Desktop or only the engine.


I might be misunderstanding how the GCP parts of this work, but if these are not required in that case, can we add a note for which requirements are needed whether you are doing this locally vs. remote?

They are all required. This document describes a non-GKE setup, so a local development setup. Local can be seen here as your own laptop or a GCE machine (which is considered a local machine in the cloud...lol).
The remote k8s development option running on GKE will be added in follow-up PRs.

I see, makes sense. Could we maybe add a note about this too to make it clear how we're using the GCE instance with minikube there locally?

Added note to the introduction section where the GCE machine is mentioned.

aarontp · 2024-07-19T00:27:21Z

docs/developer/develop-minikube.md

+---
+NOTE: This setup has been tested and is in active use by the Turbinia developers in the following configurations.
+* MacAir M3 8GB with Docker Desktop
+* GCE (e2-standard-4) Debian 12/bookworm 16GB with Docker Engine (Running on GCP using the [VSCode Remote SSH extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh))


How does this work with GCE then, is minikube still deployed locally on the GCE instance? Are there any restrictions or special config to do things like attaching GCP disks?

Yes, the GCE machine should be seen as a local machine in the cloud. The attaching of GCP disks is only possible in Turbinia when running under GKE due to the fact on how we connect disks to Nodes instead of Pods. The GKE development option will be added in a follow-up PR.

I think attaching to a GCE instance should actually work, but not sure how running under minikube might change that. I think we'd need to map in /dev if we're not already. AFAIK we actually attach to the node and not the pod which is why we need the resource tracking code so we don't detach something already attached when another worker/node is processing it.

turbinia/turbinia/processors/google_cloud.py

Line 106 in 1617dcc

instance = project.compute.GetInstance(instance_name)

turbinia/turbinia/evidence.py

Line 572 in 1617dcc

is_detachable = resource_manager.PostProcessResourceState(

Could be but I prefer a standard setup on all machines/environments for development and not customize it for different machines. If you want to develop by processing GCE disks the GKE development option is the way to go.

docs/developer/develop-minikube.md

hacktobeer · 2024-07-19T07:33:02Z

Folks, thanks for the reviews! @aarontp Thanks for the comments, I have answered them. Feel free to reply and/or resolve if all clear.

aarontp · 2024-07-20T18:22:04Z

Folks, thanks for the reviews! @aarontp Thanks for the comments, I have answered them. Feel free to reply and/or resolve if all clear.

LGTM, maybe with a couple clarifying comments as mentioned above. Thanks!

hacktobeer · 2024-07-21T08:30:15Z

@aarontp thanks for the good feedback, added some notes and explanations to the document based on your feedback. PTAL.

hacktobeer added 30 commits September 28, 2023 20:48

helm config

8fe6eac

Merge branch 'master' into skaffold

b0d5abb

Add dockerignore, skaffold config and dev Dockerfiles.

62c8fad

pvc and server dev dockerfile

dcaa2d9

port forwards and change tmp ownership to turbinia

bca3294

hot reloading through jurigged

510f654

rename skaffold file

7e18e7f

tune

d201fc7

merge dev in main Dockerfile

5d696c4

merge server dockerfile and delete dev dockerfiles

a534502

update cloudbuild config, build script and READMEs

ab0abb6

add hotreload and debug.py

2fcfee7

add debug.py source

b8a8942

devcontainer for Google Cloud Code

cdcdd8a

add GKE skaffold setup

b99eb17

cacheFrom for builds

8589e93

Merge branch 'master' into skaffold

52d8427

worker update

75c1f23

whitespace

7d8ea2b

more ignore

0fb2ad2

buildargs to build step to debug/hotreload

5ce183b

args

5723263

unwind changes

fef5d04

remove devcontainer minikube

a2b455f

cleanup

8575eec

Merge branch 'master' into skaffold

43397fa

docker env set

cd2c5b9

add debugging modules

dfcd2c8

add debug port

d6e3f19

add launch and extension vscode config

8b5b4f6

hacktobeer added 5 commits July 4, 2024 14:14

development setup docs and screenshots.

646f8f2

Forward debug ports as well for remote debugging, add some more doc d…

a272f91

…etails.

Enter component descriptions.

3269053

change note look

bec1d0f

change screenshot and fix typos

1fcac8f

hacktobeer requested review from jleaniz, wajihyassine and aarontp July 5, 2024 08:05

hacktobeer added 6 commits July 5, 2024 10:13

new debug image

28f3cd5

add to toctree

c7935c5

fix sphinx error in parsing anchors....

ba7b1e7

move devcontainer back

41659d3

remove boiletplate comments

9c1af92

make worker dockerfile consistent with server dockerfile

2db32aa

wajihyassine reviewed Jul 15, 2024

View reviewed changes

docs/developer/develop-minikube.md Show resolved Hide resolved

docs/developer/develop-minikube.md Outdated Show resolved Hide resolved

docs/developer/develop-minikube.md Show resolved Hide resolved

skaffold.yaml Show resolved Hide resolved

PR comments Wajih

5757507

hacktobeer requested a review from wajihyassine July 18, 2024 10:40

jleaniz approved these changes Jul 18, 2024

View reviewed changes

wajihyassine approved these changes Jul 18, 2024

View reviewed changes

aarontp approved these changes Jul 19, 2024

View reviewed changes

hacktobeer added 2 commits July 19, 2024 09:16

hot reload doc note added

bad43b7

use local dev version of turbinia-client instead of public pip version.

8979ac0

hacktobeer requested a review from aarontp July 19, 2024 07:33

comments added

fa59170

improvements and mention Cloud Shell Editor.

c8231b1

hacktobeer merged commit 4d3b5d0 into google:master Jul 25, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New development environment + setup document #1507

New development environment + setup document #1507

hacktobeer commented Jul 5, 2024

wajihyassine left a comment

hacktobeer commented Jul 18, 2024

jleaniz left a comment

wajihyassine left a comment

aarontp left a comment

aarontp Jul 19, 2024

hacktobeer Jul 19, 2024

aarontp Jul 20, 2024

hacktobeer Jul 21, 2024

aarontp Jul 19, 2024

hacktobeer Jul 19, 2024

aarontp Jul 20, 2024

hacktobeer Jul 21, 2024

aarontp Jul 19, 2024

hacktobeer Jul 19, 2024

aarontp Jul 20, 2024 •

edited

Loading

hacktobeer Jul 21, 2024

hacktobeer commented Jul 19, 2024

aarontp commented Jul 20, 2024

hacktobeer commented Jul 21, 2024

New development environment + setup document #1507

New development environment + setup document #1507

Conversation

hacktobeer commented Jul 5, 2024

Description of the change

Applicable issues

Additional information

Checklist

wajihyassine left a comment

Choose a reason for hiding this comment

hacktobeer commented Jul 18, 2024

jleaniz left a comment

Choose a reason for hiding this comment

wajihyassine left a comment

Choose a reason for hiding this comment

aarontp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarontp Jul 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hacktobeer commented Jul 19, 2024

aarontp commented Jul 20, 2024

hacktobeer commented Jul 21, 2024

aarontp Jul 20, 2024 •

edited

Loading