Skip to content

Commit

Permalink
pulumi for gcp gke
Browse files Browse the repository at this point in the history
  • Loading branch information
asacristani committed Oct 9, 2023
1 parent 05b78f8 commit 78a2c9e
Show file tree
Hide file tree
Showing 7 changed files with 604 additions and 20 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ node_modules
# GENEREATE SDK CLIENT
src
openapi.json

# PULUMI
*.pyc
4 changes: 4 additions & 0 deletions Pulumi.dev.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
config:
fastapi-rocket-boilerplate:nodesPerZone: "1"
gcp:project: fastapi-rocket-boilerplate
gcp:region: us-central1
6 changes: 6 additions & 0 deletions Pulumi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
name: fastapi-rocket-boilerplate
runtime:
name: python
options:
virtualenv: venv
description: A Python program to deploy a Kubernetes cluster on Google Cloud
78 changes: 62 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@
<a href="https://www.docker.com/">
<img src="https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white" alt="Docker">
</a>
<a href="https://www.docker.com/">
<img src="https://img.shields.io/badge/Pulumi-8A3391?style=for-the-badge&logo=pulumi&logoColor=white" alt="Pulumi">
</a>
</p>

<p align="center"> Also sqlmodel, pydantic, alembic, poetry, ...</p>
Expand All @@ -51,7 +54,8 @@

## 🧩 Features

- **Infrastructure**: the common services that every backend needs, served by Docker Compose.
- **Infrastructure**: the common services that every backend needs, served in local by Docker Compose
and in Google Cloud by Pulumi.
- **Easy**: all the commands ready by Makefile.
- **Fast**: thanks to Fastapi and async programming.
- **Async**: Celery using RabbitMQ as broker and Redis as backend.
Expand All @@ -65,8 +69,8 @@
## ⚙️ Requirements
- [Python 3.11](https://www.python.org/downloads/release/python-3114/)
- [Docker](https://docs.docker.com/engine/install/)
- [Node](https://nodejs.org/en) (only for SDK frontend generation)

- [Node](https://nodejs.org/en) only for SDK frontend generation
- [Pulumi](https://www.pulumi.com/) only for deploying

## 🎛️ Use
### 🔧 Installation
Expand Down Expand Up @@ -200,19 +204,61 @@ You should change the next env vars in `.env`:

Also, it is possible you want to modify the expiry time of access/refresh tokens.

## 🚀 Future features
### Admin
- Search events by model AND id
- Fix popup for reverse_delete
- Relationship of records into model details (performance)
## 🚀 Deploy
We use Pulumi for deploying.


### Others
- Deployment with Kubernetes in Google Cloud by Terraform
- Add logging
### Google Cloud
The DB will be deployed in GCP SQL service.

The rest of the services will be deployed in GCP GKE.

- Integrity tests
- Cover 100% with unit-testing
- Add mypy and pylint to the Pre-commit
- Use async/await for routes and database connections
- Authentication client with Google
1. Log in Google cloud
```shell
gcloud auth application-default login
```
2. Create a new project in Google console
3. Set the project in gcloud
```shell
gcloud config set project <YOUR_GCP_PROJECT_ID>
```
To find it, you can run:
```shell
gcloud projects list
```
4. Modify `Pulumi-dev.yaml` with your GCP project and region.
5. Modify the begging of `__main__.py` with your variables
6. run:
```
pulumi up
```

✅ It is done, your project is alive!


## 🔮 Future features

### Deployment
- [x] Deployment with Kubernetes in Google Cloud by Pulumi
- [ ] Deployment with Kubernetes in AWS by Pulumi

### Monitoring
- [ ] Add logging
- [ ] Add Sentry
- [ ] Add Flower

### Testing
- [ ] Integrity tests
- [ ] Cover 100% with unit-testing
- [ ] Add mypy and pylint to the Pre-commit

### Async
- [ ] Use 100% async/await for routes and database connections

### Auth
- [ ] Authentication client with Google

### Admin
- [ ] Search events by model AND id
- [ ] Fix popup for reverse_delete
- [ ] Relationship of records into model details (performance)
190 changes: 190 additions & 0 deletions __main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
"""A Google Cloud Python Pulumi program"""

import pulumi
import pulumi_gcp as gcp
import pulumi_kubernetes as k8s
from pulumi_gcp import storage

# SET YOUR VARIABLES
db_instance_name = "instance"
region = "us-central1"
db_tier = "db-f1-micro"
db_user = "myuser"
db_password = "mypassword"


# See versions at https://registry.terraform.io
# /providers/hashicorp/google/latest/docs/resources
# /sql_database_instance#database_version
instance = gcp.sql.DatabaseInstance(
db_instance_name,
region=region,
database_version="POSTGRES_15",
settings=gcp.sql.DatabaseInstanceSettingsArgs(
tier=db_tier,
),
deletion_protection=True,
)

database_user = gcp.sql.User(
db_user, instance=instance.name, password=db_password
)

database = gcp.sql.Database("database", instance=instance.name)

# Create a GCP resource (Storage Bucket)
bucket = storage.Bucket("my-bucket", location="US")

# Export the DNS name of the bucket
pulumi.export("bucket_name", bucket.url)


# Define your Google Cloud project
project = "your-gcp-project"
gcp_config = pulumi.Config("gcp")
project = gcp_config.get("project") or project

# Configure the GKE cluster
cluster_name = "my-gke-cluster"
cluster = gcp.container.Cluster(
cluster_name,
initial_node_count=1,
min_master_version="latest",
node_version="latest",
project=project,
location="us-central1",
)

# Use gcp.container.getClusterCredentials to get Kubernetes config
kubeconfig = pulumi.Output.all(cluster.name, project).apply(
lambda args: gcp.container.get_cluster_credentials(
cluster_name=args[0], project=args[1]
)
)

# Define the configuration for your Kubernetes services

# App Deployment
app_deployment = k8s.apps.v1.Deployment(
"app-deployment",
metadata={
"name": "app-deployment",
},
spec={
"replicas": 2,
"selector": {
"matchLabels": {"app": "app"},
},
"template": {
"metadata": {"labels": {"app": "app"}},
"spec": {
"containers": [
{
"name": "app-container",
"image": "your-app-image:latest",
"ports": [{"containerPort": 8000}],
# You can add additional
# configurations and resources here
}
]
},
},
},
)

# Celery Deployment
celery_deployment = k8s.apps.v1.Deployment(
"celery-deployment",
metadata={
"name": "celery-deployment",
},
spec={
"replicas": 2, # Adjust as needed
"selector": {
"matchLabels": {"app": "celery"},
},
"template": {
"metadata": {"labels": {"app": "celery"}},
"spec": {
"containers": [
{
"name": "celery-container",
"image": "your-celery-image:latest",
# Add any necessary command and args here
# You can add additional configurations
# and resources here
}
]
},
},
},
)

# Redis Deployment
redis_deployment = k8s.apps.v1.Deployment(
"redis-deployment",
metadata={
"name": "redis-deployment",
},
spec={
"replicas": 1, # Redis is typically deployed with a single node
"selector": {
"matchLabels": {"app": "redis"},
},
"template": {
"metadata": {"labels": {"app": "redis"}},
"spec": {
"containers": [
{
"name": "redis-container",
"image": "redis:latest",
"ports": [{"containerPort": 6379}],
# You can add additional configurations
# and resources here
}
]
},
},
},
)

# RabbitMQ Deployment
rabbitmq_deployment = k8s.apps.v1.Deployment(
"rabbitmq-deployment",
metadata={
"name": "rabbitmq-deployment",
},
spec={
"replicas": 1, # RabbitMQ is typically deployed with a single node
"selector": {
"matchLabels": {"app": "rabbitmq"},
},
"template": {
"metadata": {"labels": {"app": "rabbitmq"}},
"spec": {
"containers": [
{
"name": "rabbitmq-container",
"image": "rabbitmq:3.6-management",
"ports": [
{"containerPort": 5672},
{"containerPort": 15672},
], # RabbitMQ ports
# You can add additional configurations
# and resources here
}
]
},
},
},
)

# Export useful information
pulumi.export("cluster_name", cluster.name)
pulumi.export("kubeconfig", cluster.kube_config)

# Deploy the infrastructure
pulumi.export("app_deployment", app_deployment.metadata)
pulumi.export("celery_deployment", celery_deployment.metadata)
pulumi.export("redis_deployment", redis_deployment.metadata)
pulumi.export("rabbitmq_deployment", rabbitmq_deployment.metadata)
Loading

0 comments on commit 78a2c9e

Please sign in to comment.