Skip to content

Event-Driven Python on AWS #CloudGuruChallenge

Notifications You must be signed in to change notification settings

Naz513/ETLCovid19Project

Repository files navigation

#CloudGuruChallenge - Event-Driven Python on AWS ☁️

AWS Challenge by A Cloud Guru

Challenge Link 🔗

Cloud Guru Challenge

Challenge Goal 🥅

Automate an ETL processing pipeline for COVID-19 data using Python and AWS Cloud Services

Project Architecture 🔧

The following image displays the entire architecture developed using THE SERVERLESS FRAMEWORK

Architecture

Tech Stack 👷

The following tech stacks were used:

  1. Python
  2. AWS
  3. The Serverless Framework
  4. Github

Services 🚚

The following AWS Services are used for this project:

  1. S3 Bucket
  2. Glue Crawler
  3. Glue Database
  4. IAM Roles
  5. DynamoDB Table
  6. Cloudwatch Event Rule
  7. Lambda Function
  8. Cloudwatch Logs
  9. SNS Subscription
  10. SNS Topic

Tests 🛠️

Used Python's built-in module, Unittest, to perform some simple tests, to ensure that the code can handle unexpected situations and subsitute invalid data.

Test

SNS Email 📧

Depending on the result of the process, a notification will be sent out to all the subscribers through AWS SNS

SNS_EMAIL

QuickSight Dashboard 🏆

Dashboard developed using Quicksight and Data retrived from Athena

QuickSight Dashboard

Blog Link! 📖

https://mohdsaquib.hashnode.dev/event-driven-covid-19-data-etl-process-acloudguruchallenge

Got Questions? ❓

Feel free to email me at [email protected]