- Core Features
- API Development Life Cycle
- API Testing
- API Documenation
- CQRS Pattern
- AWS Infrastructure Design (Improvement)
- Further Improvements
- Ability to submit URL
https://really-awesome-long-url.com
to API (POST request
):
% curl -X POST \
-H "Content-Type: application/json" \
-d '{"url": "https://this-is-my-sample-original-url"}' \
https://cf20zm25j7.execute-api.us-east-1.amazonaws.com/v1/urls
Response:
736339761
-
Should receive
hashCode
associated with original URL. This approach allowsmultiple
short-url domains to interact with this API. -
hashCode
can then be used to buildhttps://<api-id>.execute-api.<region>.amazonaws.com/v1/urls/hashCode
and should return original URLhttps://really-awesome-long-url.com
(GET request
):
% curl https://<api-id>.execute-api.<region>.amazonaws.com/v1/urls/736339761
Response:
https://this-is-my-original-url
Clarification: once a Route53
record is configured with a custom domain name, the full production URL should look like this: https://shrtnr.com/hashCode
-
hashCode
generation can be as simple or complicated as required. In order to create a unique hash from a specific string, it can be implemented using:4.a. Its own
string-to-hash
converting function. It will return the hash equivalent of a string. (approach implemented)4.b.
N digits
hashCode composed of[0-9a-zA-Z]
types of characters (a-z
represent 26 characters +A-Z
represent 26 characters +0-9
equals62
characters in total).This is BASE62 encoding.
Which provides with
62^N
possibilities for IDs -> ForN = 5
-> Total amount of unique IDs:916.132.832
.For URLs that require to be
human readable
, there is a potential issue with BASE62 enconding since0
(NUMBER) andO
(LETTER) can be confused. Same applies forl
(lowercase LETTER) andI
(capital LETTER). Removing these 4 characters, leaves us with BASE58 enconding which is better forhuman readable
URLs purpose.4.c. Also, a library named
Crypto
can be used to generate various types of hashes likeSHA1
,MD5
,SHA256
, and many more. (further development)
Reference: https://www.geeksforgeeks.org/how-to-create-hash-from-string-in-javascript/
- Clone repository.
- Validate Terraform <-> Github Actions <-> AWS integration: https://developer.hashicorp.com/terraform/tutorials/automation/github-actions
- Adjuste
0-providers.tf
file to your own Terraform workspace specifications.
- Create a new branch from
main
. - Create a new
NodeJS
function folder. Runnpm init
&npm install <module>
as you need. - Create a new
Lambda function
throughTerraform
. - Create a new
Terraform Integration
for said Lambda function. - Create
unit
,integration
,load_test
tests for said Lambda function. - AWS Lambda functions can be tested locally using
aws invoke
command (https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html). - Apply
linting
best practices to new function file. - Add
unit
,integration
,load_test
steps into Github Actions (ci_cd.yml
) following the same pattern as other lambda functions. - Commit changes in your
feature branch
and create aNew Pull Request
. - Pre Deployment
Github Actions
workflow will be triggered in your new branch:
- Validate
workflow run
results. - Once everything is validated by yourself and/or colleagues, push a new commit (it could be an empty one) with the word
[deploy]
. - This will trigger pre deployment and post deployment steps within the
Github Actions
workflow:
-
Once everything is validated by yourself and/or colleagues, you can merge your branch into
main
. -
Once Github Actions workflow is successfully completed, a valuable addition is sending a notification with workflow results into Slack channel/s:
# .github/workflows/ci_cd.yml
...
send-notification:
runs-on: [ubuntu-latest]
timeout-minutes: 7200
needs: [linting, unit_tests, deployment, integration_tests, load_tests]
if: ${{ always() }}
steps:
- name: Send Slack Notification
uses: rtCamp/action-slack-notify@v2
if: always()
env:
SLACK_CHANNEL: devops-sample-slack-channel
SLACK_COLOR: ${{ job.status }}
SLACK_ICON: https://avatars.githubusercontent.com/u/54465427?v=4
SLACK_MESSAGE: |
"Lambda Functions (Linting): ${{ needs.linting.outputs.status || 'Not Performed' }}" \
"Lambda Functions (Unit Testing): ${{ needs.unit_tests.outputs.status || 'Not Performed' }}" \
"API Deployment: ${{ needs.deployment.outputs.status }}" \
"API Tests (Integration): ${{ needs.integration_tests.outputs.status || 'Not Performed' }}" \
"API Tests (Load): ${{ needs.load_tests.outputs.status || 'Not Performed' }}"
SLACK_TITLE: CI/CD Pipeline Results
SLACK_USERNAME: Github Actions Bot
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
CI/CD Slack Notification example:
Testing is conducted on 3 steps within Github Actions workflow:
- Lambda Functions (Unit testing) - Query Lambda Function
- API Testing (Integration) - Command Lambda Function
- API Testing (Load) - Query Lambda Function
-
Deployment can be triggered from
GIT commit messages
by including[deploy]
within a commit message. -
Deployment can be triggered
manually
by using Terraform CLI withinterraform
folder in repository:
% cd terraform
% terraform init
% terraform apply
-
Pre Deployment
linting
andunit_tests
steps are triggered by Github Actions. -
Post Deployment
integration_tests
andload_tests
steps are triggered by Github Actions.
- Github Actions workflow can be customized here:
# .github/workflows/ci_cd.yml
name: "CI/CD Pipeline"
on:
push:
paths:
- "terraform/**"
- ".github/workflows/**"
branches:
- main
pull_request:
...
-
Swagger / OpenAPI
YAML
documentation file (format easier to read & maintain) created following standard guidelines: https://github.com/juanroldan1989/terraform-url-shortener/blob/main/terraform/docs/api/v1/main.yaml -
YAML
file converted intoJSON
(sinceSwagger UI
script requires aJSON
file):
docs/api/v1% brew install yq
docs/api/v1% yq -o=json eval main.yml > main.json
-
JSON
file can be accessed through:3.a.
Github repository
itself as: https://raw.githubusercontent.com/github_username/terraform-url-shortener/main/docs/api/v1/main.yaml or3.b.
S3 bucket
that will containmain.yml
. Bucket created and file uploaded through Terraform. URL provided throughoutput
terraform command. Sample Terraform Code
- Both file accessibility options available within this repository.
-
static
API Documentationstandalone
HTML page generated withindocs/api/v1
folder in repository: https://github.com/swagger-api/swagger-ui/blob/master/docs/usage/installation.md#plain-old-htmlcssjs-standalone -
Within
static
API Documentation page, replaceurl
value with your ownJSON
file's URL from point3
above:
...
<script>
window.onload = () => {
window.ui = SwaggerUIBundle({
// url: "https://docs-api-v1-file-url-from-point-3.com",
dom_id: '#swagger-ui',
...
- A
static website
can also be hosted withinS3 Bucket
: https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html
- To upload files
aws sync
command is recommended. E.g.:aws s3 sync docs/api/v1 s3://$YOUR_BUCKET_NAME
Pattern implemented within REST API to handle read/write requests.
https://apisix.apache.org/blog/2022/09/23/build-event-driven-api/
CQRS stands for Command and Query Responsibility Segregation
, a pattern that separates reads and writes into different models, using commands to update data, and queries to read data.
query
and upsert
(updates or creates) responsibilities are split (segregated) into different services (e.g.: AWS Lambda Functions)
Technically, this can be implemented in HTTP so that the Command API
is implemented exclusively with POST routes
, while the Query API
is implemented exclusively with GET routes
.
For high number of POST
requests, an improvement is to decouple command
Lambda function from DynamoDB
table by adding an SQS Queue
in between.
command
Lambda function no longer writes to DynamoDB
table.
This way:
command
Lambda function sendsurl
attributes intocommand
SQS Queue as message:
// APPROACH 1
// Lambda function persists `url` record into `urls` DynamoDB Table
// await ddb.putItem(params).promise();
// APPROACH 2
// Lambda function sends `url` attributes into `command` SQS Queue as message.
const messageParams = {
MessageAttributes: {
Author: {
DataType: "String",
StringValue: "URL Shortener API - `command` Lambda Function",
},
},
MessageBody: JSON.stringify(params),
QueueUrl: "https://sqs.<region>.amazonaws.com/<account-id>/command-sqs-queue",
};
- SQS Queue message is picked up by
upsert
Lambda function. upsert
Lambda function persists record intourls
DynamoDB Table:
event.Records.forEach(async (record) => {
const message = record.body;
try {
messageJSON = JSON.parse(message);
if (!messageJSON.Item) {
throw new Error("`Item` not provided within message");
}
if (!messageJSON.Item.Id) {
throw new Error("`Item.Id` not provided within message");
}
if (!messageJSON.Item.OriginalUrl) {
throw new Error("`Item.OriginalUrl` not provided within message");
}
params["Item"] = { Id: { S: messageJSON.Item.Id.S } };
params["Item"]["OriginalUrl"] = { S: messageJSON.Item.OriginalUrl.S };
await ddb.putItem(params).promise();
} catch (err) {
statusCode = 400;
responseBody = err.message;
}
});
This set of improvements include services like:
AWS Route 53 (DNS), AWS CloudFront and S3 (storing and distributing static content: HTML/CSS/JS) and AWS Cognito (Authentication).
AWS ElastiCache is also posible to implement for READ operations within AWS Query Lambda function.
New features (or improvements) that come to mind while working on core features are placed on this list
- API Rate Limiting
In order to avoid malicious requests (e.g.: bots) attempting to:
-
Used up all possibilities for unique codes or
-
Sabotage API's availability for other users.
Rate Limiting is a good improvement to avoid those scenarios and can be accomplished by:
-
Implementing a Captcha step within frontend app.
-
Generate Free/Basic/Premium membership plans (
API Token
) within AWS API Gateway and set daily/weekly request limits for users based on membership plans.
- HTTP Redirections When returning an existing URL, we should return a 302 HTTP code for future client's request:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections#permanent_redirections
- Infrastructure code refactoring:
-
Implement
modules
with parameters alongTerraform
Lambda functions and API Gateway integrations to avoid code duplication. -
Implement a single
main.tf
Terraform file where all resources can be seen referenced and modules implemented. This brings even more clarity when reviewingTerraform
code.
GET /urls/{url}
path parameter can be sent asGET /urls?url={url}
query parameter instead. Adjustaws_api_gateway_integration
terraform resource:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_integration
https://aws.amazon.com/premiumsupport/knowledge-center/pass-api-gateway-rest-api-parameters/
-
POST /urls
request could be triggered twice (or more) with the sameurl
on its payload. Adjust backend to support this scenario. Create record in table for first request. Do not create another record for second request. -
GET /urls/{code}
request could be triggered many times. Consider caching implementation at:- AWS API Gateway level or
- AWS DynamoDB level (DAX)
-
Expand on
CQRS
pattern implementation, for high number ofPOST /urls
requests decouple them by adding anSQS Queue
instead of writing directly intoDynamoDB
table. (Improvement implemented) -
URLs shortened can be
temporal
orpermanent
ones:
-
permanent
URLs need payment by authenticated users first. -
temporal
URLs only last 24hs and can be created through a public endpoint.
-
Task running in background should remove
temporal
URLs from database after 24hs. This could be implemented through a AWSBridge Event - Schedule
rule triggered once every 24hs that is connected to aremove_temporal_urls
Lambda function. -
To increase chances of finding a URL with
GET /urls/{code}
requests, consider pre-generating records in table:
-
Once an enconding (e.g.:
BASE62
,BASE58
, etc) is decided and also -
the number (N) of maximum amount of digits a
hashCode
needs to be, -
we are able to predict the total spectrum of possible values generated,
-
therefore a task running in background to generate this
hashCode
values and insert them in the database will effectively increase chances of finding a requestedhashCode
, leaving only the task of -
associating a
long URL
with ahashCode
duringPOST /urls
requests workflow.
Frontend App can be built with:
-
Any frontend framework such as: Angular, React, NextJS.
-
With jQuery as a static HTML page.