Events Tracking Platform 🚦 Clicks & Hovers on Colors 🎯 Live Graphs & Counters

Core Features • Frontend • Backend • Development • Testing • Deployment • Wiki • Further Improvements • Contribute

Core Features

• Real-time tracking and displaying of user events (clicks and mouseover actions).
• User events information stored in database (PostgreSQL & Redis supported).
• Events stats shared via Action Cable and displayed to subscribed clients or specific clients via polling JS requests.
• Backend implementd with Ruby on Rails, Kafka and Zookeeper.
• Deployment workflow through AWS ECS script.
• Infrastructure managed through Terraform (work in progress).
• Deployment workflow through Terraform and integrated within Github Actions (CI/CD) into AWS. (work in progress)

Idea inception

Development

Relevant folder: backend

$ git clone git@github.com:juanroldan1989/color-tracking.git
$ cd color-tracking
$ cd backend

$ docker-compose up

# cleaning up
$ docker-compose down -v

Open static frontend page:

$ cd frontend
$ open index.html

Relevant folder: /backend/infrastructure/terraform

1. Enable Docker Local Development section in main.tf

2. Run commands:

$ terraform init

$ terraform plan

$ terraform apply

Application is ready

Open static frontend page:

$ cd frontend
$ open index.html

To destroy all local containers managed by your Terraform configuration:

$ terraform destroy

Testing

Backend

API tests written with RSpec

$ cd backend
$ rspec spec

Frontend

Frontend work has been purposely implemented using standalone libraries:

• jQuery (Easy DOM Handling)
• D3 (Graphs)
• Action Cable (Websockets)

── frontend
    └── static
        ├── css
        │   └── main.css
        ├── index.html
        └── js
            ├── draw_colors_table.js
            ├── events
            │   └── clicks_and_hovers.js
            ├── polling
            │   ├── draw_clicks_dashboard.js
            │   └── draw_hovers_dashboard.js
            └── websockets
                ├── action_cable.js
                ├── draw_clicks_dashboard.js
                └── draw_hovers_dashboard.js

This way anyone can use their framework of choice to implement a frontend application that interacts with the backend and write frontend tests as well.

Docker Containers

Automated Verification achieved through Chef InSpec

Chef InSpec is an infrastructure security and compliance testing framework with a human and machine-readable language for comparing actual versus desired system state.

Download and install Inspec: https://www.chef.io/downloads/tools/inspec

Source: https://joachim8675309.medium.com/docker-the-terraform-way-a7c16b5f59ed

Tests for project present in validate_containers_state.rb

PORT_DB = attribute("port", value: "5432", description: "DB port")
PORT_ZOO = attribute("port", value: "2181", description: "ZOOKEEPER port")
PORT_KAFKA = attribute("port", value: "9092", description: "KAFKA port")
PORT_API = attribute("port", value: "3000", description: "API port")
PORT_KARAFKA = attribute("port", value: "3001", description: "KARAFKA port")

control "db-container" do
  impact 1.0
  title "DB Container"
  desc "DB container should be running"

  describe docker_container "db" do
    it { should exist }
    it { should be_running }
    its("image") { should eq "postgres:9.6-alpine" }
    its("ports") { should eq "0.0.0.0:#{PORT_DB}->#{PORT_DB}/tcp" }
  end

  describe json({ command: "docker container inspect db" }) do
    its([0, "HostConfig", "NetworkMode"]) { should include "color-tracking-net" }
  end
end

control "zoopkeeper-container" do
  impact 1.0
  title "Zookeeper Container"
  desc "Zookeeper container should be running"

  describe docker_container "zookeeper" do
    it { should exist }
    it { should be_running }
    its("image") { should eq "wurstmeister/zookeeper" }
    its("ports") { should eq "22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:#{PORT_ZOO}->#{PORT_ZOO}/tcp" }
  end

  describe json({ command: "docker container inspect zookeeper" }) do
    its([0, "HostConfig", "NetworkMode"]) { should include "color-tracking-net" }
  end
end

control "kafka-container" do
  impact 1.0
  title "Kafka Container"
  desc "Kafka container should be running"

  describe docker_container "kafka" do
    it { should exist }
    it { should be_running }
    its("image") { should eq "wurstmeister/kafka" }
    its("ports") { should eq "0.0.0.0:#{PORT_KAFKA}->#{PORT_KAFKA}/tcp" }
  end

  describe json({ command: "docker container inspect kafka" }) do
    its([0, "HostConfig", "NetworkMode"]) { should include "color-tracking-net" }
  end
end

control "api-container" do
  impact 1.0
  title "API Container"
  desc "API container should be running"

  describe docker_container "api" do
    it { should exist }
    it { should be_running }
    its("image") { should eq "juanroldan1989/color-tracking-api" }
    its("ports") { should eq "0.0.0.0:#{PORT_API}->#{PORT_API}/tcp" }
  end

  describe json({ command: "docker container inspect api" }) do
    its([0, "HostConfig", "NetworkMode"]) { should include "color-tracking-net" }
  end
end

control "karafka-consumer-container" do
  impact 1.0
  title "Karafka Consumer Container"
  desc "Karafka Consumer container should be running"

  describe docker_container "karafka-consumer" do
    it { should exist }
    it { should be_running }
    its("image") { should eq "juanroldan1989/color-tracking-api" }
    its("ports") { should eq "0.0.0.0:#{PORT_KARAFKA}->3000/tcp" }
  end

  describe json({ command: "docker container inspect karafka-consumer" }) do
    its([0, "HostConfig", "NetworkMode"]) { should include "color-tracking-net" }
  end
end

Trigger tests:

$ cd backend

$ docker-compose up

$ cd backend/infrastructure/tests

$ inspec exec validate_containers_state.rb

CI/CD Integration

[Work In Progress] Provide Github Actions setup for triggering all frontend/backend and containers tests, THEN deploy.

Deployment

1. Official steps for ECS CLI Configuration: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html

2. Place both ecs-cli.asc and public_key_ecs.txt files inside backend/infrastructure/aws-ecs folder.

3. Login into AWS Console -> EC2 -> Key pairs -> Create key pair

4. Download private-ecs-key.cer and place it inside backend/infrastructure/aws-ecs folder

5. Deployment instructions:

backend/infrastructure/aws-ecs/setup-aws-ecs.sh

This script takes care of:

• Cluster configuration and creation

$ ecs-cli configure --region us-east-1 --cluster color-tracking --config-name color-tracking

• ECS Key Pair configuration, EC2 Instance Types and Cluster capability

$ ecs-cli up --keypair ecs --capability-iam --size 2 --instance-type t3.medium --cluster-config color-tracking --force

• ECS Deployment of tasks with containers

$ ecs-cli compose up --cluster color-tracking --create-log-groups

• Check containers status

$ ecs-cli ps --cluster color-tracking

6. Cluster deletion and resources cleanup

$ ecs-cli down --cluster color-tracking --force

Relevant folder: /backend/infrastructure/terraform

1. Enable AWS DEPLOYMENT section in main.tf

2. Run commands:

$ terraform init

# Observe all resources to be added
$ terraform plan

# Build & Launch infrastructure required
$ terraform apply

Application is ready

Replace http://localhost:3000 instances inside frontend/static/index.html file with Public DNS hostname:

1. Check AWS ECS console: https://console.aws.amazon.com/ecs/home#/clusters

2. Get Public DNS value provided within ECS Cluster -> Container Instance -> Public DNS

3. Open static frontend page:

$ cd frontend
$ open index.html

To destroy all remote objects managed by your Terraform configuration:

$ terraform destroy

Frontend

Frontend work has been purposely implemented using standalone libraries:

• jQuery (Easy DOM Handling)
• D3 (Graphs)
• Action Cable (Websockets)

This way anyone can use their framework of choice to implement a frontend application that interacts with the backend

Backend

Broadcasting Events from backend to frontend

• Websockets implementation in the backend achieved through Rails's own ActionCable

• Websockets implementation in the frontend achieved through action_cable Javascript library.

Database Implementation

Main tables to interact with:

User

• api_key: String

Colors

• id: Integer
• name: String (e.g.: "blue", "red")

Actions

• id: Integer
• name: String (e.g.: "hover", "click")

ActionColors

• id: Integer
• api_key: String
• action_id: Integer (e.g.: "Click")
• color_id: Integer (e.g.: "Red")
• amount: Integer (e.g.: "30")

Cable Ready (Optional)

cable_ready ruby gem is a great extension on ActionCable capabilities

Provides operations to be broadcasted to the frontend.

https://cableready.stimulusreflex.com/#what-can-i-do-with-cableready https://cableready.stimulusreflex.com/cableready-everywhere

Backend adjustments when working with cable_ready:

# Gemfile

...

gem "cable_ready", "~> 4.5.0"

...

# /v1/events_controller.rb
...

cable_ready[stream].console_log(message: { results: results })
cable_ready.broadcast

...

Frontend adjustments when working with cable_ready:

# js/websockets/draw_hovers_dashboard.js
...

console.log("Received data: ", data);

var results = data.operations.consoleLog[0].message.results;

...

Wiki

Do you need some help? Check the articles from the wiki.

Contribute

Got something interesting you'd like to add or change? Please feel free to Open a Pull Request

If you want to say thank you and/or support the active development of Color Tracking API:

1. Add a GitHub Star to the project.
2. Tweet about the project on your Twitter.
3. Write a review or tutorial on Medium, Dev.to or personal blog.