GoXCrap

X (formerly Twitter) web scrapper, written in Go

---

GoXCrap

This application collects tweets based on a defined search criteria, and save them in a database.

Set up & run (locally)

Set up

1. First of all you need to download the Chrome Web Driver that matches with the installed version of Google Chrome (the browser used for testing this project).
   You can download it from here, or you can use @puppeteer/browsers with this installation guide.
   After that, copy it inside the internal/webdriver folder.
2. Create a .env file at the root of the project, and add the following environment variables:

EMAIL=<Twitter account email>
USERNAME=<Twitter username>
PASSWORD=<Twitter password>
AHBCC_DOMAIN=<Domain of the application with the endpoint /tweets/v1> --> In this case the app AHBCC

¹

Run

In the root folder, run:

go run cmd/api/main.go --local

Setting up & run (into a Docker container)

Setup

1. Create a .env file at the root of the project, and add the following environment variables:

EMAIL=<Twitter account email>
USERNAME=<Twitter username>
PASSWORD=<Twitter password>
DRIVER_PATH=<The path to the Chrome driver> --> Example: /usr/bin/chromedriver
BROWSER_PATH=<The path to the Chrome browser> --> Example: /usr/bin/chromium
RABBITMQ_USER=<The RabbitMQ user>
RABBITMQ_PASS=<The RabbitMQ password>
AHBCC_DOMAIN=<Domain of the application with the endpoint /tweets/v1> --> In this case the app AHBCC

¹

Build & Run

docker compose up --build

---

License

MIT

Logo License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

The logo was obtained from https://github.com/ashleymcnamara/gophers, but it was slightly modified to be representative for this repository.

Footnotes

1. AHBCC: Adverse Human Behaviour Corpus Creator. More information here ↩ ↩²