This is a WIP project to build UltraStar songs scrapper from probably the biggest song database. I've done a similar project in past but I've always had a bigger idea for it and I finally have time and desire to do so.
This project is using bun as package manager alongside turborepo.
I have developed this project using [email protected], but any version above 1.1 should do.
Packages can be found in /packages
directory with the most notable being ultrastar-api
.
First, let's start with creating an account on usdb.animux.de. Go ahead and register a new account and save the credentials into .env
file.
Make sure to install yt-dlp. It is necessary for automatic video downloading.
Then you need to install dependencies with bun i
and create database using bun db:push
Then you can run this application in development mode using bun dev
.
Caution
I've had several issues with bun, turbo and Next.js causing memory leaks. It is caused probably because Node.js is used to run the dev server (app router currently relies on Node.js APIs that Bun does not yet implement). To circumvent this issue, you can run api using bun dev -- filter @ultrastar/api
and web app using cd apps/web && bun dev
.
For now, you have to downloads songs by yourself. In the future I will upload default database file, but it is over 200MB (mostly cover images).
To seed the database by yourself, start the application and head over to http://localhost:3000/api/swagger
.
Once there, head over to Sync and choose /api/sync/download
and click Test Request, then press Send Button.
It will cause to download every song (over 27000!). It will take several minutes depending on your machine and internet connection (it took ~20 minutes for me).
Important
Swagger will return timeout after some time, but it doesn't mean it isn't downloading! Once every song is downloaded, Elysia will output 🔥 Download complete
to console.
After that step is complete, head over to main page and enjoy your fresh songs.
This Turborepo includes the following packages/apps:
Apps:
@ultrastar/api
: Backend API server using Elysia.js;@ultrastar/web
: Next.js application for frontend;
Packages:
@ultrastar/libs
: Shared libraries like Elysia's Eden connector;@ultrastar/ts-config
:tsconfig.json
s used throughout the monorepo;@ultrastar/types
: Custom types used throughout the project like pieces of TS Reset or API types;@ultrastar/ui
: UI design system using shadcn/ui;@ultrastar/ultrastar-api
: Scrapper API;@ultrastar/youtube-api
: Youtube API for downloading and searching videos;
Each package/app is 100% TypeScript (besides some configs, to be fixed).
To build all apps and packages, run the following command:
bun build
Turborepo can use a technique known as Remote Caching to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.
By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can create one, then enter the following commands:
cd ultrastar
npx turbo login
This will authenticate the Turborepo CLI with your Vercel account.
Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:
npx turbo link
Learn more about the power of Turborepo: