UltraStar scrapper

This is a WIP project to build UltraStar songs scrapper from probably the biggest song database. I've done a similar project in past but I've always had a bigger idea for it and I finally have time and desire to do so.

Code structure

This project is using bun as package manager alongside turborepo. I have developed this project using [email protected], but any version above 1.1 should do. Packages can be found in /packages directory with the most notable being ultrastar-api.

How to run this application?

First, let's start with creating an account on usdb.animux.de. Go ahead and register a new account and save the credentials into .env file. Make sure to install yt-dlp. It is necessary for automatic video downloading.

Then you need to install dependencies with bun i and create database using bun db:push

Then you can run this application in development mode using bun dev.

Caution

I've had several issues with bun, turbo and Next.js causing memory leaks. It is caused probably because Node.js is used to run the dev server (app router currently relies on Node.js APIs that Bun does not yet implement). To circumvent this issue, you can run api using bun dev -- filter @ultrastar/api and web app using cd apps/web && bun dev.

Why are there no songs?

For now, you have to downloads songs by yourself. In the future I will upload default database file, but it is over 200MB (mostly cover images). To seed the database by yourself, start the application and head over to http://localhost:3000/api/swagger. Once there, head over to Sync and choose /api/sync/download and click Test Request, then press Send Button. It will cause to download every song (over 27000!). It will take several minutes depending on your machine and internet connection (it took ~20 minutes for me).

Important

Swagger will return timeout after some time, but it doesn't mean it isn't downloading! Once every song is downloaded, Elysia will output 🔥 Download complete to console.

After that step is complete, head over to main page and enjoy your fresh songs.

What's inside?

This Turborepo includes the following packages/apps:

Apps and Packages

Apps:

• @ultrastar/api: Backend API server using Elysia.js;
• @ultrastar/web: Next.js application for frontend;

Packages:

• @ultrastar/libs: Shared libraries like Elysia's Eden connector;
• @ultrastar/ts-config: tsconfig.jsons used throughout the monorepo;
• @ultrastar/types: Custom types used throughout the project like pieces of TS Reset or API types;
• @ultrastar/ui: UI design system using shadcn/ui;
• @ultrastar/ultrastar-api: Scrapper API;
• @ultrastar/youtube-api: Youtube API for downloading and searching videos;

Each package/app is 100% TypeScript (besides some configs, to be fixed).

Build

To build all apps and packages, run the following command:

bun build

Remote Caching

Turborepo can use a technique known as Remote Caching to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.

By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can create one, then enter the following commands:

cd ultrastar
npx turbo login

This will authenticate the Turborepo CLI with your Vercel account.

Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:

npx turbo link

Useful Links

Learn more about the power of Turborepo:

• Tasks
• Caching
• Remote Caching
• Filtering
• Configuration Options
• CLI Usage