Shimakaze is Vtuber Wikia scraper and REST API.
- Save vtuber details
- Vtuber data
- Vtuber channels data and videos
- Youtube
- Twitch
- Bilibili
- Niconico
- Save agency's vtuber list
- Auto update vtuber & agency data (cron)
- Interchangeable cache
- no cache
- inmemory
- Redis
- Interchangeable pubsub
- Swagger
- Docker
- Newrelic monitoring
- HTTP
- Cron
- Database
- Cache
- Pubsub
- External API
More will be coming soon...
- Go
- MongoDB
- Youtube API key
- Twitch Client id & secret
- PubSub (RabbitMQ/Redis/Google PubSub)
- (optional) Cache (Redis)
- (optional) Docker & Docker Compose
- (optional) Newrelic license key
Without Docker & Docker Compose
- Clone the repository.
git clone github.com/rl404/shimakaze
- Rename
.env.sample
to.env
and modify the values according to your setup. - Run. You need at least 2 consoles/terminals.
# Run the API.
make
# Run the consumer.
make consumer
- localhost:45001 is ready (port may varies depend on your
.env
).
# Update old vtuber data.
make cron-update
# Fill missing vtuber data.
make cron-fill
With Docker & Docker Compose
- Clone the repository.
git clone github.com/rl404/shimakaze
- Rename
.env.sample
to.env
and modify the values according to your setup. - Get docker image.
# Pull existing image.
docker pull rl404/shimakaze
# Or build your own.
make docker-build
- Run the container. You need at least 2 consoles/terminals.
# Run the API.
make docker-api
# Run the consumer.
make docker-consumer
- localhost:45001 is ready (port may varies depend on your
.env
).
# Update old vtuber data.
make docker-cron-update
# Fill missing vtuber data.
make docker-cron-fill
# Stop running containers.
make docker-stop
Env | Default | Description |
---|---|---|
SHIMAKAZE_APP_ENV |
dev |
Environment type (dev /prod ). |
SHIMAKAZE_HTTP_PORT |
45001 |
HTTP server port. |
SHIMAKAZE_HTTP_READ_TIMEOUT |
5s |
HTTP read timeout. |
SHIMAKAZE_HTTP_WRITE_TIMEOUT |
5s |
HTTP write timeout. |
SHIMAKAZE_HTTP_GRACEFUL_TIMEOUT |
10s |
HTTP graceful timeout. |
SHIMAKAZE_CACHE_DIALECT |
inmemory |
Cache type (nocache /redis /inmemory ) |
SHIMAKAZE_CACHE_ADDRESS |
Cache address. | |
SHIMAKAZE_CACHE_PASSWORD |
Cache password. | |
SHIMAKAZE_CACHE_TIME |
24h |
Cache time. |
SHIMAKAZE_DB_ADDRESS |
mongodb://localhost:27017 |
Database address with port. |
SHIMAKAZE_DB_NAME |
shimakaze |
Database name. |
SHIMAKAZE_DB_USER |
Database username. | |
SHIMAKAZE_DB_PASSWORD |
Database password. | |
SHIMAKAZE_PUBSUB_DIALECT |
rabbitmq |
Pubsub type (rabbitmq /redis /google ) |
SHIMAKAZE_PUBSUB_ADDRESS |
Pubsub address (if you are using google , this will be your google project id). |
|
SHIMAKAZE_PUBSUB_PASSWORD |
Pubsub password (if you are using google , this will be the content of your google service account json). |
|
SHIMAKAZE_CRON_UPDATE_LIMIT |
10 |
Vtuber & agency count limit when updating old data. |
SHIMAKAZE_CRON_FILL_LIMIT |
10 |
Vtuber & agency count limit when filling missing data. |
SHIMAKAZE_CRON_AGENCY_AGE |
7 |
Age of old agency data (in days). |
SHIMAKAZE_CRON_ACTIVE_AGE |
1 |
Age of old active vtuber data (in days). |
SHIMAKAZE_CRON_RETIRED_AGE |
7 |
Age of old retired vtuber data (in days). |
SHIMAKAZE_NEWRELIC_NAME |
shimakaze |
Newrelic application name. |
SHIMAKAZE_NEWRELIC_LICENSE_KEY |
Newrelic license key. | |
SHIMAKAZE_YOUTUBE_KEY |
Youtube API key. | |
SHIMAKAZE_YOUTUBE_MAX_AGE |
60 |
Age limit of youtube videos (in days). |
SHIMAKAZE_TWITCH_CLIENT_ID |
Twitch client id. | |
SHIMAKAZE_TWITCH_CLIENT_SECRET |
Twitch client secret. | |
SHIMAKAZE_TWITCH_MAX_AGE |
60 |
Age limit of twitch videos (in days). |
SHIMAKAZE_BILIBILI_MAX_AGE |
60 |
Age limit of bilibili videos (in days). |
SHIMAKAZE_NICONICO_MAX_AGE |
60 |
Age limit of niconico videos (in days). |
Shimakaze's name is taken from one of the fastest japanese destroyer. Also, exists in Kantai Collection games and manga.
Shimakaze is meant for educational purpose and personal usage only. Please use it responsibly according to Wikia License.
All data belong to their respective copyrights owners, shimakaze does not have any affiliation with content providers.
MIT License
Copyright (c) 2023 Axel