HTTP API for FGO game data. Transform the raw game data into something a bit more manageable.
View the API documentation here: https://api.atlasacademy.io.
If you are looking for only the type definitions and enums. You can download the fgo-api-types
package.
- Configuration
- Development environment set up
- Run the API server
- Architecture
- Linting
- Formatting
- Dependencies
- Testing
- Helper scripts
List of configuration variables for the main app. You can make a config.json
file at the project root with the settings. Check the config.sample.json
file for an example.
Required variables
DATA
: A JSON object with keys being region and values being gamedata's folder location and Postgresql DSN. Not all regions are required in the object. Any combination of regions is accepted.REDISDSN
: Redis DSN to a Redis server for caching.
Optional variables (click to show)
REDIS_PREFIX
: default tofgoapi
. Prefix for redis keys.CLEAR_REDIS_CACHE
: default toTrue
. If set, will clear the redis cache on start and when the webhook above is used.RAYSHIFT_API_KEY
: default to""
. Rayshift.io API key to pull quest data.RAYSHIFT_API_URL
: default to https://rayshift.io/api/v1/. Rayshift.io API URL.QUEST_CACHE_LENGTH
: default to3600
. How long to cache the quest and war endpoints in seconds. Because the rayshift data is updated continously, web and quest endpoints have lower cache time.DB_POOL_SIZE
: defaults to 3. Default pool size for SQLAlchemy connection pool. https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.QueuePool.params.pool_sizeDB_MAX_OVERFLOW
: defaults to 10. Max overflow for SQLAlchemy connection pool. https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.QueuePool.params.max_overflowWRITE_POSTGRES_DATA
: default toTrue
. Overwrite the data in PostgreSQL when importing.WRITE_REDIS_DATA
: default toTrue
. Overwrite the data in Redis when importing.ASSET_URL
: defaults to https://assets.atlasacademy.io/GameData/. Base URL for the game assets.OPENAPI_URL
: default toNone
. Set the server URL in the openapi schema export.EXPORT_ALL_NICE
: default toFalse
. If set toTrue
, at start the app will generate nice data of all servant and CE and serve them at the/export
endpoint. It's recommended to serve the files in the/export
folder using nginx or equivalent webserver to lighten the load on the API server.DOCUMENTATION_ALL_NICE
: default toFalse
. If set toTrue
, there will be links to the exported all nice files in the documentation.GITHUB_WEBHOOK_SECRET
: default to""
. If set, will add a webhook location at/GITHUB_WEBHOOK_SECRET/update
that will pull and update the game data. If it's not set, the endpoint is not created.GITHUB_WEBHOOK_GIT_PULL
: default toFalse
. If set, the app will dogit pull
on the gamedata repos when the webhook above is used.
Other ways to set the variables (click to show)
The variables can also be set as environment variables.
Secret variables can be put in the secrets folder instead of being supplied as environment variable:
> cat .\secrets\rayshift_api_key
eca334a9-3289-4ad7-9b92-1ec2bbc3fc19
> cat .\secrets\redisdsn
redis://localhost
Settings at a higher level will override the settings at a lower level.
- Secrets variable
- Enviornment variable
.env
fileconfig.json
Make sure poetry is installed: https://python-poetry.org/docs/#installation.
Docker is recommended to set up the Postgres and redis servers but those can be set up manually as well. Postgres needs the PGroonga extension installed.
git clone --depth 1 --branch JP https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-jp
git clone --depth 1 --branch NA https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-na
# It's not neccecary to clone the other regions.
git clone --depth 1 --branch CN https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-cn
git clone --depth 1 --branch KR https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-kr
git clone --depth 1 --branch TW https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-tw
git clone https://github.com/atlasacademy/fgo-game-data-api.git
cd fgo-game-data-api
# If you didn't clone other game data regions, remove them from the data field in config.json,
# and the services key in docker-compose.sample.yaml
cp config.sample.json config.json
cp docker-compose.sample.yml docker-compose.yml
docker-compose up -d
# Make sure you have the build prerequisites for psycopg2 installed if you are installing on Linux or macOS.
# https://www.psycopg.org/docs/install.html#build-prerequisites
# Debian/Ubuntu: sudo apt install libpq-dev python3-dev
# CentOS: sudo yum install python-devel postgresql-devel
poetry install
poetry shell
Run at the project root to start the API server:
> uvicorn app.main:app --reload --log-level debug --reload-dir app
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [16680] using watchgod
INFO fgoapi: Loading game data …
INFO fgoapi: Loaded game data in 15.14s.
INFO: Started server process [33312]
INFO: Waiting for application startup.
INFO: Application startup complete.
DEBUG fgoapi: Processed in 0.21ms.
INFO: 127.0.0.1:56759 - "GET / HTTP/1.1" 307 Temporary Redirect
DEBUG fgoapi: Processed in 0.24ms.
INFO: 127.0.0.1:56759 - "GET /rapidoc HTTP/1.1" 200 OK
Go to http://127.0.0.1:8000 for the API documentation.
Tips:
- Change
write_postgres_data
tofalse
after the first run to speed up reloading if it's not needed (schema doesn't change or data hasn't changed).
main.py
: Main entrypoint of the application.routers/
: Routers to deal with incoming requests. The routers call functions fromcore
to get the response data.core/
: Build response data. Get raw data from eitherdb/helpers/
orredis/helpers/
.data/
: Import translation data into memory. Preprocess data to be imported into db and redis.db/
: DB stuffs.db/helpers/
: Functions to be used bycore
to get data from Postgres.
redis/
: Redis stuffs.redis/helpers/
: Functions to be used bycore
to get data from Redis.
schemas/
: Response Pydantic models.models/
: SQLAlchemy Core Tables.
ruff and mypy are used to lint the code. ruff's configuration and mypy's configuration are in pyproject.toml.
./scripts/lint.ps1
isort and black are used to format the code. isort's configuration is in pyproject.toml and black uses default settings. prettier is used to format the json files.
./scripts/format.ps1
Use poetry to manage the dependencies. Run poetry export
after adding a production dependency.
poetry export -f requirements.txt -o requirements.txt
Run pytest at project root to run the tests or use coverage
to get coverage statistics.
./scripts/test.ps1
Format all files with black, isort and prettier.
./scripts/format.ps1
Take the dump.cs
generated by Il2CppDumper and write the gameenums.py
file.
python scripts/extract_enums.py dump.cs_path app/schemas/gameenums.py
Update translation files with new NA CEs translations. --jp-master
and --na-master
arguments are not needed if environment variables JP_GAMEDATA
and NA_GAMEDATA
are set, added to the .env
file or set in config.json
.
python scripts/update_ce_translation.py --jp-master jp_master_path --na-master na_master_path
Update the rayshiftQuest
tables with the list of available quests from Rayshift. This script should be run periodically to update the rayshiftQuest
list.
python -m scripts.load_rayshift_quest_list
Run this script when the master data changed to update the tests or when new tests are added.
python -m tests.get_test_data --raw --nice --basic
Run this script to update the static export files in export/
folder.
python -m export.niceexport
./scripts/format.ps1