Combine Add_Language & Update

[lastrucci01]

Description

Combine the steps of adding and updating loaded language models.

Tasks

• Combine add & update

Background

Masakhane Translate has a Manage CLI that let's you add language model info from the database, among other things. Language model info refers to the data stored in the database (Fig 1), the actual language models are downloaded in another step. The database seems to serve as a reference as to which database models are downloaded. The add_lang command (Fig 2) reads languages.json - which is just a file storing the language names & shorthands - and gets the name_tag composition to be stored in the database using the user-defined Language DB model. Take note of the database implementation using FlaskSQLAlchemy.

Once model info has been loaded into the database, the /update endpoint - which is linked to the AddResource Class (Fig 3) - must be hit which will go through the name_tags in the database and download the model files from Zenodo if necessary. Once the model has been downloaded it is loaded into memory with self.models which is a dictonary. It will also remove any models in memory that no longer have a reference in the DB.

Files

translate.py
model_load.py
init.py
manage.py

Screenshots

Figure 1 - Language table contents

Figure 2 - add_language command
Model info added to the database

Figure 3 - AddResource Class
Model info used to download models