English | 中文
📝Introduction | 🔨Installation | 🚀Quick Start | 📚Tutorials | 🎁Model List | 📰Dataset List | 🎉Notes
MindOCR is an open-source toolbox for OCR development and application based on MindSpore, which integrates series of mainstream text detection and recognition algorihtms/models, provides easy-to-use training and inference tools. It can accelerate the process of developing and deploying SoTA text detection and recognition models in real-world applications, such as DBNet/DBNet++ and CRNN/SVTR, and help fulfill the need of image-text understanding.
Major Features
- Modular design: We decoupled the OCR task into several configurable modules. Users can setup the training and evaluation pipelines, customize the data processing pipeline and model architectures easily by modifying just few lines of code.
- High-performance: MindOCR provides a series of pretrained weights trained with optimized configurations that reach competitive performance on OCR tasks.
- Low-cost-to-apply: Easy-to-use inference tools are provided in MindOCR to perform text detection and recognition tasks.
MindOCR is built on MindSpore AI framework, which supports CPU/GPU/NPU devices. MindOCR is compatible with the following framework versions. For details and installation guideline, please refer to the installation links shown below.
- mindspore >= 1.9 [install]
- python >= 3.7
- openmpi 4.0.3 (for distributed training/evaluation) [install]
- mindspore lite (for inference) [install]
pip install -r requirements.txt
Tips:
-
If scikit_image cannot be imported, please set environment variable
$LD_PRELOAD
as follows, (related opencv issue)export LD_PRELOAD=path/to/scikit_image.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD
git clone https://github.com/mindspore-lab/mindocr.git
cd mindocr
pip install -e .
Using
-e
for "editable" mode can help resolve potential module import issues.
pip install mindocr
As this project is under active development, the version installed from PyPI is out-of-date currently. (will update soon).
After installing MindOCR, we can run text detection and recognition on an arbitrary image easily as follows.
python tools/infer/text/predict_system.py --image_dir {path_to_img or dir_to_imgs} \
--det_algorithm DB++ \
--rec_algorithm CRNN
After running, the results will be saved in ./inference_results
by default. Here is an example result.
Visualization of text detection and recognition result
We can see that all texts on the image are detected and recognized accurately. For more usage, please refer to the inference section in tutorials.
It is easy to train your OCR model with the tools/train.py
script, which supports both text detection and recognition model training.
python tools/train.py --config {path/to/model_config.yaml}
The --config
arg specifies the path to a yaml file that defines the model to be trained and the training strategy including data process pipeline, optimizer, lr scheduler, etc.
MindOCR provides SoTA OCR models with their training strategies in configs
folder.
You may adapt it to your task/dataset, for example, by running
# train text detection model DBNet++ on icdar15 dataset
python tools/train.py --config configs/det/dbnet/db++_r50_icdar15.yaml
# train text recognition model CRNN on icdar15 dataset
python tools/train.py --config configs/rec/crnn/crnn_icdar15.yaml
Similarly, it is easy to evaluate the trained model with the tools/eval.py
script.
python tools/eval.py \
--config {path/to/model_config.yaml} \
--opt eval.dataset_root={path/to/your_dataset} eval.ckpt_load_path={path/to/ckpt_file}
For more illustration and usage, please refer to the model training section in Tutorials.
You can do MindSpore Lite inference in MindOCR using MindOCR models or Third-party models (PaddleOCR, MMOCR, etc.). Please refer to MindOCR Models Inference - Quick Start or Third-party Models Inference - Quick Start.
- Datasets
- Model Training
- Inference and Deployment
- Developer Guides
Text Detection
Text Recognition
- CRNN (TPAMI'2016)
- CRNN-Seq2Seq/RARE (CVPR'2016)
- SVTR (IJCAI'2022)
- MASTER (PR'2019)
- VISIONLAN (ICCV'2021)
- RobustScanner (ECCV'2020)
- ABINet (CVPR'2021) [coming soon]
For the detailed performance of the trained models, please refer to configs.
For details of MindSpore Lite and ACL inference models support, please refer to MindOCR Models Support List and Third-party Models Support List (PaddleOCR, MMOCR, etc.).
MindOCR provides a dataset conversion tool to OCR datasets with different formats and support customized dataset by users. We have validated the following public OCR datasets in model training/evaluation.
General OCR Datasets
- Born-Digital Images [download]
- CASIA-10K [download]
- CCPD [download]
- Chinese Text Recognition Benchmark [paper] [download]
- COCO-Text [download]
- CTW [download]
- ICDAR2015 [paper] [download]
- ICDAR2019 ArT [download]
- LSVT [download]
- MLT2017 [paper] [download]
- MSRA-TD500 [paper] [download]
- MTWI-2018 [download]
- RCTW-17 [download]
- ReCTS [download]
- SCUT-CTW1500 [paper] [download]
- SROIE [download]
- SVT [download]
- SynText150k [paper] [download]
- SynthText [paper] [download]
- TextOCR [download]
- Total-Text [paper] [download]
We will include more datasets for training and evaluation. This list will be continuously updated.
- 2023/07/06
- Add new trained models
- RobustScanner for text recognition
- 2023/07/05
- Add new trained models
- VISIONLAN for text recognition
- 2023/06/29
- 2023/06/07
- Add new trained models
- Add more benchmark datasets and their results
- Add resume training function, which can be used in case of unexpected interruption in training. Usage: add the
resume
parameter under themodel
field in the yaml config, e.g.,resume: True
, load and resume training from {ckpt_save_dir}/train_resume.ckpt orresume: /path/to/train_resume.ckpt
, load and resume training from the given path. - Improve postprocessing for detection: re-scale detected text polygons to original image space by default,
which can be enabled by add "shape_list" to the
eval.dataset.output_columns
list. - Refactor online inference to support more models, see README.md for details.
- 2023/05/15
- Add new trained models
- DBNet++ for text detection
- CRNN-Seq2Seq for text recognition
- DBNet pretrained on SynthText is now available: checkpoint url
- Add more benchmark datasets and their results
- SynthText, MSRA-TD500, CTW1500
- More benchmark results for DBNet are reported here.
- Add checkpoint manager for saving top-k checkpoints and improve log.
- Python inference code refactored.
- Bug fix: use Meter to average loss for large datasets, disable
pred_cast_fp32
for ctcloss in AMP training, fix error when invalid polygons exist.
- 2023/05/04
- Support loading self-defined pretrained checkpoints via setting
model-pretrained
with checkpoint url or local path in yaml. - Support setting probability for executing augmentation including rotation and flip.
- Add Exponential Moving Average(EMA) for model training, which can be enabled by setting
train-ema
(default: False) andtrain-ema_decay
in the yaml config. - Arg parameter changed:
num_columns_to_net
->net_input_column_index
: change the column number feeding into the network to the column index. - Arg parameter changed:
num_columns_of_labels
->label_column_index
: change the column number corresponds to the label to the column index.
- 2023/04/21
- Add parameter grouping to support flexible regularization in training. Usage: add
grouping_strategy
argument in yaml config to select a predefined grouping strategy, or useno_weight_decay_params
argument to pick layers to exclude from weight decay (e.g., bias, norm). Example can be referred inconfigs/rec/crnn/crnn_icdar15.yaml
- Add gradient accumulation to support large batch size training. Usage: add
gradient_accumulation_steps
in yaml config, the global batch size = batch_size * devices * gradient_accumulation_steps. Example can be referred inconfigs/rec/crnn/crnn_icdar15.yaml
- Add gradient clip to support training stablization. Enable it by setting
grad_clip
as True in yaml config.
- 2023/03/23
- Add dynamic loss scaler support, compatible with drop overflow update. To enable dynamic loss scaler, please set
type
ofloss_scale
asdynamic
. A YAML example can be viewed inconfigs/rec/crnn/crnn_icdar15.yaml
- 2023/03/20
- Arg names changed:
output_keys
->output_columns
,num_keys_to_net
->num_columns_to_net
- Data pipeline updated
- 2023/03/13
- Add system test and CI workflow.
- Add modelarts adapter to allow training on OpenI platform. To train on OpenI:
i) Create a new training task on the openi cloud platform.
ii) Link the dataset (e.g., ic15_mindocr) on the webpage.
iii) Add run parameter `config` and write the yaml file path on the website UI interface, e.g., '/home/work/user-job-dir/V0001/configs/rec/test.yaml'
iv) Add run parameter `enable_modelarts` and set True on the website UI interface.
v) Fill in other blanks and launch.
We appreciate all kinds of contributions including issues and PRs to make MindOCR better.
Please refer to CONTRIBUTING.md for the contributing guideline. Please follow the Model Template and Guideline for contributing a model that fits the overall interface :)
This project follows the Apache License 2.0 open-source license.
If you find this project useful in your research, please consider citing:
@misc{MindSpore OCR 2023,
title={{MindSpore OCR }:MindSpore OCR Toolbox},
author={MindSpore Team},
howpublished = {\url{https://github.com/mindspore-lab/mindocr/}},
year={2023}
}