LibSA4Py
is a static analysis library for Python, which extracts type hints and features for training ML-based type inference models.
- Python 3.7 or newer (Python 3.8 is recommended)
- Watchman (for running pyre) [Optional]
- MacOS or Linux systems
git clone https://github.com/saltudelft/libsa4py.git
cd libsa4py && pip install .
Given Python repositories, run the following command to process source code files and generate JSON-formatted outputs:
libsa4py process --p $REPOS_PATH --o $OUTPUT_PATH --d $DUPLICATE_PATH --j $WORKERS_COUNT --l $LIMIT --c --no-nlp --pyre
Description:
--p $REPOS_PATH
: The path to the Python corpus or dataset.--o $OUTPUT_PATH
: Path to store processed projects.--d $DUPLICATE_PATH
: Path to duplicate files of the given dataset (i.e. jsonl.gz file produced by the CD4Py tool). [Optional]--s
: Path to the CSV file for splitting the given dataset. [Optional]--j $WORKERS_COUNT
: Number of workers for processing projects. [Optional, default=no. of available CPU cores]--l $LIMIT
: Number of projects to be processed. [Optional]--c
: Whether to ignore processed projects. [Optional, default=False]--no-nlp
: Whether to apply standard NLP techniques to extracted identifiers. [Optional, default=True]--pyre
: Whether to runpyre
to infer the types of variables for given projects. [Optional, default=False]--tc
: Whether to type-check type annotations in projects. [Optional, default=False]
To merge all the processed JSON-formatted projects into a single dataframe, run the following command:
libsa4py merge --o $OUTPUT_PATH --l $LIMIT
Description:
--o $OUTPUT_PATH
: Path to the processed projects, used in the previous processing step.--l $LIMIT
: Number of projects to be merged. [Optional]
To apply Pyre's inferred types to projects, run the following command:
libsa4py apply --p $REPOS_PATH --o $OUTPUT_PATH
Description:
--p $REPOS_PATH
: The path to the Python corpus or dataset.--o $OUTPUT_PATH
: Path to the processed projects, used in the previous processing step.
After processing each project, a JSON-formatted file is produced, which is described here.