iscc-sct
is a proof of concept implementation of a semantic Text-Code for the ISCC
(International Standard Content Code). Semantic Text-Codes are designed to capture and represent the language
agnostic semantic content of text for improved similarity detection.
Caution
This is an early proof of concept. All releases with version numbers below v1.0.0 may break backward compatibility and produce incompatible Semantic Text-Codes.
The ISCC framework already includes a Text-Code based on lexical similarity for near-duplicate matching. The ISCC Semantic Text-Code is a planned additional ISCC-UNIT focused on capturing a more abstract and broader semantic similarity. It is engineered to be robust against a wide range of variations and, most remarkably, translations of text that cannot be matched based on lexical similarity alone.
One of the most interesting aspects of the Semantic Text-Code is its ability to generate (near)-identical codes for translations of the same text. This means that the same content, expressed in different languages, can be identified and linked, opening up new possibilities for cross-lingual content identification and similarity detection.
- Semantic Similarity: Utilizes deep learning models to generate codes that reflect the semantic essence of text.
- Translation Matching: Creates nearly identical codes for text translations, enabling cross-lingual content identification.
- Bit-Length Flexibility: Supports generating codes of various bit lengths (up to 256 bits), allowing for adjustable granularity in similarity detection.
- ISCC Compatible: Generates codes fully compatible with the ISCC specification, facilitating seamless integration with existing ISCC-based systems.
Ensure you have Python 3.9 or newer installed on your system. Install the library using:
pip install iscc-sct
For systems with GPU CUDA support, enhance performance by installing with:
pip install iscc-sct[gpu]
Generate a Semantic Text-Code using the create function:
>>> import iscc_sct as sct
>>> text = "This is some sample text. It can be a longer document or even an entire book."
>>> sct.create(text, bits=256)
{
"iscc": "ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI",
"characters": 77
}
For granular (per chunk) feature outputs:
>>> import iscc_sct as sct
>>> text = "This is some sample text. It can be a longer document or even an entire book."
>>> sct.create(text, bits=256, granular=True)
{
"iscc": "ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI",
"characters": 77,
"features": [
{
"feature": "LWMN4SPXOJLC2",
"offset": 0,
"size": 77,
"text": "This is some sample text. It can be a longer document or even an entire book."
}
]
}
The installation also provides a sct command-line tool:
sct --help
usage: sct [-h] [-b BITS] [-g] [-d] [path]
Generate Semantic Text-Codes for text files.
positional arguments:
path Path to text files (supports glob patterns).
options:
-h, --help show this help message and exit
-b BITS, --bits BITS Bit-Length of Code (default 256)
-g, --granular Activate granular processing.
-d, --debug Show debugging messages.
iscc-sct
employs the following process:
- Splits the text into semantically coherent chunks.
- Uses a pre-trained deep learning model for text embedding.
- Generates feature vectors capturing essential characteristics of the chunks.
- Aggregates these vectors and binarizes them to produce a Semantic Text-Code.
This process ensures robustness to variations and translations, enabling cross-lingual matching.
We welcome contributions to enhance the capabilities, efficiency, and compatibility of this proof of concept with the broader ISCC ecosystem. For development, install the project in development mode using Poetry:
git clone https://github.com/iscc/iscc-sct.git
cd iscc-sct
poetry install
If you have suggestions for improvements or bug fixes, please open an issue or pull request. For major changes, please open an issue first to discuss your ideas.
- Text Chunking: semantic-text-splitter
- Text Embedding: Sentence-Transformer
This project is licensed under the CC-BY-NC-SA-4.0 International License.