This repository contains the code to reproduce the experiments from our paper Generative Retrieval as Multi-Vector Dense Retrieval, which has been accepted at SIGIR'24.
Update on April 6th, 2024: we have post our paper on arXiv, you can find it here.
Note on March 29th, 2024: we will post our paper on arXiv as a pre-print version soon and provide the link to the checkpoints of our re-implemented T5-ColBERT and T5-SEAL.
Generative retrieval generates identifiers of relevant documents in an end-to-end manner using a sequence-to-sequence architecture for a given query. The relation between generative retrieval and other retrieval methods, especially those based on matching within dense retrieval models, is not yet fully comprehended. Prior work has demonstrated that generative retrieval with atomic identifiers is equivalent to single-vector dense retrieval. Accordingly, generative retrieval exhibits behavior analogous to hierarchical search within a tree index in dense retrieval when using hierarchical semantic identifiers. However, prior work focuses solely on the retrieval stage without considering the deep interactions within the decoder of generative retrieval.
In this paper, we fill this gap by demonstrating that generative retrieval and multi-vector dense retrieval share the same framework for measuring the relevance to a query of a document. Specifically, we examine the attention layer and prediction head of generative retrieval, revealing that generative retrieval can be understood as a special case of multi-vector dense retrieval. Both methods compute relevance as a sum of products of query and document vectors and an alignment matrix. We then explore how generative retrieval applies this framework, employing distinct strategies for computing document token vectors and the alignment matrix. We have conducted experiments to verify our conclusions and show that both paradigms exhibit commonalities of term matching in their alignment matrix.
Our finding applies to many generative retrieval identifier designs and provides possible explanations on how generative retrieval can express query-document relevance. As multi-vector dense retrieval is the state-of-the-art dense retrieval method currently, understanding the connection between generative retrieval and multi-vector dense retrieval is crucial for shedding light on the underlying mechanisms of generative retrieval and for developing, and understanding the potential of, new retrieval models.
pip install -r requirements.txt
common_settings/*.sh
is the dataset-specific parameters, such as max query length, etc.
mvdr.py
and faiss_index.py
reproduce ColBERT
, based on T5
, you can change it to BERT
run bash mvdr_run_scripts/run.sh
This will create an output directory in output/mvdr.DATA.train
by default, which is the $EXP_DIR
defined in mvdr_run_scripts/params.sh
All MVDR-related parameters are in mvdr_run_scripts/params.sh
.
SEAL needs a working installation of SWIG, e.g. (on Ubuntu):
sudo apt install swig
or using conda
:
conda install swig
We also assume that pytorch
is already available in your environment. SEAL has been tested with version 1.11.
Clone this repo with --recursive
so that you also include the submodule in res/external
.
git clone --recursive https://github.com/facebookresearch/SEAL.git
Compile and install sdsl-lite
:
env CFLAGS='-fPIC' CXXFLAGS='-fPIC' res/external/sdsl-lite/install.sh
Install other dependencies:
pip install -r requirements.txt
# pyserini
# pip install -r requirements_extra.txt
Now install this library.
pip install -e .
Since SEAL uses an older version of transformers, so there are a few places that need to be updated
You can directly copy our fixed version of seal/beam_search.py
to your SEAL workplace
cp beam_search.py /path/to/SEAL/seal/
And you can use git diff
to check the difference.
gr.py
reproduce SEAL using title and substring as identifiers, t5
only
the usage is similar to MVDR
run bash gr_run_scripts/run.sh
This will create an output diretory in output/gr.DATA.train
by default, which is the $EXP_DIR
defined in gr_run_scripts/params.sh
All GR related parameters are in gr_run_scripts/params.sh
.
If you find this work helpful, please cite our paper:
@article{wu2024generative,
title={Generative Retrieval as Multi-Vector Dense Retrieval},
author={Shiguang Wu and Wenda Wei and Mengqi Zhang and Zhumin Chen and Jun Ma and Zhaochun Ren and Maarten de Rijke and Pengjie Ren},
year={2024},
eprint={2404.00684},
archivePrefix={arXiv},
primaryClass={cs.IR}
}