Improving Factuality and Reasoning in Language Models through Multiagent Debate

Project Page | Paper

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch

This is a preliminary implementation of the paper "Improving Factuality and Reasoning in Language Models through Multiagent Debate". More tasks and settings will be released soon. You may see some additional debate logs here.

Also, check out gauss5930's awesome implementation of multiagent debate on opensource LLMs here!

Running experiments

The code for running arithmetic, GSM, biographies, and MMLU tasks may be found in the following subfolders

• ./math/ contains code for running math
• ./gsm/ contains code for running gsm
• ./biography/ contains code for running biographies
• ./mmlu/ contains code for running mmlu results.

Math:

To generate and evaluated answer for Math problems through multiagent debate, cd into the math directory and run: python gen_math.py

Grade School Math:

To generate answers for Grade School Math problems through multiagent debate, cd into the gsm directory and run: python gen_gsm.py

To evaluate the generated results of Grade School Math problems: python eval_gsm.py

You can download the GSM dataset here

Biography:

To generate answers for Biography problems through multiagent debate, cd into the biography directory and run: python gen_conversation.py

To evaluate the generated results for Biography problems: python eval_conversation.py

MMLU:

To generate answers for MMLU through multiagent debate, cd into the MMLU directory and run: python gen_mmlu.py

To evaluate the generated results of MMLU: python eval_mmlu.py

You can download the MMLU dataset here

If you would like to cite the paper, here is a bibtex file:

@article{du2023improving,
  title={Improving Factuality and Reasoning in Language Models through Multiagent Debate},
  author={Du, Yilun and Li, Shuang and Torralba, Antonio and Tenenbaum, Joshua B and Mordatch, Igor},
  journal={arXiv preprint arXiv:2305.14325},
  year={2023}
}