Skip to content

Commit

Permalink
Upgrade to Lucene 8 (#702) - closes #679
Browse files Browse the repository at this point in the history
  • Loading branch information
lintool committed Jun 12, 2019
2 parents a5b83a1 + fc341c8 commit 75e36f9
Show file tree
Hide file tree
Showing 52 changed files with 254 additions and 759 deletions.
4 changes: 2 additions & 2 deletions docs/regressions-car17v1.5.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2017 CAR: benchmarkY1test (v1.5)](http://trec-car.cs.unh.edu/datareleases/)| 0.1563 | 0.1295 | 0.1358 | 0.1386 | 0.1080 | 0.1048 |
[TREC 2017 CAR: benchmarkY1test (v1.5)](http://trec-car.cs.unh.edu/datareleases/)| 0.1562 | 0.1295 | 0.1358 | 0.1386 | 0.1080 | 0.1048 |


RECIP_RANK | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2017 CAR: benchmarkY1test (v1.5)](http://trec-car.cs.unh.edu/datareleases/)| 0.2336 | 0.1923 | 0.1949 | 0.2037 | 0.1599 | 0.1524 |
[TREC 2017 CAR: benchmarkY1test (v1.5)](http://trec-car.cs.unh.edu/datareleases/)| 0.2331 | 0.1923 | 0.1949 | 0.2037 | 0.1599 | 0.1524 |


2 changes: 1 addition & 1 deletion docs/regressions-core17.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2017 Common Core Track Topics](https://trec.nist.gov/data/core/core_nist.txt)| 0.2087 | 0.2823 | 0.2787 | 0.2032 | 0.2606 | 0.2613 |
[TREC 2017 Common Core Track Topics](https://trec.nist.gov/data/core/core_nist.txt)| 0.2087 | 0.2823 | 0.2788 | 0.2032 | 0.2606 | 0.2613 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-core18.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2018 Common Core Track Topics](https://trec.nist.gov/data/core/topics2018.txt)| 0.2495 | 0.3136 | 0.2920 | 0.2526 | 0.3073 | 0.2966 |
[TREC 2018 Common Core Track Topics](https://trec.nist.gov/data/core/topics2018.txt)| 0.2495 | 0.3135 | 0.2925 | 0.2526 | 0.3073 | 0.2966 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
Expand Down
24 changes: 12 additions & 12 deletions docs/regressions-cw09b.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,29 +110,29 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.1126 | 0.0933 | 0.0928 | 0.1060 | 0.1019 | 0.1086 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.1094 | 0.1081 | 0.0974 | 0.0958 | 0.0837 | 0.0879 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1106 | 0.1107 | 0.1315 | 0.1069 | 0.1059 | 0.1212 |
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.1126 | 0.0933 | 0.0929 | 0.1060 | 0.1019 | 0.1086 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.1094 | 0.1085 | 0.0975 | 0.0958 | 0.0839 | 0.0879 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1105 | 0.1107 | 0.1315 | 0.1069 | 0.1058 | 0.1212 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.2681 | 0.2389 | 0.2354 | 0.2431 | 0.2312 | 0.2618 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.2513 | 0.2467 | 0.2393 | 0.2147 | 0.2067 | 0.2167 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.2167 | 0.1920 | 0.2553 | 0.2080 | 0.1980 | 0.2140 |
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.2694 | 0.2389 | 0.2354 | 0.2431 | 0.2312 | 0.2618 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.2513 | 0.2480 | 0.2387 | 0.2147 | 0.2047 | 0.2173 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.2167 | 0.1920 | 0.2553 | 0.2080 | 0.1980 | 0.2147 |


NDCG20 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.1354 | 0.1369 | 0.1637 | 0.1143 | 0.1185 | 0.1454 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.1890 | 0.1916 | 0.1833 | 0.1619 | 0.1447 | 0.1509 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1014 | 0.0917 | 0.1441 | 0.0868 | 0.0896 | 0.1030 |
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.1354 | 0.1369 | 0.1632 | 0.1143 | 0.1182 | 0.1454 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.1890 | 0.1916 | 0.1835 | 0.1619 | 0.1449 | 0.1517 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1014 | 0.0918 | 0.1441 | 0.0868 | 0.0896 | 0.1037 |


ERR20 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.0733 | 0.0747 | 0.0981 | 0.0599 | 0.0592 | 0.0742 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.0959 | 0.0960 | 0.1091 | 0.0849 | 0.0786 | 0.0820 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1304 | 0.1493 | 0.2355 | 0.1305 | 0.1334 | 0.1558 |
[TREC 2010 Web Track: Topics 51-100](http://trec.nist.gov/data/web/10/wt2010-topics.xml)| 0.0733 | 0.0747 | 0.0977 | 0.0599 | 0.0592 | 0.0742 |
[TREC 2011 Web Track: Topics 101-150](http://trec.nist.gov/data/web/11/full-topics.xml)| 0.0959 | 0.0960 | 0.1091 | 0.0849 | 0.0787 | 0.0821 |
[TREC 2012 Web Track: Topics 151-200](http://trec.nist.gov/data/web/12/full-topics.xml)| 0.1303 | 0.1494 | 0.2355 | 0.1305 | 0.1334 | 0.1558 |


16 changes: 8 additions & 8 deletions docs/regressions-cw12.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,25 +73,25 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | QL | +RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1695 | 0.1464 | 0.1493 | 0.1291 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2469 | 0.2325 | 0.2467 | 0.2168 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1694 | 0.1464 | 0.1494 | 0.1290 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2469 | 0.2324 | 0.2466 | 0.2177 |


P30 | BM25 | +RM3 | QL | +RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2767 | 0.2387 | 0.2613 | 0.2347 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.4533 | 0.4073 | 0.4380 | 0.3793 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2773 | 0.2393 | 0.2607 | 0.2347 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.4547 | 0.4080 | 0.4380 | 0.3800 |


NDCG20 | BM25 | +RM3 | QL | +RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2086 | 0.2033 | 0.1993 | 0.1725 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2578 | 0.2530 | 0.2228 | 0.2066 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2088 | 0.2033 | 0.1993 | 0.1725 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2572 | 0.2530 | 0.2218 | 0.2083 |


ERR20 | BM25 | +RM3 | QL | +RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1284 | 0.1264 | 0.1232 | 0.1008 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1630 | 0.1655 | 0.1321 | 0.1218 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1284 | 0.1264 | 0.1233 | 0.1008 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1616 | 0.1655 | 0.1322 | 0.1245 |


16 changes: 8 additions & 8 deletions docs/regressions-cw12b13.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,26 +88,26 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0468 | 0.0412 | 0.0435 | 0.0397 | 0.0322 | 0.0359 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.0224 | 0.0210 | 0.0180 | 0.0235 | 0.0203 | 0.0186 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0468 | 0.0408 | 0.0435 | 0.0397 | 0.0322 | 0.0358 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.0224 | 0.0210 | 0.0180 | 0.0235 | 0.0203 | 0.0183 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2113 | 0.1713 | 0.1840 | 0.1767 | 0.1507 | 0.1513 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1273 | 0.1207 | 0.1107 | 0.1373 | 0.1173 | 0.1167 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2113 | 0.1673 | 0.1833 | 0.1780 | 0.1513 | 0.1507 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1273 | 0.1207 | 0.1107 | 0.1373 | 0.1173 | 0.1147 |


NDCG20 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1286 | 0.1129 | 0.1287 | 0.1107 | 0.0920 | 0.1143 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1185 | 0.1080 | 0.0964 | 0.1177 | 0.1003 | 0.1001 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1286 | 0.1119 | 0.1287 | 0.1106 | 0.0920 | 0.1141 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1183 | 0.1081 | 0.0963 | 0.1177 | 0.1004 | 0.0989 |


ERR20 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0838 | 0.0763 | 0.0943 | 0.0769 | 0.0553 | 0.0780 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1201 | 0.1065 | 0.0929 | 0.1091 | 0.0929 | 0.0896 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0838 | 0.0753 | 0.0941 | 0.0768 | 0.0553 | 0.0780 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1201 | 0.1066 | 0.0928 | 0.1092 | 0.0928 | 0.0900 |



Expand Down
8 changes: 4 additions & 4 deletions docs/regressions-gov2.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,14 +90,14 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.2689 | 0.2844 | 0.2665 | 0.2681 | 0.2708 | 0.2666 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.3390 | 0.3820 | 0.3664 | 0.3303 | 0.3559 | 0.3646 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.3080 | 0.3377 | 0.3069 | 0.2996 | 0.3154 | 0.3084 |
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.2689 | 0.2844 | 0.2669 | 0.2681 | 0.2708 | 0.2666 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.3390 | 0.3820 | 0.3666 | 0.3303 | 0.3559 | 0.3646 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.3080 | 0.3377 | 0.3069 | 0.2997 | 0.3154 | 0.3084 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.4864 | 0.5190 | 0.4986 | 0.4755 | 0.4925 | 0.4932 |
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.4864 | 0.5190 | 0.4993 | 0.4755 | 0.4925 | 0.4932 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.5540 | 0.5920 | 0.5933 | 0.5347 | 0.5620 | 0.5840 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.4907 | 0.5160 | 0.5033 | 0.4720 | 0.4847 | 0.4920 |

Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-mb11.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,6 @@ MAP | BM25 | +RM3 | +Ax | QL
P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2011 Microblog Track Topics](http://trec.nist.gov/data/microblog2011.html)| 0.3959 | 0.4170 | 0.4612 | 0.4061 | 0.4435 | 0.4408 |
[TREC 2012 Microblog Track Topics](http://trec.nist.gov/data/microblog2012.html)| 0.3316 | 0.3463 | 0.3554 | 0.3333 | 0.3520 | 0.3842 |
[TREC 2012 Microblog Track Topics](http://trec.nist.gov/data/microblog2012.html)| 0.3316 | 0.3463 | 0.3554 | 0.3333 | 0.3514 | 0.3842 |


4 changes: 2 additions & 2 deletions docs/regressions-msmarco-doc.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 |
:---------------------------------------|-----------|-----------|
[MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/TREC-2019-Deep-Learning)| 0.2308 | 0.1631 |
[MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/TREC-2019-Deep-Learning)| 0.2310 | 0.1632 |


R@1000 | BM25 | +RM3 |
:---------------------------------------|-----------|-----------|
[MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/TREC-2019-Deep-Learning)| 0.8856 | 0.8787 |
[MS MARCO Document Ranking: Dev Queries](https://github.com/microsoft/TREC-2019-Deep-Learning)| 0.8856 | 0.8785 |


2 changes: 1 addition & 1 deletion docs/regressions-msmarco-passage.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 (Default)| +RM3 | BM25 (Tuned)| +RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)| 0.1924 | 0.1661 | 0.1956 | 0.1766 |
[MS MARCO Passage Ranking: Dev Queries](https://github.com/microsoft/MSMARCO-Passage-Ranking)| 0.1926 | 0.1661 | 0.1957 | 0.1766 |


R@1000 | BM25 (Default)| +RM3 | BM25 (Tuned)| +RM3 |
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-robust04.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2004 Robust Track Topics](http://trec.nist.gov/data/robust/04.testset.gz)| 0.2531 | 0.2903 | 0.2895 | 0.2467 | 0.2747 | 0.2774 |
[TREC 2004 Robust Track Topics](http://trec.nist.gov/data/robust/04.testset.gz)| 0.2531 | 0.2903 | 0.2896 | 0.2467 | 0.2747 | 0.2774 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-robust05.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2005 Robust Track Topics](http://trec.nist.gov/data/robust/05/05.50.topics.txt)| 0.2031 | 0.2602 | 0.2584 | 0.2028 | 0.2491 | 0.2476 |
[TREC 2005 Robust Track Topics](http://trec.nist.gov/data/robust/05/05.50.topics.txt)| 0.2032 | 0.2602 | 0.2587 | 0.2028 | 0.2491 | 0.2476 |


P30 | BM25 | +RM3 | +Ax | QL | +RM3 | +Ax |
Expand Down
Loading

0 comments on commit 75e36f9

Please sign in to comment.