diff --git a/docs/fatjar-regressions/fatjar-regressions-v1.2.1.md b/docs/fatjar-regressions/fatjar-regressions-v1.2.1.md index 00647bfb4b..741819648f 100644 --- a/docs/fatjar-regressions/fatjar-regressions-v1.2.1.md +++ b/docs/fatjar-regressions/fatjar-regressions-v1.2.1.md @@ -250,7 +250,7 @@ To print out the commands that will generate the above runs without performing t ## MS MARCO V2.1 Segmented Documents -❗ Beware, running these experiments will automatically download 2 indexes totaling 145.8 GB. +❗ Beware, running these experiments will automatically download 12 indexes totaling 698.0 GB. The MS MARCO V2.1 collections were created for the [TREC RAG Track](https://trec-rag.github.io/). There were two variants: the documents corpus and the segmented documents corpus. @@ -282,7 +282,7 @@ To print out the commands that will generate the above runs without performing t ## MS MARCO V2.1 Documents -❗ Beware, running these experiments will automatically download 12 indexes totaling 698.0 GB. +❗ Beware, running these experiments will automatically download 2 indexes totaling 145.8 GB. The MS MARCO V2.1 collections were created for the [TREC RAG Track](https://trec-rag.github.io/). There were two variants: the documents corpus and the segmented documents corpus. @@ -299,7 +299,7 @@ The table below reports effectiveness (dev in terms of RR@10, DL21-DL23, RAGgy i The following command will reproduce the above experiments: ```bash -java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2.1 +java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2.1-doc ``` To print out the commands that will generate the above runs without performing the runs, use the options `-dryRun -printCommands`. diff --git a/docs/fatjar-regressions/fatjar-regressions-v1.2.2.md b/docs/fatjar-regressions/fatjar-regressions-v1.2.2.md index 5d229d19cd..a400334d49 100644 --- a/docs/fatjar-regressions/fatjar-regressions-v1.2.2.md +++ b/docs/fatjar-regressions/fatjar-regressions-v1.2.2.md @@ -20,7 +20,10 @@ See [this guide on prebuilt indexes](../prebuilt-indexes.md) for more details. ## Contents + [MS MARCO V2.1 + TREC RAG](#ms-marco-v21--trec-rag) -+ [MS MARCO V1 Passage](#ms-marco-v1-passage) ++ [MS MARCO V1 Passages](#ms-marco-v1-passages) ++ [MS MARCO V1 Documents](#ms-marco-v1-documents) ++ [MS MARCO V2 Passages](#ms-marco-v2-passages) ++ [MS MARCO V2 Documents](#ms-marco-v2-documents) + [MS MARCO V2.1 Segmented Documents](#ms-marco-v21-segmented-documents) + [MS MARCO V2.1 Documents](#ms-marco-v21-documents) + [BEIR](#beir) @@ -210,35 +213,28 @@ $ head -n 1 $OUTPUT_DIR/results.msmarco-v2.1-doc-segmented.bm25.rag24.test.jsonl To generate similar output for ArcticEmbed-L, specify the corresponding run file with `-run`. -## MS MARCO V1 Passage - -❗ Beware, running these experiments will automatically download 9 indexes totaling 203.1 GB. - -Currently, Anserini provides support for the following models: - -+ BM25 -+ SPLADE-v3: cached queries and ONNX query encoding -+ cosDPR-distil: cached queries and ONNX query encoding -+ bge-base-en-v1.5: cached queries and ONNX query encoding -+ cohere-embed-english-v3.0: cached queries and ONNX query encoding - -The table below reports the effectiveness of the models (dev in terms of RR@10, DL19 and DL20 in terms of nDCG@10): - -| | dev | DL19 | DL20 | -|:-------------------------------------------------------------|-------:|-------:|-------:| -| BM25 (k1=0.9, b=0.4) | 0.1840 | 0.5058 | 0.4796 | -| SPLADE-v3 (cached queries) | 0.3999 | 0.7264 | 0.7522 | -| SPLADE-v3 (ONNX) | 0.4000 | 0.7264 | 0.7522 | -| cosDPR-distil w/ HNSW fp32 (cached queries) | 0.3887 | 0.7250 | 0.7025 | -| cosDPR-distil w/ HNSW fp32 (ONNX) | 0.3887 | 0.7250 | 0.7025 | -| cosDPR-distil w/ HNSW int8 (cached queries) | 0.3897 | 0.7240 | 0.7004 | -| cosDPR-distil w/ HNSW int8 (ONNX) | 0.3899 | 0.7247 | 0.6996 | -| bge-base-en-v1.5 w/ HNSW fp32 (cached queries) | 0.3574 | 0.7065 | 0.6780 | -| bge-base-en-v1.5 w/ HNSW fp32 (ONNX) | 0.3575 | 0.7016 | 0.6768 | -| bge-base-en-v1.5 w/ HNSW int8 (cached queries) | 0.3572 | 0.7016 | 0.6738 | -| bge-base-en-v1.5 w/ HNSW int8 (ONNX) | 0.3575 | 0.7017 | 0.6767 | -| cohere-embed-english-v3.0 w/ HNSW fp32 (cached queries) | 0.3647 | 0.6956 | 0.7245 | -| cohere-embed-english-v3.0 w/ HNSW int8 (cached queries) | 0.3656 | 0.6955 | 0.7262 | +## MS MARCO V1 Passages + +❗ Beware, running these experiments will automatically download 9 indexes totaling 203 GB. + +Anserini provides support for a variety of models. +The table below reports the effectiveness (dev in terms of RR@10, DL19 and DL20 in terms of nDCG@10): + +| | dev | DL19 | DL20 | +|:----------------------------------------------------------------|-------:|-------:|-------:| +| BM25 (k1=0.9, b=0.4) | 0.1840 | 0.5058 | 0.4796 | +| SPLADE-v3: cached queries | 0.3999 | 0.7264 | 0.7522 | +| SPLADE-v3: ONNX | 0.4000 | 0.7264 | 0.7522 | +| cosDPR-distil: HNSW, cached queries | 0.3887 | 0.7250 | 0.7025 | +| cosDPR-distil: HNSW, ONNX | 0.3887 | 0.7250 | 0.7025 | +| cosDPR-distil: quantized (int8) HNSW, cached queries | 0.3897 | 0.7240 | 0.7004 | +| cosDPR-distil: quantized (int8) HNSW, ONNX | 0.3899 | 0.7247 | 0.6996 | +| bge-base-en-v1.5: HNSW, cached queries | 0.3574 | 0.7065 | 0.6780 | +| bge-base-en-v1.5: HNSW, ONNX | 0.3575 | 0.7016 | 0.6768 | +| bge-base-en-v1.5: quantized (int8) HNSW, cached queries | 0.3572 | 0.7016 | 0.6738 | +| bge-base-en-v1.5: quantized (int8) HNSW, ONNX | 0.3575 | 0.7017 | 0.6767 | +| cohere-embed-english-v3.0: HNSW, cached queries | 0.3647 | 0.6956 | 0.7245 | +| cohere-embed-english-v3.0: quantized (int) HNSW, cached queries | 0.3656 | 0.6955 | 0.7262 | The following command will reproduce the above experiments: @@ -248,29 +244,95 @@ java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v1-p To print out the commands that will generate the above runs without performing the runs, use the options `-dryRun -printCommands`. +## MS MARCO V1 Documents + +❗ Beware, running these experiments will automatically download 6 indexes totaling 51 GB. + +Anserini provides support for a variety of models. +The table below reports the effectiveness of selected models (dev in terms of RR@100, DL19 and DL20 in terms of nDCG@10): + +| | dev | DL19 | DL20 | +|:---------------------------------------------------------------------------|-------:|-------:|-------:| +| BM25 complete doc (k1=0.9, b=0.4) | 0.2299 | 0.5176 | 0.5286 | +| BM25 segmented doc (k1=0.9, b=0.4) | 0.2684 | 0.5302 | 0.5281 | +| BM25 complete doc with doc2query-T5 | 0.2880 | 0.5968 | 0.5885 | +| BM25 segmented doc with doc2query-T5 | 0.3179 | 0.6119 | 0.5957 | +| uniCOIL (with doc2query-T5): ONNX | 0.3531 | 0.6396 | 0.6033 | + +The following command will reproduce runs corresponding to the above models (as well as additional ones not included in the table): + +```bash +java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v1-doc +``` + +To print out the commands that will generate runs for all available models (without actually performing the runs), use the options `-dryRun -printCommands`. + +## MS MARCO V2 Passages + +❗ Beware, running these experiments will automatically download 3 indexes totaling 90 GB. + +Anserini provides support for a variety of models. +The table below reports the effectiveness of selected models (dev and dev2 in terms of RR@100, DL21-23 in terms of nDCG@10): + +| | dev | dev2 | DL21 | DL22 | DL23 | +|:-------------------------------------------------------------|-------:|-------:|-------:|-------:|-------:| +| BM25 (k1=0.9, b=0.4) | 0.0719 | 0.0802 | 0.4458 | 0.2692 | 0.2627 | +| uniCOIL (with doc2query-T5): ONNX | 0.1499 | 0.1577 | 0.6159 | 0.4614 | 0.3855 | + +The following command will reproduce runs corresponding to the above models (as well as additional ones not included in the table): + +```bash +java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2-passage +``` + +To print out the commands that will generate runs for all available models (without actually performing the runs), use the options `-dryRun -printCommands`. + +## MS MARCO V2 Documents + +❗ Beware, running these experiments will automatically download 6 indexes totaling 310 GB. + +Anserini provides support for a variety of models. +The table below reports the effectiveness of selected models (dev in terms of RR@10, DL19 and DL20 in terms of nDCG@10): + +| | dev | dev2 | DL21 | DL22 | DL23 | +|:---------------------------------------------------------------------------|-------:|-------:|-------:|-------:|-------:| +| BM25 complete doc (k1=0.9, b=0.4) | 0.1572 | 0.1659 | 0.5116 | 0.2993 | 0.2946 | +| BM25 segmented doc (k1=0.9, b=0.4) | 0.1896 | 0.1930 | 0.5776 | 0.3618 | 0.3405 | +| BM25 complete doc with doc2query-T5 | 0.2011 | 0.2012 | 0.5792 | 0.3539 | 0.3511 | +| BM25 segmented doc with doc2query-T5 | 0.2226 | 0.2234 | 0.6289 | 0.3975 | 0.3612 | +| uniCOIL (with doc2query-T5): ONNX | 0.2419 | 0.2445 | 0.6783 | 0.4451 | 0.4150 | + +The following command will reproduce runs corresponding to the above models (as well as additional ones not included in the table): + +```bash +java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2-doc +``` + +To print out the commands that will generate runs for all available models (without actually performing the runs), use the options `-dryRun -printCommands`. + ## MS MARCO V2.1 Segmented Documents -❗ Beware, running these experiments will automatically download 2 indexes totaling 145.8 GB. +❗ Beware, running these experiments will automatically download 12 indexes totaling 698 GB. The MS MARCO V2.1 collections were created for the [TREC RAG Track](https://trec-rag.github.io/). There were two variants: the documents corpus and the segmented documents corpus. The documents corpus served as the source of the segmented documents corpus, but the segmented documents corpus is the one used in official TREC RAG evaluations. The following table reports nDCG@20 scores for various retrieval conditions: -| | RAG 24 UMBRELA | RAG 24 NIST | -|-----------------------------------------------|:--------------:|:-----------:| -| baselines | 0.3198 | 0.2809 | -| SPLADE-v3 | 0.5167 | 0.4642 | -| Arctic-embed-l (`shard00`, HNSW int8 indexes) | 0.3003 | 0.2449 | -| Arctic-embed-l (`shard01`, HNSW int8 indexes) | 0.2599 | 0.2184 | -| Arctic-embed-l (`shard02`, HNSW int8 indexes) | 0.2661 | 0.2211 | -| Arctic-embed-l (`shard03`, HNSW int8 indexes) | 0.2705 | 0.2388 | -| Arctic-embed-l (`shard04`, HNSW int8 indexes) | 0.2937 | 0.2253 | -| Arctic-embed-l (`shard05`, HNSW int8 indexes) | 0.2590 | 0.2383 | -| Arctic-embed-l (`shard06`, HNSW int8 indexes) | 0.2444 | 0.2336 | -| Arctic-embed-l (`shard07`, HNSW int8 indexes) | 0.2417 | 0.2255 | -| Arctic-embed-l (`shard08`, HNSW int8 indexes) | 0.2847 | 0.2765 | -| Arctic-embed-l (`shard09`, HNSW int8 indexes) | 0.2432 | 0.2457 | +| | RAG 24 UMBRELA | RAG 24 NIST | +|---------------------------------------------------------|:--------------:|:-----------:| +| BM25 | 0.3198 | 0.2809 | +| SPLADE-v3: ONNX | 0.5167 | 0.4642 | +| Arctic-embed-l (`shard00`): quantized (int8) HNSW, ONNX | 0.3003 | 0.2449 | +| Arctic-embed-l (`shard01`): quantized (int8) HNSW, ONNX | 0.2599 | 0.2184 | +| Arctic-embed-l (`shard02`): quantized (int8) HNSW, ONNX | 0.2661 | 0.2211 | +| Arctic-embed-l (`shard03`): quantized (int8) HNSW, ONNX | 0.2705 | 0.2388 | +| Arctic-embed-l (`shard04`): quantized (int8) HNSW, ONNX | 0.2937 | 0.2253 | +| Arctic-embed-l (`shard05`): quantized (int8) HNSW, ONNX | 0.2590 | 0.2383 | +| Arctic-embed-l (`shard06`): quantized (int8) HNSW, ONNX | 0.2444 | 0.2336 | +| Arctic-embed-l (`shard07`): quantized (int8) HNSW, ONNX | 0.2417 | 0.2255 | +| Arctic-embed-l (`shard08`): quantized (int8) HNSW, ONNX | 0.2847 | 0.2765 | +| Arctic-embed-l (`shard09`): quantized (int8) HNSW, ONNX | 0.2432 | 0.2457 | The following command will reproduce the above experiments: @@ -282,7 +344,7 @@ To print out the commands that will generate the above runs without performing t ## MS MARCO V2.1 Documents -❗ Beware, running these experiments will automatically download 12 indexes totaling 698.0 GB. +❗ Beware, running these experiments will automatically download 2 indexes totaling 146 GB. The MS MARCO V2.1 collections were created for the [TREC RAG Track](https://trec-rag.github.io/). There were two variants: the documents corpus and the segmented documents corpus. @@ -299,14 +361,14 @@ The table below reports effectiveness (dev in terms of RR@10, DL21-DL23, RAGgy i The following command will reproduce the above experiments: ```bash -java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2.1 +java -cp $ANSERINI_JAR io.anserini.reproduce.RunMsMarco -collection msmarco-v2.1-doc ``` To print out the commands that will generate the above runs without performing the runs, use the options `-dryRun -printCommands`. ## BEIR -❗ Beware, running these experiments will automatically download 174 indexes totaling 391.5 GB. +❗ Beware, running these experiments will automatically download 174 indexes totaling 392 GB. Here is a selection of models that are currently supported in Anserini: @@ -317,7 +379,7 @@ Here is a selection of models that are currently supported in Anserini: + BGE (HNSW): bge-base-en-v1.5 using HNSW indexes, with ONNX query encoding > Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin. [Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses.](https://dl.acm.org/doi/10.1145/3626772.3657862) _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024)_, pages 1431-1440, July 2024, Washington, D.C. -> + The table below reports the effectiveness of the models (nDCG@10): | Corpus | BM25 (flat) | BM25 (MF) | SPLADE-v3 | BGE (flat) | BGE (HNSW) | @@ -352,17 +414,17 @@ The table below reports the effectiveness of the models (nDCG@10): | `climate-fever` | 0.1651 | 0.2129 | 0.2625 | 0.3117 | 0.3117 | | `scifact` | 0.6789 | 0.6647 | 0.7140 | 0.7408 | 0.7408 | -The following command will reproduce the above experiments (along with other experimental conditions not presented above): +The following command will reproduce runs corresponding to the above models (as well as additional ones not included in the table): ```bash java -cp $ANSERINI_JAR io.anserini.reproduce.RunBeir ``` -To print out the commands that will generate the above runs without performing the runs, use the options `-dryRun -printCommands`. +To print out the commands that will generate runs for all available models (without actually performing the runs), use the options `-dryRun -printCommands`. ## BRIGHT -❗ Beware, running these experiments will automatically download 24 indexes totaling 1.7 GB. +❗ Beware, running these experiments will automatically download 36 indexes totaling 6.8 GB. BRIGHT is a retrieval benchmark described [here](https://arxiv.org/abs/2407.12883). The following table reports nDCG@10 scores. @@ -396,10 +458,10 @@ The following table reports nDCG@10 scores. |   | | | | | **Overall average** | **0.1369** | **0.1556** | **0.1384** | -The following command will reproduce the above experiments (along with other experimental conditions not presented above): +The following command will reproduce runs corresponding to the above models (as well as additional ones not included in the table): ```bash java -cp $ANSERINI_JAR io.anserini.reproduce.RunBright ``` -To print out the commands that will generate the above runs without performing the runs, use the options `-dryRun -printCommands`. +To print out the commands that will generate runs for all available models (without actually performing the runs), use the options `-dryRun -printCommands`. diff --git a/src/main/resources/reproduce/msmarco-v1-doc.yaml b/src/main/resources/reproduce/msmarco-v1-doc.yaml index 252c180eb7..8bca8b1490 100644 --- a/src/main/resources/reproduce/msmarco-v1-doc.yaml +++ b/src/main/resources/reproduce/msmarco-v1-doc.yaml @@ -1,7 +1,7 @@ conditions: - name: bm25-doc-default - display: "BM25 (k1=0.9, b=0.4)" - display_html: "BM25 (k1=0.9, b=0.4)" + display: "BM25 complete doc (k1=0.9, b=0.4)" + display_html: "BM25 complete doc (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc -topics $topics -output $output -hits 1000 -bm25 topics: @@ -22,8 +22,8 @@ conditions: nDCG@10: 0.5286 R@1K: 0.8085 - name: bm25-d2q-t5-doc-default - display: "BM25 (k1=0.9, b=0.4) with doc2query-T5 expansions" - display_html: "BM25 with doc2query-T5 expansions (k1=0.9, b=0.4)" + display: "BM25 complete doc with doc2query-T5 (k1=0.9, b=0.4)" + display_html: "BM25 complete doc with doc2query-T5 (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc.d2q-t5 -topics $topics -output $output -hits 1000 -bm25 topics: @@ -44,8 +44,8 @@ conditions: nDCG@10: 0.5885 R@1K: 0.8403 - name: bm25-doc-segmented-default - display: "BM25 segmented (k1=0.9, b=0.4) " - display_html: "BM25 segmented (k1=0.9, b=0.4)" + display: "BM25 segmented doc (k1=0.9, b=0.4)" + display_html: "BM25 segmented doc (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc-segmented -topics $topics -output $output -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -66,8 +66,8 @@ conditions: nDCG@10: 0.5281 R@1K: 0.7755 - name: bm25-d2q-t5-doc-segmented-default - display: "BM25 segmented (k1=0.9, b=0.4) with doc2query-T5 expansions" - display_html: "BM25 segmented with doc2query-T5 expansions (k1=0.9, b=0.4)" + display: "BM25 segmented doc with doc2query-T5 (k1=0.9, b=0.4)" + display_html: "BM25 segmented doc with doc2query-T5 (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc-segmented.d2q-t5 -topics $topics -output $output -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -88,8 +88,8 @@ conditions: nDCG@10: 0.5957 R@1K: 0.8046 - name: unicoil-noexp.cached - display: "uniCOIL (no expansions) using cached queries" - display_html: "uniCOIL (no expansions) using cached queries" + display: "uniCOIL (no expansions): cached queries" + display_html: "uniCOIL (no expansions): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc-segmented.unicoil-noexp -topics $topics -output $output -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -110,8 +110,8 @@ conditions: nDCG@10: 0.5893 R@1K: 0.7623 - name: unicoil.cached - display: "uniCOIL (with doc2query-T5 expansions) using cached queries" - display_html: "uniCOIL (with doc2query-T5 expansions) using cached queries" + display: "uniCOIL (with doc2query-T5): cached queries" + display_html: "uniCOIL (with doc2query-T5): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc-segmented.unicoil -topics $topics -output $output -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -132,8 +132,8 @@ conditions: nDCG@10: 0.6033 R@1K: 0.7869 - name: unicoil.onnx - display: "uniCOIL (with doc2query-T5 expansions) using ONNX" - display_html: "uniCOIL (with doc2query-T5 expansions) using ONNX" + display: "uniCOIL (with doc2query-T5): ONNX" + display_html: "uniCOIL (with doc2query-T5): ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-doc-segmented.unicoil -topics $topics -output $output -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 -encoder UniCoil topics: diff --git a/src/main/resources/reproduce/msmarco-v1-passage.yaml b/src/main/resources/reproduce/msmarco-v1-passage.yaml index 131dd86b51..f02af108e2 100644 --- a/src/main/resources/reproduce/msmarco-v1-passage.yaml +++ b/src/main/resources/reproduce/msmarco-v1-passage.yaml @@ -23,8 +23,8 @@ conditions: nDCG@10: 0.4796 R@1K: 0.7863 - name: splade-pp-ed.cached - display: "SPLADE++ EnsembleDistil (cached queries)" - display_html: "SPLADE++ EnsembleDistil (cached queries)" + display: "SPLADE++ EnsembleDistil: cached queries" + display_html: "SPLADE++ EnsembleDistil: cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-passage.splade-pp-ed -topics $topics -output $output -impact -pretokenized -hits 1000 topics: @@ -46,8 +46,8 @@ conditions: nDCG@10: 0.7198 R@1K: 0.8995 - name: splade-pp-ed.onnx - display: "SPLADE++ EnsembleDistil (ONNX)" - display_html: "SPLADE++ EnsembleDistil (ONNX)" + display: "SPLADE++ EnsembleDistil: ONNX" + display_html: "SPLADE++ EnsembleDistil: ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-passage.splade-pp-ed -topics $topics -output $output -impact -pretokenized -hits 1000 -encoder SpladePlusPlusEnsembleDistil topics: @@ -69,8 +69,8 @@ conditions: nDCG@10: 0.7197 R@1K: 0.8998 - name: splade-v3.cached - display: "SPLADE-v3 (cached queries)" - display_html: "SPLADE-v3 (cached queries)" + display: "SPLADE-v3: cached queries" + display_html: "SPLADE-v3: cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-passage.splade-v3 -topics $topics -output $output -impact -pretokenized -hits 1000 topics: @@ -92,8 +92,8 @@ conditions: nDCG@10: 0.7522 R@1K: 0.9039 - name: splade-v3.onnx - display: "SPLADE-v3 (ONNX)" - display_html: "SPLADE-v3 (ONNX)" + display: "SPLADE-v3: ONNX" + display_html: "SPLADE-v3: ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v1-passage.splade-v3 -topics $topics -output $output -impact -pretokenized -hits 1000 -encoder SpladeV3 topics: @@ -115,8 +115,8 @@ conditions: nDCG@10: 0.7522 R@1K: 0.9039 - name: cosdpr-distil.hnsw.cached - display: "cosDPR-distil w/ HNSW fp32 (cached queries)" - display_html: "cosDPR-distil w/ HNSW fp32 (cached queries)" + display: "cosDPR-distil: HNSW, cached queries" + display_html: "cosDPR-distil: HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cosdpr-distil.hnsw -topics $topics -output $output -efSearch 1000 -hits 1000 topics: @@ -138,8 +138,8 @@ conditions: nDCG@10: 0.7025 R@1K: 0.8540 - name: cosdpr-distil.hnsw.onnx - display: "cosDPR-distil w/ HNSW fp32 (ONNX)" - display_html: "cosDPR-distil w/ HNSW fp32 (ONNX)" + display: "cosDPR-distil: HNSW, ONNX" + display_html: "cosDPR-distil: HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cosdpr-distil.hnsw -topics $topics -output $output -efSearch 1000 -hits 1000 -encoder CosDprDistil topics: @@ -161,8 +161,8 @@ conditions: nDCG@10: 0.7025 R@1K: 0.8540 - name: cosdpr-distil.hnsw-int8.cached - display: "cosDPR-distil w/ HNSW int8 (cached queries)" - display_html: "cosDPR-distil w/ HNSW int8 (cached queries)" + display: "cosDPR-distil: quantized (int8) HNSW, cached queries" + display_html: "cosDPR-distil: quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cosdpr-distil.hnsw-int8 -topics $topics -output $output -efSearch 1000 -hits 1000 topics: @@ -184,8 +184,8 @@ conditions: nDCG@10: 0.7004 R@1K: 0.8538 - name: cosdpr-distil.hnsw-int8.onnx - display: "cosDPR-distil w/ HNSW int8 (ONNX)" - display_html: "cosDPR-distil w/ HNSW int8 (ONNX)" + display: "cosDPR-distil: quantized (int8) HNSW, ONNX" + display_html: "cosDPR-distil: quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cosdpr-distil.hnsw-int8 -topics $topics -output $output -efSearch 1000 -hits 1000 -encoder CosDprDistil topics: @@ -207,8 +207,8 @@ conditions: nDCG@10: 0.6996 R@1K: 0.8538 - name: bge-base-en-v1.5.hnsw.cached - display: "bge-base-en-v1.5 w/ HNSW fp32 (cached queries)" - display_html: "bge-base-en-v1.5 w/ HNSW fp32 (cached queries)" + display: "bge-base-en-v1.5: HNSW, cached queries" + display_html: "bge-base-en-v1.5: HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.bge-base-en-v1.5.hnsw -topics $topics -output $output -efSearch 1000 -hits 1000 topics: @@ -230,8 +230,8 @@ conditions: nDCG@10: 0.6780 R@1K: 0.8483 - name: bge-base-en-v1.5.hnsw.onnx - display: "bge-base-en-v1.5 w/ HNSW fp32 (ONNX)" - display_html: "bge-base-en-v1.5 w/ HNSW fp32 (ONNX)" + display: "bge-base-en-v1.5: HNSW, ONNX" + display_html: "bge-base-en-v1.5: HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.bge-base-en-v1.5.hnsw -topics $topics -output $output -efSearch 1000 -hits 1000 -encoder BgeBaseEn15 topics: @@ -253,8 +253,8 @@ conditions: nDCG@10: 0.6768 R@1K: 0.8526 - name: bge-base-en-v1.5.hnsw-int8.cached - display: "bge-base-en-v1.5 w/ HNSW int8 (cached queries)" - display_html: "bge-base-en-v1.5 w/ HNSW int8 (cached queries)" + display: "bge-base-en-v1.5: quantized (int8) HNSW, cached queries" + display_html: "bge-base-en-v1.5: quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.bge-base-en-v1.5.hnsw-int8 -topics $topics -output $output -efSearch 1000 -hits 1000 topics: @@ -276,8 +276,8 @@ conditions: nDCG@10: 0.6738 R@1K: 0.8420 - name: bge-base-en-v1.5.hnsw-int8.onnx - display: "bge-base-en-v1.5 w/ HNSW int8 (ONNX)" - display_html: "bge-base-en-v1.5 w/ HNSW int8 (ONNX)" + display: "bge-base-en-v1.5: quantized (int8) HNSW, ONNX" + display_html: "bge-base-en-v1.5: quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.bge-base-en-v1.5.hnsw-int8 -topics $topics -output $output -efSearch 1000 -hits 1000 -encoder BgeBaseEn15 topics: @@ -299,8 +299,8 @@ conditions: nDCG@10: 0.6767 R@1K: 0.8468 - name: cohere-embed-english-v3.0.hnsw.cached - display: "cohere-embed-english-v3.0 w/ HNSW fp32 (cached queries)" - display_html: "cohere-embed-english-v3.0 w/ HNSW fp32 (cached queries)" + display: "cohere-embed-english-v3.0: HNSW, cached queries" + display_html: "cohere-embed-english-v3.0: HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cohere-embed-english-v3.0.hnsw -topics $topics -output $output -efSearch 1000 -hits 1000 topics: @@ -322,8 +322,8 @@ conditions: nDCG@10: 0.7245 R@1K: 0.8682 - name: cohere-embed-english-v3.0.hnsw-int8.cached - display: "cohere-embed-english-v3.0 w/ HNSW int8 (cached queries)" - display_html: "cohere-embed-english-v3.0 w/ HNSW int8 (cached queries)" + display: "cohere-embed-english-v3.0: quantized (int8) HNSW, cached queries" + display_html: "cohere-embed-english-v3.0: quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v1-passage.cohere-embed-english-v3.0.hnsw-int8 -topics $topics -output $output -efSearch 1000 -hits 1000 topics: diff --git a/src/main/resources/reproduce/msmarco-v2-doc.yaml b/src/main/resources/reproduce/msmarco-v2-doc.yaml index 44df461c42..5a4ac16234 100644 --- a/src/main/resources/reproduce/msmarco-v2-doc.yaml +++ b/src/main/resources/reproduce/msmarco-v2-doc.yaml @@ -1,6 +1,6 @@ conditions: - name: bm25-doc-default - display: "BM25 (k1=0.9, b=0.4)" + display: "BM25 complete doc (k1=0.9, b=0.4)" display_html: "BM25 (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc -topics $topics -output $output -hits 1000 -bm25 @@ -38,8 +38,8 @@ conditions: R@100: 0.2422 R@1K: 0.5262 - name: bm25-d2q-t5-doc-default - display: "BM25 (k1=0.9, b=0.4) with doc2query-T5 expansions" - display_html: "BM25 with doc2query-T5 expansions (k1=0.9, b=0.4)" + display: "BM25 completed doc with doc2query-T5 (k1=0.9, b=0.4)" + display_html: "BM25 with doc2query-T5 (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc.d2q-t5 -topics $topics -output $output -hits 1000 -bm25 topics: @@ -76,7 +76,7 @@ conditions: R@100: 0.2773 R@1K: 0.5549 - name: bm25-doc-segmented-default - display: "BM25 segmented (k1=0.9, b=0.4) " + display: "BM25 segmented doc (k1=0.9, b=0.4)" display_html: "BM25 segmented (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc-segmented -topics $topics -output $output -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 @@ -114,8 +114,8 @@ conditions: R@100: 0.2884 R@1K: 0.5662 - name: bm25-d2q-t5-doc-segmented-default - display: "BM25 segmented (k1=0.9, b=0.4) with doc2query-T5 expansions" - display_html: "BM25 segmented with doc2query-T5 expansions (k1=0.9, b=0.4)" + display: "BM25 segmented doc with doc2query-T5 (k1=0.9, b=0.4)" + display_html: "BM25 segmented with doc2query-T5 (k1=0.9, b=0.4)" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc-segmented.d2q-t5 -topics $topics -output $output -bm25 -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -151,9 +151,9 @@ conditions: nDCG@10: 0.3612 R@100: 0.3078 R@1K: 0.5967 - - name: unicoil-noexp-0shot.cached - display: "uniCOIL (noexp) zero-shot using cached queries" - display_html: "uniCOIL (noexp) zero-shot using cached queries" + - name: unicoil-noexp.cached + display: "uniCOIL (noexp): cached queries" + display_html: "uniCOIL (noexp): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc-segmented.unicoil-noexp-0shot -topics $topics -output $output -parallelism 16 -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -189,9 +189,9 @@ conditions: nDCG@10: 0.3898 R@100: 0.2949 R@1K: 0.5462 - - name: unicoil-0shot.cached - display: "uniCOIL (with doc2query-T5) zero-shot using cached queries" - display_html: "uniCOIL (with doc2query-T5) zero-shot using cached queries" + - name: unicoil.cached + display: "uniCOIL (with doc2query-T5): cached queries" + display_html: "uniCOIL (with doc2query-T5): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc-segmented.unicoil-0shot -topics $topics -output $output -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 topics: @@ -227,9 +227,9 @@ conditions: nDCG@10: 0.4149 R@100: 0.3101 R@1K: 0.5753 - - name: unicoil-0shot.onnx - display: "uniCOIL (with doc2query-T5) zero-shot using ONNX" - display_html: "uniCOIL (with doc2query-T5) zero-shot using ONNX" + - name: unicoil.onnx + display: "uniCOIL (with doc2query-T5): ONNX" + display_html: "uniCOIL (with doc2query-T5): ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-doc-segmented.unicoil-0shot -topics $topics -output $output -impact -pretokenized -hits 10000 -selectMaxPassage -selectMaxPassage.delimiter \# -selectMaxPassage.hits 1000 -encoder UniCoil topics: diff --git a/src/main/resources/reproduce/msmarco-v2-passage.yaml b/src/main/resources/reproduce/msmarco-v2-passage.yaml index 974d009596..a6c7a3a253 100644 --- a/src/main/resources/reproduce/msmarco-v2-passage.yaml +++ b/src/main/resources/reproduce/msmarco-v2-passage.yaml @@ -37,9 +37,9 @@ conditions: nDCG@10: 0.2627 R@100: 0.2329 R@1K: 0.4346 - - name: unicoil-noexp-0shot.cached - display: "Unicoil (no expansion) zero-shot using cached queries" - display_html: "Unicoil (no expansion) zero-shot using cached queries" + - name: unicoil-noexp.cached + display: "uniCOIL (no expansion): cached queries" + display_html: "uniCOIL (no expansion): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-passage.unicoil-noexp-0shot -topics $topics -output $output -hits 1000 -impact -pretokenized topics: @@ -75,9 +75,9 @@ conditions: nDCG@10: 0.3262 R@100: 0.2761 R@1K: 0.5070 - - name: unicoil-0shot.cached - display: "Unicoil (with doc2query-T5 expansions) zero-shot using cached queries" - display_html: "Unicoil (with doc2query-T5 expansions) zero-shot using cached queries" + - name: unicoil.cached + display: "uniCOIL (with doc2query-T5): cached queries" + display_html: "uniCOIL (with doc2query-T5): cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-passage.unicoil-0shot -topics $topics -output $output -hits 1000 -impact -pretokenized topics: @@ -113,3 +113,41 @@ conditions: nDCG@10: 0.3855 R@100: 0.3293 R@1K: 0.5541 + - name: unicoil.onnx + display: "uniCOIL (with doc2query-T5): ONNX" + display_html: "uniCOIL (with doc2query-T5): ONNX" + display_row: "" + command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2-passage.unicoil-0shot -topics $topics -output $output -hits 1000 -impact -pretokenized -encoder UniCoil + topics: + - topic_key: msmarco-v2-passage.dev + eval_key: msmarco-v2-passage.dev + scores: + - MRR@100: 0.1499 + - topic_key: msmarco-v2-passage.dev2 + eval_key: msmarco-v2-passage.dev2 + scores: + - MRR@100: 0.1577 + - topic_key: dl21 + eval_key: dl21-passage + scores: + - MAP@100: 0.2538 + MRR@100: 0.7311 + nDCG@10: 0.6159 + R@100: 0.4731 + R@1K: 0.7551 + - topic_key: dl22 + eval_key: dl22-passage + scores: + - MAP@100: 0.1050 + MRR@100: 0.5831 + nDCG@10: 0.4614 + R@100: 0.2716 + R@1K: 0.5253 + - topic_key: dl23 + eval_key: dl23-passage + scores: + - MAP@100: 0.1437 + MRR@100: 0.6424 + nDCG@10: 0.3855 + R@100: 0.3293 + R@1K: 0.5541 diff --git a/src/main/resources/reproduce/msmarco-v2.1-doc-segmented.yaml b/src/main/resources/reproduce/msmarco-v2.1-doc-segmented.yaml index 38a746c00d..280af7bc81 100644 --- a/src/main/resources/reproduce/msmarco-v2.1-doc-segmented.yaml +++ b/src/main/resources/reproduce/msmarco-v2.1-doc-segmented.yaml @@ -18,8 +18,8 @@ conditions: nDCG@100: 0.2345 R@100: 0.1698 - name: shard00.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard00): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard00): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard00.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -36,8 +36,8 @@ conditions: nDCG@100: 0.1440 R@100: 0.0746 - name: shard01.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard01): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard01): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard01.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -54,8 +54,8 @@ conditions: nDCG@100: 0.1274 R@100: 0.0690 - name: shard02.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard02): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard02): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard02.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -72,8 +72,8 @@ conditions: nDCG@100: 0.1296 R@100: 0.0720 - name: shard03.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard03): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard03): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard03.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -90,8 +90,8 @@ conditions: nDCG@100: 0.1425 R@100: 0.0805 - name: shard04.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard04): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard04): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard04.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -108,8 +108,8 @@ conditions: nDCG@100: 0.1261 R@100: 0.0654 - name: shard05.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard05): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard05): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard05.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -126,8 +126,8 @@ conditions: nDCG@100: 0.1305 R@100: 0.0638 - name: shard06.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard06): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard06): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard06.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -144,8 +144,8 @@ conditions: nDCG@100: 0.1391 R@100: 0.0715 - name: shard07.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard07): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard07): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard07.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -162,8 +162,8 @@ conditions: nDCG@100: 0.1263 R@100: 0.0587 - name: shard08.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard08): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard08): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard08.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -180,8 +180,8 @@ conditions: nDCG@100: 0.1663 R@100: 0.0893 - name: shard09.arctic-l.hnsw-int8.cached - display: "ArcticEmbed-L w/ HNSW int8 (cached queries)" - display_html: "ArcticEmbed-L w/ HNSW int8 (cached queries)" + display: "ArcticEmbed-L (shard09): quantized (int8) HNSW, cached queries" + display_html: "ArcticEmbed-L (shard09): quantized (int8) HNSW, cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard09.arctic-embed-l.hnsw-int8 -topics $topics -output $output -hits 250 -efSearch 1000 topics: @@ -198,8 +198,8 @@ conditions: nDCG@100: 0.1376 R@100: 0.0715 - name: shard00.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard00): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard00): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard00.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -216,8 +216,8 @@ conditions: nDCG@100: 0.1440 R@100: 0.0746 - name: shard01.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard01): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard01): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard01.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -234,8 +234,8 @@ conditions: nDCG@100: 0.1274 R@100: 0.0690 - name: shard02.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard02): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard02): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard02.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -252,8 +252,8 @@ conditions: nDCG@100: 0.1296 R@100: 0.0720 - name: shard03.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard03): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard03): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard03.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -270,8 +270,8 @@ conditions: nDCG@100: 0.1425 R@100: 0.0805 - name: shard04.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard04): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard04): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard04.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -288,8 +288,8 @@ conditions: nDCG@100: 0.1261 R@100: 0.0654 - name: shard05.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard05): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard05): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard05.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -306,8 +306,8 @@ conditions: nDCG@100: 0.1305 R@100: 0.0638 - name: shard06.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard06): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard06): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard06.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -324,8 +324,8 @@ conditions: nDCG@100: 0.1391 R@100: 0.0715 - name: shard07.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard07): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard07): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard07.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -342,8 +342,8 @@ conditions: nDCG@100: 0.1263 R@100: 0.0587 - name: shard08.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard08: quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard08): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard08.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -360,8 +360,8 @@ conditions: nDCG@100: 0.1663 R@100: 0.0893 - name: shard09.arctic-l.hnsw-int8.onnx - display: "ArcticEmbed-L w/ HNSW int8 (ONNX)" - display_html: "ArcticEmbed-L w/ HNSW int8 (ONNX)" + display: "ArcticEmbed-L (shard09): quantized (int8) HNSW, ONNX" + display_html: "ArcticEmbed-L (shard09): quantized (int8) HNSW, ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchHnswDenseVectors -threads $threads -index msmarco-v2.1-doc-segmented-shard09.arctic-embed-l.hnsw-int8 -topics $topics -encoder ArcticEmbedL -output $output -hits 250 -efSearch 1000 topics: @@ -378,8 +378,8 @@ conditions: nDCG@100: 0.1376 R@100: 0.0715 - name: splade-v3.cached - display: "SPLADE-v3 (cached queries)" - display_html: "SPLADE-v3 (cached queries)" + display: "SPLADE-v3: cached queries" + display_html: "SPLADE-v3: cached queries" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2.1-doc-segmented.splade-v3 -topics $topics -output $output -impact -pretokenized -removeQuery -hits 1000 topics: @@ -396,8 +396,8 @@ conditions: nDCG@100: 0.4349 R@100: 0.3198 - name: splade-v3-onnx - display: "SPLADE-v3 (ONNX)" - display_html: "SPLADE-v3 (ONNX)" + display: "SPLADE-v3: ONNX" + display_html: "SPLADE-v3: ONNX" display_row: "" command: java -cp $fatjar --add-modules jdk.incubator.vector io.anserini.search.SearchCollection -threads $threads -index msmarco-v2.1-doc-segmented.splade-v3 -topics $topics -output $output -impact -pretokenized -removeQuery -hits 1000 -encoder SpladeV3 topics: