התחל במצב לא מקוון עם האפליקציה Player FM !
ColBERT + ColBERTv2: late interaction at a reasonable inference cost
Manage episode 355037184 series 3446693
Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discus the two influential papers introducing ColBERT (from 2020) and ColBERT v2 (from 2022), which mainly propose a fast late interaction operation to achieve a performance close to full cross-encoders but at a more manageable computational cost at inference; along with many other optimizations.
📄 ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" by Omar Khattab and Matei Zaharia. https://arxiv.org/abs/2004.12832
📄 ColBERTv2: "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" by Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2112.01488
📄 PLAID: "An Efficient Engine for Late Interaction Retrieval" by Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2205.09707
📄 CEDR: "CEDR: Contextualized Embeddings for Document Ranking" by Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. https://arxiv.org/abs/1904.07094
🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ
Timestamps:
00:00 Introduction
00:42 Why ColBERT?
03:34 Retrieval paradigms recap
08:04 ColBERT query formulation and architecture
09:04 Using ColBERT as a reranker or as an end-to-end retriever
11:28 Space Footprint vs. MRR on MS MARCO
12:24 Methodology: datasets and negative sampling
14:37 Terminology for cross encoders, interaction-based models, etc.
16:12 Results (ColBERT v1) on MS MARCO
18:41 Ablations on model components
20:34 Max pooling vs. mean pooling
22:54 Why did ColBERT have a big impact?
26:31 ColBERTv2: knowledge distillation
29:34 ColBERTv2: indexing improvements
33:59 Effects of clustering compression in performance
35:19 Results (ColBERT v2): MS MARCO
38:54 Results (ColBERT v2): BEIR
41:27 Takeaway: strong specially in out-of-domain evaluation
43:59 Qualitatively how do ColBERT scores look like?
46:21 What's the most promising of all current neural IR paradigms
49:34 How come there's still so much interest in Dense retrieval?
51:09 Many to many similarity at different granularities
53:44 What would ColBERT v3 include?
56:39 PLAID: An Efficient Engine for Late Interaction Retrieval
Contact: castella@zeta-alpha.com
21 פרקים
Manage episode 355037184 series 3446693
Andrew Yates (Assistant Professor at the University of Amsterdam) and Sergi Castella (Analyst at Zeta Alpha) discus the two influential papers introducing ColBERT (from 2020) and ColBERT v2 (from 2022), which mainly propose a fast late interaction operation to achieve a performance close to full cross-encoders but at a more manageable computational cost at inference; along with many other optimizations.
📄 ColBERT: "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" by Omar Khattab and Matei Zaharia. https://arxiv.org/abs/2004.12832
📄 ColBERTv2: "ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction" by Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2112.01488
📄 PLAID: "An Efficient Engine for Late Interaction Retrieval" by Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. https://arxiv.org/abs/2205.09707
📄 CEDR: "CEDR: Contextualized Embeddings for Document Ranking" by Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. https://arxiv.org/abs/1904.07094
🪃 Feedback form: https://scastella.typeform.com/to/rg7a5GfJ
Timestamps:
00:00 Introduction
00:42 Why ColBERT?
03:34 Retrieval paradigms recap
08:04 ColBERT query formulation and architecture
09:04 Using ColBERT as a reranker or as an end-to-end retriever
11:28 Space Footprint vs. MRR on MS MARCO
12:24 Methodology: datasets and negative sampling
14:37 Terminology for cross encoders, interaction-based models, etc.
16:12 Results (ColBERT v1) on MS MARCO
18:41 Ablations on model components
20:34 Max pooling vs. mean pooling
22:54 Why did ColBERT have a big impact?
26:31 ColBERTv2: knowledge distillation
29:34 ColBERTv2: indexing improvements
33:59 Effects of clustering compression in performance
35:19 Results (ColBERT v2): MS MARCO
38:54 Results (ColBERT v2): BEIR
41:27 Takeaway: strong specially in out-of-domain evaluation
43:59 Qualitatively how do ColBERT scores look like?
46:21 What's the most promising of all current neural IR paradigms
49:34 How come there's still so much interest in Dense retrieval?
51:09 Many to many similarity at different granularities
53:44 What would ColBERT v3 include?
56:39 PLAID: An Efficient Engine for Late Interaction Retrieval
Contact: castella@zeta-alpha.com
21 פרקים
כל הפרקים
×
1 AGI vs ASI: The future of AI-supported decision making with Louis Rosenberg 54:42

1 EXAONE 3.0: An Expert AI for Everyone (with Hyeongu Yun) 24:57

1 Zeta-Alpha-E5-Mistral: Finetuning LLMs for Retrieval (with Arthur Câmara) 19:35

1 ColPali: Document Retrieval with Vision-Language Models only (with Manuel Faysse) 34:48

1 Using LLMs in Information Retrieval (w/ Ronak Pradeep) 22:15

1 Designing Reliable AI Systems with DSPy (w/ Omar Khattab) 59:57

1 The Power of Noise (w/ Florin Cuconasu) 11:45

1 Benchmarking IR Models (w/ Nandan Thakur) 21:55

1 Baking the Future of Information Retrieval Models 27:05

1 Hacking JIT Assembly to Build Exascale AI Infrastructure 38:04

1 The Promise of Language Models for Search: Generative Information Retrieval 1:07:31

1 Task-aware Retrieval with Instructions 1:11:13

1 Generating Training Data with Large Language Models w/ Special Guest Marzieh Fadaee 1:16:14

1 ColBERT + ColBERTv2: late interaction at a reasonable inference cost 57:30

1 Evaluating Extrapolation Performance of Dense Retrieval: How does DR compare to cross encoders when it comes to generalization? 58:30
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.