From the course: Advanced Guide to ChatGPT, Embeddings, and Other Large Language Models (LLMs)
Join today to access over 24,600 courses taught by industry experts.
Optimizing semantic search with cross-encoders and fine-tuning - ChatGPT Tutorial
From the course: Advanced Guide to ChatGPT, Embeddings, and Other Large Language Models (LLMs)
Optimizing semantic search with cross-encoders and fine-tuning
- So from our last section, we have our basic semantic search system set up. We made our Pinecone index, we set up a default namespace. We used the OpenAI embedder to index and retrieve a document at a time, right? And we tried it with a single query. We saw that it was a top result. Everything looked in working order, all of the parts of the machine were there. The difference now is we're going to not only add some new components, but we're also going to more rigorously test the performance of our retrieval, which is really the main part, the main crux of our semantic search system. Now, the the first secondary system that we're going to add on here is our cross-encoder. So previously in our last section, we simply used our embedding system to retrieve the top one, three, five, 10 documents. Our cross-encoder is now going to more finely re-rank those retrieved documents because generally speaking, cosine similarity works pretty well for retrieving documents. But a cross-encoder is…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.