Shadi Copty’s Post

Evaluating my fine tuned 3B model with LLM judge - many of you have asked so here goes :-). Tl;Dr - 91% accuracy vs 47% not-fine tuned; with only 140 examples and 3 minutes of training (https://lnkd.in/gHw993ZS) I started out with creating a new synthetic test data set that the fine tuned models haven't seen, using the same code from the training set (data here: https://lnkd.in/gcCSdT5e). I used Comet's Opik per Armand's recommendation (you should follow him if you aren't :-)). Then built my evaluations test for (1) JSON schema compliance, (2) distance from reference answer (3) the LLM Judge which scores first on json format, then on compliance with schema, then on entities being detected, then on not detecting more than necessary. I tried this out with a bunch of LLMs that could run on this laptop, as you can see the 3B fine tuned one performed the best, followed by the 1B, then all of the out of the box ones pretty similar and not so great. Code for you to enjoy: https://lnkd.in/guhzdUkf You'll need a Comet account, which is shockingly free. Enjoy :-) Next I'm going to try to beef up the data gen with Distilable and CrewAI, let me know if you find that interesting or have other ideas.

Evaluating my fine tuned 3B model with LLM judge - many of you have asked so here goes :-). Tl;Dr - 91% accuracy vs 47% not-fine tuned; with only 140 examples and 3 minutes of training (https://lnkd.in/gHw993ZS) I started out with creating a new synthetic test data set that the fine tuned models haven't seen, using the same code from the training set (data here: https://lnkd.in/gcCSdT5e). I used Comet's Opik per Armand's recommendation (you should follow him if you aren't :-)). Then built my evaluations test for (1) JSON schema compliance, (2) distance from reference answer (3) the LLM Judge which scores first on json format, then on compliance with schema, then on entities being detected, then on not detecting more than necessary. I tried this out with a bunch of LLMs that could run on this laptop, as you can see the 3B fine tuned one performed the best, followed by the 1B, then all of the out of the box ones pretty similar and not so great. Code for you to enjoy: https://lnkd.in/guhzdUkf You'll need a Comet account, which is shockingly free. Enjoy :-) Next I'm going to try to beef up the data gen with Distilable and CrewAI, let me know if you find that interesting or have other ideas.

Evaluating my fine tuned 3B model with LLM judge - many of you have asked so here goes :-). Tl;Dr - 91% accuracy vs 47% not-fine tuned; with only 140 examples and 3 minutes of training (https://lnkd.in/gHw993ZS) I started out with creating a new synthetic test data set that the fine tuned models haven't seen, using the same code from the training set (data here: https://lnkd.in/gcCSdT5e). I used Comet's Opik per Armand's recommendation (you should follow him if you aren't :-)). Then built my evaluations test for (1) JSON schema compliance, (2) distance from reference answer (3) the LLM Judge which scores first on json format, then on compliance with schema, then on entities being detected, then on not detecting more than necessary. I tried this out with a bunch of LLMs that could run on this laptop, as you can see the 3B fine tuned one performed the best, followed by the 1B, then all of the out of the box ones pretty similar and not so great. Code for you to enjoy: https://lnkd.in/guhzdUkf You'll need a Comet account, which is shockingly free. Enjoy :-) Next I'm going to try to beef up the data gen with Distilable and CrewAI, let me know if you find that interesting or have other ideas.

Evaluating my fine tuned 3B model with LLM judge - many of you have asked so here goes :-). Tl;Dr - 91% accuracy vs 47% not-fine tuned; with only 140 examples and 3 minutes of training (https://lnkd.in/gHw993ZS) I started out with creating a new synthetic test data set that the fine tuned models haven't seen, using the same code from the training set (data here: https://lnkd.in/gcCSdT5e). I used Comet's Opik per Armand's recommendation (you should follow him if you aren't :-)). Then built my evaluations test for (1) JSON schema compliance, (2) distance from reference answer (3) the LLM Judge which scores first on json format, then on compliance with schema, then on entities being detected, then on not detecting more than necessary. I tried this out with a bunch of LLMs that could run on this laptop, as you can see the 3B fine tuned one performed the best, followed by the 1B, then all of the out of the box ones pretty similar and not so great. Code for you to enjoy: https://lnkd.in/guhzdUkf You'll need a Comet account, which is shockingly free. Enjoy :-) Next I'm going to try to beef up the data gen with Distilable and CrewAI, let me know if you find that interesting or have other ideas.

The latest Vespa newsletter is out - highlights: 👉 RAG Vespa now provides LLM inference support, you can now implement a retrieval-augmented generation (RAG) application entirely as a Vespa application. We have added a new sample application demonstrating RAG end-to-end on Vespa: - Generation using an external LLM like OpenAI - Running an LLM locally inside the Vespa application on CPU - Running an LLM inside the Vespa application on Vespa Cloud on GPU 👉 Fuzzy Search with Prefix Match A prefix search will match “Edvard Grieg” and “Edvard Gr”. A fuzzy search matches “Edvard Grieg” with “Edward Grieg”. From Vespa 8.337 you can combine the two to match “Edvard Grieg” and “Edward Gr”. Very powerful for query completion! 👉 Pyvespa Lots of new features, including a notebook that demonstrates how the mixedbread.ai rerank models (cross encoders) can be used for global phase reranking in Vespa. 👉 Vector search performance Up to 9x faster distance calculations! Improvements for euclidean, angular, hamming and dotproduct as well as for HNSW indexing. 👉 Embeddings Since Vespa 8.329, embed the data _once_ for multiple resolutions. Store low-res in memory, hi-res on disk to optimize for cost - then use two-phase ranking for low-latency search with high precision. Get started using Vespa with LlamaIndex! Check out the new Vespa Vector Store demo notebook. Deep dive into this and more features like 10x faster data migration in the latest Vespa Newsletter:

Don't just use default settings in LLMs! Check out these 7 important LLM parameters which can help you in controlling generated responses. A detailed article is also shared below with hands-on code and examples of each of these settings! Let's understand these parameters: - max_tokens: limits the number of tokens in the response - temperature: lower values make it more consistent and higher values more creative - top_p & top_k: thresholds which can be used to select the next generated token from a subset of most probable next words at each step - frequency penalty: helps reduce same repeated tokens in response - presence penalty: encourages the model to add new diverse tokens in response - stop: standard tokens used to stop response generation The names of these parameters and typical ranges here are based on OpenAI LLM APIs, you might find slightly different names or ranges in other platforms like Huggingface but they pretty much mean and do the same thing. Do check out the article below with more details on Analytics Vidhya and share if useful!