Improving LLM Accuracy Across Diverse Text Formats

Prompt formatting can have a dramatic impact on LLM performance, but it varies substantially across models. Some pragmatic findings from a recent research paper: 💡 Prompt Format Significantly Affects LLM Performance. Different prompt formats (plain text, Markdown, YAML, JSON) can result in performance variations of up to 40%, depending on the task and model. For instance, GPT-3.5-turbo showed a dramatic performance shift between Markdown and JSON in code translation tasks, while GPT-4 exhibited greater stability. This indicates the importance of testing and optimizing prompts for specific tasks and models. 🛠️ Tailor Formats to Task and Model. Prompt formats like JSON, Markdown, YAML, and plain text yield different performance outcomes across tasks. For instance, GPT-3.5-turbo performed 40% better in JSON for code tasks, while GPT-4 preferred Markdown for reasoning tasks. Test multiple formats early in your process to identify which structure maximizes results for your specific task and model. 📋 Keep Instructions and Context Explicit. Include clear task instructions, persona descriptions, and examples in your prompts. For example, specifying roles (“You are a Python coder”) and output style (“Respond in JSON”) improves model understanding. Consistency in how you frame the task across different formats minimizes confusion and enhances reliability. 📊 Choose Format Based on Data Complexity. For simple tasks, plain text or Markdown often suffices. For structured outputs like programming or translations, formats such as JSON or YAML may perform better. Align the prompt format with the complexity of the expected response to leverage the model’s capabilities fully. 🔄 Iterate and Validate Performance. Run tests with variations in prompt structure to measure impact. Tools like Coefficient of Mean Deviation (CMD) or Intersection-over-Union (IoU) can help quantify performance differences. Start with benchmarks like MMLU or HumanEval to validate consistency and accuracy before deploying at scale. 🚀 Leverage Larger Models for Stability. If working with sensitive tasks requiring consistent outputs, opt for larger models like GPT-4, which show better robustness to format changes. For instance, GPT-4 maintained higher performance consistency across benchmarks compared to GPT-3.5. Link to paper in comments.

Achieving 3x-25x Performance Gains for High-Quality, AI-Powered Data Analysis Asking complex data questions in plain English and getting precise answers feels like magic, but it’s technically challenging. One of my jobs is analyzing the health of numerous programs. To make that easier we are building an AI app with Sapient Slingshot that answers natural language queries by generating and executing code on project/program health data. The challenge is that this process needs to be both fast and reliable. We started with gemini-2.5-pro, but 50+ second response times and inconsistent results made it unsuitable for interactive use. Our goal: reduce latency without sacrificing accuracy. The New Bottleneck: Tuning "Think Time" Traditional optimization targets code execution, but in AI apps, the real bottleneck is LLM "think time", i.e. the delay in generating correct code on the fly. Here are some techniques we used to cut think time while maintaining output quality: ① Context-Rich Prompts Accuracy starts with context. We dynamically create prompts for each query: ➜ Pre-Processing Logic: We pre-generate any code that doesn't need "intelligence" so that LLM doesn't have to ➜ Dynamic Data-Awareness: Prompts include full schema, sample data, and value stats to give the model a full view. ➜ Domain Templates: We tailor prompts for specific ontology like "Client satisfaction" or "Cycle Time" or "Quality". This reduces errors and latency, improving codegen quality from the first try. ② Structured Code Generation Even with great context, LLMs can output messy code. We guide query structure explicitly: ➜ Simple queries: Direct the LLM to generate a single line chained pandas expression. ➜ Complex queries : Direct the LLM to generate two lines, one for processing, one for the final result Clear patterns ensure clean, reliable output. ③ Two-Tiered Caching for Speed Once accuracy was reliable, we tackled speed with intelligent caching: ➜ Tier 1: Helper Cache – 3x Faster ⊙ Find a semantically similar past query ⊙ Use a faster model (e.g. gemini-2.5-flash) ⊙ Include the past query and code as a one-shot prompt This cut response times from 50+s to <15s while maintaining accuracy. ➜ Tier 2: Lightning Cache – 25x Faster ⊙ Detect duplicates for exact or near matches ⊙ Reuse validated code ⊙ Execute instantly, skipping the LLM This brought response times to ~2 seconds for repeated queries. ④ Advanced Memory Architecture ➜ Graph Memory (Neo4j via Graphiti): Stores query history, code, and relationships for fast, structured retrieval. ➜ High-Quality Embeddings: We use BAAI/bge-large-en-v1.5 to match queries by true meaning. ➜ Conversational Context: Full session history is stored, so prompts reflect recent interactions, enabling seamless follow-ups. By combining rich context, structured code, caching, and smart memory, we can build AI systems that deliver natural language querying with the speed and reliability that we, as users, expect of it.

One of the persistent challenges in using large language models (LLMs) is getting them to follow instructions reliably — especially when the instructions are subtle or domain-specific. DeepMind’s latest research introduces Symbol Tuning, a simple yet powerful fine-tuning method that significantly improves an LLM’s ability to follow symbolic prompts (e.g., bullet points, XML, Markdown, or code-like instructions) in zero-shot and few-shot settings. https://lnkd.in/gzKDdHQ2 Why this matters: 🔹 Improves instruction following in GPT-class models 🔹 Works with tiny amounts of data (just 100K tokens!) 🔹 Boosts performance in math, code, and reasoning-heavy tasks 🔹 Enhances models' ability to generalize across symbolic formats This has massive implications for building enterprise agents, RAG pipelines, and developer copilots that need high-precision, structured interaction with users or data. A great reminder: sometimes, small, well-targeted innovations create massive gains. #LLM #InContextLearning #SymbolTuning #PromptEngineering #DeepMind #GenAI #AIResearch #InstructionFollowing #EnterpriseAI #DeveloperTools

It is easy to criticize LLM hallucinations but Google researchers just made a major leap toward solving them for statistical data. In the DataGemma paper (Sep ’24), they teach LLMs when to ask an external source instead of guessing. They propose two approaches: Retrieval interleaved generation (RIG) - the model injects natural language queries into its output, triggering fact retrieval from Data Commons. Retrieval augmented generation (RAG) - the model pulls full data tables into its context and reasons over them with a long-context LLM. The results are impressive: (1) RIG improved statistical accuracy from 5–17% to ~58% (2) RAG hit ~99% accuracy on direct citations (with some inference errors still remaining) (3) Users strongly preferred the new responses over baseline answers. As LLMs increasingly rely on external tools, teaching them "when to ask" may become as important as "how to answer." Paper https://lnkd.in/gaKY_VNE

Exciting New Research: A Library of LLM Intrinsics for Retrieval-Augmented Generation I just came across a groundbreaking paper from IBM Research that introduces a novel concept for the LLM developer community: a library of LLM Intrinsics for Retrieval-Augmented Generation (RAG). In the software world, we've long benefited from reusable libraries with well-defined APIs. However, the LLM ecosystem has lacked this pattern until now. This research proposes "LLM Intrinsics" - capabilities that can be invoked through stable, well-defined APIs, independent of their implementation details. >> What are LLM Intrinsics? The researchers define an LLM intrinsic as "a capability that can be invoked through a well-defined API that is reasonably stable and independent of how the LLM intrinsic itself is implemented." Think of them as compiler intrinsics for LLMs - functions that occur frequently enough to warrant standardization. >> The RAG Intrinsics Library The library currently includes five powerful intrinsics: 1. Query Rewrite (QR) - Decontextualizes multi-turn conversation queries into standalone versions, improving retriever performance by 9 percentage points in Recall@20 and 8 points in RAGAS Faithfulness. 2. Uncertainty Quantification (UQ) - Provides calibrated certainty scores (5% to 95%) for answers, with an impressive Expected Calibration Error of just 0.064 across tasks. 3. Hallucination Detection (HD) - Analyzes responses against source documents to assign hallucination risk scores for each sentence, achieving 72.2% F1 score on the RAGTruth benchmark. 4. Answerability Determination (AD) - Determines if a query can be answered based on provided documents, achieving 77.4% weighted F1 score on SQUADRUN Dev and 86.1% on MT-RAG. 5. Citation Generation (CG) - Creates fine-grained citations for each sentence in responses, outperforming Llama-3.1-70B-Instruct with F1 scores of 62.0% to 75.4% across LongBench-Cite datasets. >> Implementation Details Each intrinsic is implemented as a LoRA adapter for IBM Granite 3.2 8b Instruct, available on HuggingFace. More importantly, they're accessible through Granite IO Processing, a framework that handles input/output transformations. The researchers also demonstrate how these intrinsics can be composed into powerful workflows. For example, combining Query Rewrite with Answerability Determination yields a 11% improvement in Joint Answerability-Faithfulness Score compared to using neither. This work represents a significant step toward standardization in the LLM ecosystem, potentially enabling the same level of collaboration and specialization we've seen in traditional software development. All models are released under Apache 2.0 license for both research and commercial use. Definitely worth exploring if you're working on RAG applications!

Thank you to the University of Toronto Machine Intelligence Student Team for inviting me to present a keynote on augmenting human-labeled datasets using Large Language Models (LLMs). Human-labeled data is crucial for testing, tuning, customizing, and validating LLMs in organizations. This is because human labeled data provides the ground truth for developing trustworthy #GenerativeAI applications and #AgenticAI systems. Yet acquiring sufficient human labeled data is often a bottleneck in many organizations. Subject matter experts and domain specialists typically have limited time for labeling tasks due to competing professional demands, making large-scale manual labeling difficult to sustain. My talk focused on how LLMs can be used not to substitute human labels, but to systematically augment them—extending the utility of existing human labeled data and improving model robustness without proportionally increasing manual labeling effort. I described practical methods for implementing two augmentation techniques with strong empirical grounding: • Negative Reinforcement with Counterfactual Examples – This technique involves analyzing labeled examples to generate counterfactual examples—outputs that are intentionally incorrect or undesirable—and using them to teach the model about what not to generate. By guiding the model using these negative samples, the model learns sharper decision boundaries, increasing robustness against hallucinations and confabulations. • Contrastive Learning with Controlled Perturbations – This technique creates diverse, label-preserving variants of human-labeled examples by introducing controlled modifications to the prompts and/or completions. These perturbations maintain core semantic meaning while varying surface-level features such as syntax, phrasing, or structure, encouraging the model to generalize beyond shallow lexical or syntactic cues. These techniques have been shown to drive measurable improvements in model behavior: • Lower Perplexity → More predictable completions and improved alignment with ground-truth targets. • Reduced Token Entropy → More focused and efficient completions, reducing inference complexity. • Higher Self-Consistency → More stable completions across repeated generations of the same prompt—a key requirement for dependable downstream use. These are not theoretical constructs—they are practical techniques for overcoming constraints in human-labeled data availability and scaling of #LLM applications with greater efficiency and rigor. Appreciate the University of Toronto Machine Intelligence Student Team (UTMIST) for a well-curated conference, and the UofT AI group for their initiatives in the space. Grateful to my research partner, Olga, for her contributions in collaboratively developing content for this presentation. Kudos to my PwC Canada teammates including Michelle B, Annie, Chris M, Michelle G, Chris D, Brenda, Bahar, Danielle, and Abhinav for their partnership on our PwC #AI portfolio.

Today, LLMs are predominantly employed for processing unstructured data inputs, such as text, images, and videos, excelling in tasks like content creation and understanding. However, their application in classic machine learning tasks—such as regression, classification, and imputation of missing values—remains relatively unexplored. This gap has led to a dichotomy in organizational model maintenance, where traditional numerical models and LLMs coexist but don't fully intersect, requiring varied expertise and creating inefficiencies. This new study developed a way to help LLMs (Llama2 7B) better at understanding and working with tables. They started by converting these tables into a format that's easier for the AI to handle, using a simple coding style called Markdown. Then, they trained the AI in two main steps. First, they played a sort of "guessing game" with the AI, where parts of the table were hidden and the AI had to predict what was missing, helping it learn from the context. Second, they trained the AI on specific tasks, like classification and regression, by showing it lots of examples. To train the AI, they gathered a huge collection of table data including about 13 billion examples covering 300 different topics, ensuring that the AI was exposed to a wide range of information. After the training, they tested the AI on imputation, classification, and regression tasks using different methods such as Zero-shot prediction, finetuning, and in-context learning. The results are outstanding: • Most importantly , the approach outperforms XGBoost with default parameters across both classification and regression tasks consistently showing higher or similar R2 and ROC-AUC across different datasets. • The approach yielded an average improvement of 8.9% in classification tasks and 10.7% in regression tasks over the Llama-2 baseline. • For missing value predictions, the model outperformed GPT-4 by 27%. • Significant advancements were observed in extreme few-shot (4-shot) predictions and tasks involving extensive context learning, with improvements of 28.8% and 25.9%, respectively, over existing models. Outperforming XGBoost is a surprising yet promising achievement, considering XGBoost's standing as the most popular and potent tree-based model for classification and regression tasks. Despite this progress, there are notable limitations including the high costs and substantial training resources required. Nonetheless, the advantages of such an approach are becoming increasingly evident. As we move towards a future where LLMs may replace traditional AI methods, we edge closer to simplifying data science tasks and achieving a unified approach. Paper link: https://lnkd.in/egNAyYCx

LLMs don’t just respond to What you ask—they respond to How you ask. If you’re still relying on basic prompting, you’re leaving a lot of performance on the table. Here’s how people are systematically optimizing prompts for higher accuracy, robustness, and efficiency in AI apps: ⭐ Few-Shot Prompting – Improve precision in classification tasks by including example inputs/outputs (e.g., for detecting jailbreak attempts, spam, or misinformation). ⭐ Meta Prompting – Use an LLM to refine its own prompts (e.g., "Given this input/output, how would you rewrite this prompt for better performance?"). This works especially well for text generation and retrieval tasks. ⭐ Gradient Prompt Optimization (GPO) – Treat prompts like trainable parameters, embedding them and optimizing with loss gradients. Think of it as fine-tuning without modifying the model itself—a game-changer for low-resource AI applications. ⭐ Prompt Optimization Libraries – Tools like DSPy automate prompt refinement, evaluating variations systematically. For production AI systems, this makes tuning scalable. The Takeaway? Prompt Optimization is a Continuous Process Real-world data shifts. New failure modes emerge. Just like model retraining, prompts need continuous iteration. What’s your go-to method for improving AI prompts?

An important lesson from working with hundreds of customers on LLM deployments: there's a **big difference** in how to evaluate and fine-tune language models based on whether your task has **one right answer** or **many**. Let me explain why this matters. Tasks with one correct answer (let's call them "deterministic") include things like classification, structured extraction, and Copilot flows that produce a single action. These are cases where you can quickly check if an output is objectively correct. In contrast, "freeform" tasks have infinitely many valid outputs - think summaries, email drafts, and chatbots. Here, correctness is more subjective, with no single "right" answer. Looking at 1,000 recent datasets on OpenPipe: ~63% were freeform ~37% deterministic. Interestingly though, among the highest-volume tasks, 60% were deterministic - likely because machine-consumed outputs tend to run at higher volume. This distinction drives three key differences in implementation: 1️⃣ Deterministic tasks usually need temperature=0 for consistent, correct outputs. Freeform tasks benefit from higher temperatures (0.7-1.0) to enable creativity and variety. 2️⃣ evaluation approaches differ. Deterministic tasks can use "golden datasets" with known-correct outputs. Freeform tasks often need vibe checks, LLM-as-judge approaches, or direct user feedback. 3️⃣ fine-tuning strategies diverge. For deterministic tasks, Reinforcement Fine-Tuning (RFT) shows promise when correctness is verifiable. For freeform tasks, preference-based methods like DPO or RLHF work better for guiding style and tone. Some practical tips for deterministic tasks: - Consider smaller, specialized models for classification/extraction - Use logprobs to measure classification confidence - You can often reduce costs significantly by going small without losing accuracy For freeform tasks: - Use DPO to train on pairs of good/bad outputs - Consider RLHF to optimize for real user feedback or business metrics - Focus on measuring and improving subjective quality The key is matching your approach to your use case. Don't automatically reach for the largest, most expensive model - sometimes a smaller, more focused solution works better! Lots more details and examples in my post here: https://lnkd.in/gFWdA7kr

Day 16/30 of LLMs/SLMs - Retrieval-Augmented Generation (RAG) Large Language Models are powerful, but they have a fixed memory. They cannot know anything that happened after their training cut-off, and they struggle with facts that were never part of their dataset. When they lack the right information, they guess. The result is fluent but unreliable text — the hallmark of hallucination. Retrieval-Augmented Generation (RAG) fixes that by giving models a way to look up information before they answer. RAG is best understood as a three-stage pipeline, and LangChain has become the de facto standard framework for building each stage efficiently. 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠 You start by collecting and preparing your documents. LangChain’s loaders handle PDFs, web pages, CSVs, and APIs. These documents are then split into smaller, semantically meaningful chunks and converted into embeddings using models like OpenAI’s text-embedding-3-small, SentenceTransformers, or InstructorXL. Those embeddings are stored in a vector database such as FAISS, Pinecone, Weaviate, or Chroma, which lets you perform similarity search later. 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 When a query arrives, LangChain converts it into an embedding and searches the vector store for the most relevant documents. Retrieval strategies vary — basic similarity search, maximal marginal relevance (MMR) to diversify context, or hybrid retrieval that mixes semantic and keyword search. The retrieved text chunks are then added to the prompt as contextual grounding. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 The LLM receives the augmented prompt containing both the user query and retrieved passages. It synthesizes an answer based on that external knowledge. LangChain manages prompt templates, context formatting, and memory across queries, making the process modular and repeatable. 𝐖𝐡𝐲 𝐑𝐀𝐆 𝐌𝐚𝐭𝐭𝐞𝐫𝐬 RAG fundamentally improves factual accuracy and trust. On benchmarks such as Natural Questions and TriviaQA, a base model like LLaMA 2-13B might achieve 45 F1, while RAG-augmented versions reach 65–70 F1 - matching much larger and costlier models. 𝐆𝐞𝐭𝐭𝐢𝐧𝐠 𝐒𝐭𝐚𝐫𝐭𝐞𝐝 𝐰𝐢𝐭𝐡 𝐋𝐚𝐧𝐠𝐂𝐡𝐚𝐢𝐧 𝐑𝐀𝐆 If you want to experiment, LangChain makes it approachable. A minimal prototype takes fewer than 20 lines of code. Here’s a good progression 👉 Start with the LangChain tutorial: https://lnkd.in/gUpHpkKT 👉 Add a vector store: Try Chroma for local experiments or Pinecone for scalable hosting. 👉 Experiment with retrieval methods: compare similarity search vs. MMR. 👉 Integrate your own data: ingest PDFs, database exports, or web content. 👉 Deploy a chain: connect your retriever, model, and prompt template into a single workflow. Tune in tomorrow for more SLM/LLMs deep dives. -- 🚶➡️ To learn more about LLMs/SLMs, follow me - Karun! ♻️ Share so others can learn, and you can build your LinkedIn presence!