I think both have a point.

The comparison Jev has been missing

Classifying text, estimating probabilities, and returning labels are not new capabilities. We have built systems like this for years. But making them reliable in production requires more than a capable model. Infrastructure, reliability, and API shape matter just as much.

There is also a straightforward build-versus-buy argument. Being able to train a classifier does not mean every team wants to. We do not expect every company to operate bare-metal simply because servers have existed for decades. Cloud platforms exist for a reason.

Still, a convenient API needs a capable model behind it.

Most Jev comparisons so far have focused on general-purpose LLMs. They demonstrate its advantages in speed and cost, but leave a second question unanswered: how does Jev compare with traditional classifiers fine-tuned for specific tasks?

The Jev team has deliberately avoided benchmaxxing, and they have not published any comparisons against established zero-shot or supervised classifiers.

So I built a small benchmark suite to answer two questions:

1. How much does Jev improve on existing zero-shot approaches?
2. How close can it get to models trained specifically for a task?

Benchmark methodology

The benchmark covers five established datasets and all three Jev output types: Choice, Noul, and Score.

For each task, Jev 1.13.0 receives an instruction and the candidate labels or scoring rubric. It does not receive any few-shot examples.

The supervised baselines are task-specific models: SPACE-2 for the three intent datasets, RoBERTa-large for SST-2, and a RoBERTa-large cross-encoder for STS-B. The zero-shot reference is DeBERTa-v3-large-zeroshot-v2.0, an established NLI-based classifier. There is no zero-shot baseline for STS-B in this comparison.

I reproduced every result locally on the same evaluation examples rather than copying reported figures from papers. The dataset versions, prompts, checkpoints, and results are recorded in the repository. In total, the benchmark contains 10,907 examples across five datasets.

The evaluation metrics are straightforward:

• Choice: Accuracy, since each example has one correct label and the classes are balanced.
• Noul: I ask whether a sentence expresses positive sentiment, threshold the returned probability at 0.5, and measure accuracy.
• Score: STS-B assigns human similarity ratings from 0 to 5. I map Jev onto the same six-level scale and measure Spearman correlation with the human ratings. This tests whether Jev ranks sentence pairs similarly to people, not whether its absolute scores are calibrated.

Jev materially improves zero-shot classification

The most striking result is Jev’s improvement over zero-shot NLI on intent classification. It leads by 12.76 percentage points on BANKING77, 24.38 on CLINC150, and 26.77 on HWU64.

In practice, the NLI baseline often identified the general topic but missed the distinction between related options. For example, it classified “How can I prevent my credit score from dropping?” as credit_score, while Jev selected the more specific improve_credit_score.

The example-level comparison makes the difference even clearer. Across the four classification datasets, Jev was correct when the NLI baseline was wrong on 2,058 examples. The reverse happened on only 267 examples.

I tested each difference with 100,000 paired-bootstrap resamples: the confidence intervals exclude zero for all three intent datasets, which suggests that Jev’s improvement is unlikely to be due to chance.

Jev does not fully close the supervised accuracy gap

When compared to the task-specific models, Jev does not perform as well. The gap is relatively small on SST-2 and STS-B, but widens to 5.84 percentage points on CLINC150, 11.15 on HWU64, and 14.87 on BANKING77.

Whether those gaps are acceptable depends on the cost of being wrong. Jev’s 91.96% accuracy on CLINC150 looks reasonably close to the supervised model’s 97.80%. In error-rate terms, however, the comparison is 8.04% vs. 2.20% (3.7x more mistakes). This difference becomes important when a prediction triggers an automated action.

However, the accuracy gap is not surprising. The supervised models have learned from labelled examples, including information that may be impossible to infer from the label names alone.

For example, when asked “What do I do with my card PIN?”, Jev made the very reasonable choice of change_pin. However, the correct label was get_physical_card, which sounds as though it should cover ordering a card instead. In fact, all 40 test examples assigned to it are questions about finding a card PIN. Jev missed this dataset-specific nuance and scored 0/40 on that class. Conversely, SPACE-2 got 39/40 right because it had learned what that label means within this particular dataset.

Interestingly, Jev performed closest to the supervised model on tasks with more labels. It performed best on CLINC150 which had 150 labels, as compared to 77 for BANKING77 and 64 for HWU64. This suggests that label cardinality alone does not determine task difficulty. Distinguishing between closely related labels may be harder than choosing among a larger set of more distinct requests.

Scaling to many labels without external fan-out

A standard NLI model is not inherently a multiclass zero-shot classifier. To use it as one, we must build a layer around the model, by turning each candidate label into a hypothesis, pairing every hypothesis with the input, running inference over all the resulting pairs, and combining their scores into a final decision.

For CLINC150, each input fans out into 150 {premise, hypothesis} pairs. Across 4,500 examples, the model evaluates 675,000 pairs. Batching makes this slightly more efficient, but we still need to construct and manage this structure outside the model.

Jev provides the multiclass abstraction directly. The caller supplies the shared state, question, and complete candidate set in one request, and the model returns a distribution over the options.

Its internal architecture has not been published, so we cannot say exactly how that request is processed. However, Archer’s black-box experiments suggest that Jev might be jointly encoding the state, question, and candidate answers into a shared hidden-state representation, then making its output decision from there. If so, computation over the shared state could be reused across candidates, rather than independently pairing and processing the state with every option.

His latency experiments offer further support. Adding questions scaled differently from repeatedly extending the shared state, which is consistent with shared-state computation and batched branch processing.

The benchmark cost less than fifty cents

Across the five completed runs, Jev processed 11,286,248 input tokens and returned 9,068,060 output tokens. At TypeSafe’s published price of $0.042 per million input tokens (output tokens are free), the total calculated API cost was $0.47.

This does not prove Jev is cheaper than every self-hosted alternative. Such a comparison would depend on hardware, utilisation, engineering effort, and serving costs.

But it does show that API spend was a negligible barrier to running the benchmark.

The real value is faster iteration

For me, Jev’s practical appeal is how quickly it can turn a proposed decision into something testable.

Imagine designing a new support-routing flow. Before you have a stable taxonomy or enough labelled data to train a model, you can describe the possible destinations, run real requests through them, and inspect where the distinctions break down.

You may discover that two queues overlap, a label is poorly defined, or the underlying business rules are unclear.

That work remains valuable even if you eventually replace Jev with a trained model. It helps you decide what to train, which data to collect, and what level of accuracy the application actually requires.

These results make Jev an attractive starting point for that process. On the intent datasets, it delivers a meaningful improvement over the zero-shot NLI baseline without requiring task-specific training. Where the remaining accuracy gap matters, the supervised results show why investing in specialisation can still be worthwhile.

How far can zero-shot take you?

There are limits to what this benchmark can establish.

These are public datasets, so prior exposure during training cannot be ruled out. SST-2 uses its validation split, which was also used while developing the supervised checkpoint. CLINC150 excludes out-of-scope requests, so the results tell us nothing about rejecting an intent that the model was not offered. And testing three interfaces across five datasets is still far from evaluating every workflow those interfaces could support.

Even with those limitations, there is plenty to like.

Established machine-learning capabilities become much more useful when they are accessible through a well-designed API, inexpensive to test, and accurate enough to give a new workflow a credible first implementation.

Jev makes that combination extremely compelling.

Try it yourself

Code, pinned checkpoints, results: github.com/thisisandreeeee/jev-benchmarks.

How close does Jev get to task-specific models? was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

But SFT also exposed a harder problem. Improving one behaviour can distort others. In our case, it sometimes made the model’s output distribution too sharp, leading to repetitive loops especially under greedy decoding strategies.

This article is a more technical follow-up to the original post. We will look at the failure modes encountered after SFT, why common inference-time fixes only treat the symptoms, and how Group Relative Policy Optimisation (GRPO) was used to optimise the model against explicit rewards for correctness, simplicity, clarity, and ASD-STE100 compliance.

SFT worked… Until it didn’t

Supervised fine-tuning produced a meaningful shift in behaviour. Semantic simplicity improved from 65.82% to 85.46%, while technical adequacy slightly improved from 68.30% to 70.15%. The model also preserved most of its task-fulfilment and clarity scores. In other words, SFT largely achieved what we wanted: simpler responses without an obvious collapse in correctness.

The problem appeared during generation.

Under some prompts, especially with greedy decoding, the SFT model could fall into repetitive loops. For example:

…from a single exit point from a single next hop from a single next hop from a single next hop…

Once the model entered one of these loops, it often had difficulty escaping.

This was not enough to invalidate the SFT result. Most generations were fine, and the aggregate evaluation metrics were strong. But it suggested that the behavioural shift we had learned came with an undesirable side effect.

What went wrong?

SFT trains the model to increase the probability of the target tokens in the training examples.

At a high level, it minimises the negative log-likelihood:

If the training data consistently contains shorter, simpler, more direct answers, the model learns to assign more probability to those kinds of continuations.

But the objective does not directly care about the shape of the full output distribution.

In our case, the looping behaviour was consistent with the fine-tuned model becoming too confident in a small set of continuations under certain contexts.

This becomes especially problematic with greedy decoding:

Greedy decoding always picks the single highest-probability next token. That works well when the distribution is well behaved, but it can amplify local pathologies.

If one continuation produces a context that again strongly favours a similar continuation, the process can become self-reinforcing: context → high-probability continuation → similar context → same continuation → ⋯

The model had learned the right general behaviour, but in some local regions of the generation space, the learned update appeared to be too aggressive.

A practical workaround

Rather than retrain immediately, I first tried to make generation more robust at inference time.

First, I moved away from purely greedy decoding and introduced stochastic decoding.

Instead of always choosing the highest-probability token, sampling allows lower-ranked but still plausible tokens to be selected. This gives the model a way to escape deterministic cycles rather than repeatedly following the same local maximum.

Second, I added repetition and presence penalties.

These reduce the attractiveness of tokens or phrases that have already appeared in the generated text. They directly discourage the model from repeatedly returning to the same continuation.

Third, I reduced the LoRA adapter scale.

A LoRA-adapted layer can be written approximately as:

Reducing s weakens the contribution of the adapter while keeping the underlying base model unchanged.

This was particularly interesting because it suggested something about what the SFT adapter had learned. The direction of the update appeared useful: the adapter clearly pushed the model toward simpler language. But the full-strength update seemed too strong in some contexts.

Scaling the adapter down preserved much of the desired behaviour while reducing the tendency to fall into repetitive loops.

But this was still unsatisfying

These inference time fixes worked, but it felt like an operational hack.

Stochastic decoding, repetition penalties, and adapter scaling were effectively acting as guardrails around the fine-tuned model. Ideally, the model should learn the desired behavioural shift without needing those guardrails.

We could probably have squeezed out more performance from SFT by experimenting with a lower learning rate or earlier stopping. More diverse training data could also reduce the chances of over-specialising on a narrow set of response patterns.

But I decided to try a slightly different approach.

SFT only tells the model to imitate the examples it is given. What if we could instead tell the model directly which outputs were better?

Enter, GRPO

Group Relative Policy Optimisation (GRPO) is a reinforcement-learning method for post-training language models.

The key difference from SFT is the learning signal. SFT learns by imitation: given a prompt and a target response, it increases the probability of the target tokens. GRPO does not require a single “correct” target response. Instead, it generates several candidate responses for the same prompt, scores them with a reward function, and updates the model to make higher-reward responses more likely relative to the rest of the group.

This makes GRPO useful when the desired behaviour is easier to evaluate than to demonstrate perfectly in a dataset.

A reward function with the wrong incentives

The first reward function that we tried treated correctness as a multiplier over the style objectives:

where C is correctness, S is simplicity, A is ASD-STE100 compliance, and the weights control the relative importance of the style objectives.

The intuition was that incorrect answers should receive little or no reward, while correct answers should be rewarded for writing simply.

In practice, however, correctness was something the model could improve just by adding more detail. Longer answers could include extra explanations and caveats that made them appear more technically complete, even when that detail was unnecessary. The result was a model that became more verbose instead of simpler.

Compared with the SFT baseline, semantic simplicity fell from 85.46% to 77.25%, while average sentence length increased slightly from 15.32 to 15.85 words. Technical adequacy was essentially unchanged at 70.15% vs. 70.50%, and task fulfilment also stayed roughly flat. The multiplicative reward added optimisation pressure without improving correctness, while making the model noticeably less simple.

Treating correctness as a constraint

The second reward formulation treated correctness as a constraint, rather than an objective:

The idea was simple: first be correct enough, then optimise for simplicity.

Once a response passed the correctness threshold, adding more technically correct detail no longer increased its reward. The model instead had to compete on simplicity, ASD-STE100 compliance, and the other style objectives.

The TensorBoard logs suggested that this produced a much healthier optimisation signal: overall reward improved over training, while correctness, simplicity, ASD-STE100 compliance, and clarity generally improved alongside it.

The best model so far

Compared with the previous SFT setup, semantic simplicity improved from 85.46% to 95.83%, while technical adequacy improved from 70.15% to 72.42%. Average sentence length fell from 15.32 to 11.80 words, and mean output length dropped from 622 to 331 tokens.

The progression is clear. SFT produced the largest initial improvement in simplicity, while GRPO pushed it further and also achieved the highest technical adequacy. The final model wrote much shorter sentences and almost eliminated long sentences. This came with a modest trade-off in task fulfilment and clarity, suggesting that the reward function still has room for refinement.

Importantly, the looping problem did not return, even with the GRPO adapter at full strength. We kept the same repetition and presence penalties for consistency, so GRPO alone should not get all the credit.

Try it yourself

If you wish to reproduce the results or explore the artifacts:

• GitHub — code and evaluation harness
• HF Dataset — GRPO training and validation data
• HF Model — trained GRPO LoRA adapter and model details

Making LLMs Simpler: From SFT to GRPO was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Eleven weeks ago, I set up my own personal AI agent.

I did not have a grand plan for it.

Sure, it would be nice if it made me more productive. Maybe it could help me learn faster, stay organised, or automate a few annoying tasks.

But I did not really know what I wanted it to do.

So I installed Hermes on a small DigitalOcean VPS, connected it to WhatsApp and Telegram, and gave it access to the things I used regularly: files, code, GitHub, Google Drive, Gmail, cron jobs, persistent memory, and reusable skills.

Because Hermes is designed to extend itself, its capabilities could grow with how we used it.

Naturally, I asked my wife to try it too.

Her response was very reasonable:

“What am I supposed to do with this?”

I did not have a good answer.

At first, it looked like a learning tool

Fast forward eleven weeks, we had sent more than 1,000 messages to Hermes.

My first uses of Hermes were fairly predictable.

Like many others, I use AI to help answer questions about difficult concepts I am learning. One day, I found myself reading more about the attention mechanisms in LLMs.

After we had discussed the topic, I asked Hermes to turn our conversation into a set of koans: small exercises designed to make me reason through the ideas myself.

Hermes:

• identified the key concepts;
• designed the koans;
• wrote and tested the exercises;
• set up the repository and environment.

All I needed to do was focus on solving the koans. If you’re feeling adventurous, you can try them out here.

What I appreciated was how easy this was. I was already talking to Hermes about attention. Turning that conversation into a learning system took just one additional message.

Later, I asked Hermes to send me revision questions every day.

Hermes figured out how to schedule the task using cron, remembered what I had been learning, and started sending me Socratic-style questions regularly.

I did not need to create a new app, configure a learning platform, or explain what I wanted to revise. Hermes already had the context.

Then I started using it as a knowledge system

For a while, I had wanted to build my own personal AI wiki, as inspired by Karpathy’s llm-wiki.

But I did not want to spend my weekends manually adding random things to a wiki just for the sake of having one. So I parked the idea.

Then, during one particularly busy period at work, I noticed that my inbox had accumulated a large backlog of newsletters.

Typically, I’d try to read them every week but this time I had fallen behind. And the longer the backlog grew, the less likely I was to start.

So I thought, what if the newsletters became the input to the wiki?

Hermes helped turn that fuzzy idea into an automated workflow.

Each week, it would process the newsletters, update the wiki, and send me a pull request to review. It would also send me a digest highlighting the most useful ideas and how they connected to concepts I had already encountered.

Because the wiki was just Markdown, I could also open it in Obsidian and explore the connections later. This made staying up to date considerably easier.

Then my wife started finding uses for it

A few months ago, my wife and I got a puppy.

And because dogs are pack animals, our puppy hated being alone.

So we started doing separation training: progressively increasing the duration the dog spends alone without pushing beyond what she can tolerate.

The annoying part was keeping track of everything.

How long was the previous session?
Which set did she struggle with?
Should today’s session be harder or easier?
Were we progressing too quickly?

My wife thought, why don’t we just ask Hermes to manage it?

We told Hermes to consume literature on separation training and had it design a program. After each session, we scored our puppy’s performance, and Hermes used that feedback to adjust the next one.

This was probably the first real aha moment for my wife.

“Let me know when this movie comes out”

Getting a puppy also meant we spent much more time at home.

There were plenty of movies we wanted to watch but did not want to see badly enough to go to the cinema. Hence, we would usually wait for the movie to come out on streaming.

The problem was that we would often forget. A few months later, the movie would disappear into the endless pile of things we vaguely intended to watch.

Once again, my wife suggested using Hermes.

Now, whenever there is a movie we are interested in, we ask Hermes to track it. When it becomes available on a streaming service we can access, Hermes lets us know.

That is how Hermes reminded us about Project Hail Mary.

It turned out to be one of our favourite movies of the year.

And there is a decent chance we would have forgotten about it otherwise.

Next, we gave it our household accounting

My wife and I track our household budget with YNAB.

I love the philosophy behind it: give every dollar a job and make spending decisions deliberately.

What I do not love is reconciling transactions.

YNAB’s automatic bank integrations are much better supported in the US than in Singapore, so a lot of our transaction management was manual.

Every month, we would sit down on a Sunday and work through everything over 3 hours.

That was pre-Hermes.

Now, we export our transactions as CSV files and send them over. Hermes reviews everything, resolves the straightforward cases, flags anything ambiguous, and inserts the prepared transactions into YNAB for us to review.

The process that used to take us 3 hours now takes around 30 minutes. And much of those 30 minutes is simply waiting for Hermes to work.

The interesting part wasn’t productivity

Did Hermes make our lives more productive?

Maybe.

Some workflows clearly save time.

Others took more time to build than they will ever save.

But after eleven weeks, I do not think productivity is the most meaningful way to evaluate the experiment.

The thing that changed was how use cases appeared.

I did not sit down on day one and decide to:

1. Build a puppy separation-training system
2. Track movie streaming releases
3. Reconcile household finances

Those ideas emerged because Hermes was already present when the problems appeared.

That led me to three lessons.

1. You discover agent use cases by using them

There is a temptation with new technology to first ask:

What is the killer use case?

I am increasingly unsure that this is the right question for personal agents.

Many of our useful workflows were things I would never have put on a requirements document.

They surfaced naturally.

A problem appeared.

We realised Hermes already had most of what it needed.

We asked it to help.

Then the workflow came to life.

2. Proximity is a key enabler

I think one reason this worked is extremely mundane:

Hermes lives in WhatsApp and Telegram.

Those are already places where my wife and I talk.

Movie ideas appear there.

Dog-training discussions happen there.

Random household problems appear there.

There is no need to remember to open an AI app and ask: “What processes in my life should I automate today?”

The agent is simply nearby when the problem occurs.

The best interface for a personal agent might be wherever your life is already happening.

3. Tiny use cases matter

Many of the use cases we found meaningful were actually quite small.

Tracking a movie release is a good example.

Nobody is going to buy an “AI movie reminder system.”

But I think that is exactly why it mattered to us.

It was a tiny annoyance that would normally never be worth solving properly. Because Hermes was already there, we could just ask it to handle it.

The same was true for puppy training, revision questions, and parts of our YNAB workflow.

None of these felt transformative on their own.

What I found interesting was how easy it became to keep adding small things that were specific to how we lived.

Over time, those small conveniences started to accumulate.

I am not sure any single one of them changed our lives very much.

But together, they did make parts of everyday life noticeably easier.

So what is Hermes actually good for?

Eleven weeks ago, my wife asked:

“What am I supposed to do with this?”

I still do not have a particularly good answer.

The difference is that I no longer think that is a problem.

A personal AI agent might be difficult to understand before you use it precisely because its most valuable uses are contextual.

They depend on your routines.

Your frustrations.

Your hobbies.

Your family.

The random problems that happen to appear on a Tuesday evening.

The useful thing about Hermes is not any single workflow.

It is that it sits close to our real lives, carries context forward, accumulates capabilities, and is ready when those workflows emerge.

So if you ask me today what my personal agent is for, my answer is still:

I don’t know.

But now it is followed by:

Let’s see.

What I Learned from Running Hermes for 11 Weeks was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ask an LLM a technical question and it will try to be precise. The answer is usually correct, but often more complicated than it needs to be.

For example:

Why does my model perform worse on new data?

A typical answer might look like this:

This behaviour is likely caused by overfitting, where the model learns patterns specific to the training dataset that do not generalise well to unseen examples. This can occur when model capacity is high relative to dataset size, training continues for too long, or regularisation is insufficient.

A simpler answer says nearly the same thing:

Your model is probably overfitting. It learned the training data too closely, so it does not work as well on new data.

Simpler does not mean less useful

The second answer gives up some detail, but it is much easier to understand. That led to the central question behind this project:

Can we post-train a small language model to produce answers that are both correct and simple?

A frontier LLM with a well written system prompt could probably do this already.

But where is the fun in that?

Simplicity is hard to measure

We start by defining what success looks like.

Shorter does not always mean simpler. Response length is a useful signal, but optimising for it too aggressively can produce fragmented answers that are harder to understand.

Simplicity also depends on less obvious details: sentence structure, ambiguous words, excessive contractions, semicolons, and abbreviations such as i.e. and e.g.

So, what does “simple” actually mean?

Fortunately, some very smart people have already thought deeply about this and came up with ASD-STE100 Simplified Technical English, a controlled form of English originally developed for aircraft maintenance documentation. The goal is to make technical instructions clear, precise, and difficult to misinterpret, especially for non-native English speakers.

We use these rules as the basis for our deterministic checks.

Building an evaluation harness

Defining simplicity is only half the problem. We also need a way to measure it without rewarding answers that are short but wrong.

An answer can follow every style rule and still be incomplete, misleading, or unhelpful. So we complement our deterministic checks with an LLM judge across four dimensions:

• Technical adequacy: Is the answer correct, complete, safe, and appropriately qualified?
• Task fulfilment: Does it fully address what the user asked?
• Clarity and coherence: Is it unambiguous, logically structured, and internally consistent?
• Semantic simplicity: Is it direct and proportionate, without unnecessary repetition or complexity?

For example, here is the rubric used to judge semantic simplicity, including the criteria and evaluation steps given to the LLM judge:

"semantic_simplicity": {
    "criteria": (
        "Judge whether the answer expresses ideas directly and simply "
        "without removing necessary meaning or technical precision."
    ),
    "steps": (
        "Evaluate simplicity at the level of the whole answer, "
        "not only individual sentences.",
        "Check whether the amount of detail and structure is "
        "proportionate to the user's request.",
        "Penalize repetition, fragmented or choppy presentation, "
        "unnecessary headings, excessive itemization, and redundant summaries.",
        "Do not award the highest score merely because sentences are short.",
    ),
}

The final evaluation combines:

• Deterministic checks → How is the answer written?
• LLM-as-a-judge → Is the answer actually good?

Together, they let us measure simplicity without accidentally rewarding answers that are merely short. We use this evaluation harness to measure each experiment against a holdout set of 100 curated prompts spanning multiple domains.

Building the SFT dataset

The training dataset was generated using a two-phase approach:

1. Generate diverse prompts — Build a stratified mix of prompts across 20 domains, multiple task intents, target audiences, and expected response lengths, then filter out malformed or duplicate examples.
2. Generate simple and correct answers — Use a teacher model guided by ASD-STE100-inspired rules and correctness checks to simplify each answer without dropping important caveats, qualifications, or technical meaning.

This process produced 1,000 training examples.

Dataset generation used a mix of DeepSeek-V4-Pro at xhigh reasoning and GPT-5.6 Sol at medium reasoning. The prompts for both phases were refined iteratively through manual review of the teacher outputs.

Generating answers that were both simple and correct required explicit correctness instructions, with correctness taking priority over style. This ensured the teacher preserved important caveats, failure conditions, and technical terms before simplifying language and structure.

The Hugging Face dataset card can be viewed here.

The training recipe

We use Qwen3.5-4B as our starting point, as it is small enough to fine-tune cheaply, yet capable enough to be interesting.

Each training example contains a user prompt and an assistant answer. The prompt is provided as context, but loss is computed only on the assistant tokens, so the model learns to produce the target answer rather than reproduce the prompt. We use Unsloth for parameter efficient fine-tuning, with TRL configured to compute completion-only loss.

Fine-tuning was done using LoRA, freezing the base model and training only a small adapter with rank 16 and alpha 16. The model was kept at bf16 precision rather than quantising with QLoRA since a 4B model fits comfortably on an NVIDIA L4.

Training and inference both run on Modal, which gives us serverless access to L4 GPUs without needing to manage infrastructure.

Training runs for two epochs on 900 examples, 100 for validation, and a separate 100 holdout set. The remaining settings are deliberately conservative: an effective batch size of 8, a learning rate of 1e-4, 8-bit AdamW, a linear learning-rate schedule, and 0.01 weight decay.

The result is a LoRA adapter, which we merge with the original base model for inference.

The Hugging Face model card can be viewed here.

The experiment

We compare three approaches of increasing complexity:

1. Base model — Qwen3.5-4B with no enhancements.
2. Prompt engineering — Before changing the model, we test how far we can get with simple prompt engineering. If a good system prompt solves the problem, there is little value in post-training.
3. LoRA fine-tuning — Using the SFT adapter trained on our curated dataset, scaled by 0.25 and merged into the base model.

These results exposed an important trade-off: prompting made the model simpler, but often at the expense of being right.

Prompting alone was too aggressive

Prompt engineering had the strongest effect on simplicity:

• Average sentence length fell by 38%
• Long sentences fell by 83%
• Sentence mechanics improved by 13%
• Verb-form and modal compliance improved by 13%
• Semantic simplicity improved by 14%

But the prompt-tuned model also scored worse on:

• Technical adequacy: -13%
• Task fulfilment: -10%
• Clarity and coherence: -7%

This demonstrates that responses can be concise and readable while being wrong. Simplicity is useful only when the answer remains correct.

SFT found a better balance

Relative to the baseline, SFT improved semantic simplicity by almost 30%, while technical adequacy increased slightly.

It also produced:

• 15% shorter sentences
• 40% fewer long sentences
• Better terminology consistency
• Almost unchanged task fulfilment and clarity

That is arguably the more interesting result: the model learned to simplify the answer without merely optimising for shorter sentences or stricter writing rules.

Prompting made the model write less.

SFT made it say more with less.

It also had an operational benefit: mean output length fell by about half, while generation time fell by 37%. Simpler answers were not only easier to read; they were cheaper to produce.

But the model almost forgot how to stop

The first training runs exposed an unexpected failure mode, where a phrase could begin repeating until the model hit its token limit.

…from a single exit point from a single next hop from a single next hop from a single next hop…

This is known as degenerate looping. The model gets trapped in a self-reinforcing sequence and effectively forgets how to stop.

Fine-tuning changes the model’s token probabilities. In some contexts, the LoRA adapter can make a narrow set of continuations much more likely than others. With greedy decoding, the model always selects the highest-probability token, so once it enters a repetitive pattern, there is no randomness to help it escape.

Greedy decoding was initially used to make the experiments deterministic, but it ended up amplifying a failure mode that appeared after fine-tuning.

Fixing degenerate loops at inference time

I tested three decoding changes:

1. Stochastic decoding —Sample from the token distribution instead of always taking the highest-probability token. This produced the largest improvement by giving the model a way to escape repetitive trajectories.
2. Presence penalty — Apply a fixed penalty on tokens that have already appeared. This reduced repetition but degraded answers where repeating the same term is necessary.
3. Repetition penalty — Reduce the probability of tokens that have already appeared using a multiplicative penalty.

The best result came from combining stochastic decoding, a modest repetition penalty, and a lower presence penalty. This eliminated the looping behaviour (in our holdout set) while preserving simplicity and correctness.

Try it yourself

If you wish to reproduce the results or explore the artifacts:

• GitHub — code and evaluation harness
• HF Dataset — SFT training and validation data
• HF Model — trained LoRA adapter and model details

Get LLMs to say more with less was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

My excitement was short-lived, however, when I realized that support for taking TF 2.0 models to production was not what I imagined it to be. Sure, TensorFlow Extended was an option, but it wasn’t a particularly pleasant one considering that the documentation had yet to catch up.

Still, I was impatient; and, eager to execute. What ensued was an arduous journey filled with tears and cheers, all in search of the one true prize that is a working machine learning pipeline. I present to you my r̶a̶n̶t̶s̶ findings in the hope that you will not have to go through what I did. Well, until the next release anyway.

What this is (not)

This article is in no way an authoritative piece on the way to build production-ready TF 2.0 pipelines. It illustrates one possible workflow for deploying ML models that accounts for memory constraints and training-serving skew (amongst other things). If these concepts sound foreign, I recommend reading Google’s Rules of Machine Learning. Prior experience with neural networks and the machine learning lifecycle would certainly be helpful.

You will learn how to:

• Use TF Transform to perform feature imputation and scaling
• Build models using the Keras functional API and feature columns
• Export a model for TF Serving that reuses the Transform graph

To keep things short, I will only be showing code snippets.

1. Install dependencies

You will need the following Python packages. Since we will be using Apache Beam to run our TF Transform pipelines, let’s install that too.

pip install apache-beam==2.16.0 tensorflow==2.0.0 tensorflow-transform==0.15.0

2. Preprocess data using TF Transform

This guide assumes familiarity with TF Transform, which involves writing transformations in a preprocessing_fn to be executed by a Beam pipeline. Further information on how to get started can be found here.

Let’s write a simple transformation for a numerical variable age that imputes missing values with the mean and applies feature scaling.

import tensorflow as tf
import tensorflow_transform as tft

def preprocessing_fn(inputs):
    outputs = inputs.copy()
    
    age = outputs["age"]
    mean_age = tft_mean(age)
    age = impute(age, -1, mean_age)
    outputs["age"] = tft.scale_to_z_score(age)
    
    return outputs

Hold up! Doesn’t TF Transform support calculation of means using thetft.mean() Analyzer? Why do we need to write our own implementation of tft_mean? This is because TF Transform has a known issue where NaNs may produce unexpected results for some Analyzers. To address this, let’s only calculate the mean using non-null values.

def tft_mean(tensor):
    finite_indices = tf.math.is_finite(tensor.values)
    finite_values = tf.boolean_mask(tensor.values, finite_indices)
    return tft.mean(finite_values)

Now, let’s impute missing values using the mean age. There are two ways of representing missing values: first, by exclusion through the use of a SparseTensor; second, by using some arbitrary value (e.g. -1 for numerical variables or NULL for categorical variables). Let’s account for both scenarios.

def impute(tensor, missing, replacement):
    sparse = tf.sparse.SparseTensor(
        tensor.indices, tensor.values, [tf.shape(tensor)[0], 1]
    )
    dense = tf.sparse.to_dense(sp_input=sparse, default_value=replacement)
    return tf.where(tf.equal(tensor, missing), replacement, dense)

You can then run your TF Transform pipeline, don’t forget to export the Transform graph so that we can reuse it later!

import apache_beam as beam 
import tensorflow_transform.beam as tft_beam
import tempfile

with beam.Pipeline() as pipeline:
    with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
        # read raw data
        raw_data = pipeline >> beam.io.ReadFromTFRecord(...)
        
        # apply transformation
        transformed_data, transform_fn = (
            (raw_data, raw_metadata) >> tft_beam.AnalyzeAndTransformDataset(preprocessing_fn)
        )
        
        # export transform graph
        _ = (
            transform_fn >> tft_beam.WriteTransformFn("data/tft/")
        )

3. Build a model using Keras layers and feature columns

It’s time for some feature engineering. TensorFlow feature columns provide a concise API for common operations such as one-hot encoding. Let’s define feature columns for a numerical variable age and a categorical variablecountry.

features = [
    tf.feature_column.numeric_column("age"),
    tf.feature_column.indicator_column(
        tf.feature_column.categorical_column_with_vocabulary_list(
            "country", ["America", "Japan", "China"],
        )
    ),
]

Using feature columns with the Keras functional API can be a little tricky. You will need to instantiate a Keras tensor using an Input layer.

from tensorflow.keras.layers import Input

feature_inputs = {
    "age": Input(name="age", shape=(), dtype=tf.float32),
    "country": Input(name="country", shape=(), dtype=tf.string),
}

We can now feed this into a Keras DenseFeatures layer and proceed to define our neural network architecture.

from tensorflow.keras.layers import Dense, DenseFeatures
from tensorflow.keras import Model

output_1 = DenseFeatures(features)(feature_inputs)
output_2 = Dense(16, activation="relu")(output_1)
preds = Dense(10, activation="softmax")(output_2)
model = Model(inputs=feature_inputs, outputs=preds)

model = model.compile(...)
model.fit(...)

4. Export model and call Transform graph

Congratulations on making it this far! After training our model, we will export our model as a SavedModel to deploy it. To prevent skew between training and serving, we’ll need to load the exported Transform graph to run the same transformations on serving inputs.

tft_dir = "data/tft/"  # location of exported transform_fn

tft_output = tft.TFTransformOutput(tft_dir)
tft_output.transform_raw_features(raw_serving_inputs)

In Tensorflow 1.x, you would include the above logic in the serving_input_receiver_fn when exporting your model. However, as we move away from the Estimator API in TF 2.x, this functionality is being deprecated. Instead, we will need to modify the serving signature of the Keras model by overriding the save() method.

class ExportModel(tf.keras.Model):
    def __init__(self, model, tft_dir):
        super().__init__(self)
        self.model = model
        self.tft_output = tft.TFTransformOutput(tft_dir)
        
    @tf.function(input_signature=[SERVING_FEATURE_SPEC])
    def serving_fn(self, inputs):
        transformed = self.tft_output.transform_raw_features(inputs)
        return {"preds": self.model(transformed)}

    def save(self, output_dir):
        signatures = {"serving_default": self.serving_fn} 
        tf.saved_model.save(self, output_dir, signatures)

ExportModel(model, "data/tft/").save("data/model/1/")

We can then serve our model using TF Serving.

Hope this helps. Ciao!

End-to-end Machine Learning with TFX on TensorFlow 2.x was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Competition and cooperation are typically viewed to be mutually exclusive, particularly so within academic institutions that practice curve grading. It pays to cooperate insofar as individual performance is not affected, and many adopt competitive behaviors as examinations draw near. This doesn’t necessarily have to be the case; individuals can be competitive and cooperative at the same time, as I will demonstrate through the use of agent-based modeling and deep learning technology.

I was introduced to agent-based modeling in a class I took in the University Scholars Programme at NUS, on computational thinking. During one particular lecture, we studied the evolution of altruistic and cooperative habits using an agent-based model, that simulates the behavior of altruistic and selfish agents.

The model is first randomized with each patch in a field occupied by an agent. At each time step, an agent will try to plant “seeds” around its neighboring patches, which will then grow into an altruistic or selfish agent. The probability of each new agent being altruistic or selfish depends on the cost and benefits of being altruistic, which is a factor modeled between 0 and 1. Under these conditions, I ran two simulations to visualize the spread of altruism and selfishness in this imaginary world.

Simulations are a great tool for understanding relationships within complex systems. Through them, we visualize how changing parameters can affect outcomes, which can be more enriching than learning by hearing or reading instead. In this simulation, I modify the stress imposed on agents and observed the following.

Contrasting these two simulations helped me to arrive at a rather puzzling conclusion. The selfish agents were more competitive, and this competitive nature did not always translate to good results especially under stressful conditions. So, what does healthy competition actually mean? When we eventually step out into society as working professionals, is it more important to be cooperative, or competitive?

I desired an alternative perspective on competition, which came in the form of a paper by artificial intelligence research scientist Ian Goodfellow. Goodfellow proposes a novel approach to designing generative models using mathematical abstractions of the brain. The mathematics behind these Generative Adversarial Networks (GANs) isn’t so important here, but know that it’s a straightforward approach involving an adversarial process.

Where previous advancements in deep learning revolved around discriminative models, GANs were generative in nature. On the one hand, discriminative tasks such as image recognition involve machines trying to differentiate between images of objects, and technological advancements have made such tasks relatively trivial. On the other hand, generative tasks are significantly harder because they require a machine to be creative, and to produce unique images of new objects.

GANs generate new images by learning and generalizing features from existing images, before piecing them together much like an artist would. I chanced upon a project by members of the open source community where they tried to create new Pokemon designs using GANs, and the results left me pretty amazed.

While the generated Pokemon are admittedly far from complete, the GANs ability to learn and generalize the spiral patterns on Poliwhirl demonstrates marked improvement over its predecessors. The relationship between the Generator and Discriminator is analogous to other iconic rivalries, such as that between soccer athletes Cristiano Ronaldo and Lionel Messi. Both are widely regarded as two of the all-time best players, and constantly try to best each other to establish themselves as the undisputed number one. Through that process, they are motivated to train harder to become the best versions of themselves.

“We push each other sometimes, this is why the competition is so high.” — Cristiano Ronaldo

In a similar fashion, the Generator and Discriminator in a GAN work in opposition to one another and try to outsmart each other, improving very rapidly in the process. Over time, Generators create more realistic images that are harder for Discriminators to differentiate, who are then forced to get better to catch the Generators.

As it turns out, competition and cooperation might not be too different after all. GANs offer an alternate view on relationships in the workplace that does not binarize competition and cooperation as separate concepts, but instead proposes that competition and cooperation are essentially the same. Generators and Discriminators alone are not powerful as compared to its predecessor neural networks, but together they cooperate and contribute to the overall success of the system through recursive competition.

While the real world is arguably more complex than can be modeled by an adversarial process, the understanding that competition and cooperation can coexist in a single relationship is a powerful fact — one that a prospective member of the workforce, or an aspiring manager, could adopt to better maximize the value of their teams.

Andre is an aspiring data scientist. Expected to graduate in May 2018, he is currently on the lookout for opportunities to break into the field. Check out his snazzy personal website, or reach him via LinkedIn! You can also view his resume here.

Join 30,000+ people who read the weekly 🤖Machine Learnings🤖 newsletter to understand how AI will impact the way we work and live.

I first set foot in Silicon Valley during the summer of 2016. Having taken part in a couple of hackathons back in Singapore, I was pretty excited to explore the bay area hacker scene. I had heard stories of triumph through my fellow Singaporeans, and was on the look out for upcoming hackathons to get my hands dirty.

3 months later, I was on track to achieve what I had set out to do. I got to work with many different talented individuals, and had won 75% of the hackathons we had taken part in, with over $8,000 in winnings.

I started to notice a trend. Each hackathon that we had taken part in, fell into one of three categories:

1. The “Build whatever you want, as long as it’s cool.” hackathon
2. The “We built some new tech, please use it?” hackathon
3. The “Please solve my problem with tech!” hackathon

My first exposure to hackathons in Singapore were of the first category. It was mostly about a bunch of programmers getting together for 24 hours of code, caffeine, and pizza. I realized teams that win at these hackathons aren’t always the ones using the hottest new technology, but instead, are exceedingly creative in how they use existing technology for the craziest ideas. Check out this amazing hack by the recent grand prize winners of PennApps XIII, who were able to secretly transmit data out of your computer using basic hardware.

There are also those hackathons, where startups offer free pizza and company swag in exchange for you building applications with their technology and tweeting about it. If you’re lucky, you get laptop stickers and a nifty certificate too! While not particularly enriching, you cannot doubt their effectiveness as a growth hack.

The last category of hackathons, that I am starting to see increasingly more of, revolve around solving business problems with technical solutions. The winners aren’t necessarily the ones with the coolest hack. Rather, they win because they offer real solutions to a real problem that the organizers actually have. Hackathons attract large amounts of technical talent, and companies are starting to recognize them for the ability to generate interesting new perspectives with a working prototype.

Based on my personal experience, and through conversations I’ve had with other winners, I’ve distilled the winning process into pseudo-code.

prototype = None
while hackathon.in_progress():
  requirements = ask_judges(prototype)
  if is_fulfilled(requirements):
    collect_your_prize_money()
  else:
    prototype = build(requirements)

The hackathon process is an iterative one. You come up with an initial idea, and check with the judges if they like it. If so, your chances of taking home some moolah are pretty high, and you just need to execute well. Chances are they won’t, in which case you’d have gathered some feedback to refine your idea. You build out a better prototype, and consult with the judges 2 hours later. You repeat this process as many times as necessary, until you satisfy the judges or run out of time. The result is a prototype that the judges will most probably like, considering how it’s as much their brainchild as it is yours.

Such hackathons aren’t always fair, in the sense that the strongest or most creative developers may not win if their hack does not adequately address the business problem. But that’s the reality, because if you’re a company that is forking out money to organize a hackathon, you’d want someone to solve your problems.

Now, what does any of this have to do with actually running a startup?

I spent the last 3 years exploring the startup space, learning about why startups succeed and fail. While I may not know enough to succeed at the startup game yet, I have learnt a lot about understanding customers through countless previous failed ventures. I believe that there are similarities between winning a hackathon and building a product that consumers love.

The hackathon process begins with figuring out what the judges want. You build a quick and dirty prototype, before checking back with the judges if this is what they had in mind. Rinse and repeat.

Similarly…

The startup process begins with figuring out what your customers want. You build a quick and dirty Minimum Viable Product, before checking back with your customers if this is what they had in mind. Rinse and repeat.

There is a lot that a hackathon can teach you about running a startup. You learn to work well with people under very stressful conditions. You learn about all the amazing ways technology can transform the world. And perhaps most importantly, you learn to listen to your customers.

This is the very first post for the “Everyone Can Write” Initiative by the NUS Entrepreneurs’ Association. Andre is an aspiring data scientist, and can be reached via LinkedIn or Github.

What hackathons can teach us about running a startup was originally published in NUS Silicon Valley on Medium, where people are continuing the conversation by highlighting and responding to this story.