When Cloudflare released Clef on October 1, its launch post carried a table of ten public benchmarks, seven of them led by a Clef model. The Hacker News thread passed 600 points in a day, and one of the first replies was six words long: “Public benchmarks are easy to cheat” (Traictory, Hacker News).

Fair. So we ran the test nobody could have trained for. Construct’s agent has about 40 small decisions written for a decision model, and every one of them was written around Jev: the prompts, the questions, and the thresholds, all built against Jev over the last two weeks. That is Jev’s home turf. We pointed the same requests at Clef and Clef-flash without changing a line, and scored all three against labelled cases.

Clef is Cloudflare’s open-weight decision model, a drop-in alternative to TypeSafe’s Jev. On 117 labelled cases across five of our production decisions, Clef reached the right outcome on 92.3% of calls using thresholds set for Jev, against 88.6% for Jev and 82.1% for Clef-flash. Both Clef models returned the identical probability on every repeated call, and Jev did not.

We build Construct, an AI employee with its own computer. Every number below comes from our own runs on October 4, 2026 (UTC): 117 cases, three calls each, three models, 1,053 calls in the main run, plus about 800 more in follow-up controls. Facts about the models themselves come from Cloudflare’s and TypeSafe’s documentation, checked the same day. The full method, confidence intervals and per-case results are in the paper: Compatible APIs, Incompatible Thresholds: A Drop-In Replacement Study of Clef and Jev on Decision Specifications from a Production Agent (PDF, 13 pages).

• Accuracy, drop-in: Clef 92.3%, Jev 88.6%, Clef-flash 82.1%, all on thresholds set for Jev. The gap between Clef and Jev is not statistically significant.
• Accuracy, own thresholds: Clef 93.2%, Jev 90.9%, Clef-flash 90.6%, with each threshold chosen on held-out cases.
• Repeatability: Clef and Clef-flash gave the same probability on all 351 calls each. The probability Jev’s decisions act on moved by up to 0.10, and 2 cases changed verdict.
• Code changed to swap models: one string.
• Failed calls: 0 of 1,053.
• Cost: $23 per million decisions on Jev, $35 on Clef-flash, $93 on Clef. The main run cost about five cents.

What are Clef and Clef-flash?

Clef and Clef-flash are decision models from Cloudflare: you send a state and a set of typed questions, and they return a probability for each allowed answer instead of generating text. Clef is a 27 billion parameter model built on Qwen3.8, and Clef-flash is a 9 billion parameter model built on Qwen3.5. Both are released under Apache 2.0 with weights on Hugging Face, both run on Workers AI, and both accept the same request format as Jev (Cloudflare).

If decision models are new to you: they are the fast, cheap judgment calls around a language model. Is this the same person? Does this email need a reply? Should this tool call wait for a human? TypeSafe calls the category System One models, after Kahneman’s fast, intuitive System 1. Our first post on Jev explains the idea and how we put one inside an AI agent’s memory.

Jev, Clef and Clef-flash compared, October 2026

Two details in that table matter later. Clef reads images, which Jev does not today. And the prices are per input token, so the model that counts fewer tokens for the same request closes some of the gap.

How we benchmarked Clef against Jev

We did not write a new test for Clef. We took five decisions that already exist in Construct’s code and ran each one’s real request builder and real interpreter, the same functions production calls.

The five production decisions in the benchmark

Entity match is live in production. The other four are built and switched off, waiting on shadow data. Their thresholds were set by judgment with Jev in mind and have never been fitted to data, for Jev or anything else.

Swapping the model was the least interesting part of the work. This is the whole change:

// Before
await env.AI.run("typesafe/jev", { state, questions }, options);

// After
await env.AI.run("@cf/cloudflare/clef", { state, questions }, options);

Cloudflare says Clef is “fully Jev-API compatible”. On our requests that held: Choice, Score and Noul questions, JSON state, and the response shape all worked unchanged, and all 702 Clef calls returned valid answers.

The rest of the method, briefly:

• Cases: 117, all synthetic, weighted toward the edges of each decision. We drafted and labelled them with an AI assistant (Claude) before any of the three models saw them. No label came from Jev or Clef, and the labels have not yet been independently reviewed by a second person.
• Runs: each case three times per model, one call at a time, models interleaved so they shared the same network conditions, with the gateway cache skipped.
• Scoring: a call is correct when the production interpreter reaches the labelled outcome at the production threshold. We also report each model’s best threshold, found on the same cases, which flatters all three equally.
• Route: Cloudflare’s REST API through our AI Gateway, from a laptop. Fine for accuracy. Wrong for latency, which gets its own section.

Clef vs Jev accuracy: the results

Overall accuracy across five production decisions, 351 calls per model. Left: the thresholds configured today, set for Jev. Right: each model’s own best threshold, found in-sample.

With no retuning at all, Clef had the highest score of the three, and the highest or equal-highest on four of the five decisions, on thresholds that were set for a different model.

Accuracy by decision at the production threshold, with accuracy at each model’s own best threshold in brackets

Accuracy per decision at today’s thresholds. Jev keeps email triage. Clef takes entity matching and notification priority.

The honest size of this result: Clef leads Jev by 3.7 points on 117 cases, with a 95% interval of -1.1 to +8.8 points. That is not statistically significant, and we would not stretch it into “Clef beats Jev”. Three of the entity cases are ones our product resolves in code before any model is asked (identical names, and names that differ only in case). Jev did worst on exactly those, and without them the lead shrinks to 2.0 points (92.1% against 90.1%). What we can say is narrower and more useful. Cloudflare’s public numbers predicted Clef would be at least as good as Jev on decision tasks, and on private prompts it had never seen, written for its competitor, it was not measurably worse.

Jev deserves its due here too. It won email triage outright, it never made a wrong merge, and it was perfect on the extract gate. This is a close race between two good models.

Is Clef deterministic?

Yes, on everything we sent it. We called each model three times with the same request for all 117 cases. Clef returned the same probability, to four decimal places, on all 351 calls. So did Clef-flash. To rule out a cache, we sent the same requests again with changes that do not touch the model’s input (a unique query string, reformatted JSON, reordered keys): same answers, a cache miss on every response, and a new inference ID each time.

Largest change in probability between three identical calls. Clef and Clef-flash sit at zero on every decision.

The probability each Jev decision acts on moved between identical calls by up to 0.10, and two cases changed verdict from one call to the next. Across every probability Jev returned, the largest move was 0.16, and 100 of the 117 cases came back with at least one number different. TypeSafe describes Jev as returning “similar answers for similar inputs”, which is a promise of consistency, not determinism, and a wobble of this size is consistent. But a decision layer acts when a probability crosses a line, so any wobble means a case near the line is decided by which call you happened to make.

Cloudflare’s post explains why Clef behaves differently: the decision step is non-autoregressive, with choices derived directly from the model’s internal representations and no text sampled along the way. We can confirm the practical result. For anyone who has debugged an agent that did something different the second time, this is the finding that matters most, more than a few points of accuracy.

Jev’s scores moved under the same version name

This one surprised us. On September 20 we calibrated our live entity matcher on Jev. Real matches scored 0.71 to 0.82, everything else scored 0.34 or less, and we put the merge threshold at 0.70, in the gap.

Two weeks later we asked the same three calibration probes again.

Jev’s probability for the correct match on our three calibration probes: one call on September 20, and the mean of 20 calls on October 4 through Workers AI and on TypeSafe’s own API with the version pinned. All were served as jev-1.13.0.

All three now fall under the bar. To be sure this was not a noisy afternoon, we asked each probe 20 more times: none of the 60 calls reached 0.70, and the highest was 0.66. Clef, asked the same 20 times, returned the identical number every time. The served version was `jev-1.13.0` both times, so the alarm we built to catch version changes stayed quiet. On this benchmark Jev still never merged two different people, which is the expensive mistake, but it made only 44% of the merges it should have. In production, a missed merge becomes a duplicate.

We do not know why the scores moved. Our first suspect was our own route: we call Jev through Cloudflare’s Workers AI, which cannot pin a version. So we repeated everything against TypeSafe’s own API with the version pinned to `jev-1.13.0`. Same result: none of 60 calls reached 0.70, and Jev’s answers still varied between identical calls on 104 of 117 cases. The route is not the cause. Three probes are still three probes, and our September baseline is one call each. We are not claiming Jev got worse in general. The lesson is about our own setup: a threshold is only as stable as the model under it, and a version string is not a guarantee. Open weights can change that, if you use them: a model you pin or host yourself cannot move under its thresholds. The hosted Clef alias we tested carries no version either, so we will re-ask these same probes on Clef in two weeks.

Entity matching: where the three models differ most

Entity matching is the decision where a mistake is permanent. A wrong merge fuses two real people in the agent’s memory, so precision matters more than recall. AI agent memory covers what that memory holds and how you can correct it.

Entity matching at the production threshold of 0.70: 15 pairs that should merge and 15 that should not.

• Clef caught the aliases this decision exists for: “Jose Garcia Marquez” against the accented spelling at 0.98, “Stripe, Inc.” against “Stripe” at 0.98, “I.B.M.” against “IBM” at 0.99. Its one wrong merge was “Acme” into a lone “Acme Corporation” at 0.78, a case reasonable people would argue about.
• At a 0.80 threshold, Clef made 10 of 15 correct merges and no wrong ones. That is our starting point for shadow mode, not a recommendation: it was picked on these same 30 cases, and “Acme” against “Acme Corporation” sits at 0.78, just under it.
• Jev is not worse at telling people apart. Drop its threshold to 0.50 and it makes 64% of the correct merges, still with no wrong ones, close to Clef’s 67% at 0.80. Its problem at 0.70 is that its scores moved, not that it lost the ability to rank the pairs.
• Clef-flash is quick to say yes. It matched “Acme Labs Europe” to “Acme Labs” at 0.76 and the bare concept “pricing” to “pricing model” at 0.89. For a merge, that is too eager. For a gate that only skips work, it is fine.
• All three refused a one-letter surname typo, “Mendelsohn” against “Mendelson”. Good. Those can be two people.

Thresholds do not transfer between decision models

If you take one thing from this post into your own migration, take this. A threshold is a property of a model and a prompt together. Move the model and the threshold is wrong, even when the new model is better.

The three models are not far apart in ability. Asked to rank our cases from “should act” to “should not”, all three do it well: on the four decisions we could measure this way, the ranking score (AUROC) is 0.93 or higher for every model and a perfect 1.00 on two decisions. What differs is the number each model attaches to the same judgment.

The most accurate threshold per model on two decisions, against the configured threshold

Clef-flash shows the effect best. On Jev’s thresholds it scored 82.1%. With thresholds chosen for it on held-out cases it scored 90.6%, level with Jev’s 90.9% on the same footing, and Clef stayed ahead at 93.2%. The 9B model was never bad at these decisions. It was being read with another model’s ruler.

The thresholds in the table were found on the same cases they are scored on, which is why the bracketed figures in the results table (96.6%, 94.9%, 93.7%) run higher than the held-out ones. Treat them as a direction and not as settings. The practical rule is simple: swap the model in shadow, log probabilities for a week, and set the threshold from that.

How fast is Clef? What we could and could not measure

Cloudflare reports a median of 209 ms for Clef and 38.8 ms for Clef-flash, against 524 ms for Jev. We are not going to confirm or dispute those numbers, because our harness was the wrong instrument.

We called all three models through Cloudflare’s REST API from a laptop. The fastest call we saw for any model was 340 ms, and for every model the floor sat between 340 and 384 ms. That floor is the path: authentication, the public API, the gateway, and the round trip. You cannot see a 38 ms model through a 340 ms window.

Latency per call through the REST API from a laptop, 351 calls per model. Not a production measurement.

For transparency, those are the raw numbers. On this path Clef-flash matched Jev at the median, Clef was slower, and both Clef models had longer tails, three days after launch. None of that tells you what a Worker sees when it calls the model through the AI binding, next to the GPU, which is how we call it in production. That measurement is next, and we will publish it here.

What does Clef cost compared with Jev?

Cost per million decisions at list price, using the mean input tokens each model counted for our requests.

Clef’s list price is 5.7 times Jev’s, which is the first thing the Hacker News thread did arithmetic on. Two things soften that.

First, Clef counted 28% fewer tokens than Jev for the same request: 387 per call against 541. Per decision, the gap is about 4 times, not 5.7.

Second, look at the unit. A million Clef decisions cost $93. All 1,053 calls in this benchmark cost about five cents. An agent that makes a thousand decisions a day spends about nine cents a day on Clef and two on Jev, next to a language model bill many times that. If you run billions of decisions a day, price it carefully, and look at Clef-flash at $35 per million. For everyone else, accuracy and repeatability are worth more than the difference.

Clef vs Jev: which decision model should you use?

Which decision model fits which job, based on our benchmark and each vendor’s documentation

The broader field is moving fast. Besides Jev and Clef there are open decision models such as Laya, Kev and OpenDecider, and Cloudflare’s own table includes several of them. We have only tested the three in this post.

What we are switching, and why Cloudflare makes it easy

Construct already runs on Cloudflare: Workers, Durable Objects, AI Gateway, and Workers AI for the small models. How our agents get computers we mostly do not pay for describes that stack. So Clef is not a new vendor for us. It is a new model ID on a binding we already call.

• Entity matching goes to Clef in shadow first, starting at 0.80. Jev keeps deciding while Clef’s answers are recorded beside it. The threshold we ship will come from that data, and we switch only if real traffic agrees with this benchmark.
• Background decisions default to Clef, each with its own threshold measured in shadow.
• Interactive decisions wait for the latency test. If Clef-flash is as fast from a Worker as Cloudflare reports, it is the natural fit for anything a person is waiting on.
• Clef-flash stays away from anything that merges or deletes until it has its own thresholds.

There is a quieter benefit. In our Jev post we said that decisions which read your conversations or email “wait for the paperwork”, because sending that text to another provider is a privacy decision. A first-party Cloudflare model keeps that text on the platform that already runs the rest of Construct, which removes the main reason most of our decisions are still switched off.

We are also watching Cloudflare’s new fine-tuning work, which it calls Reinforcement Learning for Calibrated Decisions. Our decision layer already records the label, the probability and what the product did for every decision, with no user text. That is close to the dataset such a system wants.

Read next: We put Jev inside our AI employee

How to try Clef

• On Workers AI: call `@cf/cloudflare/clef` or `@cf/cloudflare/clef-flash` through the AI binding or the REST API. The request is a `state` plus a map of `questions`, the same as Jev.
• Coming from Jev: change the model ID and keep everything else. Then re-measure every threshold.
• Self-hosted: the weights are at `Cloudflare/clef` and `Cloudflare/clef-flash` on Hugging Face under Apache 2.0.
• Read the probabilities, not the `confidence` field. It means different things on Clef and Jev, and on Jev it changed meaning between our September and October runs. Read `probabilities` and both models behave the same.
• Pace a REST benchmark. Cloudflare’s REST API throttled our token at roughly 90 calls a minute. The Workers binding is the route for real traffic.

Where Construct fits

Construct is an AI employee with its own computer: a workspace filesystem, long-term memory, a schedule, email, and workflows you can read before you run them. Decision models make the small calls around the edges, such as whether a person your agent just read about is one it already knows. Planning and writing run on general-purpose language models.

Two boundaries, stated plainly. As of this post, Jev still makes the live entity-matching decision and Clef has only run in this benchmark; this page will say when that changes. And the other four decisions in this benchmark are built but switched off in production until shadow data supports them.

If decision models are new to you, start with we put Jev inside our AI employee. For why small, repeatable judgments matter so much in long tasks, read your agent has a half-life. And if the category itself is new, begin with what is an AI employee.

Originally published at construct.computer

Construct is an AI employee. You hand it a job. It goes and does the job, without you having to babysit it.
To do that with agents like Hermes or OpenClaw, you spin up a VPS. Real CPU. Real disk. A full Linux machine that stays on between turns. That costs real money, and the moment more people sign up to poke around and never come back, it murders your infrastructure bill.

< Meet Construct >

We give every Agent that same kind of computer.
Don’t get excited, we are not paying for all of them.

The computers work. When someone asks Construct to do something, the machine wakes up and does it. When the tab goes quiet, nothing stays on just to cosplay as useful. Idle boxes are a kink we refuse to subsidize.

We like edging. So we put the whole stack on the edge. The agent loop in a Durable Object. Linux summoned for one tool call, then gone. Disk that is not a disk. The bill only finishes when somebody actually works.

This post is how we built that. Behaviour, constants, and tradeoffs are production.

The bill that scaled with people who never came back

The default way to run an AI agent is to hand it a Linux box and walk away.
Not stupid. Agents are trained on bash. Bash wants a filesystem and a process table. The lazy move is a machine that stays up between turns. Lazy is usually correct.

Until the invoice arrives.

We know that invoice because we signed it first. Construct’s original backend was a Bun and Elysia monolith on a VPS. SQLite on local disk. nginx out front. Roughly 1,300 lines of container management whose only job was handing every user their own box. It worked. Growing the product meant growing a fleet of real machines with real disks, billed through the silent hours where nobody typed a character.

That is the expensive version of “every user gets a computer”. Technically true. Financially a treadmill. Someone who signed up once and ghosted cost the same as a power user. The product could not scale if the bill scaled with accounts instead of work.

We killed it on 30 March 2026 with the commit title: “migrate backend from VPS monolith to Cloudflare Workers.” Everything in this post starts there.

Hono on Workers. Per-user WebSocket hub in a Durable Object with hibernation. Container object that archived to R2 on sleep and restored on wake. D1 for relational state. R2 for files. Frontend on Workers assets. Two days later we dragged the leftover container infrastructure into a legacy folder and wrote the plan we still run: the agent runs headlessly inside Durable Objects.

Containers and Sandboxes went GA on 13 April 2026. We moved onto the Sandbox SDK once it had the shape we wanted.

We did not see the future. We stared at a cost curve that scaled with signups instead of usage, bet on Durable Objects before there was a tidy product name for the thing, and watched the platform walk to the same answer. camelAI hit the same wall and reached nearly the same conclusion independently. Validation, or a warning about people who write posts like this. Maybe both.

Only half a computer deserves to be awake

“The agent needs a computer” is two unrelated things taped together.

Agent loop: transcript, tool routing, model calls, memory recall. This is how Construct thinks across a turn.

Machine: the thing that runs pdftotext, converts a spreadsheet, compiles a project, unzips the archive somebody should not have uploaded. This is how Construct touches a real filesystem.

Only the second needs Linux.

The first needs durable state and a socket. Cloudflare sells exactly that. Split the two and the expensive half stops needing to be alive at all.

Everybody argues about which model to use. Almost nobody argues about which half of their stack is getting paid to nap. That is the whole post, in one sentence.

Linux shows up, edges the job, and leaves 💦

This is the section the title is about.
Each Construct agent gets one Durable Object, resolved by name. Sessions are rows inside it, not objects of their own. Temporary subagents run as delegated sessions on the same parent. Fan out a job , buy concurrency, not instances. That is how Construct can split a big ask without spinning up a fleet.

The transcript lives in that object’s SQLite. Two things, kept apart on purpose. One table is the append-only message log the user scrolls through. A separate record holds the compaction summary plus a watermark of how far it has already consumed. That second one is what reaches the model. Conflate them and you pay, per turn, to re-send an afternoon of conversation to a model that did not ask for it. People do this. At scale.

Hibernation is where the bill actually moves. We use the WebSocket hibernation API, stash per-connection state on the socket itself, and let the runtime answer keepalives without ever waking the object.

A tab open all afternoon is not a running process. It is a socket the runtime holds and a handful of SQLite rows. The object wakes when something happens. Goes back down when it does not. Held, not running. Your Construct session feels continuous. Our invoice does not.

Eviction can happen any moment, so every transcript row also archives to D1. Empty on start? Restore from the archive. Make death cheap and constant and it stops being an incident.

One annoying edge case: replayed reconnect events can include a turn-started with no matching completion, because the object got evicted mid-turn. Client reconnects, nothing is running , we send terminal idle status so nobody sits watching a spinner think about a thought that died twenty minutes ago.

When the agent calls the terminal tool, we resolve a sandbox:

sandbox = getSandbox(SANDBOX_BINDING, instanceKeyFor(user, workspace), {
 transport: rpc,
 enableDefaultSession: false, # every exec is a fresh shell
 sleepAfter: idleWindowFor(plan),
})

Instance key scoped to user, or workspace when we know it. Not per agent. Not per session. Live containers track active humans, not agent count. Matters the moment somebody spawns twelve agents and goes to lunch.

sleepAfter is the pivot. Ten minutes idle and the box is gone. We call that a feature. You can call it edging. Same config value. The idle window is the plan’s command ceiling in minutes, clamped by a global 300-second cap, plus a five minute buffer. That resolves to a ten minute sleep on every plan today. Pair with active CPU billing on Cloudflare Containers , charged for CPU consumed, not wall-clock uptime , and an idle sandbox stops being something you manage.

Difference between “we should build a reaper for idle machines” and “there is nothing to reap.” One is a roadmap item forever. The other is a config value.

Unglamorous consequence: every exec is a fresh shell. Working directory does not persist. Agents hate this personally. We wrote a hint into the failure path explaining that each terminal call starts clean, relative paths resolve against the scratch tree, and any directory change or exported variable from an earlier call is gone forever. Grief counselling in a string in an error branch.

Before each exec: kill leftover processes, then mount, then run. Kill after mount and you drop the local bucket’s inotify watch and can leave a stale read-only workspace behind. Learned that the hard way.

She wrangler on my D1 till I R2 😳

Construct’s durable workspace is an R2 bucket mounted into the container over s3fs. Explicit uid and gid so the non-root user can actually write.
She wrangler on my D1 till I R2, The files finish. The machine does not have to.

This is the second half of the cost story, and the half people skip. Quotas of 100 MB, 1 GB, and 3 GB are quotas on bytes stored, not volumes kept spun up. When Construct saves a report, a spreadsheet, or a research brief, those files outlive the machine by construction. They were never on the machine. Nobody’s work sits on provisioned block storage waiting for their owner to remember the product exists.

Contrast again: provisioned volume waiting patiently versus bytes in R2 that do not care if the box is awake.

How we verify the mount matters more than the mount itself. Checking the directory exists is useless. Passes on a stale read-only mount every time. So we make the non-root user touch a probe file and delete it, with a ten second timeout. Exit code zero means the mount is actually writable.

Probe fails: unmount, remount, fix ownership, probe again, only then give up. Debounced to once a minute. Isolate-level mount cache key carries a version string we bump when mount policy changes, so a warm isolate can never silently skip a policy change.

Honest footnote: local development uses plain directory sync, not FUSE. Files land root-owned. We chown on mount. That local/prod divergence cost us an afternoon. It is why we trust a write probe over any status output that claims everything is fine.

What living on the edge actually costs

No sales pitch from here down. Living on the edge has a price and we pay it.

You cannot set `USER` in a Cloudflare Sandbox image. Control plane intercepts HTTPS and needs root. Privilege drop moves to the point of execution. Dom energy stays with the control plane. The agent bottoms out as `runuser`. Elevated commands stay root through a wrapper. Everything else is `runuser -u <agent user>`.

No sudo. No passwordless sudoers. Process and FD limits by the wrapper. Elevated access per command or until the container sleeps. Grants live inside the container, not Worker memory. Once the machine is disposable, the container is the correct place for short-lived trust. Permission that dies with the box is permission you never have to remember to revoke.

Memory and a registry catalog used to be separate Workers behind service bindings. We folded them into one. Never a latency optimisation. Service bindings are already efficient in-account transport. The cost was operational: two more deployments, more local processes, more domains, more generated binding contracts, releases that had to be choreographed. Microservice theatre for 41 KiB. We stopped. Nothing came.

Memory was about 41 KiB compressed. Registry under 1 KiB. Combined Worker about 1,002 KiB against a 2 MiB gate. 500 ms startup stays a cutover check because Wrangler dry runs do not report startup. We wrote down when to undo it: compressed bundle past 2 MiB, startup past 500 ms, own release cadence needed, or failures widen the API blast radius. Four triggers. Before anyone got attached.

Web app served by a Worker with static assets rather than Pages, because we needed custom domains to manage their own DNS and Pages OAuth could not.

Living on the edge means never assuming you have time. Construct’s memory recall runs on a budget. Whole recall: 800 ms. Inside that, remote embedding plus Vectorize: 250 ms. Those are configured budgets, not measured percentiles. We do not have deployed latency dashboards yet. Docs say so.

SQLite full-text search and graph traversal are the floor and always return, local to the Durable Object. Vectorize is best-effort on top. Vector path drags: slightly worse answer, not a hang. On one big warm box you would simply wait. Waiting is exactly what you cannot afford when nothing is warm by default. Warmth is a luxury. Edging is the default.

Embedding model was the one place with real measurements. 18-query set with close distractors, on Workers AI:

|      Model           | recall@1  |  mrr  |
| - - - - - - - - - - -| - - - - - | - - - |
| qwen3-embedding-0.6b | 15/18     | 0.892 |
| bge-m3               | 14/18     | 0.889 |
| bge-large-en-v1.5    | 13/18     | 0.852 |
| embeddinggemma-300m  | 12/18     | 0.824 |
[man, medium should have more tabular data support]

Qwen3 is trained for asymmetric retrieval. Tell it whether it holds a question or a stored record and ranking tightens. Query-side instruction moved mrr from 0.892 to 0.924 on the same set. Free accuracy for one string. Take it.

A 0.6b model also keeps this inside budget. Memory pipeline gate: USD 1 per 1,000 turns, excluding the primary chat model. Do not swap embedding models casually. Vectorize indexes are built at those exact dimensions. Different model means recreate and re-embed everything.

Three scars from the platform:

setTimeout versus setAlarm has a boundary around ten seconds. Discord gateway reconnects with backoff. Anything beyond ten seconds goes on an alarm. Timers do not survive eviction. That object holds one WebSocket for every guild we serve. Split it when Discord reports more than one shard, somewhere around 2,500 guilds. Better to schedule that than discover it at 3am.

A fallback on the same backend as its primary is not a fallback. Same outage, different name. Compaction runs the same Gemma build on Workers AI and AI Studio. Failover costs the route. Every other slot crosses providers.

Alarms hand you a bare Durable Object id. `ctx.id.name` only exists when you reached the object through a named stub. Alarm-triggered instantiation does not. Ours addressed outbound messages. Queue rejected them. Repair alarm re-queued forever, at cost. Infinite unfinished business. Fix: persist the name on first use. Nothing warns you.

The container is real. It is not free. Image lands between 1.5 and 1.9 GB, which forces `standard-1`, because disk allocation is effectively the image size limit. LibreOffice needs 4 GiB or it gets OOM killed. Production configured for up to 100 concurrent instances. Command execution capped at 300 seconds on every plan today. That is a real product limit this architecture caused, not a feature. s3fs is not real POSIX. It will surprise you on a weekend. Fresh shells irritate agents trained on bash. We pay for that in prompt tokens.

Flat version: if your workload is one long-lived process per user that truly never idles, none of this helps you. Buy a VM. Simpler. Happier. Nobody will make edging jokes at you.

What we run today

Workers. Durable Objects with SQLite storage. Containers through the Sandbox SDK. D1. R2. Queues with dead letter queues. Vectorize. Workers AI. AI Gateway. Email Routing inbound with the send binding outbound. Rate limiting bindings. Cache API. Turnstile. Cron triggers. Static assets. Workers Logs. Agents SDK.

No KV. No Hyperdrive. No machines of our own. Just Construct users with computers that mostly sleep.

All our Agents get computers, we pay for almost none.
We like edging. So we put the whole stack on the edge.

That is how an AI employee can feel always-on without an always-on bill.

If you want the product side rather than the plumbing, what an AI employee actually does covers the work it completes, inspectable agent memory covers what it remembers and how you correct it, and building your own agent stack is the honest version of what you would be operating instead.

Originally posted on construct.computer

All our Agents get computers, we pay for almost none

One of the improvements was the introduction of OpenID Connect (OIDC), which allows developers to automate package publishing through github actions, without having to setup any kind of api key or access token, and today we are having a look at how you can implement it for your packages as well.

Step 1: setup a node package (or skip if you already have one)

As an example, we’ll create a fresh package which just logs hello world to console

NOTE: this package is only meant as a demo for this blog. It is not recommended to create such packages for yourself and you should just setup OIDC on your existing packages :)

mkdir npm-oidc-demo && cd npm-oidc-demo
npm init -y
touch index.js

export function helloWorld() {
    console.log("Hello World");
}

Update your package.json to look like this:

{
  "name": "npm-oidc-demo",
  "version": "1.0.1",
  "description": "A simple hello world package",
  "main": "index.js",
  "type": "module",
  "exports": {
    ".": "./index.js"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/ankushKun/npm-oidc-demo"
  },
  "keywords": [
    "npm",
    "oidc"
  ],
  "author": "",
  "license": "ISC"
}

an already published package is required to use OIDC (might change in the future, who knows…)

Run npm login and follow the steps to link the npm cli with your npmjs account, when successful, run npm whoami and it should print your username.

and we are ready for publishing, just run npm publish

NOTE: If you ignored the previous note and still followed this guide to publish the demo package yourself, you might need to change your package name as I have already used “npm-oidc-demo”

$ npm whoami
ankushkun

$ npm publish
npm notice
npm notice 📦  npm-oidc-demo@1.0.0
npm notice Tarball Contents
npm notice 428B README.md
npm notice 64B index.js
npm notice 239B package.json
npm notice Tarball Details
npm notice name: npm-oidc-demo
npm notice version: 1.0.0
npm notice filename: npm-oidc-demo-1.0.0.tgz
npm notice package size: 602 B
npm notice unpacked size: 731 B
npm notice shasum: 338ccdfe6ad57442bef858098fc247680fb23c9e
npm notice integrity: sha512-XAKUT4avwrfcV[...]lEjW6hNUMJieQ==
npm notice total files: 3
npm notice
npm notice Publishing to https://registry.npmjs.org/ with tag latest and default access
Authenticate your account at:
https://www.npmjs.com/auth/cli/5f8fa35b-c07a-4af8-9851-831f191efc3e
Press ENTER to open in the browser...

+ npm-oidc-demo@1.0.0

The package is now available at https://www.npmjs.com/package/npm-oidc-demo

Step 2: Update OIDC details on package settings

Head over to the settings tab on our freshly published package and select GitHub Actions as the trusted publisher

I have already created a github repo at ankushKun/npm-oidc-demo, now we just need to create a workflow file. Put this inside .github/workflows/publish.yml

name: Publish to npm

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20.x"
          registry-url: "https://registry.npmjs.org"

      - name: Update npm
        run: npm install -g npm@latest

      - name: Publish to npm
        run: npm publish

Push the yml file to the repo and update the OIDC details on the package settings. I have kept the environment name empty because I don’t have any explicit environment setup for actions.

Now when you create a new release on the github repo, a workflow will start running which will publish the package to npm with provenance.

And that’s how one can setup cicd to publish npm packages from github actions without having to create any access tokens themselves!

An API — Application Programming Interface, defines how different applications can talk to each other and exchange data through structures requests and responses

API Types?
1. Web APIs — the one we will be focusing on today
2. OS APIs — Windows APIs, Android APIs, etc
3. Library APIs — Like a python library
4. Hardware APIs — allow accessing hardware functions (embedded libraries?)

RESTful Web APIs

A RESTful Web API is one that follows the “REST” (Representational State Transfer) principles, a software architecture style that ensures scalability, simplicity, and stateless communication between servers and clients.

Common REST HTTP Methods
There are 4 common methods/requests to call REST APIs
1. GET — to get some data from the server
2. POST — to send/create some data from the server and get a response
3. PUT — to update existing data on a server
4. DELETE — to remove some data form a server

When browsing the web, GET request is used to fetch the client side content that gets rendered by the browser

API response codes
Whenever an API is called, it will always return a status code (a number) to signify the result of the request. Categorised into 1XX (informational response), 2XX (success codes), 3XX (redirects), 4XX (client errors), 5XX (server errors)

200 — OK, successful request
301 — Moved Permanently
400 — Bad Request, invalid inputs
401 — Unauthorised
404 — Not Found
500 — Internal Server Error

Lets write an API endpoint ourselves

Make sure you have NodeJS + NPM already installed and follow these steps

mkdir api-test
cd api-test
npm init -y
npm i express
touch index.js

This will create a folder for our express API backend and create a JS file to write code in

Edit the index.js file and add this code

const express = require('express');
const app = express();

app.get('/', (req, res) => {
    res.send('Hello World');
});

app.listen(3000, () => {
    console.log('Server is running on port 3000');
});

We have just initialised an express backend which is serving the “Hello World” string at root (/) endpoint. i.e. on sending a GET request to /, Hello World will be returned

Run this file with node index.js and open http://localhost:3000/ on any browser. It should show Hello World

Try changing the string in res.send('this one') to a simple HTML like <h1>Hi there</h1> and restart the express server (ctrl+c and node index.js again), check if something changes on the served page

But this is just for GET requests, we cannot make POST requests to this endpoint

As you can see it gives an error if we try to make a POST request to the GET endpoint we created

Lets define a POST endpoint now

// add this after defining defining the app
app.use(express.json());

// POST endpoint
app.post('/', (req, res) => {
    const name = req.body.name
    res.send(`Hello ${name}!`);
});

restart the app and let’s try sending a post request to the server

You can use any HTTP client like Thunder Client on VSCode or Postman. I am going to use cURL coz simple

curl http://localhost:3000/ -d '{ "name": "Ankush" }'

this should print the response as Hello Ankush!

BOOM! You just created a GET and POST request! A majority of web APIs are comprised of just GET and POST requests

Time to write some frontend

<html>
  <body>
    <input type="text" id="name" name="name" placeholder="Enter your name">
    <button onclick="handleSubmit()">Submit</button>
    <script>
      async function handleSubmit() {
        const name = document.getElementById('name').value;
        const response = await fetch('/', {
          method: 'POST',
          headers: {'Content-Type': 'application/json'},
          body: JSON.stringify({ name })
        });
        const data = await response.text();
        document.body.innerHTML = data;
      }
    </script>
  </body>
</html>

We will serve this HTML through the GET endpoint we created earlier

app.get('/', (req, res) => {
    res.send(`
        <html>
            <body>
                <input type="text" id="name" name="name" placeholder="Enter your name">
                <button onclick="handleSubmit()">Submit</button>
                <script>
                    async function handleSubmit() {
                        const name = document.getElementById('name').value;
                        const response = await fetch('/', {
                            method: 'POST',
                            headers: {
                                'Content-Type': 'application/json'
                            },
                            body: JSON.stringify({ name })
                        });
                        const data = await response.text();
                        document.body.innerHTML = data;
                    }
                </script>
            </body>
        </html>
    `);
});

Restart the express app and refresh localhost:3000, and try entering your name

Now if you open developer tools and go into the network tab, you can see the POST request being sent and the different data associated with it

Congrats you just sent a POST request with some data, received some data back in response and showed it on the frontend!

Final Code:

const express = require('express');
const app = express();

app.use(express.json());

// To test this endpoint using curl, you can use the following command:
// curl http://localhost:3000/ 
app.get('/', (req, res) => {
    res.send(`
        <html>
            <body>
                <input type="text" id="name" name="name" placeholder="Enter your name">
                <button onclick="handleSubmit()">Submit</button>
                <script>
                    async function handleSubmit() {
                        const name = document.getElementById('name').value;
                        const response = await fetch('/', {
                            method: 'POST',
                            headers: {
                                'Content-Type': 'application/json'
                            },
                            body: JSON.stringify({ name })
                        });
                        const data = await response.text();
                        document.body.innerHTML = data;
                    }
                </script>
            </body>
        </html>
    `);
});

// To test this endpoint using curl, you can use the following command:
// curl http://localhost:3000/ -d '{ "name": "Ankush" }'
app.post('/', (req, res) => {
    const name = req.body.name;
    res.send(`Hello ${name}!`);
});

app.listen(3000, () => {
    console.log('Server is running on port http://localhost:3000');
});

What’s this about?

I had recently attended a workshop by Intel on their oneAPI programming model which was conducted by Abhishek Nandy at IIT Roorkee during the Cognizance 2023. I attended this workshop with my friend, Krish Agrawal who is also the co-author of this blog. The workshop was held for 2 days in which Abhishek Sir on the first day told about this new model by Intel and how to get started with its implementation with the help of JupyterLab. Before ending the day, we were told that there is a hackathon where he will take ideas by all the students in the form of teams as to how we use Intel oneAPI in real-life projects.

SYCL (/sɪkl/) is a programming model for high-performance computing that allows developers to write code for heterogeneous systems that use accelerators such as GPUs, FPGAs, and other specialised processing units. It is an open standard developed by the Khronos Group, an industry consortium that also develops other widely used graphics and compute APIs such as OpenGL and Vulkan. SYCL code can be executed on a variety of devices, including CPUs, GPUs, and FPGAs, without modification.

Fractals are fascinating geometric shapes that repeat themselves infinitely on different scales. They are created through a process of repeating a simple mathematical equation or algorithm to form increasingly intricate patterns. These fractal shapes are naturally occurring and can be observed in many aspects of nature, such as the patterns of a fern leaf, the coastlines of continents, and even the formation of clouds. One of the most well-known and captivating fractals is the Mandelbrot fractal. This type of fractal is generated through an iterative process using complex numbers, resulting in a shape that is renowned for its intricate detail and infinite complexity. It’s a true marvel of mathematics!

Now that we know what our project revolves around, let’s see its implementation in JupyterLab.

Getting started on DevCloud

DevCloud is sandbox that intel provides to learn and try out their oneAPI ecosystem. We will be using the devcloud jupyter lab to run our SYCL fractal generation project. It is way easier and quicker to run SYCL projects on the sandbox than on your local machine.

So head over to https://devcloud.intel.com/oneapi/ and create an account. Navigate to the ‘Getting Started’ tab, scroll down to ‘Connect with JupyterLab’ and Launch a JupyterLab instance. It might take some time or get loaded instantly depending on the server load so have some patience.

Once launched, you will find a bunch of folders in the file explorer, including a ‘mandelbrot’ folder.

Open the Terminal from the landing page, cd into the mandelbrot directory and run the build script.

cd mandelbrot
chmod +x build.sh
./build.sh

This will compile and run the project and create a ‘mandelbrot.png’ file inside the ‘build’ folder. As you can see the image is 1024x1024 pixels and it took just 190 milliseconds to generate the image (serial time + parallel time). Now if we were to do the same in python or regular C++ without SYCL, it would take seconds to generate the same result. So that’s the power of SYCL.

If you checkout the ‘src/mandel.hpp’ file, you can find 4 variables row and column size, max iterations and repetitions. We can modify these values to generate Mandelbrot sets of different resolutions and qualities.

Try changing these values and see how much time it takes for the images of different configurations to generate.

Automate it

Let’s now write a bash script that automatically changes the values in the C++ file and builds the project.

Create a file ‘autogen.sh’ and inside it write the following lines.

#!/bin/bash

PASSES=(10 100 500 1000 5000)
SIZES=(1024 2048 4096 8192 16384)

EDIT_FILE="src/mandel.hpp"
BUILD_FILE="build.sh"

prev_size=100
prev_pass=100
let count=0

The variable ‘PASSES’ contains a list of iterations that will be used in each image of the image generated and sizes contains the list of resolutions for the images in pixels. ‘prev_size’ and ‘prev_pass’ is a placeholder that we will use later. Count is the number of images generated.

for SIZE in ${SIZES[@]}; do
    for PASS in ${PASSES[@]}; do
        echo ""
        echo "Generating fractal for $SIZE x $SIZE with $PASS passes"
        echo ""

        ...

    done
done

We create a nested loop for the values of SIZES and PASSES, so we can have all the combinations of sizes and passes to generate the images with. In the body of the loop we print a message to display to the user.

In order to edit the values of the 3 variables in the .hpp file we will use the sed command.

sed -ie "s/row_size=$prev_size/row_size=$SIZE/g" $EDIT_FILE
sed -ie "s/col_size=$prev_size/col_size=$SIZE/g" $EDIT_FILE
sed -ie "s/max_iterations=$prev_pass/max_iterations=$PASS/g" $EDIT_FILE
sed -ie "s/repetitions=$prev_pass/repetitions=$PASS/g" $EDIT_FILE

prev_size=$SIZE
prev_pass=$PASS
count=$((count+1))

This uses find and replace to replace the previous values of the variables with new values and updates the count in each iteration of the bash loop.

To run the build command we simply do

./$BUILD_FILE

echo ""
echo "Generated for $SIZE x $SIZE with $PASS passes"
echo ""

The project is built and image is saved in the build folder, so we will have to copy it from there into a location of our choice. Create a folder images in the mandelbrot folder and add the following copy command to the autogen.sh script

cp "./build/mandelbrot.png" "./images/$count.mandelbrot_${SIZE}_${PASS}.png"

This copies the image from build into images folder and also renames it to contain useful information such as size and number of passes and the count.

Run the autogen script and let the fractals be created!

chmod +x autogen.sh
./autogen.sh

Here is the full autogen script

#!/bin/bash

PASSES=(10 100 500 1000 5000)
SIZES=(1024 2048 4096 8192 16384)

EDIT_FILE="src/mandel.hpp"
BUILD_FILE="build.sh"

prev_size=100
prev_pass=100
let count=0

for SIZE in ${SIZES[@]}; do
    for PASS in ${PASSES[@]}; do
        echo ""
        echo "Generating fractal for $SIZE x $SIZE with $PASS passes"
        echo ""
        sed -ie "s/row_size=$prev_size/row_size=$SIZE/g" $EDIT_FILE
        sed -ie "s/col_size=$prev_size/col_size=$SIZE/g" $EDIT_FILE
        sed -ie "s/max_iterations=$prev_pass/max_iterations=$PASS/g" $EDIT_FILE
        sed -ie "s/repetitions=$prev_pass/repetitions=$PASS/g" $EDIT_FILE

        prev_size=$SIZE
        prev_pass=$PASS
        count=$((count+1))

        ./$BUILD_FILE

        echo ""
        echo "Generated for $SIZE x $SIZE with $PASS passes"
        echo ""

        cp "./build/mandelbrot.png" "./images/$count.mandelbrot_${SIZE}_${PASS}.png"
        
    done
done

More iterations means better quality of the image, you can zoom in and see the small details and tiny fractal repetitions, but to see these clearly the resolution must be increased.

The winning moment

It was Day 2 and we both had absolutely nothing with us. We sat at the back thinking what we would present the crowd with. We had about 3 hours in our hands to build something. While brainstorming, we were randomly playing with the Mandelbrot directory already given by Intel in their Virtual Private Server. We ran the code with the proper command and was later told to rerun using qbus -I which then redirected usto another portal maybe through some kind of routing. We then decided to create a bash script to randomise four values in the original program i.e. row_size, column_size, max_iterations and repetitions. We were able to do this and now we had started generating images in a folder. Some images with higher resolution or more repetitions took more time than the rest. The longest time was taken by 16K resolution image with 1000 repetitions which took approximately 50 minutes (still wayy less than an average C++ or Python script generating the same iterations and resolution).

We presented our solution made on a presentation made on Keynote. We remarkably remember that we were the only team with the minimal interaction with Abhishek Sir and also the only ones to receive a round of applause from him. At the end of all prizes, 6 stood out of 13 and were congratulated on stage for the same. Then came the cash prizes for the Top 3. As soon as second was done, we thought the claps were consolatory of nature and we were already packing. But then we realised, we were indeed 1st. (Out of excitement, I slammed the top of Krish’s Macbook Pro 😂). In the end we got ₹12000 as a prize in the form of Amazon Vouchers.

If you scrolled till this point, you may as well checkout my social handles and leave a follow, Thanks for reading ;)