AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

New hardware is fun. Rewriting a working application to accommodate it is not. That trade shows up in almost every accelerator demo I have sat through: the board goes live, and quietly the app grows a special client library, then a special request shape, then a special set of apologies for whenever the board is having a bad day. If adding an accelerator means rewriting the app, the hardware lane has become an island.

Demo 6 keeps the application completely unchanged. Tenstorrent - or a mock accelerator that behaves like one - registers as another backend behind the AI on the Edge Gateway, and the routing policy decides what that is worth.

How Hardware Becomes an Island

Accelerator demos tend to drift toward hardware-specific code paths:

• A different client library.
• A different request shape.
• A different health check.
• A different dashboard.
• A different fallback story.

Each step is defensible on its own during bring-up, and together they are bad for application architecture. The application should ask for a workload. The router should decide whether an accelerator is eligible, healthy, and preferred.

Nobody here is claiming one hardware path is always faster. The claim is narrower and more useful: specialized hardware can join the governed edge platform without becoming a separate product.

One More Backend in the Registry

The whole trick is that the accelerator lane is just a backend registration, using the same registry schema as every other backend:

{
  "id": "tenstorrent-edge",
  "kind": "OpenAICompatible",
  "baseUrl": "http://tenstorrent-edge:8000/v1",
  "models": ["accelerator-chat"],
  "location": "edge",
  "capabilities": ["chat", "streaming"],
  "tags": ["local", "accelerator", "specialized-hardware"],
  "priority": 5
}

From there the gateway treats Tenstorrent like any other OpenAI-compatible backend until it has a reason not to:

• Discover models through /v1/models or a configured model list.
• Record capabilities explicitly.
• Test chat, streaming, and embeddings independently.
• Track latency, status codes, saturation, selected count, and fallback count.
• Expose compatibility failures in the route trace.

The routing policy is AcceleratorPreferred, not AcceleratorOnly. If the accelerator is healthy and supports the requested model family, it wins. If it is unhealthy or saturated, the router picks another eligible local backend, and the audience never has to think about it.

One status caveat I want on the record: as of August 2026, Tenstorrent’s documentation describes TT-Inference-Server for deploying LLM serving on its hardware, including container management, model downloads, serving configuration, and an OpenAI-compatible API endpoint. Model support still depends on the validated hardware and software combination, which is exactly why the gateway discovers and displays capability rather than assuming it.

What the Demo Has to Prove

The proof the audience needs is that no application code changes. Everything else is staging:

1. Show the backend registry: Foundry Local, Azure Foundry, Tenstorrent, mock accelerator.
2. Ask the same operational question used in Demo 1.
3. Add workload metadata: AcceleratorPreferred.
4. Show the router selects tenstorrent-edge or mock-accelerator.
5. Stream the response through the same app and gateway.
6. Show metrics and backend health.
7. Mark the accelerator saturated.
8. Ask again.
9. Show fallback to another eligible local backend.
10. Show the app request did not change.

Step ten is the entire point. Steps two through nine exist to make step ten believable.

I also avoid benchmark claims unless the evaluation harness produced them. The architecture claim here is portability and governance, not a vague performance win - vague wins are exactly what skeptical rooms smell first.

What Is Actually Left to Build

Most of this lane is assembly, not invention. The private cloud buildout already describes the Tenstorrent lab path and the mixed accelerator environment. The Demo 1 gateway already provides the abstraction that keeps the app unchanged, and the Demo 5 design supplies saturation and fallback controls. What remains is adapter hardening:

• Tenstorrent backend config profile.
• Capability discovery and display.
• Health check and timeout behavior.
• Compatibility tests for the supported API paths.
• Accelerator-preferred routing policy.
• Saturation fallback behavior.
• Optional hardware metrics later, starting with endpoint-level metrics now.

The mock accelerator is required, not nice-to-have. Real hardware is valuable, but the session still has to work when the board is offline, in use by something else, or running a different model than the demo expects. A dependency I cannot reproduce on demand is not a demo plan; it is a coin flip.

Where Overpromising Starts

The failure mode is overpromising hardware support. OpenAI-compatible does not mean feature-complete: chat, streaming, embeddings, tool calls, batch behavior, and error semantics all need separate smoke tests, because each one is its own little contract the hardware may or may not honor.

So the gateway displays capability and denial explicitly instead of discovering the gap mid-request:

{
  "backendId": "tenstorrent-edge",
  "eligible": false,
  "reason": "Backend does not advertise embeddings capability."
}

An honest “no” from the router beats a mystery timeout from the hardware every time.

What Done Looks Like

Demo 6 is complete when:

1. Tenstorrent or mock accelerator can be registered as a backend.
2. The router can prefer it for tagged workloads.
3. The app does not change when the backend changes.
4. Fallback works when the accelerator is unhealthy or saturated.
5. Metrics show selected backend, latency, errors, and fallback.
6. No performance claims appear unless produced by the evaluation harness.

Criterion three is the one I watch. The moment swapping a backend means touching the application again, this stopped being an architecture and became a science project.

Related Posts

This post connects the Tenstorrent private cloud buildout to the AI on the Edge gateway. The hardware is part of the architecture, but not the whole architecture - that distinction is why the application survived this demo untouched.

References

• Tenstorrent tools documentation
• Deploy LLMs with TT-Inference-Server

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

The edge has to stand alone sometimes. Internet links fail. Cloud services throttle. A local model crashes mid-request, an accelerator endpoint saturates, and sooner or later a response adapter meets a payload that is almost, but not quite, OpenAI-compatible. None of that is hypothetical; it is just a Tuesday.

Demo 5 is the failure lab - a controlled set of ways for all of that to happen on demand - built to prove the architecture fails in visible, policy-correct ways. Happy-path demos are easy to fake, and failure behavior is part of the product here, so I would rather show it on purpose than meet it for the first time on stage.

Why Build a Failure Lab

Happy-path AI demos are easy to fake. Real systems need answers to harder questions:

• What happens when the cloud is unavailable?
• What happens when the local model is unavailable?
• What happens when the best backend is slow or saturated?
• What happens when a backend returns malformed output?
• What happens when a restricted request has no eligible backend?
• Can the presenter reset the demo without debugging state live?

The failure behavior is part of the product, so the demo has to be able to show it on purpose rather than by accident.

Inside the Failure Lab

In the design, AiOnTheEdge.DemoScenarios owns deterministic failures and reset controls behind the shared demo endpoints. Fault types are posted to the single faults endpoint, and reset is the shared endpoint every service exposes:

POST /admin/demo/faults   { "fault": "backend-down",      "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "backend-slow",      "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "backend-malformed", "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "network-cloud-blocked" }
POST /admin/demo/faults   { "fault": "policy-local-only" }
POST /admin/demo/reset    clears active faults and restores the seeded state

The dashboard needs a Failure Lab page:

Faults should be simple and explicit - every one maps to something boring that can actually happen:

Staging a Fail-Closed Denial

A fail-closed denial is the moment worth staging. The sequence I use:

1. Start with all backends healthy.
2. Ask a normal public question and show a valid route.
3. Mark the prompt restricted and show LocalOnly.
4. Disable the local backend.
5. Ask again and show denial because cloud is not eligible.
6. Change policy to CloudAllowed for nonrestricted content.
7. Show fallback to an allowed backend.
8. Block cloud.
9. Show local RAG and camera incident summaries still work with cached models and local data.
10. Run make demo-smoke-test.
11. Reset the demo.

The edge does not have to answer every request. It has to answer the requests it is allowed to answer and refuse the rest clearly. That sentence is the whole design review, and everything else in this post is machinery for proving it.

Parts That Can Already Break

The earlier demos already provide the components that can fail: gateway route selection, RAG retrieval and generation, camera event replay, Azure-backed telemetry, and accelerator backend registration. Demo 5 turns those components into test cases instead of ad hoc failure stories.

The New Work Is the Harness

The new build is the failure harness itself:

• Central fault state.
• Reset and seed endpoints.
• Smoke-test runner.
• Failure Lab UI.
• Deterministic mock backend behaviors.
• Local telemetry when Azure is unavailable.
• Dashboard panels for denials, fallbacks, adapter errors, and reset status.

The smoke test should validate the contract the presenter needs:

make demo-reset
make demo-seed
make demo-smoke-test

That test should cover local route, cloud-allowed fallback, local-only denial, malformed response handling, camera replay, RAG citation, and reset. When it passes, I stop worrying about whatever the venue’s Wi-Fi is doing.

Too Many Faults Ruin the Show

The failure lab can become too noisy. The audience should see two or three failures live, not every possible fault; a parade of toggles is its own kind of confusion. The best live sequence is:

1. Local backend down.
2. Restricted prompt denied.
3. Cloud blocked but local RAG still answers.

Everything else is useful for validation and backup, but not every control needs to be shown in a 45-minute talk.

It pays to be precise about what offline actually means here: local application behavior can continue after warmup, while live Azure management and cloud telemetry depend on connectivity and prior setup. Anything vaguer and someone walks away convinced the whole stack runs air-gapped forever.

What Done Looks Like

Demo 5 is complete when:

1. Every fault can be enabled and reset through API and UI.
2. The gateway never routes restricted prompts to cloud.
3. Backend failures produce clear user-facing and telemetry-facing reasons.
4. make demo-smoke-test covers every primary failure path.
5. At least one complete demo path runs with no internet connection after warmup.
6. Reset is designed to return the system to a known state in under one minute, fast enough to run between talk segments.

That reset budget is not vanity. A demo you cannot recover from between segments is a demo you will only run once.

Related Posts

This is the credibility test for the gateway, RAG assistant, operations assistant, and Azure operations layer. It should be built before depending on real hardware or live cloud services in a session - credibility is much cheaper to install up front.

References

• What is Foundry Local?
• Best practices and troubleshooting guide for Foundry Local CLI
• Azure IoT Operations overview
• Process and route data with data flows

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

Answering prompts is not the same as operating a system, a distinction I did not fully appreciate until the demos started stacking up. Teams also have to deploy it, secure it, observe it, rotate its secrets, understand its failures, and prove policy decisions after the fact. That is what Demo 4 sets up with Azure. The model may run locally, but the estate should still be visible and governable.

Local AI Goes Invisible

Local AI can become invisible infrastructure, and it happens without anyone deciding anything. A model server starts on a developer machine. A Kubernetes deployment gets copied to an edge node. A gateway ends up with API keys in a config file. Logs stay local. Metrics are whatever the process prints. Nobody can tell which requests went where. I have watched perfectly serious systems drift into exactly that state one shortcut at a time, and invisibility is not an operating model an enterprise can run on.

For AI on the Edge, every local execution path needs an operations path:

• How was it deployed?
• Which version is running?
• Which backends are healthy?
• Which requests fell back?
• Which requests were denied?
• Where are secrets stored?
• Which policies are being enforced?

The Control Plane Around Local Execution

The Azure-governed mode uses Azure as the control plane around local execution. Each layer gets exactly one job:

The Kubernetes deployment should be conventional:

infra/
  azure/
  kubernetes/
  arc/
  dashboards/
  policies/

Cluster resources should include:

The dashboards should make the architecture measurable:

• Requests by backend.
• Fallback count.
• Denied count.
• P50/P95/P99 latency.
• Prompt body logging posture.
• Local-only request count.
• Backend health.
• Camera events by site and type.
• Active incidents and summaries.

If a number is not on a dashboard, it does not exist during an outage.

How the Talk Actually Runs

Policy evidence has to land in three places at once: the app response, the dashboards, and the query history. Any one of them alone is a story; together they are proof. So the talk runs like this:

1. Show the app running at the edge.
2. Show the Arc-connected cluster in Azure.
3. Show GitOps sync status.
4. Send requests through the gateway.
5. Open Grafana and show backend selection, latency, fallback, and denial metrics.
6. Open Log Analytics and run a KQL query over routing events.
7. Trigger a LocalOnly denial.
8. Show the denial in the app, dashboard, and KQL.
9. Rotate or change a backend secret if implemented, then show config reload or rollout status.

One caveat before anyone pastes these into their own workspace: the queries below target the planned Log Analytics schema. AiRoutingEvents, AiInferenceRequests, and CameraEvents exist once the ingestion pipeline ships, so read them as the observability contract rather than as queries against a live workspace:

AiRoutingEvents
| summarize count() by selectedBackend, policy, decision

AiRoutingEvents
| where policy == "LocalOnly" and decision == "Denied"

AiInferenceRequests
| summarize p95Latency=percentile(latencyMs, 95) by backendId, bin(timestamp, 5m)

CameraEvents
| summarize count() by siteId, cameraId, eventType, bin(timestamp, 5m)

Signals That Already Exist

The earlier demos already emit the signals worth collecting: gateway routing decisions, RAG retrieval and privacy decisions, camera incident summaries, and backend health and failures. The private cloud buildout already frames Arc and GitOps as the management overlay for the local Kubernetes environment. Demo 4 turns those signals into operational evidence instead of log lines nobody reads.

What Had to Be Built

The new work is infrastructure and observability - none of it glamorous, all of it load-bearing:

• Bicep or Terraform for the Azure resources.
• Helm or Kustomize for Kubernetes deployment.
• Managed Prometheus scrape config.
• Grafana dashboard JSON.
• KQL saved queries.
• Key Vault secret integration for the Azure-governed path.
• Application policy events that match dashboard and query fields.

Policy examples, split by the layer that actually enforces them. This split matters more than it looks, because people assume Azure Policy reaches inside applications, and it does not:

To say it plainly: Azure Policy governs Azure resources such as tags and secret sources; it cannot block a gateway from enabling an unknown model backend. The routing boundaries in this demo are application-level policy events, surfaced through the same evidence flow as the Azure-side rules.

The Brittle Demo Trap

The failure mode here is demo brittleness from trying to install cloud operations live during a talk. Do not do that. Preflight the Azure path, keep a recorded or screenshot backup for portal views, and run the local dashboard live.

Arc and Azure Monitor require outbound connectivity and setup. Disconnected demos should show local behavior and buffered telemetry, not pretend Azure is live while the edge is offline. Nobody in the audience can see your resource group anyway; everybody can see a stalled terminal.

What Done Looks Like

Demo 4 is complete when:

1. make infra-plan and make infra-apply produce repeatable Azure resources.
2. Gateway, assistants, and dashboard deploy through Helm or Kustomize.
3. Metrics are visible locally and in Managed Prometheus when Azure is connected.
4. Grafana dashboards can be imported automatically.
5. Log Analytics receives routing and incident events.
6. Policy denial is visible in app logs, dashboard, and KQL.
7. make teardown removes demo resources safely.

That last one matters more than it sounds. Leaving resources behind is how a demo quietly becomes a bill.

Related Posts

This post makes the prior demos operational. It depends on the gateway routing events from One App, Many Places to Run AI and the camera incident events from From Camera Events to Operator Guidance.

References

• Azure Arc-enabled Kubernetes overview
• Application deployments with GitOps using Flux v2
• Enable monitoring for Arc-enabled Kubernetes clusters
• Use Azure Key Vault Secrets Provider extension with Arc-enabled Kubernetes

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

Back in 2019 a vendor shut down the cloud behind my cameras and bricked them; I repurposed what was left instead of throwing the hardware out, and the fleet never left. So when this series needed a real workload instead of a generic AI sample, the camera fleet was the obvious pick. Availability, RTSP loss, stale heartbeats, motion bursts, command acknowledgements, config drift, and connector-fronted cameras all flow through the existing cameras/# contract already.

The goal is not asking a model to read random logs. The goal is turning structured edge events into operator guidance without changing the camera control plane. Those are two very different projects, and I only signed up for one of them.

What an Operator Actually Needs

Edge operators do not need another dashboard full of raw events. I have stared at enough of those to know they mostly teach you where to look next. What an operator wants when a site goes quiet is simple: what happened, what is impacted, what evidence supports that conclusion, and what action should happen first.

The camera control plane already has the right shape for this:

• Cameras or gateways connect outbound.
• They publish status, inventory, availability, events, metrics, command acknowledgements, and logs.
• The topic tree is consistent: cameras/<site>/<camera>/<channel>.
• Azure IoT Operations can sit under that topic contract as the MQTT broker and edge data plane.
• A data flow can forward the same cameras/# stream northbound without changing producers.

So the operations assistant builds on that contract. It does not create a second camera model. One camera model in this house is plenty.

Three Ways In, One Event Shape

In the design, AiOnTheEdge.OperationsAssistant takes input three ways. Simulated mode replays JSON from samples/camera-events, which is the reliable presentation path. MQTT mode subscribes to cameras/#, which is the local edge path. Event Hubs mode consumes forwarded camera telemetry, which is the cloud analytics path.

Every input mode normalizes events into one shape. The sample below is synthetic:

{
  "eventId": "evt-001",
  "timestampUtc": "2026-06-30T12:00:00Z",
  "siteId": "remote1",
  "cameraId": "garage-east",
  "channel": "event",
  "eventType": "rtsp_loss",
  "severity": "warning",
  "payload": {
    "streamUrl": "rtsp://camera/stream1",
    "durationSeconds": 90,
    "lastFrameUtc": "2026-06-30T11:58:30Z"
  }
}

From there the service builds incidents from facts. Six pieces split the work:

• CameraEventIngestWorker reads replay files, MQTT, or Event Hubs.
• FleetStateStore tracks latest status, inventory, availability, version, config hash, and recent events.
• IncidentBuilder groups related events by site, camera, event type, time window, and severity.
• RunbookRetriever pulls relevant local runbook chunks from the Knowledge Assistant.
• OperatorPromptBuilder creates a compact prompt from structured facts and runbook snippets.
• OperationsAssistantController exposes incident summaries and fleet questions.

And whatever the assistant answers, evidence rides along with it:

{
  "summary": "Remote property 1 likely has a site-level network issue.",
  "impact": [
    "garage-east offline",
    "driveway-west offline",
    "front-door heartbeat stale"
  ],
  "evidence": [
    {
      "timestampUtc": "2026-06-30T11:58:30Z",
      "cameraId": "garage-east",
      "eventType": "rtsp_loss"
    }
  ],
  "recommendedActions": [
    "Check the remote1 camera VLAN gateway and VPN tunnel.",
    "Verify NTP and DNS availability for the camera VLAN.",
    "Avoid rebooting individual cameras until site connectivity is confirmed."
  ],
  "routeDecision": {
    "policy": "LocalOnly",
    "selectedBackend": "foundry-local"
  }
}

How the Talk Actually Runs

The moment worth staging takes a raw event through incident grouping to a first action. Here is the run order I actually use:

1. Start replay: site-network-partition.
2. Show raw events arriving with their original cameras/<site>/<camera>/<channel> topics.
3. Show normalized events updating fleet state.
4. Ask: What happened at remote1 in the last 15 minutes?
5. Show the assistant summary, impacted cameras, evidence, runbook citations, and first actions.
6. Open the route trace and show local-only inference due to operational camera telemetry.
7. Trigger config-drift.
8. Ask: Which cameras need config remediation?
9. Show desired versus reported config hash and the apply_config recommendation.
10. Show that the same raw events can also flow through Azure IoT Operations and Event Hubs in the full Azure path.

Notice what the model never does: invent cameras, sites, or causes. It summarizes only from structured incident facts and retrieved runbook text. If the facts do not name a cause, neither does the assistant.

Standing on the Camera Series

The three camera posts already define the control plane, and this demo does not reopen it:

• Part 1 defines the topic tree, network model, command set, desired/reported state, and camera classes.
• Part 2 swaps the broker under the same contract to Azure IoT Operations.
• Part 3 forwards cameras/# to Event Hubs and adds the ONVIF connector bridge path.

Those posts are prerequisites here; this one adds a local AI operations layer on top and stops arguing about cameras. The new build is just the operations surface:

POST /camera-events
GET  /fleet/sites
GET  /fleet/sites/{siteId}
GET  /fleet/cameras/{siteId}/{cameraId}
GET  /incidents
GET  /incidents/{incidentId}
POST /incidents/{incidentId}/summarize
POST /fleet/query
POST /admin/demo/replay/{scenarioName}

Six seed scenarios give the demo its plot:

Each scenario lives in AiOnTheEdge.DemoScenarios as a deterministic script of normalized events, so the demo behaves the same way in a hotel ballroom as it did on my desk. For example, the shape of the site-network-partition scenario file is:

{
  "scenarioName": "site-network-partition",
  "description": "Several cameras at one site lose connectivity inside a 90-second window.",
  "events": [
    {
      "offsetSeconds": 0,
      "event": {
        "eventId": "evt-partition-001",
        "siteId": "remote1",
        "cameraId": "garage-east",
        "channel": "event",
        "eventType": "rtsp_loss",
        "severity": "warning"
      }
    },
    {
      "offsetSeconds": 45,
      "event": {
        "eventId": "evt-partition-002",
        "siteId": "remote1",
        "cameraId": "driveway-west",
        "channel": "availability",
        "eventType": "offline",
        "severity": "error"
      }
    }
  ]
}

The Hallucination Trap

The failure to avoid is a confident hallucinated incident. An assistant that infers a site outage because it sounds plausible is worse than no assistant at all; somebody will act on that answer and start rebooting the wrong things. It should only name a cause when the structured facts and runbook snippets support it.

For a live demo, the simulator is the default. Real cameras and Azure IoT Operations are valuable, but the presentation should not depend on a camera or a VPN behaving perfectly. Talks provide enough surprises on their own.

What Done Looks Like

Demo 3 earns its place when:

1. Camera event replay produces deterministic fleet state.
2. MQTT mode can subscribe to cameras/# when a broker is present.
3. Event Hubs mode can consume forwarded telemetry when Azure is connected.
4. The assistant summarizes at least three seeded incident types.
5. The assistant cites structured evidence and runbook snippets.
6. The UI shows original topic, normalized event, incident, and AI answer.
7. The answer never invents cameras or sites not present in the event store.
8. The full demo runs without real cameras.

That last item is the point. If the demo needed real cameras, I would be doing IT support on stage instead of showing an architecture.

Related Posts

This is the direct continuation of the Azure IoT Operations camera series. The existing control-plane post owns the cameras/# contract; this post uses that contract as the AI workload.

References

• Azure IoT Operations overview
• Azure IoT Operations built-in local MQTT broker
• Process and route data with data flows
• Configure the connector for ONVIF

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

Private data is the easiest reason to care about edge AI. If the data cannot leave the site, then the answer cannot depend on a cloud fallback that nobody noticed.

Demo 2 extends the platform with a Knowledge Assistant. In the design, it ingests local documents, builds embeddings locally when available, retrieves relevant chunks, generates grounded answers, and sends every generation request through the same routing policy engine from Demo 1. The RAG part is nearly conventional. The part I actually care about is that the privacy promise gets enforced by routing instead of by hope.

Where most RAG demos leak

RAG demos tend to blur the privacy boundary in the same comfortable way: the documents are local, but answer generation quietly calls a cloud model. For public docs, nobody gets hurt. For camera runbooks, incident notes, network diagrams, customer data, or site-specific operational history, that silent hop is the whole ballgame.

So the privacy promise has to be testable:

• What classification was assigned to the document?
• Which chunks were retrieved?
• Was the prompt body logged?
• Which backend generated the answer?
• What would happen if the local backend failed?

Each of those answers has to be visible in the demo itself, or the demo is proving nothing.

Four jobs and a deliberately boring vector store

AiOnTheEdge.KnowledgeAssistant has four jobs:

1. Ingest documents with metadata and classification.
2. Chunk and embed the documents.
3. Retrieve relevant context for a question.
4. Ask the gateway for an answer under an explicit policy.

The first version should keep storage boring: one local vector store, wrapped behind an interface, with the retrieval trace visible. The citations come from retrieval metadata, not from model-generated prose - if the model is writing its own bibliography, the citations are fiction with good formatting.

A seeded document looks like this:

{
  "documentId": "garage-camera-runbook",
  "title": "Garage Camera Runbook, Remote Property 1",
  "sourcePath": "samples/documents/garage-camera-runbook.md",
  "classification": "Restricted",
  "tags": ["iot", "cameras", "runbook"],
  "createdUtc": "2026-05-11T09:00:00Z"
}

One honesty note before anyone asks for the dataset: the documents, queries, and responses in this post are synthetic samples rather than captured traffic. Site ids follow the camera series naming - remote1 and remote2 are the remote properties in the Part 1 network model, and garage-east is the Class B agent camera configured on remote1 in Part 2.

The query shape makes policy explicit:

{
  "question": "What should I check first if the remote1 garage camera is offline?",
  "classification": "Restricted",
  "policy": "LocalOnly",
  "topK": 5,
  "includeCitations": true
}

The response carries everything those five testability questions need: the answer, citations, route decision, privacy decision, and retrieval trace:

{
  "answer": "Start with the remote1 site network path before rebooting individual cameras...",
  "citations": [
    {
      "documentId": "garage-camera-runbook",
      "heading": "Offline camera checklist",
      "score": 0.86
    }
  ],
  "routeDecision": {
    "policy": "LocalOnly",
    "selectedBackend": "foundry-local",
    "fallbackUsed": false
  },
  "privacyDecision": {
    "promptBodyLogged": false,
    "reason": "Restricted content suppresses prompt body logging."
  }
}

Foundry Local is a good fit for the laptop version because Microsoft documents local embedding generation and RAG-style workflows that run on device. The implementation still supports mock embeddings, because the demo has to run even before every model is cached.

Running the denial on purpose

What the audience watches for is a denied fallback.

1. Show the document library: camera runbooks, deployment notes, incident notes, and the existing camera architecture posts.
2. Ask: Which cameras are offline, what probably caused it, and what should the operator check first?
3. Show retrieved chunks and citation metadata.
4. Show the grounded answer.
5. Open the route trace: classification Restricted, policy LocalOnly, selected local backend.
6. Disable the local backend.
7. Ask again.
8. Show the request fails closed because cloud fallback is not allowed.
9. Change the classification to Internal.
10. Ask again with a policy that allows fallback.
11. Show cloud or mock-cloud fallback only after the policy changes.

That last step is the point of the whole demo: the privacy boundary is enforced by the platform, not by hope.

Seed documents I already own

The existing posts pull double duty as seed documents:

• The Tenstorrent private cloud buildout gives hardware and environment context.
• The Azure IoT Operations camera posts give camera topology, MQTT topics, broker security, data-flow routing, and connector notes.

Those posts get ingested as samples, but this demo does not retell them. It proves that private operational knowledge can be queried locally with citations.

The new build is AiOnTheEdge.KnowledgeAssistant:

POST /documents/ingest
GET  /documents
POST /query
POST /query/explain
POST /admin/reindex
POST /admin/demo/seed-rag

Implementation requirements:

• Ingest Markdown, PDF text, JSON, CSV, and plain text.
• Chunk by heading, paragraph, and token budget.
• Preserve source filename, heading path, and classification.
• Use Foundry Local embeddings where available.
• Support local or mock embeddings for fallback.
• Generate citations from retrieval metadata.
• Send answer generation through the AI on the Edge Gateway.
• Evaluate known questions with expected citations.

The outage that proves the boundary

The critical failure for this design is a local model outage while the data is restricted.

Cloud fallback is not a recovery path for restricted RAG. The system returns a clear denial instead:

{
  "decision": "Denied",
  "policy": "LocalOnly",
  "reason": "Restricted content must stay on edge and no eligible local backend is healthy."
}

The UI shows the retrieval trace and the route denial side by side, which makes the behavior legible: the system found useful context and correctly refused to send it to an ineligible backend.

What done looks like

Demo 2 earns its number when:

1. Sample documents can be ingested with metadata and classification.
2. Queries return grounded answers with deterministic citations.
3. Restricted content routes only to local or edge backends.
4. Cloud fallback is visibly denied for LocalOnly.
5. Retrieval traces are visible in the dashboard.
6. An evaluation script can run 10 known questions and produce pass/fail results.
7. The demo runs without internet after models and packages are pre-cached.

Related Posts

This demo consumes the existing camera and lab posts as private knowledge sources, and it depends on the gateway from One App, Many Places to Run AI, because the RAG assistant should not own fallback policy itself. The moment a component owns fallback policy, it starts making exceptions.

References

• Tutorial: Build a RAG application with Foundry Local
• Generate text embeddings with Foundry Local
• What is Foundry Local?

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

One application asks one question and gets one useful answer. Where the model actually ran - the laptop, an edge cluster, a cloud endpoint, a Tenstorrent-backed server, or a mock backend - is none of the application’s business. Keeping that placement choice invisible to the app is the entire job of the AI on the Edge Gateway, and this first demo exists to make that abstraction obvious end to end.

Why hardcoded endpoints stop scaling

Hardcoding model endpoints is fine for a prototype and bad for a platform. Once an application knows too much about the model runtime, every placement decision turns into an app change:

• Moving from a laptop runtime to an edge server changes code.
• Adding a cloud fallback changes code.
• Testing a specialized accelerator changes code.
• Denying cloud fallback for restricted data becomes app-specific retry logic.

The app should express what it needs. The platform should decide where that request is allowed to run.

What the gateway looks like

On the outside, the API surface is deliberately boring - it is the surface applications already speak:

POST /v1/chat/completions
POST /v1/embeddings
GET  /v1/models
GET  /healthz
GET  /readyz
GET  /metrics
POST /admin/route/explain
POST /admin/backends/{backendId}/enable
POST /admin/backends/{backendId}/disable
POST /admin/demo/faults

Routing those endpoints is a backend registry:

{
  "Backends": [
    {
      "id": "foundry-local",
      "kind": "FoundryLocal",
      "baseUrl": "http://localhost:5273/v1",
      "models": ["local-chat", "local-embed"],
      "location": "device",
      "capabilities": ["chat", "embeddings", "streaming"],
      "tags": ["local", "offline", "private"],
      "priority": 10
    },
    {
      "id": "azure-foundry",
      "kind": "OpenAICompatible",
      "baseUrl": "https://example.services.ai.azure.com/openai/v1",
      "apiKeySecretName": "azure-foundry-key",
      "models": ["cloud-chat"],
      "location": "cloud",
      "capabilities": ["chat", "streaming"],
      "tags": ["cloud", "managed"],
      "priority": 50
    },
    {
      "id": "mock-accelerator",
      "kind": "OpenAICompatible",
      "baseUrl": "http://localhost:5199/v1",
      "models": ["accelerator-chat"],
      "location": "edge",
      "capabilities": ["chat", "streaming"],
      "tags": ["local", "accelerator", "specialized-hardware"],
      "priority": 20
    },
    {
      "id": "mock-cloud",
      "kind": "OpenAICompatible",
      "baseUrl": "http://localhost:5188/v1",
      "models": ["cloud-chat"],
      "location": "cloud",
      "capabilities": ["chat", "streaming"],
      "tags": ["cloud", "mock"],
      "priority": 40
    }
  ]
}

Every backend gets normalized into the same internal shape: health, supported model aliases, capabilities, location, tags, priority, observed latency, and current fault state. From there on, the router does not care whether a backend is a laptop runtime or a rack in the basement.

Then comes my favorite line in the whole codebase: the router evaluates policy before it evaluates convenience.

The policies are small enough to keep in your head:

Foundry Local belongs in this first demo as the device-local runtime. Microsoft positions it as an on-device AI runtime and SDK, with an optional local server aimed at development and integration scenarios - which is a different job from being the shared multi-user server inference layer for the whole edge estate. For server-style edge inference, the demo keeps Azure Local, other OpenAI-compatible services, and accelerator endpoints behind the same registry.

Walking through the demo

Route explanation carries this demo. This is the sequence I rehearse until it gets boring:

1. Open the AI on the Edge Console.
2. Ask: Summarize the current edge site health and recommend the next action.
3. Show the response streaming through the gateway.
4. Open the route trace and show selectedBackend: foundry-local.
5. Disable foundry-local.
6. Ask again with CloudAllowed.
7. Show fallback to azure-foundry or mock-cloud.
8. Ask again with LocalOnly.
9. Show the request is denied instead of silently falling back to cloud.
10. Re-enable the local backend and show the metrics panel.

The line I want the audience walking out with is simple: same app, same API, different placement decision.

Borrowing the lab instead of rebuilding it

The private cloud lab already supplies the edge environment and the Tenstorrent hardware lane, and the camera series already supplies an operational workload. Neither one needs rebuilding just to prove a gateway. Demo 1 can run entirely with a local runtime and deterministic mock backends.

The new implementation is the gateway foundation:

• OpenAI-compatible request and response normalization.
• Backend registry and health checks.
• Route explanation endpoint.
• Policy evaluation before fallback.
• Prometheus metrics.
• Route decision events.
• Admin controls for enable, disable, and fault injection.
• A dashboard panel for backend health, latency, selected backend, and denial reasons.

Before any of this depends on real cloud or real hardware, the implementation should include a mock local backend, the mock-cloud backend from the registry above, and a mock accelerator. Deterministic mocks are not a shortcut; they are what make the conference demo reliable.

When the local box dies

The first failure worth showing is local backend loss:

And the rule underneath every row of that table: the gateway must never treat “cloud is healthy” as permission to send restricted content there.

What done looks like

Demo 1 earns its number when:

1. One app can call /v1/chat/completions on the gateway.
2. The gateway can route to local, cloud/mock-cloud, and accelerator/mock-accelerator backends.
3. The dashboard shows selected backend, policy, denial reason, fallback reason, and latency.
4. Disabling a backend changes the routing decision.
5. LocalOnly prevents cloud fallback.
6. Metrics include request count, latency, selected backend, fallback count, and denied count.
7. make demo-laptop runs the whole demo without a cloud dependency.

Related Posts

This post is the platform foundation for the private RAG demo, the camera operations assistant, the failure lab, and the accelerator lane that follow. The existing Tenstorrent buildout is the hardware background; the existing camera control-plane post is the workload background. Neither is required reading, but both are why this post stayed short.

References

• What is Foundry Local?
• Best practices and troubleshooting guide for Foundry Local CLI
• What is Foundry Local on Azure Local?
• Endpoints for Microsoft Foundry Models
• Understanding deployment types in Microsoft Foundry Models

AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

1. AI on the Edge: Local AI Without Local Chaos
2. One App, Many Places to Run AI
3. Private RAG That Cannot Leave the Edge
4. From Camera Events to Operator Guidance
5. Edge AI You Can Actually Operate
6. When the Edge Has to Stand Alone
7. Specialized Hardware Without an App Rewrite

Background posts:

• Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos
• Cameras on Azure IoT Operations, Part 1: The Control Plane and Network Model
• Cameras on Azure IoT Operations, Part 2: Swapping in the AIO MQTT Broker
• Cameras on Azure IoT Operations, Part 3: Data Flows, Connectors, and the Cloud

Project page:
AI on the Edge

Years ago a vendor shut down the cloud service behind my home security cameras and left me holding hardware I owned but could no longer use. I salvaged what I could and kept the lesson: anything I actually depend on should run where I can reach it. That instinct is most of why local AI appeals to me - privacy, latency, cost control, disconnected operation, and the simple fact that some data already lives at the edge. The hard part was never getting one model to answer one prompt on one box. The hard part is making local AI behave like a platform instead of a pile of model servers hiding under desks.

That is the point of AI on the Edge. The project is a governable edge AI system where applications call one API, policy decides where inference is allowed to run, Azure provides the management plane, and the local environment keeps working when the network is not perfect.

The tagline is not decoration. It is the operating model:

Cloud-governed, locally executed.

Why local AI goes sideways

Model sprawl never announces itself. A developer points an app at a local runtime. An operations team deploys a different endpoint on a Kubernetes node. A hardware experiment bolts on an accelerator-specific API. A cloud team wants a managed endpoint for approved workloads. Every one of those decisions is reasonable on its own. Stacked together, they become an unmanaged surface:

• Applications hardcode model endpoints.
• Sensitive prompts can fall back to cloud by accident.
• Local model failures are invisible to operations teams.
• Hardware-specific demos turn into one-off branches.
• Edge environments have no consistent reset, smoke test, or dashboard story.

AI on the Edge treats those as platform problems. The model runtime matters, but the project is really about routing, policy, observability, secrets, fallback, repeatability, and deployment.

One gateway, every backend

The center of the system is a reusable .NET gateway that exposes an OpenAI-compatible API to the application. Behind that gateway sit a backend registry and a policy-driven router. Depending on the request, inference can land on a laptop-local runtime, an Azure Local deployment, an Azure AI Foundry model endpoint, a Tenstorrent-backed endpoint, or a deterministic mock backend that exists purely for conference safety.

The important part is that the application never picks the backend directly. It sends the request with metadata, and that metadata is the contract: the workload (chat, embeddings, RAG answer, incident summary), the classification (public, internal, restricted, secret), the policy (local only, prefer local, cloud allowed, accelerator preferred), and the operational context (latency target, streaming requirement, fallback rules).

For every request, the gateway emits a route decision event. That event records the selected backend, the denied backends, the policy, the classification, the fallback behavior, latency, token counts, and whether prompt bodies were logged, redacted, or suppressed. When somebody asks why an answer came from where it did, the answer lives in the telemetry, not in my memory.

Azure provides the control plane wrapped around that local execution:

• Azure Arc brings the edge Kubernetes cluster into Azure management.
• GitOps applies the desired state.
• Azure Monitor, Managed Prometheus, and Grafana make behavior visible.
• Key Vault handles secrets and certificates where the Azure-governed path is active.
• Azure Policy and application policy events make denied routes explicit.
• Azure IoT Operations gives the camera workload a real edge data plane.

How the demos stack up

The series maps straight onto how I am building the talk:

The final two posts turn the demo system into an implementation guide and an evergreen reference architecture.

Standing on work I already published

The private cloud and physical lab are already covered in Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos. That post owns the hardware story: basement studio, private cloud, Tenstorrent paths, NVIDIA systems, Proxmox, TrueNAS, Kubernetes, and multi-cloud demo intent.

The camera control plane is already covered in the three-part Azure IoT Operations series. Those posts own the camera details: outbound MQTT, the cameras/<site>/<camera>/<channel> topic tree, TLS and X.509 on the AIO MQTT broker, data flows to Event Hubs, and the ONVIF connector bridge.

AI on the Edge builds on both instead of repeating either. The lab is the environment, the cameras are the workload, and the new thing is the AI platform that sits between them.

What I am actually building here

The demo system is the new work:

• AiOnTheEdge.Gateway for the OpenAI-compatible facade.
• AiOnTheEdge.Routing for backend selection, fallback, and policy.
• AiOnTheEdge.KnowledgeAssistant for private RAG over local documents.
• AiOnTheEdge.OperationsAssistant for camera event triage.
• AiOnTheEdge.ControlDashboard for health, route traces, privacy posture, and failure controls.
• AiOnTheEdge.DemoScenarios for deterministic seed, reset, replay, and smoke tests.
• Azure and Kubernetes deployment assets for the governed edge path.

One status note, stated here once so the whole series inherits it: these components are part of my private reference implementation for the talk. The build is in progress and the repo is not published, so the posts that follow present API surfaces, registries, and acceptance criteria as the design the demos target - not as downloadable software. Nobody will be cloning their way into this series, and pretending otherwise would just waste your afternoon.

The first version must run in laptop mode with mock or local backends. Azure-governed mode is the richer path, not the live-demo dependency.

The failure that proves the design

The failure this project protects against is quiet policy drift. A local backend goes down, a cloud endpoint happens to be healthy, and a restricted prompt silently leaves the edge because the only trick the application knows is retry.

That is unacceptable behavior for this architecture, full stop. Restricted data must fail closed. Cloud fallback must be a policy decision, not a retry side effect. And the audience should see the denial, the reason, and the telemetry - not take my word for any of it.

What done looks like

By the end of the series, all of this had better be true:

• A reader can understand why local AI still needs governance.
• One application can call one gateway and reach multiple backend types.
• Restricted RAG content never routes to cloud.
• Camera telemetry can drive an operator assistant without reworking the existing MQTT contract.
• Azure operations can see routing, failures, policy denials, and backend health.
• The demo runs in laptop mode with no cloud dependency yet can still be reset, seeded, and smoke-tested before a session.

Related Posts

Start with the Tenstorrent buildout if you want the hardware context. Start with the Azure IoT Operations camera series if you want the workload context. Start here if you want the AI platform and presentation system. Everything after this post either goes through the gateway or fails because of it.

References

• What is Foundry Local?
• Best practices and troubleshooting guide for Foundry Local CLI
• What is Foundry Local on Azure Local?
• Azure Arc-enabled Kubernetes overview
• Azure IoT Operations overview
• Tenstorrent tools documentation

• Part 1 (this post): the control plane and network model - what the system is, how the cameras work, and how the networks work.
• Part 2: swapping in the Azure IoT Operations MQTT broker - TLS, X.509 camera identity, and topic authorization on Arc-enabled Kubernetes.
• Part 3: data flows, connectors, and the cloud - forwarding telemetry to Event Hubs and onboarding ONVIF cameras as AIO assets.

I built a central camera control plane in .NET 10, then made Azure IoT Operations an opt-in broker and edge data plane for it. The interesting part of that story is that AIO does not replace the design - it slots into an existing, opinionated one. So before the AIO specifics in Parts 2 and 3, this post covers the foundation AIO plugs into: the control plane, the two camera integration classes, the MQTT wire contract, and the network model.

Why a control plane, not a pile of cameras

The usual way people “network” cameras is to put a pile of cheap IP cameras on a LAN, port-forward an NVR, and hope. That model rots: every camera is a little server with a web UI, a Telnet port, and firmware from 2017, and every one is an inbound attack surface.

The control-plane model inverts that. A camera is treated like a managed IoT node:

• It connects outbound to an MQTT broker.
• It publishes its status, inventory, and events.
• It subscribes for commands from a central server.

There is no inbound SSH or Telnet to a camera in normal operation. The central server owns identity, configuration, status, commands, updates, health, and stream registration. The camera (or a per-site gateway) owns capture, health checks, watchdog behavior, and safe command execution. That split is the whole point: the control surface is a closed message contract, not a shell.

The .NET solution behind this is a standard clean-architecture layout (.NET 10 + Aspire): CameraNetwork.Contracts holds the wire types, CameraNetwork.Controller is the Blazor Server dashboard + REST API + /metrics, and two lightweight Worker services run at the edge - CameraNetwork.Agent and CameraNetwork.SiteGateway. Those two are how the fleet splits into two integration classes. The solution is part of my private reference implementation and is not published, so treat these project names as documentation of the design rather than pointers to a cloneable repo.

Two kinds of cameras, one control plane

Real fleets are never homogeneous. Some cameras can run our code; most cannot. The design absorbs that with two classes that look identical to the control plane:

• Class B - agent-managed. The camera runs our sidecar, the izon-agent (CameraNetwork.Agent), and speaks MQTT directly. This is for cameras you can get a process onto: iZON, Axis ACAP, OpenIPC, Thingino. The agent reports health, accepts the closed command set, watches the RTSP stream, and updates itself.
• Class A - gateway-managed. A plain RTSP/ONVIF PoE camera (Amcrest, Reolink, TP-Link VIGI, and similar) that cannot run our code. A per-site CameraNetwork.SiteGateway speaks for it: it probes the camera and publishes availability, inventory, and status on the camera’s behalf, so a “dumb” camera shows up in the dashboard like any other.

The Class A probe is real network work, not a TCP ping. It does an ICMP reachability check, an RTSP OPTIONS handshake, an ONVIF query (make, model, firmware, MAC, stream and snapshot URLs, plus WS-Discovery), and a validated JPEG snapshot pull. That probing sits behind an ICameraProbe seam, with a NetworkCameraProbe for real cameras and a SimulatedCameraProbe for dev and CI.

The payoff: the controller code, the dashboard, the API, and the metrics never branch on camera class. A Class A camera fronted by a gateway and a Class B camera running the agent produce the same records. That uniformity is exactly what lets Part 3 add a third producer - an Azure IoT Operations connector - without the controller noticing.

The wire contract: one MQTT topic tree

Everything rides on one topic tree, rooted at cameras and shaped cameras/<site>/<camera>/<channel>. The controller subscribes to cameras/#; an agent publishes its own channels and subscribes only to its own cmd topic. The contract lives in code in CameraNetwork.Contracts, so the controller and the agents cannot drift apart.

Two design choices matter here. First, availability is retained and backed by an MQTT Last-Will: the agent sets a retained Last-Will of {"state":"offline"} on connect, so an unexpected drop flips the camera offline with no active reporting. Second, the command set is closed. An agent rejects anything outside this list, so the channel can never become arbitrary remote code execution across the fleet:

restart_rtsp   restart_network   reboot       identify
privacy_on     privacy_off       snapshot     get_status
get_logs       apply_config      update_agent rollback_agent

Acks come back as accepted (non-terminal), succeeded, failed, or unsupported. The closed set is the security model for the command channel - it is the reason a compromised broker session cannot tell a camera to do something arbitrary.

The controller reasons in terms of desired state versus reported state. An operator sets a desired agent version and config; the controller hashes the config into a stable 16-character DesiredConfigHash. Pushing apply_config carries that config and hash to the agent, which applies it and echoes the hash back in its status as ReportedConfigHash. Config drift is then just DesiredConfigHash != ReportedConfigHash, and an outdated agent is DesiredAgentVersion != ReportedAgentVersion. Both surface on the dashboard and in /metrics.

This contract is the seam that makes the rest of the series possible. Because it is single-sourced and broker-agnostic, the broker underneath it can change without touching either end - which is exactly what Part 2 does.

The network model

The control plane assumes unique routed subnets per property over a site-to-site routed VPN (WireGuard or pfSense), not client NAT. You do not overlap 192.168.1.0/24 everywhere; each property gets its own space and the central peer holds routes to each remote camera subnet.

Central property:   LAN 10.10.0.0/16   Server VLAN 10.10.10.0/24   Camera VLAN 10.10.30.0/24
Remote property 1:  LAN 10.20.0.0/16                               Camera VLAN 10.20.30.0/24
Remote property 2:  LAN 10.30.0.0/16                               Camera VLAN 10.30.30.0/24
VPN transit:        10.255.0.0/24

camera-control.internal   10.10.10.20      mqtt.camera.internal   10.10.10.20

The camera VLANs are default-deny. Treat every old camera as compromised until proven otherwise, and only allow what is required:

The important property falls out of this directly: control commands never require inbound SSH or Telnet to a camera. The agent (or gateway) holds an outbound MQTT connection, and the controller publishes commands the camera receives over that already-open connection. Cameras get no Internet, no camera-to-camera traffic, and no path to the normal LAN. Vendor cloud domains stay blocked; old web and Telnet services stay firewalled.

Procurement note. For client-, business-, or government-adjacent work, avoid Hikvision and Dahua (FCC Covered List) and prefer Axis, Hanwha, Bosch, i-PRO, TP-Link VIGI, Amcrest, Reolink, or UniFi.

Where Azure IoT Operations comes in

Notice what the broker actually is in all of this: a trusted MQTT bus on a private routed VPN. The default deployment uses Eclipse Mosquitto, hardened with per-camera username/password and ACLs and reachable only from the controller’s IP. That works, and the rest of the series leaves it as the byte-identical default.

But the broker is also a seam. The agents, the gateway, and the controller all connect through one shared set of connection options, and the topics and JSON payloads are defined once in CameraNetwork.Contracts. That means the broker can be swapped for something with a richer edge story without changing a single line of camera or controller logic - and Azure IoT Operations is exactly that something:

• AIO ships an enterprise-grade MQTT broker that runs on Arc-enabled Kubernetes at the edge. Part 2 swaps Mosquitto for it, with TLS and per-camera X.509 identity, and shows how one authorization rule reproduces the entire Mosquitto ACL.
• AIO ships data flows that route from that broker to the cloud. Part 3 forwards the same cameras/# tree to Azure Event Hubs with no application change, then onboards ONVIF cameras as AIO assets through a connector and a small bridge.

The key idea, and the reason this is worth three posts: AIO is introduced as an opt-in parallel path. The control plane, the two camera classes, the wire contract, and the network model in this post all stay exactly as described. AIO changes what sits under the contract and what happens after the message reaches the broker - not the contract itself.

Continue to Part 2: swapping in the Azure IoT Operations MQTT broker.

References

• Azure IoT Operations overview
• Azure IoT Operations MQTT broker overview
• Azure Arc-enabled Kubernetes overview
• MQTT v5 specification
• ONVIF specifications
• FCC Covered List

• Part 1: the control plane and network model - what the system is and how the cameras and networks work.
• Part 2 (this post): swapping in the Azure IoT Operations MQTT broker - TLS, X.509 camera identity, and topic authorization.
• Part 3: data flows, connectors, and the cloud - forwarding telemetry to Event Hubs and onboarding ONVIF cameras as AIO assets.

In Part 1 the broker was just “a trusted MQTT bus” - Eclipse Mosquitto by default. This post replaces it with the Azure IoT Operations MQTT broker without touching a single line of camera or controller logic, and gets stronger security for free in the process. The whole point of AIO here is that it gives you an enterprise MQTT broker that runs on Kubernetes at the edge, with cloud-managed identity, authorization, and (in Part 3) data flows - while the application keeps speaking the exact same topics and payloads.

The seam that makes the swap possible

The control plane has three MQTT clients: the controller’s ingest service, the Class B agent, and the Class A site gateway. All three connect through one shared project, CameraNetwork.Mqtt, which exposes a MqttConnectionOptions record and a single extension method:

// Every client builds its options the same way:
var options = new MqttClientOptionsBuilder()
    .WithClientId(clientId)
    .Apply(connection)   // CameraNetwork.Mqtt resolves profile -> TLS / MQTT v5 / auth
    .Build();

MqttConnectionOptions carries a profile, an auth mode, and the TLS and credential details. There are two profiles:

• Mosquitto (the default) - plain TCP on 1883, username/password, MQTT 3.1.1. Nothing in Part 1 changes.
• AzureIotOperations - turns on TLS and MQTT v5 automatically, with Auth=X509 for clients outside the cluster or Auth=Sat (Kubernetes service-account token) for in-cluster components.

Because the profile resolves the transport details, swapping brokers is a configuration change, not a code change. The MQTT library already does v5, TLS, and X.509 client certificates - there is no new client dependency to swap a fleet onto AIO.

Standing up Azure IoT Operations

AIO runs on an Azure Arc-enabled Kubernetes cluster. For a sandbox that is a single-node k3s box; in production it is whatever Arc-enabled cluster you run at the edge. The bring-up is a sequence of az commands (wrapped in a setup script in the private repo behind this series, not published here), and it is deliberately a parallel sandbox - it never touches the existing Docker Compose or TrueNAS deployment, which keep using Mosquitto.

The command shapes below match the AIO CLI as of June 2026. The schema-registry prerequisite in particular is version-sensitive, so re-check the deployment doc if az iot ops create rejects any arguments.

# Arc-connect the cluster and enable the features AIO needs
az connectedk8s connect --name cameranetwork-k3s --resource-group cameranetwork-aio --location eastus
az connectedk8s enable-features --name cameranetwork-k3s --resource-group cameranetwork-aio \
  --custom-locations-oid "$CL_OID" --features cluster-connect custom-locations

# A storage account + schema registry are required before 'az iot ops create'
az iot ops schema registry create --name cameranetwork-sr --resource-group cameranetwork-aio \
  --registry-namespace cameranetwork-sr-ns --sa-resource-id "$SA_ID"

# Initialize and create the AIO instance (this is the broker, dataflows, and the rest)
az iot ops init   --cluster cameranetwork-k3s --resource-group cameranetwork-aio
az iot ops create --cluster cameranetwork-k3s --resource-group cameranetwork-aio \
  --name cameranetwork-aio --sr-resource-id "$SR_ID"

When this finishes you have an AIO instance with an MQTT broker. AIO installs a default in-cluster listener (aio-broker:18883, TLS + service-account-token auth) for its own components. We leave that one alone and add a second listener for the cameras, which live outside the cluster.

A listener for cameras outside the cluster

Cameras connect from remote properties over the VPN, so they need a listener exposed off the cluster. On k3s, a LoadBalancer service picks up the node IP out of the box. The listener terminates TLS and references the X.509 authentication and authorization policies we define next.

apiVersion: mqttbroker.iotoperations.azure.com/v1
kind: BrokerListener
metadata:
  name: camera-external
  namespace: azure-iot-operations
spec:
  brokerRef: default
  serviceType: LoadBalancer
  serviceName: aio-broker-external   # do not clash with the default 'aio-broker' service
  ports:
    - port: 8883
      protocol: Mqtt
      authenticationRef: camera-x509-authn
      authorizationRef: camera-authz
      tls:
        mode: Automatic
        certManagerCertificateSpec:
          issuerRef:
            name: azure-iot-operations-aio-certificate-issuer   # the issuer AIO installs by default
            kind: ClusterIssuer
            group: cert-manager.io

tls.mode: Automatic lets cert-manager mint the listener’s server certificate from AIO’s default cluster issuer. That is fine for a sandbox; to tighten it you add the external IP or DNS name as a SAN so clients can do strict server-cert validation instead of trusting the CA directly.

Per-camera identity with X.509

This is where AIO earns its keep over plain Mosquitto. Each camera presents a client certificate, and the broker validates it against a CA we control. The repo’s cert helper generates a CA and a per-camera client cert whose common name equals the MQTT client id:

./make-camera-cert.sh izon-remote1-garage-east     # Class B agent
./make-camera-cert.sh site-gateway-remote2         # Class A gateway

The CA certificate is imported into the cluster as a ConfigMap, and a BrokerAuthentication resource trusts it. The clever bit is the root-subject mapping: every certificate signed by our CA inherits the attribute role: camera, so the entire fleet shares one identity policy with no per-device wiring.

apiVersion: mqttbroker.iotoperations.azure.com/v1
kind: BrokerAuthentication
metadata:
  name: camera-x509-authn
  namespace: azure-iot-operations
spec:
  authenticationMethods:
    - method: X509
      x509Settings:
        trustedClientCaCert: camera-client-ca
        authorizationAttributes:
          cameras:
            subject: CN = CameraNetwork Camera CA   # must match the CA subject exactly
            attributes:
              role: camera

One rule that reproduces the whole ACL

With every camera certificate carrying role: camera, authorization is a single allow rule. AIO authorization policies are allow-only - anything not granted is denied - so this one rule lets the whole fleet connect and use the cameras/# topic tree, which is exactly the wire contract from Part 1.

apiVersion: mqttbroker.iotoperations.azure.com/v1
kind: BrokerAuthorization
metadata:
  name: camera-authz
  namespace: azure-iot-operations
spec:
  authorizationPolicies:
    cache: Enabled
    rules:
      - principals:
          attributes:
            - role: camera
        brokerResources:
          - method: Connect
          - method: Publish
            topics: [ "cameras/#" ]
          - method: Subscribe
            topics: [ "cameras/#" ]

This is the AIO equivalent of the Mosquitto ACL. On Mosquitto you hand every camera a username/password and an ACL file entry; here you hand every camera a certificate from one CA and write one attribute-based rule. The trust anchor moves from a shared secret to a certificate authority, which is a real upgrade: revoking a camera is a CRL or re-issue operation, not a password edit pushed to a config file.

If you later want per-camera isolation - a camera that can only touch its own channels - you give each certificate site and camera attributes in the authentication resource, then scope the authorization topics by token substitution, e.g. cameras/{principal.attributes.site}/{principal.attributes.camera}/+. That mirrors a per-camera Mosquitto ACL, but it is optional; the single fleet-wide rule is the starting point.

Pointing a component at AIO

Nothing about the swap is code. You flip the profile and point at the external listener. A Class B agent’s settings:

{
  "Agent": {
    "SiteId": "remote1",
    "CameraId": "garage-east",
    "MqttProfile": "AzureIotOperations",
    "MqttHost": "<EXTERNAL-IP>",
    "MqttPort": 8883,
    "MqttAuth": "X509",
    "MqttClientCertPfxPath": "/certs/izon-remote1-garage-east.pfx",
    "MqttCaCertPath": "/certs/camera-ca.crt",
    "MqttAllowUntrustedCertificates": true
  }
}

The controller is the same idea with its own config prefix (environment variables shown):

Mqtt__Profile=AzureIotOperations
Mqtt__Host=<EXTERNAL-IP>
Mqtt__Port=8883
Mqtt__Auth=X509
Mqtt__ClientCertPfxPath=/certs/camera-controller.pfx
Mqtt__CaCertPath=/certs/camera-ca.crt

MqttProfile=AzureIotOperations turns on TLS and MQTT v5 automatically. MqttAllowUntrustedCertificates=true skips server-cert validation and is a sandbox-only shortcut - drop it once the listener’s server cert carries the external IP as a SAN and rely on the CA file instead. The controller can also run inside the cluster with Auth=Sat and Host=aio-broker, using the default internal listener.

Verify the fleet landed on the broker

Check the broker resources, then watch a real camera connect over TLS with its client certificate:

# Broker resources are healthy
kubectl get brokerlistener,brokerauthentication,brokerauthorization -n azure-iot-operations
az iot ops check

# Subscribe from your workstation with the CA + a camera cert/key
mosquitto_sub -h <EXTERNAL-IP> -p 8883 -V mqttv5 -t 'cameras/#' -v \
  --cafile certs/camera-ca.crt \
  --cert certs/izon-remote1-garage-east.crt \
  --key certs/izon-remote1-garage-east.key

Start the agent with the AIO config and you see its retained availability and inventory, then periodic status. Point the controller at the same listener and the camera appears on the dashboard exactly as it did on Mosquitto. Kill the agent and its availability flips to offline through the Last-Will - the retained-message and Last-Will-Testament behavior carries across brokers, which is the real test that the swap is transparent.

Why this matters

The Mosquitto deployment trusts the private VPN and VLAN: the broker is never exposed to untrusted networks, hardening is per-camera username/password plus ACLs, and the firewall makes port 1883 reachable only from the controller. That is a perfectly good model on a trusted segment.

Moving to the AIO broker upgrades the trust model without changing the application:

• Identity is a certificate from a CA you control, not a shared secret.
• TLS is on by the profile, so cameras can cross less-trusted segments.
• Authorization is attribute-based and cloud-managed, with one rule for the fleet and an obvious path to per-camera scoping.
• The broker is now a managed Kubernetes workload with health you can query (az iot ops check), not a single container.

And critically, the message contract from Part 1 is untouched. That is what sets up Part 3: once the fleet is publishing to the AIO broker, AIO can route that traffic to the cloud and onboard new kinds of cameras as assets - again with no application change.

Continue to Part 3: data flows, connectors, and the cloud.

References

• Azure IoT Operations MQTT broker overview
• Configure broker listeners in Azure IoT Operations
• Configure broker authentication (X.509)
• Configure broker authorization
• Deploy Azure IoT Operations to an Arc-enabled Kubernetes cluster
• Azure Arc-enabled Kubernetes overview

• Part 1: the control plane and network model - what the system is and how the cameras and networks work.
• Part 2: swapping in the Azure IoT Operations MQTT broker - TLS, X.509 camera identity, and topic authorization.
• Part 3 (this post): data flows, connectors, and the cloud - forwarding telemetry to Event Hubs and onboarding ONVIF cameras as AIO assets.

Part 2 put the fleet on the Azure IoT Operations MQTT broker. Now the broker is no longer just a message bus - it is an edge data plane, and two AIO features turn that into real leverage: data flows route telemetry to the cloud, and connectors bring in cameras that cannot speak our protocol at all. Both land without changing the control plane, which is the recurring theme of this series.

Forwarding telemetry to the cloud, for free

In Part 1, every producer publishes to one topic tree: cameras/<site>/<camera>/<channel>. That single fact makes cloud egress almost trivial. An AIO data flow has a source and a destination; point the source at the local broker on the cameras/# tree - the exact tree everything already uses - and point the destination at Azure. No producer changes, no new topics, nothing republished.

The destination here is Azure Event Hubs, addressed through its Kafka surface, authenticated with the AIO instance’s managed identity (no connection strings on disk):

apiVersion: connectivity.iotoperations.azure.com/v1
kind: DataflowEndpoint
metadata:
  name: cameranetwork-eventhub
  namespace: azure-iot-operations
spec:
  endpointType: Kafka
  kafkaSettings:
    host: "<EVENTHUB_NAMESPACE>.servicebus.windows.net:9093"
    authentication:
      method: SystemAssignedManagedIdentity
      systemAssignedManagedIdentitySettings: {}
    tls:
      mode: Enabled

The data flow itself wires the local broker to that endpoint. The source endpointRef: default is AIO’s built-in local MQTT broker (aio-broker), and one side of every data flow must be that local broker:

apiVersion: connectivity.iotoperations.azure.com/v1
kind: Dataflow
metadata:
  name: cameras-to-eventhub
  namespace: azure-iot-operations
spec:
  profileRef: default
  mode: Enabled
  operations:
    - operationType: Source
      sourceSettings:
        endpointRef: default          # the local broker (host aio-broker)
        dataSources: [ "cameras/#" ]   # the same tree from Part 1
    - operationType: Destination
      destinationSettings:
        endpointRef: cameranetwork-eventhub
        dataDestination: camera-telemetry   # the Event Hub / Kafka topic name

Before applying it, you create the Event Hubs namespace and hub and grant the AIO managed identity the Azure Event Hubs Data Sender role - that role grant is what the managed-identity auth above relies on. Telemetry is forwarded as JSON pass-through, so no schema registry is needed; the payloads landing in Event Hubs are the unchanged CameraNetwork.Contracts JSON from Part 1.

As agents heartbeat, the Event Hubs “incoming messages” graph rises, carrying availability, inventory, status, and events for the whole fleet. From there, Fabric Real-Time Intelligence or Azure Data Explorer are the natural next stops for dashboards and historical analytics; those destinations add a schema-registry reference, but the edge side stays exactly as shown. The reason this is so cheap is structural: because Part 1 single-sourced the topic tree, cloud egress is one data flow over cameras/#, not an integration per producer.

Bringing in cameras that cannot speak the protocol

Data flows handle the outbound story. The inbound story is connectors. AIO ships an ONVIF connector and a media connector that can talk to standards-based cameras directly - discover them, model them as assets, and publish their telemetry into the broker. That is a different path from the Class A site gateway in Part 1, and it is interesting precisely because it lets AIO itself onboard a camera.

The connectors are preview (per the Microsoft Learn docs linked at the bottom of this post, as retrieved in June 2026) and their custom resources are version- and install-specific, so they are deployed through the supported tooling rather than checked-in YAML. The flow is:

1. Register the camera as an Azure Device Registry device (its ONVIF endpoint plus credentials).
2. Let discovery enumerate the camera’s capabilities and profiles.
3. Create assets for the streams and snapshots you care about.

At that point the connector is publishing asset telemetry into the AIO broker - but on its topic and in its shape, not on the cameras/<site>/<camera>/... contract the controller understands. Something has to translate. That something is a small bridge.

The AioBridge: a gateway whose probe is a connector

CameraNetwork.AioBridge is an in-cluster Worker service. Conceptually it is a site gateway whose probe is the AIO connector instead of a direct ONVIF query. It subscribes to the connector’s telemetry, maps each asset to a (site, camera) pair, and republishes onto cameras/<site>/<camera>/... using the same CameraJson serializer the rest of the fleet uses. To the controller, the result is indistinguishable from a Class A camera behind a gateway - which is the whole reason Part 1 insisted the controller never branch on camera class.

The bridge connects to AIO’s internal listener (aio-broker:18883, TLS plus a Kubernetes service-account token), so it never leaves the cluster. It gets its SAT through a projected volume and trusts the broker via the AIO CA bundle:

# 50-aiobridge-deployment.yaml (trimmed)
containers:
  - name: aio-bridge
    # image lives in a private registry; substitute your own build of the bridge
    image: <your-registry>/cameranetwork-aiobridge:0.1.0
    env:
      - { name: AioBridge__Host,  value: "aio-broker" }
      - { name: AioBridge__Port,  value: "18883" }
      - { name: AioBridge__UseTls, value: "true" }
      - { name: AioBridge__CaFile, value: "/var/run/certs/ca.crt" }
      - { name: AioBridge__SatAuthFile, value: "/var/run/secrets/tokens/broker-sat" }
      - { name: AioBridge__ConnectorTelemetryTopic, value: "azure-iot-operations/data/#" }
      - { name: AioBridge__DefaultSiteId, value: "aio-edge" }
volumes:
  - name: broker-sat
    projected:
      sources:
        - serviceAccountToken: { path: broker-sat, audience: aio-internal, expirationSeconds: 86400 }

Asset-to-camera identity is explicit so cameras do not collide: the preferred path is custom attributes on each asset (cameraNetwork.site and cameraNetwork.camera), with a fallback of a default site id plus a slug of the asset name. Connectors are preview, so the two things most likely to differ between installs - the connector’s telemetry topic and its payload field names - are isolated in one class, ConfigurableAssetTelemetryMapper, and covered by tests in the private reference implementation. Like the rest of the camera solution, the bridge itself is not published, so treat that class and its tests as design detail rather than downloadable code. If your connector publishes elsewhere or names fields differently, you adjust that single mapper, not the controller.

Commands back to connector-fronted cameras

A camera that only reports is half a control plane. The bridge also answers commands, behind a toggle (AioBridge:HandleCommands, on by default). It subscribes to cameras/+/+/cmd and answers only for cameras it has actually seen - it tracks the assets it has observed, so it never steals a command meant for a Class B agent that happens to share the topic space.

Command handling sits behind an IBridgeCommandExecutor seam, mirroring the site gateway’s policy. The default executor acks get_status as succeeded and acks everything else - including snapshot - as unsupported, until you plug in a real executor backed by media-connector capture or ONVIF control. The ack comes back on cameras/<site>/<camera>/cmd_ack, the same channel from Part 1. In the dashboard, you click get-status on an AIO camera and watch the cmd_ack arrive, exactly as you would for an agent-managed camera.

This is the closed command set from Part 1 doing its job again: the bridge cannot be told to do anything arbitrary, only to attempt actions from the fixed vocabulary, and it honestly reports unsupported for the ones it cannot yet perform.

The whole picture

Put the three posts together and the shape is a single contract with interchangeable parts underneath it:

Three different kinds of cameras, one wire contract, one broker, one data flow to the cloud - and the controller treats all of them identically. Azure IoT Operations did not reshape the application; it slotted in underneath the contract (the broker, Part 2) and alongside it (the connectors and data flows, Part 3). That is the argument for AIO in a camera control plane: it is an enterprise-grade edge MQTT broker and cloud data plane that you can adopt incrementally, behind a seam, without rewriting the system that already works.

Start at Part 1 if you came in here first - the contract and the network model are what make all of this hold together.

References

• Azure IoT Operations data flows overview
• Configure a data flow endpoint for Azure Event Hubs
• Connector for ONVIF (preview)
• Media connector (preview)
• Azure Device Registry overview
• Send data to Microsoft Fabric Real-Time Intelligence