I build & scalethe infrastructureAI platformsrun on.
I’m Sunil, Founding Lead DevOps & MLOps Engineer at ORGN. For 9+ years I have built and run the platforms AI products depend on, from multi-cloud Kubernetes and confidential GPU compute to secure delivery and cost governance, so that what runs on them never has to think about them.
$kubectl diagnose -n kd-liveness-wrong-port1 problem: 1 warning.1.WARNINGLiveness probe kills the containerKD112 Deployment kd-liveness-wrong-port/webCause: container web fails its liveness probe (GET / on port 9090, but the container declares port 8080), so the kubelet restarts it (5 restarts)Evidence: [1] Pod …/web-67fbcb44db-snwhf: event Unhealthy (6 times)│ Liveness probe failed: … dial tcp 10.244.0.26:9090: connect: connection refused$ kubectl get events -n kd-liveness-wrong-port --field-selector … [2] Pod …/web-67fbcb44db-snwhf: event Killing (6 times)│ Container web failed liveness probe, will be restartedFix: Point the probe at a port and path the app serves, or give it a startupProbe or longer initialDelaySeconds if it starts slowly.
$gpucost scan --prometheus http://localhost:9090 --prices prices.csvGPU fleet 3 nodes · 9 GPU pods · 481.7 GPU-hours offered · 421.6 requested (87.5%) GPU activity 37.2%, over the 100.0% of requested GPU-hours DCGM measured Recoverable waste $750.64 · 221.4 GPU-hours · 7 findingsFindingsHIGHGPU005 More replicas than the peak load needed · 96.3 GPU-hours · $385.33HIGHGPU004 GPUs no pod requested · 60.1 GPU-hours · $240.33HIGHGPU002 Model served no traffic · 24.1 GPU-hours · $16.86HIGHGPU001 GPUs held but never used · 24.1 GPU-hours · $96.33MEDIUMGPU003 Engine asleep while holding GPUs · 16.8 GPU-hours · $11.78 LOW GPU006 Serving pod busy for a small share of its time · 21.0 idle GPU-hoursModelsMODEL … GPU-HOURS COST … PER 1M TOKENS meta-llama/Llama-3.3-70B-Instruct … 289.0 $1,156.00 … $31.87
kube-diagnose on the fault suite’s recording of a wrong liveness port; gpucost on its synthetic demo fleet. Some lines are left out, and “…” marks a cut within a line.
I make the systems under AI products dependable, and a little boring.
↗Based in Gurugram, India · open to remote
I’m Sunil Mehta, Founding Lead DevOps & MLOps Engineer at ORGN, where I run 10+ Kubernetes clusters, 300+ Intel TDX confidential GPU nodes and the Rust LLM gateways that serve 100M+ requests a month. Twice now I have been the founding DevOps lead.
I care about the point where infrastructure stops being a project and becomes something a team can trust: keyless delivery, policy at admission, signed artifacts, alerts that mean something, and a cloud bill someone actually owns.
My path runs through CloudInnov8, To The New, Zeta and Infra360. In the open I build benchmarks that measure Kubernetes scheduling, routing and autoscaling for AI workloads, a scanner for wasted GPUs and a troubleshooter that shows its evidence, and I send fixes upstream: merged in Argo CD, MLflow, NVIDIA KAI Scheduler and llm-d, in review at KServe, Tesla and xAI.
Intel TDX confidential GPU nodes: hardware-isolated sandboxes for AI workloads, with attestation services delivered through GitOps.
Experience
2017 — now · 9.5 years
Hiring for
Feb 2025 — Present
Orgn.comCurrent
Founding Lead DevOps & MLOps Engineer
Gurugram, India
300+
Intel TDX GPU nodes
100M+
LLM requests a month
55%
lower cloud cost
Architected dev, staging, and production GKE across 10+ Kubernetes clusters with Argo CD, Kyverno policy enforcement, Grafana HA, and protected Terraform workflows.
Operated 300+ Intel TDX confidential GPU nodes for hardware-isolated AI sandboxes, Rust LLM gateways serving 100M+ requests per month, and attestation services delivered through GitOps.
Owned $70K–$90K in monthly cloud spend across 10+ clusters, running the platform for an engineering org of 50+.
Built an end-to-end DevSecOps pipeline: secret scanning (TruffleHog, git-secrets), SAST (SonarQube), SCA (Snyk), image scanning (Trivy, Clair), IaC scanning (Checkov, Terrascan), cosign signing, Kyverno and OPA admission control, and Falco runtime detection.
Migrated AWS workloads to GCP and DigitalOcean, lowering infrastructure costs by 55% through right-sizing, environment consolidation, and scale-to-zero policies.
Delivered zero-downtime blue-green migration of central OAuth 2.1 / OIDC SSO onto GKE and Cloud SQL with Private Service Connect.
Established keyless CI/CD pipelines (GitHub Actions WIF → GAR → Argo CD) for Rust gateways and NestJS services, lifting release velocity by 70%.
Maintained 99.95% uptime across 200+ services using Istio, Prometheus, Loki, and OpenTelemetry, compressing alert noise by 60%.
Led a team of 15+ DevOps engineers across 8 enterprise client accounts, boosting delivery efficiency by 45%.
Designed multi-region EKS and AKS clusters with Terraform to sustain 99.99% uptime for business-critical workloads.
Engineered CI/CD with GitHub Actions and Jenkins, curbing rollback incidents by 40%.
Unified OpenTelemetry and Elastic APM across 100+ microservices, halving debugging and triage time.
Executed cost-optimization roadmaps with executive leadership, saving $140K in annual cloud overhead.
Integrated Istio and mutual TLS for multi-tenant clients, handling 50M+ monthly requests securely.
Ran vulnerability management across client accounts with SAST, DAST and SCA, remediating critical and high findings and hardening cloud infrastructure.
Open-source benchmarks and tools for AI infrastructure on Kubernetes. Each one asks a question a platform team has to answer, replays the same workload through the real components in CI, and claims only the differences that held.
From training to serving, nothing unsignedEvery stage runs in CI on a real kind cluster
24admission, network, monitoring, load and GitOps checks on every push
prepare
train
gate
sign
attest
verify
admit
serve
The gate blocked a 0.909 challenger against the 0.938 champion.
MLOps · Supply chainv0.8.0
kubernetes-secure-mlops-pipeline
Every model in production is reproducible, signed and verified.
Kubeflow Pipelines and MLflow train and gate each model against the one it replaces; models ship as signed KServe images with SLSA provenance and an ML-BOM, and Kyverno refuses anything else at admission. One Helm chart reaches dev, staging and prod through Argo CD. Live drift and accuracy alerts start retraining, and every pin is held to a 14-day release age.
View on GitHub ↗Waste found in a day, in GPU-hoursA synthetic fleet backfilled into a real Prometheus in CI
Extra replicas96.3
Unrequested GPUs60.1
Model with no traffic24.1
Idle notebook24.1
Engine asleep16.8
221.4 GPU-hours, $750.64 at the demo's prices, each hour counted once.
GPU FinOps · LLM servingv0.1.0
k8s-gpu-cost-scanner
Which GPUs is the cluster paying for and not using?
gpucost reads what an inference cluster already sends to Prometheus, from kube-state-metrics, the DCGM exporter, vLLM, SGLang and OpenCost, and finds idle and stranded GPUs, models with no traffic, engines asleep and replicas beyond the peak need, each with its evidence, cost and fix, and what each model costs per million tokens. Read-only, one Go binary with no dependencies.
GoPrometheusDCGMvLLMOpenCostFinOps
View on GitHub ↗Root cause named, of 87 broken namespacesThe same faults on a real kind cluster in CI, judged by the same words
kube-diagnose87
k8sgpt33
k8sgpt reports symptoms: for a crash or a wrong liveness port it said only that the termination reason is Error, and for a deny-all NetworkPolicy it reported nothing.
Kubernetes troubleshooting · SREv0.5.0
kube-diagnose
Why is this Kubernetes workload broken?
Reads a namespace through kubectl and names each root cause, the object to fix, and the evidence, with the read-only kubectl command behind every fact: 102 rules from image pulls and crash logs to unschedulable pods, Services that select nothing, NetworkPolicies that close them or drop their traffic, RBAC denials, Kueue scheduling gates, volumes that cannot bind or be written, and CronJobs that skip runs. It can only read, Secret values never reach it, and record --redact writes a snapshot with names, IPs and images replaced, safe to attach to an issue. A fault suite breaks a kind cluster 87 ways on every push and scores it against k8sgpt.
Gokubectl pluginkindk8sgptSRE
View on GitHub ↗Cost per useful GPU-hourReal schedulers on kind + kwok, the same 12-job trace, 30 CI runs
Volcano, as installed$5.33
kube-scheduler$5.23
Kueue$5.21
KAI Scheduler$3.97
Volcano + GPU bin-packing$3.96
The usual outcome over 30 runs. One setting made Volcano 26% faster and 26% cheaper, run after run.
GPU scheduling · FinOpsv0.5.0
k8s-gpu-scheduler-benchmark
Which Kubernetes GPU scheduler should a platform run?
Replays the same AI workload through the default kube-scheduler, Kueue, Volcano and KAI Scheduler, in a reference simulation up to a 1,024-GPU fleet and on the real schedulers in CI with kwok GPU nodes. Out of the box only KAI bin-packs GPU pods by their GPUs; one setting fixes each of the others.
KueueVolcanoKAI SchedulerkwokFinOps
View on GitHub ↗Prefix-cache hits, by routerMedian of eight runs, each on its own cluster
Service (round-robin)63.9%
Load-aware64.3%
Prefix-aware79.9%
Prefix + load79.8%
Median first token fell from 121 ms to 88 ms, in all eight runs.
LLM serving · Routingv0.6.0
k8s-inference-routing-benchmark
How much does the router in front of LLM servers matter?
Replays the same chat and RAG traffic through a Kubernetes Service and the llm-d Router on kind clusters with simulated vLLM pods, claiming only differences that held in every run. Prefix-aware routing finds the cache but piles requests onto two of four pods, so load-aware routing keeps the best tail. Under load the router matters more and the routers differ less.
llm-d RouterGateway API Inference ExtensionvLLMPrefix caching
View on GitHub ↗Requests served on time, by ramp lengthFirst token within 2 s · real HPA on kind, three seeds per ramp
30 s ramp73.5%
120 s ramp82.9%
300 s ramp89.9%
600 s ramp89.8%
The gain stops at 300 s. On the cluster, no ramp is slow enough.
LLM serving · Autoscalingv0.8.0
k8s-inference-autoscaling-benchmark
What should an autoscaler follow when it scales LLM inference?
The Kubernetes HorizontalPodAutoscaler’s algorithm, modelled step by step on four signals and checked against the real HPA, Prometheus and prometheus-adapter on a kind cluster. Pod startup is the binding constraint and a small warm pool is the cheapest fix. KEDA on the same signals showed no difference that held over two runs, and in the model a forecast that scales one pod startup ahead helps on slow ramps only.
HPAKEDAPrometheuskindSimulationForecasting
View on GitHub ↗Noise removed before a rule shipsFindings on whole open-source repositories, before and after the fix
Secret rule · before1,556
Secret rule · after101
eval/exec rule · before1,124
eval/exec rule · after8
128 rules, each swept over real repositories before release.
AI security · Code reviewv0.11.1
agentic-code-review-tool
Scanner-grounded AI review for the whole pull request.
128 rules across app code, Kubernetes, Docker, Terraform, GitHub Actions and AI/agent code (OWASP LLM Top 10, MCP, agent skills and plugins), including the credential-stealing workflow shape of this year’s GhostAction attacks. A Claude or local-LLM agent verifies findings through read-only tools. Reports to GitHub, GitLab, SARIF and JUnit.
PythonClaude APIOWASP LLMSARIF
View on GitHub ↗Scan, test, sign, gate, deployA delivery path for LLM and agent services
14AI security test cases, injection and leakage, on every pull request
commit
scan
test
sign
gate
deploy
SBOM and AI-BOM in CycloneDX 1.6, signed keylessly with cosign.
DevSecOps · LLM servicesReference
ai-devsecops-cicd
A secure delivery path for LLM and agent services.
Prompt-injection tests mapped to the OWASP LLM Top 10, SBOM and AI-BOM generation, Trivy and secret scanning, keyless cosign signing with attestations, and Kyverno admission policies.
GitHub ActionsTrivycosignKyverno
View on GitHub ↗Output tokens per secondQwen2.5-7B 4-bit · vLLM-Metal · Apple M5 Pro
1 request at a time40.3
4 at a time60.1
The cost: median first token rises from 92 ms to 273 ms.
LLM serving · Apple SiliconPlatform
llm-inference-platform
GPU-backed LLM serving you can measure.
vLLM-Metal on Apple Silicon behind a Python streaming gateway, with Prometheus and Grafana monitoring, reproducible benchmarks and Kubernetes deployment templates.
vLLMPythonPrometheusKubernetes
View on GitHub ↗Median routing latency, lower is better24 labelled questions, local Qwen2.5-7B on an M5 Pro
Jev + routing policy387 ms
LLM classification1,693 ms
Keyword rules also matched 22 of 24: a small suite, not a general claim.
LLM routing · RAGRouter
jev-inference-router
An observable routing layer for local GPU inference.
Retrieval, Prometheus metrics, non-root containers, and reproducible evaluations against rule-based and classifier baselines.
Security fix: every serving image’s lock still trusted the compromised emSign root CAs; moved all fifteen Python locks to the fixed certifi, with both test suites passing.