<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: DevOps Daily</title>
    <description>The latest articles on DEV Community by DevOps Daily (@devopsdaily).</description>
    <link>https://dev.to/devopsdaily</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F382434%2F66e04ef9-7e4f-491c-997a-30f4a999d40c.jpg</url>
      <title>DEV Community: DevOps Daily</title>
      <link>https://dev.to/devopsdaily</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/devopsdaily"/>
    <language>en</language>
    <item>
      <title>When the Clock Is Wrong: TLS, JWT and Signed-Request Failures From Clock Skew</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Wed, 18 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/when-the-clock-is-wrong-tls-jwt-and-signed-request-failures-from-clock-skew-18b2</link>
      <guid>https://dev.to/devopsdaily/when-the-clock-is-wrong-tls-jwt-and-signed-request-failures-from-clock-skew-18b2</guid>
      <description>&lt;p&gt;A certificate can be freshly renewed and still be "not yet valid." A JWT can be correctly signed and still fail &lt;code&gt;iat&lt;/code&gt; or &lt;code&gt;nbf&lt;/code&gt;. An AWS request can have the right key and receive &lt;code&gt;RequestTimeTooSkewed&lt;/code&gt;. When those symptoms arrive together, stop rotating credentials and check time first. A clock is an input to each decision, and a host that is only slightly early or late can turn an infrastructure problem into an authentication-looking incident.&lt;/p&gt;

&lt;p&gt;This post maps the time check behind each failure, shows safe reproductions without changing a machine clock, and gives a Linux response path that makes loss of sync visible before users find it.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;TLS validation compares the verifier's current time with &lt;code&gt;notBefore&lt;/code&gt; and &lt;code&gt;notAfter&lt;/code&gt;. It has no protocol-level grace period.&lt;/li&gt;
&lt;li&gt;JWT leeway is a verifier policy. PyJWT accepts a &lt;code&gt;leeway&lt;/code&gt; argument, and its default is zero when you do not pass one.&lt;/li&gt;
&lt;li&gt;S3 documents a 15-minute request timestamp limit. MIT Kerberos defaults to 300 seconds. TOTP recommends 30-second steps and at most one step for network delay.&lt;/li&gt;
&lt;li&gt;On Linux, first check &lt;code&gt;timedatectl&lt;/code&gt;, then use &lt;code&gt;chronyc tracking&lt;/code&gt; and &lt;code&gt;chronyc sources -v&lt;/code&gt; when chrony is your time daemon.&lt;/li&gt;
&lt;li&gt;Correct the host clock, not a container clock. Alert on sync state and offset, then preserve UTC timestamps and offsets in incident logs.&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;&lt;em&gt;Press play to watch a server's clock fall 1, 6 and then 20 minutes behind: a fresh JWT fails first, then a certificate issued 2 minutes ago and Kerberos (5 minutes allowed), and finally an S3 signed request (15 minutes allowed).&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Shell access to a Linux host or VM.&lt;/li&gt;
&lt;li&gt;Permission to inspect the active time service. Correcting time needs an approved change path because a clock step can surprise applications.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;chrony&lt;/code&gt; installed if you use the chrony commands and configuration below.&lt;/li&gt;
&lt;li&gt;OpenSSL, Python with &lt;code&gt;cryptography&lt;/code&gt; for the TLS fixture, and PyJWT for the JWT demo.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The time window behind the error
&lt;/h2&gt;

&lt;p&gt;The same skew produces different errors because each protocol asks a different question. &lt;a href="https://www.rfc-editor.org/rfc/rfc5280" rel="noopener noreferrer"&gt;RFC 5280&lt;/a&gt; requires certificate path validation to include the current time in the certificate validity period. &lt;a href="https://www.rfc-editor.org/rfc/rfc7519" rel="noopener noreferrer"&gt;RFC 7519&lt;/a&gt; defines &lt;code&gt;nbf&lt;/code&gt;, &lt;code&gt;exp&lt;/code&gt;, and &lt;code&gt;iat&lt;/code&gt; as NumericDate-based claims. SigV4 includes a UTC request timestamp in the signature input. TOTP derives its moving value from time.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Time comparison&lt;/th&gt;
&lt;th&gt;Documented tolerance&lt;/th&gt;
&lt;th&gt;Typical symptom&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;TLS certificate&lt;/td&gt;
&lt;td&gt;Local verifier time versus &lt;code&gt;notBefore&lt;/code&gt; and &lt;code&gt;notAfter&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;None in the certificate validity check&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;certificate is not yet valid&lt;/code&gt; or expired&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JWT&lt;/td&gt;
&lt;td&gt;Verifier time versus &lt;code&gt;nbf&lt;/code&gt;, &lt;code&gt;exp&lt;/code&gt;, and possibly &lt;code&gt;iat&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Library policy. &lt;a href="https://github.com/jpadilla/pyjwt/blob/2.15.1/jwt/api_jwt.py" rel="noopener noreferrer"&gt;PyJWT 2.15.1 defaults to 0&lt;/a&gt;; RFC 7519 permits small leeway, usually no more than a few minutes&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ImmatureSignatureError&lt;/code&gt;, expired token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS SigV4&lt;/td&gt;
&lt;td&gt;Signed request timestamp versus AWS time&lt;/td&gt;
&lt;td&gt;S3 rejects a difference greater than 15 minutes&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;RequestTimeTooSkewed&lt;/code&gt;; other signature paths can report signature mismatch or expiry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MIT Kerberos&lt;/td&gt;
&lt;td&gt;Client or service time versus KDC time&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;clockskew&lt;/code&gt; default: 300 seconds&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Clock skew too great&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TOTP&lt;/td&gt;
&lt;td&gt;Prover time step versus validator time step&lt;/td&gt;
&lt;td&gt;RFC 6238 recommends a 30-second step and at most one step for network delay&lt;/td&gt;
&lt;td&gt;A correct-looking code is rejected&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The S3 limit and error name come from &lt;a href="https://docs.aws.amazon.com/AmazonS3/latest/developerguide/RESTAuthentication.html" rel="noopener noreferrer"&gt;AWS's request authentication documentation&lt;/a&gt;. AWS also documents &lt;code&gt;SignatureDoesNotMatch&lt;/code&gt; and the &lt;code&gt;Signature expired&lt;/code&gt; diagnostic for SigV4 troubleshooting in its &lt;a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-troubleshooting.html" rel="noopener noreferrer"&gt;IAM guide&lt;/a&gt;. MIT documents its 300-second default in &lt;a href="https://web.mit.edu/Kerberos/www/krb5-current/doc/admin/conf_files/krb5_conf.html" rel="noopener noreferrer"&gt;&lt;code&gt;krb5.conf&lt;/code&gt;&lt;/a&gt;. RFC 6238 calls for a &lt;a href="https://www.rfc-editor.org/rfc/rfc6238#section-5.2" rel="noopener noreferrer"&gt;30-second default step and at most one step of delay&lt;/a&gt;. Those numbers are useful triage boundaries, not a reason to let time drift until the next boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove TLS time failures without changing the host clock
&lt;/h2&gt;

&lt;p&gt;Do not set a workstation's clock backward to test certificate behavior. OpenSSL's &lt;a href="https://docs.openssl.org/3.0/man1/openssl-verification-options/" rel="noopener noreferrer"&gt;&lt;code&gt;-attime&lt;/code&gt; verification option&lt;/a&gt; supplies the verification time instead. That isolates the test from package updates, log ordering, token expiry, and every other process on the machine.&lt;/p&gt;

&lt;p&gt;This run creates a local CA that began 30 days earlier and two leaf certificates issued at the run time. One leaf starts at issuance. The other has &lt;code&gt;notBefore&lt;/code&gt; one hour earlier. The verifier clock is then simulated 10 minutes behind, and then 8 days ahead. The fixtures come from this short script, which uses the Python &lt;code&gt;cryptography&lt;/code&gt; package so it can set &lt;code&gt;notBefore&lt;/code&gt; freely (&lt;code&gt;python3 mkcerts.py&lt;/code&gt; writes &lt;code&gt;ca.pem&lt;/code&gt;, &lt;code&gt;leaf-now.pem&lt;/code&gt; and &lt;code&gt;leaf-backdated.pem&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Build a demo CA (valid since 30 days ago) and two leaf certs issued "now":&lt;/span&gt;
&lt;span class="c1"&gt;# leaf-now.pem has notBefore = issuance time, leaf-backdated.pem has notBefore = issuance time - 1 hour.&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;cryptography&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;x509&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;cryptography&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;NameOID&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;cryptography&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hazmat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;primitives&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;serialization&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;cryptography&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hazmat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;primitives&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;asymmetric&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ec&lt;/span&gt;
&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;microsecond&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NameAttribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;NameOID&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;COMMON_NAME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;ca_key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate_private_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SECP256R1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;ca&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CertificateBuilder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;subject_name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Demo CA"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;issuer_name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Demo CA"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca_key&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;serial_number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random_serial_number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;not_valid_before&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;not_valid_after&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;365&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_extension&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BasicConstraints&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;critical&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sign&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SHA256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ca.pem"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"wb"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;serialization&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Encoding&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;PEM&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;fname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;back&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"leaf-now.pem"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"leaf-backdated.pem"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate_private_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SECP256R1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CertificateBuilder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;subject_name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"api.example.com"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;issuer_name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;subject&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;serial_number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random_serial_number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;not_valid_before&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seconds&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;back&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;not_valid_after&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_extension&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SubjectAlternativeName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x509&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DNSName&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"api.example.com"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;critical&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
         &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sign&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ca_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SHA256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"wb"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;serialization&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Encoding&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;PEM&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then verify with OpenSSL 3.0, passing &lt;code&gt;-attime&lt;/code&gt; as a Unix timestamp:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;verify certificates at simulated times&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ openssl x509 -in leaf-now.pem -noout -startdate
&lt;span class="nv"&gt;notBefore&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;Nov 11 13:33:26 2026 GMT
$ openssl x509 -in leaf-backdated.pem -noout -startdate
&lt;span class="nv"&gt;notBefore&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;Nov 11 12:33:26 2026 GMT
$ openssl verify -CAfile ca.pem -attime &lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;date +%s&lt;span class="o"&gt;)&lt;/span&gt; - 600 &lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; leaf-now.pem &lt;span class="c"&gt;# verifier clock 10 min behind&lt;/span&gt;
CN &lt;span class="o"&gt;=&lt;/span&gt; api.example.com
error 9 at 0 depth lookup: certificate is not yet valid
error leaf-now.pem: verification failed
$ openssl verify -CAfile ca.pem -attime &lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;date +%s&lt;span class="o"&gt;)&lt;/span&gt; - 600 &lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; leaf-backdated.pem &lt;span class="c"&gt;# same clock, notBefore 1 h earlier&lt;/span&gt;
leaf-backdated.pem: OK
$ openssl verify -CAfile ca.pem -attime &lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;date +%s&lt;span class="o"&gt;)&lt;/span&gt; + 8*86400 &lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; leaf-now.pem &lt;span class="c"&gt;# verifier clock 8 days ahead&lt;/span&gt;
CN &lt;span class="o"&gt;=&lt;/span&gt; api.example.com
error 10 at 0 depth lookup: certificate has expired
error leaf-now.pem: verification failed

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result is not a claim about every public CA's issuance policy. It shows the direct consequence of the two validity timestamps: a verifier behind the issuing clock rejects the first leaf, while the locally backdated leaf passes. OpenSSL documents that it reports not-yet-valid or expired when the verification time is outside the validity window.&lt;/p&gt;

&lt;p&gt;Shorter certificate lifetimes mean renewal happens more often. That increases the frequency of encounters with newly issued certificates, not the size of the clock error needed to fail. Check your renewal automation as described in &lt;a href="https://dev.to/devopsdaily/lets-encrypt-goes-to-64-day-certificates-when-does-your-acme-client-actually-renew-347l"&gt;our 64-day certificate renewal check&lt;/a&gt;, and use the &lt;a href="https://devops-daily.com/games/ssl-tls-handshake" rel="noopener noreferrer"&gt;SSL/TLS Handshake Simulator&lt;/a&gt; to review where certificate validation belongs in the handshake.&lt;/p&gt;

&lt;h2&gt;
  
  
  JWTs: a small leeway is a policy, not a clock fix
&lt;/h2&gt;

&lt;p&gt;RFC 7519 says an implementation &lt;strong&gt;may&lt;/strong&gt; provide small leeway, usually no more than a few minutes, for clock skew. It does not choose a value for you. PyJWT's &lt;a href="https://pyjwt.readthedocs.io/en/stable/usage.html" rel="noopener noreferrer"&gt;&lt;code&gt;decode()&lt;/code&gt; examples&lt;/a&gt; show that &lt;code&gt;leeway&lt;/code&gt; can be seconds or a &lt;code&gt;datetime.timedelta&lt;/code&gt;; its &lt;a href="https://github.com/jpadilla/pyjwt/blob/2.15.1/jwt/api_jwt.py" rel="noopener noreferrer"&gt;2.15.1 source&lt;/a&gt; sets the default to zero.&lt;/p&gt;

&lt;p&gt;Here is the complete local demonstration. The issuer clock is 90 seconds ahead of the verifier. The token has &lt;code&gt;iat&lt;/code&gt;, &lt;code&gt;nbf&lt;/code&gt;, and &lt;code&gt;exp&lt;/code&gt;, so a validator that checks issued-at time rejects it before it is valid.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Clock skew demo for JWTs: the issuer's clock runs 90 seconds ahead of the API that validates the token.&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jwt&lt;/span&gt;

&lt;span class="no"&gt;SECRET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"demo-secret-not-for-production-use-only"&lt;/span&gt;
&lt;span class="n"&gt;skew&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt; &lt;span class="c1"&gt;# issuer clock minus verifier clock, in seconds&lt;/span&gt;
&lt;span class="n"&gt;issuer_now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;skew&lt;/span&gt;
&lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;jwt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"sub"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"svc-billing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"iat"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;issuer_now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"nbf"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;issuer_now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"exp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;issuer_now&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;SECRET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;algorithm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"HS256"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;leeway&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;jwt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;SECRET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;algorithms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"HS256"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;leeway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;leeway&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"leeway={leeway:&amp;gt;3}s accepted"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;jwt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;PyJWTError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"leeway={leeway:&amp;gt;3}s rejected: {type(e). __name__ }: {e}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


$ &lt;span class="n"&gt;python&lt;/span&gt; &lt;span class="n"&gt;jwt_skew&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt; &lt;span class="c1"&gt;# PyJWT 2.15.1, issuer clock 90 s ahead of the verifier&lt;/span&gt;
&lt;span class="n"&gt;leeway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0s&lt;/span&gt; &lt;span class="n"&gt;rejected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ImmatureSignatureError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;The&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;yet&lt;/span&gt; &lt;span class="n"&gt;valid&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iat&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;leeway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;30s&lt;/span&gt; &lt;span class="n"&gt;rejected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ImmatureSignatureError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;The&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;yet&lt;/span&gt; &lt;span class="n"&gt;valid&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iat&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;leeway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120s&lt;/span&gt; &lt;span class="n"&gt;accepted&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The 120-second value makes the test pass. It does not make a 90-second clock difference healthy. For a service, use a deliberately small policy allowance, such as 30 to 60 seconds if the threat model permits it, alert when it is consumed, and fix time sync. Never use a client-supplied timestamp as the authority for a security decision. Signed webhook schemes have the same replay-window concern, so keep the sender timestamp, receive time, and verifier decision in the delivery record. See &lt;a href="https://dev.to/devopsdaily/what-it-actually-takes-to-deliver-a-webhook-in-production-4ghe"&gt;what production webhook delivery needs&lt;/a&gt; for the surrounding signature and retry design.&lt;/p&gt;

&lt;h2&gt;
  
  
  Inspect the Linux time path before restarting auth services
&lt;/h2&gt;

&lt;p&gt;Start with the service-independent view:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timedatectl

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inspect &lt;code&gt;System clock synchronized&lt;/code&gt; and &lt;code&gt;NTP service&lt;/code&gt;. On a host using systemd-timesyncd, &lt;code&gt;timedatectl timesync-status&lt;/code&gt; reports the selected server, leap state, offset, delay, jitter, and poll interval. The &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/timedatectl.html" rel="noopener noreferrer"&gt;systemd &lt;code&gt;timedatectl&lt;/code&gt; manual&lt;/a&gt; documents those fields.&lt;/p&gt;

&lt;p&gt;On a chrony host, inspect both the remaining system correction and the sources:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;chronyc tracking
chronyc sources -v

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In &lt;code&gt;tracking&lt;/code&gt;, &lt;code&gt;System time&lt;/code&gt; is the remaining offset between chronyd's NTP clock and the system clock. &lt;code&gt;Leap status: Not synchronised&lt;/code&gt; is a bad state. In &lt;code&gt;sources -v&lt;/code&gt;, &lt;code&gt;Reach&lt;/code&gt; is an eight-bit octal reachability register, so &lt;code&gt;377&lt;/code&gt; means a valid response on each of the last eight polls. A &lt;code&gt;?&lt;/code&gt; source is not selectable, including when it is unreachable. Chrony's &lt;a href="https://chrony-project.org/doc/latest/chronyc.html" rel="noopener noreferrer"&gt;&lt;code&gt;chronyc(1)&lt;/code&gt; manual&lt;/a&gt; defines these fields and their selection markers.&lt;/p&gt;

&lt;p&gt;The common failure modes are mundane:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A device without a working RTC can boot at an old time and needs a usable time source before time-sensitive work starts. systemd-timesyncd records a synchronized time to help advance a machine without a battery-backed RTC on a later boot, but does not hold all units until accurate synchronization by default. See the &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-timesyncd.service.html" rel="noopener noreferrer"&gt;systemd-timesyncd manual&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;A paused or restored VM can resume with stale time. &lt;a href="https://chrony-project.org/faq.html" rel="noopener noreferrer"&gt;Chrony's FAQ&lt;/a&gt; calls out a virtual machine that is suspended and resumed with an incorrect time as a case that may need stepping on any clock update, not only at boot.&lt;/li&gt;
&lt;li&gt;A firewall rule can block NTP's UDP port 123. When &lt;code&gt;chronyc sources&lt;/code&gt; shows no valid responses from a server, the &lt;a href="https://chrony-project.org/faq.html" rel="noopener noreferrer"&gt;chrony FAQ&lt;/a&gt; points first at a firewall blocking requests to UDP port 123 or the replies.&lt;/li&gt;
&lt;li&gt;A Linux container shares the host kernel, so the container cannot independently repair the host's system clock. Fix the node or VM that owns the kernel. &lt;a href="https://docs.docker.com/learn/content/what-is-a-container/" rel="noopener noreferrer"&gt;Docker's container explanation&lt;/a&gt; describes that shared-kernel model.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Correct time safely, then make loss of sync noisy
&lt;/h2&gt;

&lt;p&gt;Chrony normally slews a correction, which avoids a sudden jump but can take a long time for a large offset. Its &lt;a href="https://chrony-project.org/doc/latest/chrony.conf.html" rel="noopener noreferrer"&gt;&lt;code&gt;makestep threshold limit&lt;/code&gt; directive&lt;/a&gt; controls when it may jump instead. A useful boot-time policy is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Add to chronyd's active configuration file
# Step offsets over 1 second during the first three clock updates.
&lt;/span&gt;&lt;span class="n"&gt;makestep&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;

&lt;span class="c1"&gt;# Keep the kernel's synchronization flag (and the RTC) updated.
# node_exporter's node_timex_sync_status depends on it.
&lt;/span&gt;&lt;span class="n"&gt;rtcsync&lt;/span&gt;

&lt;span class="c1"&gt;# Replace these with separate, approved NTS-capable sources.
&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="n"&gt;ntp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;net&lt;/span&gt; &lt;span class="n"&gt;iburst&lt;/span&gt; &lt;span class="n"&gt;nts&lt;/span&gt;
&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="n"&gt;ntp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;net&lt;/span&gt; &lt;span class="n"&gt;iburst&lt;/span&gt; &lt;span class="n"&gt;nts&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the sources support NTS and &lt;code&gt;chronyd&lt;/code&gt; was built with NTS support, the &lt;code&gt;nts&lt;/code&gt; option enables Network Time Security; support was added in chrony 4.0. If a host is currently wrong and the operational impact of a jump has been assessed, &lt;code&gt;chronyc makestep&lt;/code&gt; immediately applies the remaining correction. Chrony's manual warns that time jumps can seriously affect some software. Confirm the source state first, make the correction in the approved window, then re-run &lt;code&gt;chronyc tracking&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For Prometheus, node_exporter's timex collector exposes &lt;code&gt;node_timex_offset_seconds&lt;/code&gt; and &lt;code&gt;node_timex_sync_status&lt;/code&gt;. Its &lt;a href="https://github.com/prometheus/node_exporter/blob/master/collector/timex.go" rel="noopener noreferrer"&gt;source&lt;/a&gt; defines the former as offset between the local and reference clocks and the latter as &lt;code&gt;1&lt;/code&gt; for synchronized and &lt;code&gt;0&lt;/code&gt; for unsynchronized. Start with a rule that catches loss of sync or a sustained offset, then tune the threshold to the most time-sensitive system you operate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;clock-health&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NodeClockNotSynchronized&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;node_timex_sync_status == 0&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Node clock is not synchronized'&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NodeClockOffsetHigh&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;abs(node_timex_offset_seconds) &amp;gt; 0.05&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Node clock offset exceeds 50 ms'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;node_timex_sync_status&lt;/code&gt; depends on the kernel time synchronization flag. Node exporter's &lt;a href="https://github.com/prometheus/node_exporter/blob/master/docs/TIME.md" rel="noopener noreferrer"&gt;time notes&lt;/a&gt; say every chrony version needs the &lt;code&gt;rtcsync&lt;/code&gt; option to maintain that flag, so on chrony hosts the sync alert is only meaningful with &lt;code&gt;rtcsync&lt;/code&gt; set. Pair the alert with the daemon's own status and &lt;code&gt;chronyc tracking&lt;/code&gt; during investigation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Record the failing host's UTC time, offset, synchronization state, source reachability, and the exact verifier error before changing anything.&lt;/li&gt;
&lt;li&gt;Compare issuer, verifier, and identity-provider times. Do not assume that a successful &lt;code&gt;date&lt;/code&gt; command on one pod says anything about the node behind another service.&lt;/li&gt;
&lt;li&gt;Repair the source or network path, then use a safe step policy when the offset is too large to slew promptly.&lt;/li&gt;
&lt;li&gt;Retest one TLS verification with &lt;code&gt;-attime&lt;/code&gt;, one JWT with its configured leeway, and one signed request after the host reports synchronized.&lt;/li&gt;
&lt;li&gt;Keep logs in UTC, include an offset where available, and return the server time in safe error responses. This makes cross-host event ordering possible without exposing sensitive details.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Time is shared infrastructure. Treat it like DNS or certificate trust: monitor it, make dependencies wait where they need accurate time, and investigate divergence before widening every authentication window.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/clock-skew-incidents-tls-jwt-chrony" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>linux</category>
      <category>ntp</category>
      <category>chrony</category>
      <category>tls</category>
    </item>
    <item>
      <title>Learn DevOps From Your Terminal: The DevOps Daily CLI and SSH App</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Tue, 17 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/learn-devops-from-your-terminal-the-devops-daily-cli-and-ssh-app-3g46</link>
      <guid>https://dev.to/devopsdaily/learn-devops-from-your-terminal-the-devops-daily-cli-and-ssh-app-3g46</guid>
      <description>&lt;p&gt;You spend most of your day in a terminal: SSH sessions, &lt;code&gt;kubectl&lt;/code&gt;, &lt;code&gt;git&lt;/code&gt;, logs. Switching to a browser to learn something new breaks that flow. So we put DevOps Daily in the terminal too. You can read the posts, play the quizzes, study flashcards, tick off checklists and solve the incident of the day from the same window where you run your commands.&lt;/p&gt;

&lt;p&gt;There are two ways in. &lt;code&gt;ssh ssh.devops-daily.com&lt;/code&gt; works right now with nothing to install. The &lt;code&gt;devops-daily&lt;/code&gt; CLI is a small open source binary for macOS, Linux and Windows that keeps your quiz scores and flashcard progress and works offline. This post shows both, what you can do with them, and how they are built.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Try it now: &lt;code&gt;ssh ssh.devops-daily.com&lt;/code&gt;. No account, no password, any SSH key or none.&lt;/li&gt;
&lt;li&gt;Install the CLI with Homebrew, npm, Go or a one-line install script. The command is &lt;code&gt;devops-daily&lt;/code&gt;; Homebrew, npm and the install script also add the short alias &lt;code&gt;ddy&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;It has 369 posts, 47 quizzes, 27 flashcard decks, 24 checklists, a daily incident, and links to our 68 games and simulators.&lt;/li&gt;
&lt;li&gt;No telemetry and no account. After the first sync it works offline.&lt;/li&gt;
&lt;li&gt;The code is open source under the MIT license: &lt;a href="https://github.com/The-DevOps-Daily/devops-daily-cli" rel="noopener noreferrer"&gt;The-DevOps-Daily/devops-daily-cli&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A terminal with an SSH client for the SSH app. Every macOS and Linux machine has one, and so does Windows 10 and later.&lt;/li&gt;
&lt;li&gt;For the CLI: macOS, Linux or Windows on amd64 or arm64, plus Homebrew, npm or Go if you want to install it that way.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it over SSH first
&lt;/h2&gt;

&lt;p&gt;Run this from any terminal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh ssh.devops-daily.com

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You get the home screen of the app:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr1ofqxv3k1kuyp9pupei.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr1ofqxv3k1kuyp9pupei.webp" alt="The DevOps Daily SSH app home screen with five sections: latest posts, daily quiz, incident of the day, newsletter and links" width="799" height="393"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Use the arrow keys or the numbers to move, Enter to open a section and &lt;code&gt;q&lt;/code&gt; to quit. Over SSH, the latest posts show a summary and a link (the CLI reads them in full), the daily quiz has one question that is the same for everyone that day, and the incident of the day is a real-world problem with four possible fixes.&lt;/p&gt;

&lt;p&gt;The first time you connect, SSH asks you to confirm the server's host key. Check that the fingerprint matches this one before you type &lt;code&gt;yes&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ED25519 key fingerprint is SHA256:foWFYF2OcGurRwunAlzGTHrpARFN+VfBRDZdmiZ0zf8.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is an app, never a shell. The server accepts any key or no key, does not store or log keys, and refuses everything except an interactive session: no commands, no SFTP or scp, and no port forwarding. Sessions end after 10 minutes without a key press or 30 minutes in total, and each address can open 10 new connections per minute. If you have not thought about how SSH can be exposed safely before, the &lt;a href="https://devops-daily.com/checklists/ssh-hardening" rel="noopener noreferrer"&gt;SSH hardening checklist&lt;/a&gt; is a good companion read.&lt;/p&gt;

&lt;h2&gt;
  
  
  Install the CLI
&lt;/h2&gt;

&lt;p&gt;Pick the way you install other tools.&lt;/p&gt;

&lt;p&gt;Homebrew on macOS or Linux:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;brew install The-DevOps-Daily/tap/devops-daily

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;npm, or &lt;code&gt;npx&lt;/code&gt; to try it without installing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm install -g @devopsdaily/devops-daily-cli
npx @devopsdaily/devops-daily-cli

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Go:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go install github.com/The-DevOps-Daily/devops-daily-cli/cmd/devops-daily@latest

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On macOS or Linux, the install script downloads the release for your system and checks it against the published checksum. On Windows, use npm, Go or the ZIP file from the releases page.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fsSL https://raw.githubusercontent.com/The-DevOps-Daily/devops-daily-cli/main/install.sh &lt;span class="o"&gt;|&lt;/span&gt; sh

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As with any &lt;code&gt;curl | sh&lt;/code&gt;, read the script first if you prefer; it is short. Every release on GitHub has a &lt;code&gt;checksums.txt&lt;/code&gt; and a build provenance attestation, so you can check where a binary came from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh attestation verify devops-daily_0.1.0_linux_arm64.tar.gz -R The-DevOps-Daily/devops-daily-cli

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What you can do
&lt;/h2&gt;

&lt;p&gt;Run &lt;code&gt;devops-daily&lt;/code&gt; with no command for the same home menu as the SSH app. Every section also has its own command:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;posts [--tag k8s]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;List posts, newest first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;read &amp;lt;slug&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Read a post in the terminal, with its links at the end&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;quiz [topic]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Play one of the site's quizzes; your best score is kept&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;flashcards [deck]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Study a deck with simple spaced repetition&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;checklist [name]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Tick off a checklist; your progress is saved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;incident [--fix]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Today's real-world incident, then the fix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;play [simulator]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Open a simulator, a game or Opsville in your browser&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sync&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Download the latest content now&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Names are forgiving. &lt;code&gt;devops-daily quiz docker&lt;/code&gt;, &lt;code&gt;devops-daily play dns&lt;/code&gt; and &lt;code&gt;devops-daily read terraform-state&lt;/code&gt; find the closest match, or list the candidates when there is more than one.&lt;/p&gt;

&lt;p&gt;The first sync downloads everything at once, then posts are a list away:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;devops-daily&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ devops-daily sync
Downloaded 23 files (7.6 MB).
369 posts, 47 quizzes, 27 flashcard decks, 24 checklists, 16 incidents, 68 games
$ devops-daily posts --limit 3
2026-11-13 CKNE Is Live: What the Kubernetes Network Engineer Exam Tests...
            ckne-kubernetes-network-engineer-exam-guide
2026-11-13 Five Production Bugs You Can Reproduce in a Browser Tab
            five-production-bugs-you-can-reproduce-in-a-browser
2026-11-12 We Tested AWS Emulator: It Blocks More Than Its Docs Say, and...
            aws-emulator-hands-on-test
Read one: devops-daily read &amp;lt;slug&amp;gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The incident of the day describes a real production problem and asks what you would do:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;devops-daily incident&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ devops-daily incident
Four nodes free, pod still Pending
Intermediate · Kubernetes, AWS EBS, StatefulSets, Cluster Autoscaler

An EKS cluster spans three availability zones with one node group across all
three. The only node in us-east-1a fails. Your StatefulSet pod, with a 200 GiB
EBS volume, is rescheduled and stays Pending. The cluster autoscaler adds a
node, in us-east-1b. The pod is still Pending: 0/4 nodes are available: 4
node(s) had volume node affinity conflict. The four nodes have plenty of free
CPU and memory. What is the right fix?
A) Delete the PVC so Kubernetes provisions a new volume in a zone that has free nodes
B) Change the StorageClass to volumeBindingMode: WaitForFirstConsumer
C) Get a node into us-east-1a, and split the node group so each zone has its own
D) Give the pod a higher PriorityClass so it can preempt pods on other nodes
Run with --fix to see the answer.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Make your pick, then run &lt;code&gt;devops-daily incident --fix&lt;/code&gt; for the answer, why each wrong option is wrong, and a link to a post that goes deeper. For hands-on practice, the &lt;a href="https://devops-daily.com/simulators" rel="noopener noreferrer"&gt;simulators&lt;/a&gt; cover many of the same problems.&lt;/p&gt;

&lt;p&gt;Quizzes, flashcards and checklists are the same ones as on the site: the &lt;a href="https://devops-daily.com/quizzes" rel="noopener noreferrer"&gt;quizzes&lt;/a&gt;, the &lt;a href="https://devops-daily.com/flashcards" rel="noopener noreferrer"&gt;flashcard decks&lt;/a&gt; and the &lt;a href="https://devops-daily.com/checklists" rel="noopener noreferrer"&gt;checklists&lt;/a&gt;. In the terminal, your best quiz scores, flashcard schedule and checklist ticks are saved on your machine, so you can stop and pick up where you left off.&lt;/p&gt;

&lt;p&gt;When something is easier to learn by doing, &lt;code&gt;devops-daily play&lt;/code&gt; opens it in your browser: &lt;code&gt;play dns&lt;/code&gt; opens the &lt;a href="https://devops-daily.com/games/dns-simulator" rel="noopener noreferrer"&gt;DNS simulator&lt;/a&gt;, and &lt;code&gt;play opsville&lt;/code&gt; opens &lt;a href="https://devops-daily.com/opsville" rel="noopener noreferrer"&gt;Opsville&lt;/a&gt;, our DevOps town game.&lt;/p&gt;

&lt;h2&gt;
  
  
  Offline and private
&lt;/h2&gt;

&lt;p&gt;After the first sync, the CLI works without a network connection. Content lives in a cache directory (&lt;code&gt;~/.cache/devops-daily&lt;/code&gt; on Linux) and your progress in a state directory (&lt;code&gt;~/.local/state/devops-daily&lt;/code&gt;). Set &lt;code&gt;DEVOPS_DAILY_OFFLINE=1&lt;/code&gt; to turn off the automatic content refresh (an explicit &lt;code&gt;sync&lt;/code&gt;, &lt;code&gt;update&lt;/code&gt; or &lt;code&gt;play&lt;/code&gt; still uses the network), and &lt;code&gt;NO_COLOR&lt;/code&gt; if you want plain output.&lt;/p&gt;

&lt;p&gt;There is no telemetry and no account, and your progress never leaves your machine. The CLI downloads public content files from devops-daily.com, and asks GitHub for the latest version only when you run &lt;code&gt;devops-daily update&lt;/code&gt;. Like any download, those requests carry your IP address and the CLI version, and nothing else about you.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it is built
&lt;/h2&gt;

&lt;p&gt;The CLI and the SSH app are one Go codebase with two front doors. Both use the same screens, built with &lt;a href="https://github.com/charmbracelet/bubbletea" rel="noopener noreferrer"&gt;Bubble Tea&lt;/a&gt; and Lip Gloss. Posts render in the terminal with Glamour, and the SSH server runs on Wish, which serves a Bubble Tea app to every SSH session instead of a shell.&lt;/p&gt;

&lt;p&gt;The site is static, so there is no API server to run. When devops-daily.com builds, it also writes small JSON files under &lt;code&gt;/cli/v1/&lt;/code&gt;: the post index, post bodies, quizzes, decks, checklists and incidents. The CLI downloads those files and caches them. The format is documented in the repository, so you can build your own tools on top of it.&lt;/p&gt;

&lt;p&gt;Releases are built with GoReleaser for six platforms, with checksums and build provenance attestations. The npm package is published from GitHub Actions with npm trusted publishing, so the release workflow stores no long-lived npm token.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/devops-daily-cli" rel="noopener noreferrer"&gt;The-DevOps-Daily/devops-daily-cli on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is next
&lt;/h2&gt;

&lt;p&gt;This is version 0.1.0. Next on the list:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sign in with your DevOps Daily account&lt;/strong&gt; , so quiz XP, roadmap progress and Opsville stats follow you between the browser and the terminal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small checks you can use every day&lt;/strong&gt; : TLS certificates, DNS records, HTTP security headers and email authentication (SPF, DKIM and DMARC) for a host or domain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A terminal game&lt;/strong&gt; in the spirit of Opsville, with incidents to fix from your keyboard.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you find a bug or want a command, open an issue on the &lt;a href="https://github.com/The-DevOps-Daily/devops-daily-cli/issues" rel="noopener noreferrer"&gt;GitHub repository&lt;/a&gt;. And if you want a new post, quiz or incident in your terminal every week, &lt;a href="https://devops-daily.com/newsletters" rel="noopener noreferrer"&gt;subscribe to the newsletter&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ssh ssh.devops-daily.com&lt;/code&gt; is the fastest way to try it, with nothing to install.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;brew install The-DevOps-Daily/tap/devops-daily&lt;/code&gt; (or npm, Go or the install script) gives you the CLI, with saved progress and offline use.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;devops-daily&lt;/code&gt; for the menu, or jump straight in with &lt;code&gt;posts&lt;/code&gt;, &lt;code&gt;quiz&lt;/code&gt;, &lt;code&gt;flashcards&lt;/code&gt;, &lt;code&gt;checklist&lt;/code&gt;, &lt;code&gt;incident&lt;/code&gt; and &lt;code&gt;play&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;It is open source, has no telemetry, and needs no account.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/devops-daily-cli-learn-devops-from-your-terminal" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>cli</category>
      <category>terminal</category>
      <category>go</category>
    </item>
    <item>
      <title>RDS MySQL 8.0 Has Been Billing Extended Support Since August: Find It and Price It</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Tue, 17 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/rds-mysql-80-has-been-billing-extended-support-since-august-find-it-and-price-it-599g</link>
      <guid>https://dev.to/devopsdaily/rds-mysql-80-has-been-billing-extended-support-since-august-find-it-and-price-it-599g</guid>
      <description>&lt;p&gt;If an RDS for MySQL 8.0 instance is still running, its bill changed on August 1, 2026. RDS ended standard support for 8.0 on July 31 and now bills Extended Support by vCPU-hour in addition to the DB instance, storage, backup, and transfer charges. The easy mistake is treating that as an upgrade decision for later. It is already a recurring line item.&lt;/p&gt;

&lt;p&gt;This post gives you a per-Region inventory, a billing-data path, a cost model, and the decision points for an 8.0 to 8.4 upgrade. The same inventory also exposes older PostgreSQL and Aurora versions that have crossed their own support boundaries.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;RDS MySQL 8.0 entered paid Extended Support on &lt;strong&gt;August 1, 2026&lt;/strong&gt;. Year 3 pricing begins August 1, 2028, and support ends July 31, 2029. &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html" rel="noopener noreferrer"&gt;AWS's MySQL release calendar&lt;/a&gt; also says MySQL 5.7 and 8.0 are now only available under Extended Support.&lt;/li&gt;
&lt;li&gt;Use &lt;code&gt;EngineLifecycleSupport&lt;/code&gt; to see enrollment, but combine it with engine version and support dates. Enrollment alone does not mean a charge until standard support has ended.&lt;/li&gt;
&lt;li&gt;RDS charges provisioned MySQL Extended Support per vCPU-hour. AWS says it also applies to Multi-AZ standby instances and to read replicas on the same version, so count every one.&lt;/li&gt;
&lt;li&gt;A new CLI or API-created instance defaults to Extended Support enabled if you omit the setting. Specify &lt;code&gt;open-source-rds-extended-support-disabled&lt;/code&gt; if automatic upgrade or creation failure is preferable to silent enrollment.&lt;/li&gt;
&lt;li&gt;EKS has the same financial shape at the control-plane level: $0.10 per cluster-hour during standard support and $0.60 in extended support. Set its upgrade policy to &lt;code&gt;STANDARD&lt;/code&gt; only when you accept AWS automatically upgrading it at the end of standard support.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;AWS CLI v2 credentials able to call RDS, EC2, EKS, and Cost Explorer in the intended accounts.&lt;/li&gt;
&lt;li&gt;Billing permission for &lt;code&gt;ce:GetCostAndUsage&lt;/code&gt;. For organization-wide results, run it from the AWS Organizations management account or a role there; member accounts see only their own &lt;a href="https://docs.aws.amazon.com/cost-management/latest/userguide/ce-access.html" rel="noopener noreferrer"&gt;Cost Explorer data&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;A current RDS pricing view for each Region you operate in. Rates and support calendars are Region and date sensitive.&lt;/li&gt;
&lt;li&gt;An upgrade test environment and an application owner before changing an engine version or lifecycle setting.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What changed, and what RDS is charging
&lt;/h2&gt;

&lt;p&gt;AWS lists these RDS for MySQL dates: MySQL 8.0 community end of life was April 30, 2026, RDS standard support ended July 31, 2026, Year 1 pricing began August 1, 2026, Year 3 begins August 1, 2028, and Extended Support ends July 31, 2029. &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.html" rel="noopener noreferrer"&gt;MySQL 5.7's calendar&lt;/a&gt; is already in Year 3: it began charging March 1, 2024, stepped up March 1, 2026, and ends June 30, 2029. MySQL 8.4 remains in standard support until July 31, 2029.&lt;/p&gt;

&lt;p&gt;The other calendars deserve the same review. RDS for PostgreSQL 11, 12, and 13 started paid Extended Support on April 1, 2024, March 1, 2025, and March 1, 2026. PostgreSQL 14 reaches the boundary after standard support ends February 28, 2027, with Year 1 pricing from March 1, 2027. Those dates come from the &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/PostgreSQLReleaseNotes/postgresql-release-calendar.html" rel="noopener noreferrer"&gt;RDS PostgreSQL release calendar&lt;/a&gt;. Aurora has a different schedule: Aurora MySQL 3, compatible with MySQL 8.0, remains in standard support until April 30, 2028, while &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQLReleaseNotes/aurorapostgresql-release-calendar.html" rel="noopener noreferrer"&gt;Aurora PostgreSQL's calendar&lt;/a&gt; shows the same paid dates as RDS PostgreSQL for 11 through 14. Do not apply the RDS MySQL 8.0 deadline to Aurora MySQL 3.&lt;/p&gt;

&lt;p&gt;For provisioned RDS for MySQL, the pricing page describes Extended Support as &lt;strong&gt;per vCPU per hour&lt;/strong&gt;. The vCPU count is the instance class's published vCPU count, not the CPU percentage you happened to use. AWS's &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.Summary.html" rel="noopener noreferrer"&gt;DB instance class tables&lt;/a&gt; are the source of record for the class. The charge applies to Multi-AZ standby instances, according to the &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/extended-support-charges.html" rel="noopener noreferrer"&gt;Extended Support charges guide&lt;/a&gt;, so a traditional Multi-AZ deployment has more billable vCPUs than its primary alone. Include every read replica too: the &lt;a href="https://aws.amazon.com/rds/mysql/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; says replicas running a version past standard support are charged as well, each with its own class and vCPU count.&lt;/p&gt;

&lt;p&gt;AWS documents Aurora Serverless v2 capacity in ACUs, but the RDS MySQL pricing page's vCPU formula is for provisioned RDS for MySQL. Do not convert ACUs to vCPUs for an invoice forecast unless the Aurora price page for the exact engine and Region supplies that rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Inventory every Region before estimating
&lt;/h2&gt;

&lt;p&gt;Start with a read-only snapshot of RDS instances and clusters. &lt;code&gt;EngineLifecycleSupport&lt;/code&gt; reports enrollment using &lt;code&gt;open-source-rds-extended-support&lt;/code&gt; or &lt;code&gt;open-source-rds-extended-support-disabled&lt;/code&gt;; it is not a substitute for checking the version.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; region &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;aws ec2 describe-regions \
  --query &lt;span class="s1"&gt;'Regions[].RegionName'&lt;/span&gt; --output text --no-cli-pager&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"### $region: DB instances"&lt;/span&gt;
  aws rds describe-db-instances --region &lt;span class="s2"&gt;"$region"&lt;/span&gt; \
    --query &lt;span class="s1"&gt;'DBInstances[].{Id:DBInstanceIdentifier,Engine:Engine,Version:EngineVersion,Class:DBInstanceClass,MultiAZ:MultiAZ,Source:ReadReplicaSourceDBInstanceIdentifier,Support:EngineLifecycleSupport}'&lt;/span&gt; \
    --output table --no-cli-pager

  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"### $region: clusters, including Aurora"&lt;/span&gt;
  aws rds describe-db-clusters --region &lt;span class="s2"&gt;"$region"&lt;/span&gt; \
    --query &lt;span class="s1"&gt;'DBClusters[].{Id:DBClusterIdentifier,Engine:Engine,Version:EngineVersion,Mode:EngineMode,Members:DBClusterMembers[].DBInstanceIdentifier,Support:EngineLifecycleSupport}'&lt;/span&gt; \
    --output table --no-cli-pager
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;describe-db-clusters&lt;/code&gt; covers Aurora and Multi-AZ DB clusters. For Aurora, lifecycle setting lives at the cluster level, then the &lt;code&gt;Members&lt;/code&gt; list tells you which DB instances to count. The &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/extended-support-creating-db-instance.html" rel="noopener noreferrer"&gt;RDS creation documentation&lt;/a&gt; confirms that members of a cluster share this setting.&lt;/p&gt;

&lt;p&gt;Use the service's support-date response rather than copying a calendar into automation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws rds describe-db-major-engine-versions \
  --engine mysql \
  --major-engine-version 8.0 \
  --query &lt;span class="s1"&gt;'DBMajorEngineVersions[].{Engine:Engine,Major:MajorEngineVersion,Lifecycles:SupportedEngineLifecycles}'&lt;/span&gt; \
  --output json --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/extended-support-viewing-support-dates.html" rel="noopener noreferrer"&gt;support-date API guide&lt;/a&gt; documents &lt;code&gt;SupportedEngineLifecycles&lt;/code&gt;, including its support name and start and end timestamps. Repeat it for &lt;code&gt;postgres&lt;/code&gt;, &lt;code&gt;aurora-mysql&lt;/code&gt;, and &lt;code&gt;aurora-postgresql&lt;/code&gt; where those engines are in scope.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the charge in Cost Explorer
&lt;/h2&gt;

&lt;p&gt;First discover the usage-type values in &lt;em&gt;your&lt;/em&gt; bill, and do not hard-code a filter before you have seen the returned values. This query groups all RDS usage by usage type, then &lt;code&gt;jq&lt;/code&gt; narrows the returned group names locally:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ce get-cost-and-usage \
  --time-period &lt;span class="nv"&gt;Start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2026-10-01,End=2026-11-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost UsageQuantity \
  --filter &lt;span class="s1"&gt;'{"Dimensions":{"Key":"SERVICE","Values":["Amazon Relational Database Service"]}}'&lt;/span&gt; \
  --group-by &lt;span class="nv"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;DIMENSION,Key=USAGE_TYPE \
  --output json \
&lt;span class="o"&gt;|&lt;/span&gt; jq &lt;span class="s1"&gt;'.ResultsByTime[].Groups[]
      | select(.Keys[0] | test("extended|support"; "i"))'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inspect the result before adding a Cost Category, budget, CUR query, or saved Cost Explorer filter. The point is to prove the actual usage type in the billing account, not to make a dashboard look complete with a guessed string. For detailed allocation, query the cost and usage data export using the service, usage-type, usage amount, and cost columns that appear in your exported schema. Data Exports schemas vary by export choice, so copy column names from the &lt;a href="https://docs.aws.amazon.com/cur/latest/userguide/dataexports-table-dictionary.html" rel="noopener noreferrer"&gt;table's own definition&lt;/a&gt; rather than borrowing an Athena query from a different export. Pair this with &lt;a href="https://devops-daily.com/posts/cloud-cost-allocation-tags-aws-gcp-azure" rel="noopener noreferrer"&gt;cost-allocation tags&lt;/a&gt; if a team needs to own the upgrade decision.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Three ways to keep the inventory current&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CLI inventory&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws rds describe-db-instances --region us-east-1 \
  --query &lt;span class="s1"&gt;'DBInstances[].{Id:DBInstanceIdentifier,Engine:Engine,Version:EngineVersion,Class:DBInstanceClass,Support:EngineLifecycleSupport}'&lt;/span&gt; \
  --output table --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Cost Explorer&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws ce get-cost-and-usage \
  --time-period &lt;span class="nv"&gt;Start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;2026-10-01,End=2026-11-01 \
  --granularity MONTHLY --metrics UnblendedCost \
  --group-by &lt;span class="nv"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;DIMENSION,Key=USAGE_TYPE

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Terraform&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="n"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_db_instance"&lt;/span&gt; &lt;span class="s2"&gt;"app"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;identifier&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"app-mysql"&lt;/span&gt;
  &lt;span class="n"&gt;engine&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"mysql"&lt;/span&gt;
  &lt;span class="n"&gt;engine_version&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"8.4"&lt;/span&gt;
  &lt;span class="n"&gt;instance_class&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"db.r6g.large"&lt;/span&gt;
  &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"appadmin"&lt;/span&gt;
  &lt;span class="n"&gt;manage_master_user_password&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="n"&gt;engine_lifecycle_support&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"open-source-rds-extended-support-disabled"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Put a monthly number on MySQL 8.0
&lt;/h2&gt;

&lt;p&gt;The formula is simple once the inventory is right:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Extended Support per month = billable vCPUs x billed hours x rate per vCPU-hour

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As checked November 11, 2026, AWS's &lt;a href="https://aws.amazon.com/rds/mysql/pricing/" rel="noopener noreferrer"&gt;RDS for MySQL pricing page&lt;/a&gt; shows $0.100 per vCPU-hour for Years 1 and 2 and $0.200 for Year 3 in its US East rate table, and its FAQ example for a US East (N. Virginia) db.r5.large uses the same $0.100 Year 1 rate. AWS says the actual price depends on MySQL version, Region, and calendar year, so confirm the selected Region and engine in the current pricing view before treating this as a purchase forecast.&lt;/p&gt;

&lt;p&gt;For a transparent planning example, assume two &lt;strong&gt;single-AZ&lt;/strong&gt; &lt;code&gt;db.r6g.2xlarge&lt;/code&gt; MySQL 8.0 instances in US East (N. Virginia), 8 vCPUs each, at 730 hours per month. That is 16 billable vCPUs. The Extended Support addition is &lt;code&gt;16 x 730 x $0.100 = $1,168&lt;/code&gt; in Years 1 and 2. At the Year 3 rate it is &lt;code&gt;$2,336&lt;/code&gt;. This excludes the instance's normal price, storage, I/O, backups, and transfer. Add the standby vCPUs before using this calculation for Multi-AZ.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monthly RDS Extended Support addition for two db.r6g.2xlarge instances&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Years 1-2&lt;/td&gt;
&lt;td&gt;1168$&lt;/td&gt;
&lt;td&gt;Extended Support&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Year 3&lt;/td&gt;
&lt;td&gt;2336$&lt;/td&gt;
&lt;td&gt;Extended Support&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;16 billable vCPUs x 730 hours. Uses the US East (N. Virginia) $0.100 Year 1-2 and $0.200 Year 3 per-vCPU-hour example on AWS RDS for MySQL pricing, checked November 11, 2026. Single-AZ only, excludes normal RDS charges.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  EKS has a smaller, separate support timer
&lt;/h2&gt;

&lt;p&gt;EKS charges per cluster-hour, not per vCPU. AWS lists $0.10 per cluster-hour during standard support and $0.60 in extended support. A Kubernetes version gets 14 months of standard support and 12 months of extended support. Check each cluster's version and policy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws eks describe-cluster --name platform-prod --region us-east-1 \
  --query &lt;span class="s1"&gt;'cluster.{version:version,support:upgradePolicy.supportType}'&lt;/span&gt; \
  --output table --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;supportType&lt;/code&gt; field is &lt;code&gt;STANDARD&lt;/code&gt; or &lt;code&gt;EXTENDED&lt;/code&gt;. AWS's &lt;a href="https://docs.aws.amazon.com/eks/latest/userguide/view-upgrade-policy.html" rel="noopener noreferrer"&gt;upgrade policy guide&lt;/a&gt; says &lt;code&gt;STANDARD&lt;/code&gt; causes an automatic upgrade at the end of standard support, while &lt;code&gt;EXTENDED&lt;/code&gt; keeps the version in extended support. Change it only after validating the next Kubernetes version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws eks update-cluster-config --name platform-prod --region us-east-1 \
  --upgrade-policy &lt;span class="nv"&gt;supportType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;STANDARD \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://aws.amazon.com/eks/pricing/" rel="noopener noreferrer"&gt;EKS pricing page&lt;/a&gt; has the current rates. This setting avoids the extended-support choice by accepting automatic control-plane version movement, not by making an old version free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide whether to upgrade or pay knowingly
&lt;/h2&gt;

&lt;p&gt;For RDS MySQL, 8.0 to 8.4 is a supported major upgrade path. Outside the lifecycle behavior described below, AWS does not automatically apply major upgrades, because they can break compatibility. Its &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MySQL.Major.html" rel="noopener noreferrer"&gt;major-upgrade guide&lt;/a&gt; says RDS runs mandatory 8.0 to 8.4 prechecks before stopping the instance. Failures are logged in &lt;code&gt;PrePatchCompatibility.log&lt;/code&gt;, so treat that log as an upgrade backlog, not a reason to ignore the charge.&lt;/p&gt;

&lt;p&gt;Test three classes of problems before a production cutover:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;mysql_native_password&lt;/code&gt; is disabled by default in MySQL 8.4, so accounts and old clients that rely on it need a deliberate plan. &lt;a href="https://dev.mysql.com/doc/refman/8.4/en/native-pluggable-authentication.html" rel="noopener noreferrer"&gt;MySQL's documentation&lt;/a&gt; describes the change and its connection failures.&lt;/li&gt;
&lt;li&gt;MySQL 8.4 changes defaults including &lt;code&gt;innodb_adaptive_hash_index&lt;/code&gt; from &lt;code&gt;ON&lt;/code&gt; to &lt;code&gt;OFF&lt;/code&gt;, &lt;code&gt;innodb_change_buffering&lt;/code&gt; from &lt;code&gt;all&lt;/code&gt; to &lt;code&gt;none&lt;/code&gt;, and &lt;code&gt;innodb_io_capacity&lt;/code&gt; from 200 to 10000. Review parameter groups and performance expectations against &lt;a href="https://dev.mysql.com/doc/refman/8.4/en/upgrading-from-previous-series.html" rel="noopener noreferrer"&gt;MySQL's changed-defaults table&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;RDS prechecks reject obsolete types or functions, invalid trigger definers or creation context, new reserved-word collisions, overlapping &lt;code&gt;mysql&lt;/code&gt; system-table names, and obsolete &lt;code&gt;sql_mode&lt;/code&gt; settings.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a production path, use an RDS Blue/Green deployment when its prerequisites fit the database. It creates a green staging copy that stays in sync with blue, permits testing before cutover, and AWS says switchover is usually under one minute, although workload conditions can make it longer. See the &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-overview.html" rel="noopener noreferrer"&gt;RDS Blue/Green overview&lt;/a&gt; and &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-creating.html" rel="noopener noreferrer"&gt;creation guide&lt;/a&gt;. The method is supported for RDS MySQL 8.0 and 8.4. It is a better place to find application and parameter incompatibilities than the maintenance window.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Disabling Extended Support on an existing instance or cluster already past standard support does not leave it on the old version without a charge. AWS states that it automatically upgrades to the next supported major version. Treat that change as an upgrade operation, with testing and an owner, not as a FinOps toggle.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Prevent the next surprise
&lt;/h2&gt;

&lt;p&gt;For new CLI-created RDS DB instances and Multi-AZ DB clusters, omitting lifecycle support defaults to enabled. Add the explicit opt-out to creation automation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws rds create-db-instance \
  --db-instance-identifier app-mysql \
  --engine mysql \
  --engine-version 8.4 \
  --db-instance-class db.r6g.large \
  --allocated-storage 100 \
  --master-username appadmin \
  --manage-master-user-password \
  --no-publicly-accessible \
  --db-subnet-group-name app-private \
  --vpc-security-group-ids sg-0123456789abcdef0 \
  --engine-lifecycle-support open-source-rds-extended-support-disabled \
  --no-cli-pager

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After a support boundary has passed, that setting makes a new instance creation fail rather than enroll it. Before the boundary, a disabled instance is automatically upgraded on or shortly after the end of standard support. The &lt;a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/extended-support-creating-db-instance.html" rel="noopener noreferrer"&gt;RDS lifecycle documentation&lt;/a&gt; defines both behaviors.&lt;/p&gt;

&lt;p&gt;Terraform exposes the equivalent &lt;code&gt;engine_lifecycle_support&lt;/code&gt; argument on &lt;a href="https://github.com/hashicorp/terraform-provider-aws/blob/main/website/docs/r/db_instance.html.markdown" rel="noopener noreferrer"&gt;&lt;code&gt;aws_db_instance&lt;/code&gt;&lt;/a&gt; and &lt;a href="https://github.com/hashicorp/terraform-provider-aws/blob/main/website/docs/r/rds_cluster.html.markdown" rel="noopener noreferrer"&gt;&lt;code&gt;aws_rds_cluster&lt;/code&gt;&lt;/a&gt;. Make it explicit in modules, then add a policy check that rejects the enabled value where the organization expects an upgrade instead. The site's &lt;a href="https://devops-daily.com/posts/terraform-best-practices" rel="noopener noreferrer"&gt;Terraform best-practices guide&lt;/a&gt; has useful patterns for keeping that module input visible in review. For enforcement outside Terraform, use a custom AWS Config rule that you have tested in your own account.&lt;/p&gt;

&lt;p&gt;The decision record should have three fields: the affected version, the monthly Extended Support addition, and either a named upgrade date or an approved end date for paid support. That turns an automatic charge into a time-bounded engineering choice. If you need a rehearsal pattern for the release itself, the site's &lt;a href="https://devops-daily.com/posts/deployment-strategies-guide" rel="noopener noreferrer"&gt;blue-green deployment guide&lt;/a&gt; is a useful companion.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/aws-rds-extended-support-mysql-8-0-charges" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>finops</category>
      <category>aws</category>
      <category>rds</category>
      <category>mysql</category>
    </item>
    <item>
      <title>AI Agent Sandboxes Compared: Render, Vercel, Cloudflare, E2B and Daytona</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Mon, 16 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/ai-agent-sandboxes-compared-render-vercel-cloudflare-e2b-and-daytona-5eoe</link>
      <guid>https://dev.to/devopsdaily/ai-agent-sandboxes-compared-render-vercel-cloudflare-e2b-and-daytona-5eoe</guid>
      <description>&lt;p&gt;A one-shot code interpreter can disappear the moment it has returned a result. A coding agent often cannot: it may hold a development server, wait for a human to review a diff, or return to the same repository tomorrow. That difference turns timeout semantics, snapshot contents, idle billing, and egress controls into architecture decisions.&lt;/p&gt;

&lt;p&gt;This is a sandbox-only comparison of Render, Vercel, Cloudflare, E2B, and Daytona. If you are choosing a broader agent runtime with identity, memory, tool gateways, and an agent loop, start with &lt;a href="https://dev.to/devopsdaily/where-to-run-ai-agents-8-managed-agent-runtimes-compared-mgo"&gt;our managed-agent runtime comparison&lt;/a&gt;. Here, the question is narrower: where should your agent's shell, files, and processes live?&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Short tool calls:&lt;/strong&gt; Cloudflare fits event-driven work when a Worker already owns the request. Its instance can stop after the work, but save files yourself before it does. &lt;a href="https://developers.cloudflare.com/sandbox/concepts/lifetime/" rel="noopener noreferrer"&gt;Cloudflare documents the lifecycle and snapshot boundary.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long-running coding agents:&lt;/strong&gt; Render is the clearest fit when a process must survive a human wait. It can disable both timeouts and its runtime snapshots retain memory and processes. &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;It is a public beta.&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pause for review, then resume in place:&lt;/strong&gt; Daytona VM sandboxes and E2B can preserve memory and processes through a pause. Daytona charges disk while paused; E2B says a pause can preserve filesystem and memory. &lt;a href="https://www.daytona.io/docs/billing" rel="noopener noreferrer"&gt;Daytona billing&lt;/a&gt; and &lt;a href="https://e2b.dev/security" rel="noopener noreferrer"&gt;E2B security documentation&lt;/a&gt; describe those states.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cheapest idle:&lt;/strong&gt; E2B's &lt;a href="https://e2b.dev/" rel="noopener noreferrer"&gt;homepage&lt;/a&gt; labels a paused sandbox as unbilled. Render stops CPU and memory charges when a sandbox is suspended, leaving snapshot storage at &lt;a href="https://render.com/pricing" rel="noopener noreferrer"&gt;$0.08 per GB-month&lt;/a&gt;; Vercel bills its filesystem snapshot at the same rate, and Daytona VM pauses bill reserved disk. All of these are cents for an overnight wait; plan minimums matter more.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A workload description: how long a task actively computes, how long it waits, and whether a running process must survive the wait.&lt;/li&gt;
&lt;li&gt;An account with the plan and regional availability needed by the provider you choose.&lt;/li&gt;
&lt;li&gt;A place to persist the work product outside the sandbox when deletion would be unacceptable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What we compared
&lt;/h2&gt;

&lt;p&gt;The four dimensions in the table below are deliberately operational: how long the machine exists, what a pause means, what is billed, and what survives. “Not documented” means we could not find the detail in current official vendor material, not that the vendor necessarily lacks the feature.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;Runtime limits&lt;/th&gt;
&lt;th&gt;Suspend or resume&lt;/th&gt;
&lt;th&gt;Pricing and idle state&lt;/th&gt;
&lt;th&gt;What persists&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Render&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Default runtime timeout &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;24 hours&lt;/a&gt;; set &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;60 seconds to 30 days&lt;/a&gt; (&lt;a href="https://render.com/docs/cli-reference" rel="noopener noreferrer"&gt;24 hours on Hobby workspaces&lt;/a&gt;), or &lt;code&gt;0&lt;/code&gt; to disable. Default idle timeout &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;15 minutes&lt;/a&gt;. Up to &lt;a href="https://render.com/docs/sandboxes-limits" rel="noopener noreferrer"&gt;8 CPU, 16 GB RAM, 40 GB writable disk&lt;/a&gt;, &lt;a href="https://render.com/docs/sandboxes-limits" rel="noopener noreferrer"&gt;1,000 concurrent&lt;/a&gt; per group.&lt;/td&gt;
&lt;td&gt;Idle suspension keeps the filesystem and can resume within &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;30 days&lt;/a&gt;. Runtime snapshots keep filesystem, memory, and CPU state; filesystem snapshots keep files only.&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://render.com/blog/introducing-render-sandboxes" rel="noopener noreferrer"&gt;Active CPU plus allocated memory&lt;/a&gt;; suspended means snapshot-storage only. List rates: &lt;a href="https://render.com/pricing" rel="noopener noreferrer"&gt;$0.12 per active CPU-hour plus $0.038 per hour per 2 GB of memory ($0.076/hour for 2c-4g), and $0.08 per GB-month of snapshot storage&lt;/a&gt;.&lt;/td&gt;
&lt;td&gt;Runtime snapshot preserves files, processes, and memory. Termination permanently deletes the filesystem unless exported or snapshotted. &lt;a href="https://render.com/blog/introducing-render-sandboxes" rel="noopener noreferrer"&gt;Outbound policy can allow all, allow a domain list, or deny all.&lt;/a&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Vercel&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Session default &lt;a href="https://vercel.com/kb/guide/vercel-sandbox-duration-and-persistence" rel="noopener noreferrer"&gt;5 minutes&lt;/a&gt;, maximum &lt;a href="https://vercel.com/kb/guide/vercel-sandbox-duration-and-persistence" rel="noopener noreferrer"&gt;45 minutes on Hobby and 24 hours on Pro and Enterprise&lt;/a&gt;. Up to &lt;a href="https://vercel.com/sandbox" rel="noopener noreferrer"&gt;4, 8, or 32 vCPU by plan&lt;/a&gt;, with &lt;a href="https://vercel.com/sandbox" rel="noopener noreferrer"&gt;2 GB per vCPU&lt;/a&gt;.&lt;/td&gt;
&lt;td&gt;There is no in-memory pause. Persistent sandboxes stop, snapshot the filesystem, then boot a new session on use.&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://vercel.com/docs/sandbox/pricing" rel="noopener noreferrer"&gt;$0.128 active CPU-hour and $0.0212 GB-hour provisioned memory&lt;/a&gt;, plus &lt;a href="https://vercel.com/docs/sandbox/pricing" rel="noopener noreferrer"&gt;$0.08 GB-month snapshot storage&lt;/a&gt;. CPU waiting on I/O is not billed, but memory is billed while the session runs.&lt;/td&gt;
&lt;td&gt;Persistent is the default: filesystem and installed packages return, processes do not. A &lt;a href="https://vercel.com/kb/guide/vercel-sandbox-duration-and-persistence" rel="noopener noreferrer"&gt;Drive&lt;/a&gt; can outlive a sandbox. A &lt;a href="https://vercel.com/docs/sandbox/concepts/firewall" rel="noopener noreferrer"&gt;sandbox firewall&lt;/a&gt; can allow all, deny all, or apply a user-defined policy, changeable at runtime.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cloudflare Sandboxes&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No guaranteed runtime duration. Inactivity can be set to &lt;a href="https://developers.cloudflare.com/sandbox/concepts/lifetime/" rel="noopener noreferrer"&gt;up to 6 hours&lt;/a&gt;; a platform event can stop an instance. Largest container is &lt;a href="https://developers.cloudflare.com/containers/platform/pricing/" rel="noopener noreferrer"&gt;4 vCPU, 12 GiB, 20 GB&lt;/a&gt;.&lt;/td&gt;
&lt;td&gt;No pause. Stop ends memory, processes, and disk. A beta snapshot is filesystem-only and must be restored by your application.&lt;/td&gt;
&lt;td&gt;Containers bill per &lt;a href="https://developers.cloudflare.com/containers/platform/pricing/" rel="noopener noreferrer"&gt;10 ms while running&lt;/a&gt;: active CPU, provisioned memory, and disk. Workers Paid includes &lt;a href="https://developers.cloudflare.com/containers/platform/pricing/" rel="noopener noreferrer"&gt;375 vCPU-minutes, 25 GiB-hours, and 200 GB-hours monthly&lt;/a&gt;.&lt;/td&gt;
&lt;td&gt;A Durable Object keeps its storage until code deletes it. Snapshots expire &lt;a href="https://developers.cloudflare.com/sandbox/concepts/lifetime" rel="noopener noreferrer"&gt;30 days after creation or last restore&lt;/a&gt;; mounted R2 objects outlive instances. Network egress pricing is &lt;a href="https://developers.cloudflare.com/containers/platform/pricing/" rel="noopener noreferrer"&gt;documented by region&lt;/a&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;E2B&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Hobby sessions up to &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;1 hour&lt;/a&gt;; Pro up to &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;24 hours&lt;/a&gt;. Hobby has &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;10 concurrent&lt;/a&gt;; Pro has &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;100, expandable to 1,100&lt;/a&gt;. Default size is &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;2 vCPU and 4 GiB&lt;/a&gt;; a larger official per-sandbox maximum is not documented.&lt;/td&gt;
&lt;td&gt;Pause can preserve filesystem and memory for resume.&lt;/td&gt;
&lt;td&gt;E2B charges per second of a &lt;strong&gt;running&lt;/strong&gt; sandbox: &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;2 vCPU is $0.000028/s and 4 GiB is $0.000018/s&lt;/a&gt;. Pro has a &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;$150/month minimum&lt;/a&gt;. Its &lt;a href="https://e2b.dev/" rel="noopener noreferrer"&gt;homepage&lt;/a&gt; labels the paused state as unbilled.&lt;/td&gt;
&lt;td&gt;E2B says &lt;a href="https://e2b.dev/security" rel="noopener noreferrer"&gt;timeout or shutdown destroys sandboxes&lt;/a&gt;; pause can preserve filesystem and memory. Retention and storage pricing for paused sandboxes: not documented in the sources reviewed.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Daytona&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Default auto-stop is &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;15 minutes&lt;/a&gt;; VM sandboxes default to &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;60-minute auto-pause&lt;/a&gt; if neither interval is set. Both can be disabled. Sandboxes default to &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;1 vCPU, 1 GB RAM, and 3 GiB disk&lt;/a&gt;, with a per-sandbox maximum of 4 vCPUs, 8 GB RAM, and 10 GB disk; organization limits are on the &lt;a href="https://www.daytona.io/docs/limits" rel="noopener noreferrer"&gt;limits page&lt;/a&gt;.&lt;/td&gt;
&lt;td&gt;VM &lt;code&gt;pause()&lt;/code&gt; freezes running processes and retains memory. Containers do not support pause.&lt;/td&gt;
&lt;td&gt;Per-second resources: &lt;a href="https://www.daytona.io/" rel="noopener noreferrer"&gt;$0.0504/vCPU-hour, $0.0162/GiB-hour, and $0.000108/GiB-hour storage after the first 5 GiB&lt;/a&gt;. Paused and stopped VM sandboxes bill disk only.&lt;/td&gt;
&lt;td&gt;VM pause preserves memory and filesystem; stop clears memory. Containers can archive filesystem to object storage. &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;TTL and auto-delete can remove a sandbox.&lt;/a&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Checked against official docs on 2026-11-11; prices in USD, list prices.&lt;/p&gt;

&lt;p&gt;All five expose a programmatic sandbox, but their isolation differs where documented: Render and Vercel use &lt;a href="https://render.com/blog/introducing-render-sandboxes" rel="noopener noreferrer"&gt;Firecracker microVMs&lt;/a&gt; and &lt;a href="https://vercel.com/docs/sandbox" rel="noopener noreferrer"&gt;Firecracker microVMs&lt;/a&gt; respectively, E2B uses a &lt;a href="https://e2b.dev/security" rel="noopener noreferrer"&gt;Firecracker microVM with its own kernel&lt;/a&gt;, and Cloudflare describes a container instance as a Linux VM. &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;Daytona publishes container, Linux VM, Windows, and GPU sandbox classes.&lt;/a&gt; SDK coverage differs too: Render and Vercel document TypeScript and Python SDKs, and Daytona documents &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;Python, TypeScript, Ruby, Go, and Java&lt;/a&gt;. Render is &lt;a href="https://render.com/docs/sandboxes-limits" rel="noopener noreferrer"&gt;Oregon-only&lt;/a&gt; during the beta, Daytona documents &lt;code&gt;us&lt;/code&gt; and &lt;code&gt;eu&lt;/code&gt; targets, and Vercel lists its sandbox regions on its &lt;a href="https://vercel.com/docs/sandbox/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Render: a runtime snapshot is a real checkpoint
&lt;/h2&gt;

&lt;p&gt;Render Sandboxes entered &lt;a href="https://render.com/blog/introducing-render-sandboxes" rel="noopener noreferrer"&gt;public beta on November 4, 2026&lt;/a&gt;, which is the newest official status even though older pages may describe early access. The distinction that matters is its two snapshot kinds. A runtime snapshot captures writable files, memory, and CPU state, so a restored sandbox resumes running processes. A filesystem snapshot captures files and installed tools but not a process. &lt;a href="https://render.com/docs/sandboxes-snapshots" rel="noopener noreferrer"&gt;A runtime snapshot must restore to the same compute plan.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That makes Render unusually direct for a coding agent waiting on a reviewer: its default 15-minute idle timer snapshots and suspends, a command or file transfer resumes it, and background processes do &lt;strong&gt;not&lt;/strong&gt; count as activity. &lt;a href="https://render.com/docs/sandboxes-creating" rel="noopener noreferrer"&gt;Render defines idle as no command execution or file transfer.&lt;/a&gt; If a test run is working only inside the sandbox, either keep an external interaction alive or disable idle suspension. A runtime timeout also resets after a suspend and resume.&lt;/p&gt;

&lt;p&gt;There are beta gaps to plan for. Sandboxes do not yet have private networking to other Render services or public URLs, and credential brokering is not generally available. &lt;a href="https://render.com/docs/sandboxes" rel="noopener noreferrer"&gt;Those are listed as beta limitations.&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Vercel: persistence is filesystem persistence
&lt;/h2&gt;

&lt;p&gt;Vercel separates a long-lived named sandbox from the VM session inside it. When a persistent sandbox stops, it snapshots the filesystem, and the next operation boots a new session from that snapshot. &lt;a href="https://vercel.com/kb/guide/vercel-sandbox-duration-and-persistence" rel="noopener noreferrer"&gt;The default is persistent.&lt;/a&gt; It is convenient for a workspace that needs &lt;code&gt;node_modules&lt;/code&gt;, source edits, and configuration to return. It is not a pause: a development server, process memory, open terminal, and connection state are gone.&lt;/p&gt;

&lt;p&gt;Vercel's &lt;a href="https://vercel.com/docs/sandbox/concepts/firewall" rel="noopener noreferrer"&gt;sandbox firewall&lt;/a&gt; can switch from allow-all to deny-all or a user-defined policy at runtime without restarting the process, and it can broker credentials so secrets never enter the sandbox. For cost work, do not treat active CPU as the whole bill. Vercel excludes I/O wait from CPU metering, but provisioned memory is still measured for the time the session is running. &lt;a href="https://vercel.com/docs/sandbox/pricing" rel="noopener noreferrer"&gt;Its own example explains both meters.&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Cloudflare: an instance is not the durable object
&lt;/h2&gt;

&lt;p&gt;Cloudflare's design is useful when the agent's durable control plane is a Worker and Durable Object, and a Linux machine is an attached execution unit. The Durable Object can retain a session record or snapshot identifier after the instance stops, but the instance's files, memory, and processes do not survive that stop. &lt;a href="https://developers.cloudflare.com/sandbox/concepts/lifetime/" rel="noopener noreferrer"&gt;Cloudflare documents this split explicitly.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The sharp edge is activity. A Worker request, stream, WebSocket, or alarm can keep the Durable Object active. Work running only &lt;em&gt;inside&lt;/em&gt; the instance does not. So a build can receive &lt;code&gt;SIGTERM&lt;/code&gt; when the inactivity timer ends even while it is still working. &lt;a href="https://developers.cloudflare.com/sandbox/concepts/lifetime/" rel="noopener noreferrer"&gt;Use an alarm that checks work more frequently than the inactivity timeout, or arrange for the process to finish before the timeout.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Snapshot when files matter, and restart the process after restore. Cloudflare says snapshots are public beta, filesystem-only, application-managed, and not restored automatically. Mounted R2 is the better fit for files that must outlive a workspace or be shared.&lt;/p&gt;

&lt;h2&gt;
  
  
  E2B: pause holds the machine state
&lt;/h2&gt;

&lt;p&gt;E2B is a direct microVM sandbox service. Its official security material says each sandbox has its own Firecracker microVM and that filesystem and memory state can be preserved through pause and resume. &lt;a href="https://e2b.dev/security" rel="noopener noreferrer"&gt;That is the capability a human-review workflow needs.&lt;/a&gt; Its pricing page bills a running sandbox per second, rather than publishing separate active-CPU and provisioned-memory notions.&lt;/p&gt;

&lt;p&gt;The important plan boundary is time: the free Hobby tier allows sessions up to one hour, while Pro allows up to 24 hours and starts at $150 per month. E2B's Enterprise offering is where it advertises multi-day and persistent sessions. &lt;a href="https://e2b.dev/pricing" rel="noopener noreferrer"&gt;Those are plan statements, not an implied retention guarantee for every paused sandbox.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;E2B's homepage marks the paused state as unbilled, which makes it a cheap place for a sandbox to wait. What the official material reviewed here does not give is how long a paused sandbox is retained, or a per-sandbox maximum above the listed configuration options. Ask before you build a long review queue on paused sandboxes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Daytona: choose VM if you mean pause
&lt;/h2&gt;

&lt;p&gt;Daytona has more lifecycle states than the others, and the sandbox class changes what each state means. A Linux VM sandbox can pause, freezing processes and retaining memory. A container sandbox cannot pause, but can stop and later archive its filesystem. &lt;a href="https://www.daytona.io/docs/sandboxes" rel="noopener noreferrer"&gt;The lifecycle documentation distinguishes the two.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The default timer is another operational trap. Daytona's auto-stop fires after 15 minutes of no qualifying events, and an internal long-running process does not count. VM-capable sandboxes instead default to a 60-minute auto-pause when neither timer is supplied. &lt;a href="https://www.daytona.io/docs/en/troubleshooting/" rel="noopener noreferrer"&gt;Daytona calls out this exact failure mode in troubleshooting.&lt;/a&gt; Have the controller call &lt;code&gt;refreshActivity()&lt;/code&gt; or configure the lifecycle deliberately.&lt;/p&gt;

&lt;p&gt;Paused VM state is not billed for CPU or RAM, but it is billed for reserved disk. That makes Daytona a strong review-pause option, not automatically the cheapest idle option. Use archive for a container filesystem that can tolerate a cold return; archived containers stop billing, while snapshots remain billed storage. &lt;a href="https://www.daytona.io/docs/billing" rel="noopener noreferrer"&gt;Daytona's billing table is explicit on each state.&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Worked cost: 40 active minutes, then an 8-hour review
&lt;/h2&gt;

&lt;p&gt;Assume the agent actively uses both vCPUs for 30 minutes, pauses or stops for 8 hours, then uses both vCPUs for 10 more minutes. The arithmetic below excludes network, creation, and storage retained beyond the stated resources. It also assumes no included allowance or account credit.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Render: $0.21 plus snapshot storage.&lt;/strong&gt; &lt;code&gt;(2 vCPU x 2/3 h x $0.12) + (2/3 h x $0.076 memory for 2c-4g) = $0.2107&lt;/code&gt;. The 8-hour suspended period has no CPU or memory charge; snapshot storage is $0.08 per GB-month, so a 10 GB snapshot held for 8 hours adds under one cent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vercel: $0.23 plus snapshot storage and one creation event.&lt;/strong&gt; &lt;code&gt;(2 x 2/3 x $0.128) + (4 x 2/3 x $0.0212) = $0.2272&lt;/code&gt;. Stop, then let persistence restore the filesystem for the second session. Its 8-hour stored snapshot is charged by size at &lt;a href="https://vercel.com/docs/sandbox/pricing" rel="noopener noreferrer"&gt;$0.08/GB-month&lt;/a&gt;, under one cent for 10 GB over 8 hours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cloudflare: not computable for exactly 2 vCPU and 4 GiB.&lt;/strong&gt; Cloudflare sells fixed instance types, and its 2-vCPU type has &lt;a href="https://developers.cloudflare.com/containers/platform/pricing" rel="noopener noreferrer"&gt;8 GiB and 16 GB disk&lt;/a&gt;, not 4 GiB. A saved workspace also needs a separately sized snapshot or R2 object.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;E2B: $0.1104 in running compute, plus the Pro plan minimum.&lt;/strong&gt; &lt;code&gt;2,400 seconds x ($0.000028 + $0.000018) = $0.1104&lt;/code&gt;; E2B labels the 8-hour pause as unbilled. We assume Pro because the workflow spans more than Hobby's one-hour session limit; check how your plan counts a paused-and-resumed sandbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Daytona: $0.1104 for CPU and RAM, plus disk during all 8 hours and 40 minutes.&lt;/strong&gt; &lt;code&gt;(2 x 2/3 x $0.0504) + (4 x 2/3 x $0.0162) = $0.1104&lt;/code&gt;. The result cannot include disk without choosing a disk size. The first 5 GiB are free, then the published disk rate applies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Known compute cost for the review-pause scenario&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E2B compute only&lt;/td&gt;
&lt;td&gt;0.1104$&lt;/td&gt;
&lt;td&gt;known compute&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Daytona CPU + RAM&lt;/td&gt;
&lt;td&gt;0.1104$&lt;/td&gt;
&lt;td&gt;known compute&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Render before snapshot storage&lt;/td&gt;
&lt;td&gt;0.2107$&lt;/td&gt;
&lt;td&gt;known compute&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vercel before snapshot storage&lt;/td&gt;
&lt;td&gt;0.2272$&lt;/td&gt;
&lt;td&gt;known compute&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;40 active minutes at 2 vCPU and 4 GiB, then 8 hours paused or stopped. Values exclude snapshot and disk storage, which depend on size (under one cent per 10 GB for 8 hours at Render and Vercel). Render assumes both vCPUs are active. E2B excludes its $150/month Pro minimum. Official list prices checked November 11, 2026.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The chart is not a lowest-price ranking. It is a warning that idle semantics move cost between CPU, memory, snapshot storage, disk storage, and plan minimums. A model call that waits for a response can be cheap in active-CPU systems, while a running session with provisioned memory continues to accrue memory cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  The smallest useful SDK shape
&lt;/h2&gt;

&lt;p&gt;The following calls follow current official examples. They are deliberately lifecycle examples, not a production authorization model. Cloudflare and E2B are omitted because their current examples require application-specific Durable Object setup, or because we could not verify a current TypeScript pause call without relying on a beta SDK reference.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Create, run, then suspend or stop&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Render&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;Render&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@renderinc/sdk'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sandboxes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sandboxes&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandboxes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;timeoutSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandboxes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'echo hello'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandboxes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;terminate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Vercel&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;Sandbox&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@vercel/sandbox'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;runCommand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;cmd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'node'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'-e'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'console.log(\"hello\")'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// Persistent sandboxes snapshot their filesystem on stop.&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Daytona&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;Daytona&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@daytona/sdk'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;daytona&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Daytona&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;daytona&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'daytona-vm-medium'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;autoStopInterval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;executeCommand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'echo hello'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// VM sandboxes preserve memory and running processes on pause.&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pause&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  How to pick
&lt;/h2&gt;

&lt;p&gt;Choose &lt;strong&gt;Render&lt;/strong&gt; if preserving a process through an idle wait is the defining requirement, and the beta status plus Oregon placement are acceptable. Choose &lt;strong&gt;Vercel&lt;/strong&gt; if you need a named workspace whose filesystem resumes cleanly, and a restarted process is fine. Choose &lt;strong&gt;Cloudflare&lt;/strong&gt; when the Worker and Durable Object are your durable agent control plane, with the sandbox treated as replaceable compute. Choose &lt;strong&gt;E2B&lt;/strong&gt; when you want a focused microVM service with pause and resume, but price the Pro minimum and verify persistence retention. Choose &lt;strong&gt;Daytona VM sandboxes&lt;/strong&gt; when a true pause matters and disk-only pause billing fits the workload.&lt;/p&gt;

&lt;p&gt;Do not use an auto-stop default as job completion. For every option, persist a task checkpoint outside process memory before a timeout, a stop, or a platform event. If an agent can reach a production system, also constrain egress and give it a narrowly scoped identity. &lt;a href="https://dev.to/devopsdaily/we-hid-96-instructions-in-the-logs-an-ops-agent-reads-here-is-what-stopped-them-3fec"&gt;Our guide to agent prompt-injection controls&lt;/a&gt; explains why that boundary belongs outside the model prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not confirm
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Render retention time for an automatically suspended sandbox beyond the stated 30-day resumability period.&lt;/li&gt;
&lt;li&gt;A Vercel in-memory pause (its persistence is filesystem-only).&lt;/li&gt;
&lt;li&gt;A Cloudflare fixed maximum instance lifetime or an exact 2 vCPU / 4 GiB configuration.&lt;/li&gt;
&lt;li&gt;E2B's retention period for paused sandboxes and a public per-sandbox maximum above the listed sizes.&lt;/li&gt;
&lt;li&gt;Daytona's retention duration for a paused VM sandbox.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/ai-agent-sandboxes-compared-render-vercel-cloudflare-e2b-daytona" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>cloud</category>
      <category>aiagents</category>
      <category>sandboxes</category>
      <category>e2b</category>
    </item>
    <item>
      <title>The Agentic Loop Explained: Step Through the Plan, Build, Judge and Repeat Loop Behind AI Coding Agents</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Mon, 16 Nov 2026 10:52:54 +0000</pubDate>
      <link>https://dev.to/devopsdaily/the-agentic-loop-explained-step-through-the-plan-build-judge-and-repeat-loop-behind-ai-coding-36g2</link>
      <guid>https://dev.to/devopsdaily/the-agentic-loop-explained-step-through-the-plan-build-judge-and-repeat-loop-behind-ai-coding-36g2</guid>
      <description>&lt;p&gt;Ask a chat model for a signup endpoint and you get one answer. A coding agent works differently: it can write the route, run the tests, read the failure, fix it, and keep going until some check says the job is done. The model inside can be the same. The difference is a loop wrapped around it, and that loop is what this post covers: the agentic loop explained one step at a time, with an animated simulator you can pause, step through and break on purpose.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://devops-daily.com/games/agentic-loop-simulator" rel="noopener noreferrer"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsdkbm6v943c62rmud0si.jpg" alt="Agentic Loop Simulator on DevOps Daily" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Disclosure: I help build the simulator; it is free, runs in the browser, and needs no signup.&lt;/p&gt;

&lt;h2&gt;
  
  
  One call versus a loop
&lt;/h2&gt;

&lt;p&gt;A single model call reads a prompt and writes a reply. An agentic loop turns that into a cycle: gather context, take one action with a tool, check the result against the goal, and go again until the goal is met or a budget runs out. The model does the thinking inside each step. The harness around it owns the loop, the tools and, most important, the rules for when to stop.&lt;/p&gt;

&lt;p&gt;Anthropic's &lt;a href="https://www.anthropic.com/engineering/building-effective-agents" rel="noopener noreferrer"&gt;Building effective agents&lt;/a&gt; describes a close relative, the evaluator-optimizer workflow: "one LLM call generates a response while another provides evaluation and feedback in a loop." One possible design takes that further and splits the work into three roles.&lt;/p&gt;

&lt;h2&gt;
  
  
  Plan, build, judge
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://devops-daily.com/games/agentic-loop-simulator" rel="noopener noreferrer"&gt;Agentic Loop Simulator&lt;/a&gt; plays one task through those three roles: "add a working /signup endpoint (hash the password, return 201)". Be clear about what you are watching: it is a scripted, illustrative run. No model, editor or test suite runs in your browser. Every step is written in advance to show the pattern. Three cards light up in turn:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Plan&lt;/strong&gt; (planner agent): decide the single next step toward the goal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build&lt;/strong&gt; (builder agent plus tools): do it, here an edit to &lt;code&gt;routes/signup.ts&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge&lt;/strong&gt; (a separate judge agent): grade the result against the goal and the spec.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A gate under the cards shows the decision after each judge step: "goal not met ✗ loop back to plan" or "goal met ✓ stop". With the judge turned off, the builder grades itself and the gate says "self-check says done ✓ stop", even though the goal is not met. Press Play (it is slow on purpose, about 2.3 seconds a step at 1x), or step with Prev and Next or the arrow keys. Speed goes from 0.5x to 2x.&lt;/p&gt;

&lt;p&gt;With the separate judge on, which is the default, the script takes three loops:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The builder writes the handler. The judge step shows &lt;code&gt;npm test&lt;/code&gt; failing and rejects the work: the password is stored in plain text.&lt;/li&gt;
&lt;li&gt;The builder adds bcrypt hashing. The tests now pass, and the judge still rejects it, because the endpoint returns 200 and the spec says 201. The caption puts it plainly: "Tests green is not the same as goal met."&lt;/li&gt;
&lt;li&gt;The builder returns 201 Created. The judge approves, and the loop stops.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Watch the two panels under the loop while it runs. Loop state tracks the iteration, a token counter with a meter, whether the goal is met, and whether the test suite is passing. The context window panel adds a chip for each plan and edit, so you can see the context get longer every loop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turn the judge off
&lt;/h2&gt;

&lt;p&gt;Now flip the "Separate judge agent" toggle off and press Play again. This time the builder grades its own work. Loop one fails its own tests and goes round again. Loop two adds the hashing, the tests go green, and the builder decides it is done. The run stops with the card "Stopped by its own self-check", and one more step changes it to "Shipped a confident bug": the endpoint returns 200, not 201, and nothing in the loop compared the work with the spec.&lt;/p&gt;

&lt;p&gt;That is the case for an independent check. A self-check can miss requirements the builder never tested for, and a reviewer with its own instructions and the spec in front of it may catch them.&lt;/p&gt;

&lt;p&gt;Real agents also stop on checks that do not measure the goal. In a &lt;a href="https://github.com/The-DevOps-Daily/do-agent-issue-to-pr/tree/main/runs/issue-2-attempt-1" rel="noopener noreferrer"&gt;recorded run&lt;/a&gt;, an agent appended two tests below &lt;code&gt;if __name__ == "__main__":&lt;/code&gt;, where unittest never collects them, and reported "All 8 tests pass (6 pre-existing + 2 new ones)". The same suite, run again outside the agent, printed &lt;code&gt;Ran 6 tests&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The token counter ends at 91k with the judge and 56k without, but those are authored values in the script, not measurements of what verification costs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How context can raise loop cost
&lt;/h2&gt;

&lt;p&gt;The hint under the token meter says: "The whole context is re-sent every loop, so each loop costs more than the one before." The script's three judged loops add 27k, 31k and 33k. In a real harness, keeping history makes each request bigger, while compaction, selective context and caching pull the other way. Either way, a loop needs a stop condition the judge can check, plus a hard cap on turns in case it never gets there. Anthropic's guide calls these stopping conditions, "such as a maximum number of iterations". In Claude Code's headless mode the cap is &lt;code&gt;--max-turns&lt;/code&gt;, and there is no limit by default.&lt;/p&gt;

&lt;h2&gt;
  
  
  A toy loop you can run
&lt;/h2&gt;

&lt;p&gt;You can see the harness logic without any model at all. In this Python toy, the planner and builder are stubs that apply a fixed list of edits. The part worth reading is the loop, the two judges and the &lt;code&gt;max_loops&lt;/code&gt; cap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# a toy harness: the "agents" are stubs, the loop is real&lt;/span&gt;
&lt;span class="no"&gt;SPEC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"hashed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;201&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="no"&gt;FIXES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"route"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"hashed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;201&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="no"&gt;FIXES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;FIXES&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tests_pass&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;          &lt;span class="c1"&gt;# what the builder checks&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"route"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"hashed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;meets_spec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;          &lt;span class="c1"&gt;# what a separate judge checks&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;tests_pass&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;SPEC&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_loops&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;loop&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_loops&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;loop&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"stopped after {loop} loops, status {code['status']}"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"hit max_loops"&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"self-check:     "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tests_pass&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"separate judge: "&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;meets_spec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;self-check:      stopped after 2 loops, status 200
separate judge:  stopped after 3 loops, status 201
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same outcome as the simulator, with one honest caveat: the toy's judge wins because it checks one more requirement, the status code. Add that check to &lt;code&gt;tests_pass&lt;/code&gt; and the self-check stops in the right place too. So the toy does not prove you need a second agent. It shows that a loop stops wherever its definition of done lets it stop, and that definition has to cover the spec, not only the tests you already have.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it maps to a real coding agent
&lt;/h2&gt;

&lt;p&gt;The bottom of the simulator maps each card to Claude Code. Plan is a planner subagent or a plan-mode turn that writes the next step. Build is the main agent using its Read, Edit and Bash tools. Judge is a separate reviewer subagent, ideally on a different model. Treat that last part as an option to test, not a rule: the &lt;a href="https://code.claude.com/docs/en/sub-agents" rel="noopener noreferrer"&gt;Claude Code subagents docs&lt;/a&gt; show how to define a reviewer with its own system prompt, tools and &lt;code&gt;model&lt;/code&gt;, plus a &lt;code&gt;maxTurns&lt;/code&gt; cap. Whether a different model helps depends on your task.&lt;/p&gt;

&lt;p&gt;For the wider picture, including the fix-until-green, experiment and overnight triage loops and the executor and advisor split for cost, read the companion post, &lt;a href="https://devops-daily.com/posts/stop-prompting-start-looping" rel="noopener noreferrer"&gt;Stop Prompting, Start Looping: Agentic Loops Explained&lt;/a&gt;. Then run the simulator twice, once with the judge and once without, and compare what each run counted as done. That definition is the part of the loop you design.&lt;/p&gt;

&lt;p&gt;Does your agent setup use a separate judge step, or does one agent check its own work? Tell us in the comments.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>llm</category>
      <category>programming</category>
    </item>
    <item>
      <title>ndots:5 in Kubernetes: Why One DNS Lookup Turns Into Ten Queries</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Mon, 16 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/ndots5-in-kubernetes-why-one-dns-lookup-turns-into-ten-queries-2g15</link>
      <guid>https://dev.to/devopsdaily/ndots5-in-kubernetes-why-one-dns-lookup-turns-into-ten-queries-2g15</guid>
      <description>&lt;p&gt;Your application asks for &lt;code&gt;api.example.com&lt;/code&gt;, CoreDNS looks healthy, and yet DNS request volume jumps by a multiple of outbound HTTP traffic. The surprising part is often not a retry or a bad cache. It is the Pod resolver doing exactly what its search path tells it to do. In a measured glibc run, that one two-dot external name produced eight DNS queries, or ten when the Pod also inherited a node search domain.&lt;/p&gt;

&lt;p&gt;This post shows the resolver rule, reproduces the count, and gives you a safe way to decide between a trailing dot, a lower &lt;code&gt;ndots&lt;/code&gt; value, and DNS-side improvements. The numbers below are a controlled resolver experiment, not a promise about every language runtime.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes documents &lt;code&gt;ndots:5&lt;/code&gt; and a three-part Service search list for a normal Linux Pod. A name with fewer than five dots is searched under those suffixes before glibc tries it as absolute.&lt;/li&gt;
&lt;li&gt;In our glibc 2.36 measurement, &lt;code&gt;api.example.com&lt;/code&gt; generated eight A and AAAA queries with three search suffixes. An inherited fourth suffix made it ten. &lt;code&gt;api.example.com.&lt;/code&gt; and &lt;code&gt;ndots:2&lt;/code&gt; each made two.&lt;/li&gt;
&lt;li&gt;Do not lower &lt;code&gt;ndots&lt;/code&gt; blindly. &lt;code&gt;my-svc.other-ns&lt;/code&gt; made six queries with &lt;code&gt;ndots:1&lt;/code&gt;, because glibc tried the absolute name first; it made four with &lt;code&gt;ndots:2&lt;/code&gt; and then found the Service through &lt;code&gt;svc.cluster.local&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Start with the names your workload actually resolves, then measure CoreDNS requests and NXDOMAIN responses. For external configuration, an absolute FQDN with a trailing dot is the narrowest change. For a Pod-wide change, test &lt;code&gt;ndots: "2"&lt;/code&gt; against every short and cross-namespace Service name it uses.&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;&lt;em&gt;Press play to watch one lookup of api.example.com from a Pod with four search suffixes: five names, each asked as A and AAAA, so ten queries reach CoreDNS and eight of them ask for names that do not exist. A trailing dot or ndots:2 takes it down to two.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Access to a non-production namespace and permission to inspect Pods and the &lt;code&gt;kube-system&lt;/code&gt; namespace.&lt;/li&gt;
&lt;li&gt;A workload inventory that includes external hostnames, same-namespace Services, and cross-namespace Services.&lt;/li&gt;
&lt;li&gt;CoreDNS metrics in your monitoring system, or approval to enable short-lived query logging during a quiet test window.&lt;/li&gt;
&lt;li&gt;A Linux test environment if you want to reproduce the glibc experiment. Other resolver implementations can behave differently.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What Kubernetes puts in a Pod
&lt;/h2&gt;

&lt;p&gt;For a &lt;code&gt;ClusterFirst&lt;/code&gt; Linux Pod, Kubernetes shows this resolver shape in its &lt;a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="noopener noreferrer"&gt;DNS for Services and Pods documentation&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="n"&gt;nameserver&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;dns&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="no"&gt;IP&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="n"&gt;search&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;namespace&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;svc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt; &lt;span class="n"&gt;svc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt; &lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;
&lt;span class="n"&gt;options&lt;/span&gt; &lt;span class="n"&gt;ndots&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual cluster domain and nameserver differ by installation. The first search suffix lets &lt;code&gt;payments&lt;/code&gt; find &lt;code&gt;payments&lt;/code&gt; in the caller's namespace. The second is why &lt;code&gt;payments.finance&lt;/code&gt; can resolve as &lt;code&gt;payments.finance.svc.cluster.local&lt;/code&gt;. Kubernetes also allows additional search domains to be merged into Pod DNS configuration, subject to documented list limits, so inspect the running Pod instead of assuming this three-entry example is complete.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n app &lt;span class="nb"&gt;exec&lt;/span&gt; deploy/api -- cat /etc/resolv.conf

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kubernetes explains its choice in the &lt;a href="https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/" rel="noopener noreferrer"&gt;service debugging guide&lt;/a&gt;: &lt;code&gt;ndots:5&lt;/code&gt; is high enough for the DNS names Kubernetes generates. That is the compatibility goal. It lets names such as &lt;code&gt;service.namespace.svc&lt;/code&gt; participate in search-path resolution, rather than treating any dotted name as immediately external.&lt;/p&gt;

&lt;p&gt;The important glibc rule comes from &lt;a href="https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html" rel="noopener noreferrer"&gt;resolv.conf(5)&lt;/a&gt;: a name with fewer dots than &lt;code&gt;ndots&lt;/code&gt; is tried against each search suffix in order. At &lt;code&gt;ndots:5&lt;/code&gt;, &lt;code&gt;api.example.com&lt;/code&gt; has two dots, so it is below the threshold. With three suffixes, the resolver asks for three search-expanded names before its final absolute lookup. When the application asks for both IPv4 and IPv6 addresses, glibc sends an A and an AAAA question for each candidate, which doubles four candidate names into eight DNS requests. With &lt;code&gt;AI_ADDRCONFIG&lt;/code&gt; set by the caller on a host without an IPv6 address, glibc skips AAAA, so an IPv4-only setup can see half the count.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;api.example.com&lt;/strong&gt; 2 dots, ndots:5&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;3 search suffixes&lt;/strong&gt; 6 A and AAAA NXDOMAINs&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Absolute lookup&lt;/strong&gt; A and AAAA answer&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The resolver is not proving that the external domain is broken. It is asking whether strings such as &lt;code&gt;api.example.com.default.svc.cluster.local&lt;/code&gt; are internal names first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement: eight, ten, and two
&lt;/h2&gt;

&lt;p&gt;We ran this test on 2026-11-11 on Debian 12 with glibc 2.36. A small UDP DNS stub listened on &lt;code&gt;127.0.0.2:53&lt;/code&gt;, logged every question, returned addresses only for &lt;code&gt;api.example.com&lt;/code&gt; and &lt;code&gt;my-svc.other-ns.svc.cluster.local&lt;/code&gt;, and returned NXDOMAIN for everything else. Each run bind-mounted a Pod-style &lt;code&gt;resolv.conf&lt;/code&gt; and &lt;code&gt;hosts: files dns&lt;/code&gt; &lt;code&gt;nsswitch.conf&lt;/code&gt; over &lt;code&gt;/etc&lt;/code&gt; in &lt;code&gt;unshare -rm&lt;/code&gt;, then called &lt;code&gt;getent ahosts &amp;lt;name&amp;gt;&lt;/code&gt;. That exercises the glibc resolver path used by many Linux applications. &lt;code&gt;getent ahosts&lt;/code&gt; sets &lt;code&gt;AI_ADDRCONFIG&lt;/code&gt;, and our test host had an IPv6 address, so every candidate got an A and an AAAA question; on an IPv4-only host, newer glibc versions offer &lt;code&gt;getent --no-addrconfig ahosts&lt;/code&gt; to see the same pairs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Queries per lookup in the glibc resolver test&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;api.example.com, ndots:5, 3 suffixes&lt;/td&gt;
&lt;td&gt;8 queries&lt;/td&gt;
&lt;td&gt;external name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;api.example.com, ndots:5, 4 suffixes&lt;/td&gt;
&lt;td&gt;10 queries&lt;/td&gt;
&lt;td&gt;external name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;api.example.com., trailing dot&lt;/td&gt;
&lt;td&gt;2 queries&lt;/td&gt;
&lt;td&gt;absolute name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;api.example.com, ndots:1 or ndots:2&lt;/td&gt;
&lt;td&gt;2 queries&lt;/td&gt;
&lt;td&gt;absolute name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;my-svc.other-ns, ndots:1&lt;/td&gt;
&lt;td&gt;6 queries&lt;/td&gt;
&lt;td&gt;Service name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;my-svc.other-ns, ndots:2&lt;/td&gt;
&lt;td&gt;4 queries&lt;/td&gt;
&lt;td&gt;Service name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;nonexistent.example.org, ndots:5&lt;/td&gt;
&lt;td&gt;8 queries&lt;/td&gt;
&lt;td&gt;NXDOMAIN&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Measured with the stub, glibc 2.36, date 2026-11-11. Each bar counts logged A and AAAA questions.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here are two of the runs. &lt;code&gt;getent&lt;/code&gt; prints the address the application would get; the stub's log, cleared before each run, shows every question the resolver sent to get there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;glibc resolver trace&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# ndots:5, three Kubernetes search suffixes&lt;/span&gt;
$ : &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; q.log
$ getent ahosts api.example.com &lt;span class="o"&gt;|&lt;/span&gt; head -n 1
192.0.2.10 STREAM api.example.com
&lt;span class="c"&gt;# what the stub saw: 8 questions&lt;/span&gt;
$ cut -d&lt;span class="s1"&gt;' '&lt;/span&gt; -f2- q.log
A api.example.com.default.svc.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
AAAA api.example.com.default.svc.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
A api.example.com.svc.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
AAAA api.example.com.svc.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
A api.example.com.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
AAAA api.example.com.cluster.local. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NXDOMAIN
A api.example.com. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NOERROR
AAAA api.example.com. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NOERROR
&lt;span class="c"&gt;# the trailing dot makes the name absolute&lt;/span&gt;
$ : &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; q.log
$ getent ahosts api.example.com. &lt;span class="o"&gt;|&lt;/span&gt; head -n 1
192.0.2.10 STREAM api.example.com
&lt;span class="c"&gt;# what the stub saw: 2 questions&lt;/span&gt;
$ cut -d&lt;span class="s1"&gt;' '&lt;/span&gt; -f2- q.log
A api.example.com. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NOERROR
AAAA api.example.com. -&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; NOERROR

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The four-suffix case adds &lt;code&gt;eu-west-1.compute.internal&lt;/code&gt; and produces two more NXDOMAINs. A nonexistent name produces eight NXDOMAINs under the three-suffix &lt;code&gt;ndots:5&lt;/code&gt; configuration. Negative answers are still work for the client and resolver, even if a cache later reduces repeats.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reproduce the resolver test
&lt;/h3&gt;

&lt;p&gt;Run the stub as root on a Linux host with Python 3 (it binds port 53). The test also needs a Pod-style &lt;code&gt;resolv.conf&lt;/code&gt; and an &lt;code&gt;nsswitch.conf&lt;/code&gt; containing &lt;code&gt;hosts: files dns&lt;/code&gt;; &lt;code&gt;attempts:1 timeout:1&lt;/code&gt; only keep a failed run short:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="c1"&gt;# resolv-5.conf
&lt;/span&gt;&lt;span class="n"&gt;nameserver&lt;/span&gt; &lt;span class="mf"&gt;127.0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.2&lt;/span&gt;
&lt;span class="n"&gt;search&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;svc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt; &lt;span class="n"&gt;svc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt; &lt;span class="n"&gt;cluster&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;
&lt;span class="n"&gt;options&lt;/span&gt; &lt;span class="n"&gt;ndots&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# nsswitch.conf
&lt;/span&gt;&lt;span class="n"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt; &lt;span class="n"&gt;dns&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The stub is deliberately tiny so its log is the evidence, not a timing benchmark.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# stub.py, run with: sudo python3 stub.py 127.0.0.2 q.log&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="n"&gt;answers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"api.example.com."&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"192.0.2.10"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"2001:db8::10"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"my-svc.other-ns.svc.cluster.local."&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"10.96.12.34"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"2001:db8::34"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;types&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"A"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"AAAA"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;qname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="n"&gt;sock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AF_INET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SOCK_DGRAM&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;bind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"127.0.0.2"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recvfrom&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;txid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;H"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;qname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;qtype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unpack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;H"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;answers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;qtype&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;"\xc0\x0c"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;HHIH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;inet_aton&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;qtype&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="s2"&gt;"\xc0\x0c"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;HHIH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;inet_pton&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AF_INET6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rcode&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
    &lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;HHHHHH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;txid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mh"&gt;0x8180&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;rcode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;sock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;question&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peer&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"/dev/stdout"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{time.time():.3f} {types.get(qtype, qtype):&amp;lt;4} {name} -&amp;gt; {'NOERROR' if hit else 'NXDOMAIN'}\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="c1"&gt;# In a second shell. No host root is needed where unprivileged user&lt;/span&gt;
&lt;span class="c1"&gt;# namespaces are enabled: -r maps you to root inside a new user namespace,&lt;/span&gt;
&lt;span class="c1"&gt;# and -m gives that shell a private mount table.&lt;/span&gt;
&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="c1"&gt;# clear the stub's log before each run&lt;/span&gt;
&lt;span class="n"&gt;unshare&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rm&lt;/span&gt; &lt;span class="n"&gt;sh&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="s1"&gt;'
  mount --bind "$PWD/resolv-5.conf" /etc/resolv.conf
  mount --bind "$PWD/nsswitch.conf" /etc/nsswitch.conf
  getent ahosts api.example.com | head -n 1
'&lt;/span&gt;
&lt;span class="n"&gt;cut&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="s2"&gt;" "&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;f2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="c1"&gt;# the stub's log: one line per question&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not copy this mounting pattern onto a node that is serving workloads. It is a disposable namespace experiment. In a cluster, inspect the effective configuration first and measure the real workload separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the extra requests can hurt
&lt;/h2&gt;

&lt;p&gt;The count is not automatically an incident. A cache may answer many repeated questions, and an internal name that matches early stops the search. It becomes expensive when a high-QPS workload resolves external or missing names often enough that the multiplier turns into CoreDNS work, NXDOMAIN volume, and upstream traffic. The latency risk is most visible when one of the search candidates waits for a timeout: &lt;a href="https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html" rel="noopener noreferrer"&gt;resolv.conf(5)&lt;/a&gt; warns that search processing can be slow and create substantial network traffic when the listed-domain servers are not local.&lt;/p&gt;

&lt;p&gt;Cloud limits can turn the multiplier into a capacity concern. AWS documents a hard limit of &lt;a href="https://docs.aws.amazon.com/whitepapers/latest/hybrid-cloud-dns-options-for-vpc/constraints.html" rel="noopener noreferrer"&gt;1,024 packets per second per network interface to the Amazon-provided DNS server&lt;/a&gt;. That is not a Kubernetes-wide query limit, and it only applies when traffic reaches that resolver. It is still a reason to count the forwarded requests on EKS rather than treating redundant lookups as free. AWS's &lt;a href="https://docs.aws.amazon.com/eks/latest/best-practices/monitoring_eks_workloads_for_network_performance_issues.html" rel="noopener noreferrer"&gt;EKS guidance&lt;/a&gt; names CoreDNS replica placement and NodeLocal DNSCache as responses to DNS throttling.&lt;/p&gt;

&lt;p&gt;There is a second, separate failure mode worth keeping distinct from &lt;code&gt;ndots&lt;/code&gt;: parallel UDP A and AAAA requests can interact badly with connection tracking on some Kubernetes paths. The Kubernetes &lt;a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/" rel="noopener noreferrer"&gt;NodeLocal DNSCache documentation&lt;/a&gt; says skipping DNAT and conntrack helps reduce conntrack races, and that its cache can use TCP for misses to CoreDNS. The NodeLocal DNSCache &lt;a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1024-nodelocal-cache-dns/README.md" rel="noopener noreferrer"&gt;KEP&lt;/a&gt; describes DNS timeouts often up to 30 seconds from retries, including 5-second A and AAAA timeouts. Do not infer a universal kernel fix from that history. Diagnose your dataplane and kernel instead.&lt;/p&gt;

&lt;p&gt;For glibc clients, &lt;code&gt;single-request-reopen&lt;/code&gt; changes how it reopens the socket when paired A and AAAA requests are mishandled, while &lt;code&gt;use-vc&lt;/code&gt; forces TCP. Both are documented in &lt;a href="https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html" rel="noopener noreferrer"&gt;resolv.conf(5)&lt;/a&gt;. They are targeted compatibility options, not an &lt;code&gt;ndots&lt;/code&gt; optimization, so test them with the actual client image and DNS path before adding them to a Pod spec.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pick the smallest safe fix
&lt;/h2&gt;

&lt;p&gt;The first question is what kind of name you are changing.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Name pattern&lt;/th&gt;
&lt;th&gt;Lowest-risk first move&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;External hostname in static configuration&lt;/td&gt;
&lt;td&gt;Use its trailing-dot FQDN&lt;/td&gt;
&lt;td&gt;Bypasses search expansion for that one name.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External hostnames throughout one workload&lt;/td&gt;
&lt;td&gt;Test &lt;code&gt;ndots: "2"&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Two-dot external names are tried absolute first.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Same-namespace Service&lt;/td&gt;
&lt;td&gt;Keep the short name&lt;/td&gt;
&lt;td&gt;The first search suffix is useful.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;service.namespace&lt;/code&gt; cross-namespace name&lt;/td&gt;
&lt;td&gt;Test carefully with &lt;code&gt;ndots:2&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;It retains search-first behavior in the measured run.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A fleet with repetitive lookup traffic&lt;/td&gt;
&lt;td&gt;Evaluate NodeLocal DNSCache and CoreDNS cache&lt;/td&gt;
&lt;td&gt;These reduce resolver-path work without changing application spelling.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Use an absolute FQDN when you control the hostname
&lt;/h3&gt;

&lt;p&gt;The trailing dot says the DNS name is already absolute:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;API_BASE_URL&lt;/span&gt;
    &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'https://api.example.com.'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That made two queries in the measurement. It can be an excellent narrow fix for a configured hostname, but test it end to end. HTTP client URL handling, the &lt;code&gt;Host&lt;/code&gt; header, certificate matching, proxy routing, and TLS SNI handling are library and proxy specific. Test the exact client, proxy, and certificate path rather than assuming every component treats the final dot the same way.&lt;/p&gt;

&lt;h3&gt;
  
  
  Set &lt;code&gt;ndots: "2"&lt;/code&gt; only after a service-name test
&lt;/h3&gt;

&lt;p&gt;The Kubernetes API represents DNS option values as strings. This Deployment applies &lt;code&gt;ndots: "2"&lt;/code&gt; to its Pods:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;apps/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Deployment&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;dnsConfig&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ndots&lt;/span&gt;
            &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'2'&lt;/span&gt; &lt;span class="c1"&gt;# DNS option values are strings in the Pod API.
&lt;/span&gt;      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;registry.example.com/api:2026-11-16&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kubernetes documents that &lt;code&gt;dnsConfig.options&lt;/code&gt; are merged with the DNS policy configuration and shows the same quoted &lt;code&gt;value: "2"&lt;/code&gt; form in its &lt;a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="noopener noreferrer"&gt;custom DNS example&lt;/a&gt;. In the test, &lt;code&gt;ndots:2&lt;/code&gt; made &lt;code&gt;api.example.com&lt;/code&gt; absolute first, while &lt;code&gt;my-svc.other-ns&lt;/code&gt; searched &lt;code&gt;my-svc.other-ns.default.svc.cluster.local&lt;/code&gt; then found &lt;code&gt;my-svc.other-ns.svc.cluster.local&lt;/code&gt;. By contrast, &lt;code&gt;ndots:1&lt;/code&gt; first sent &lt;code&gt;my-svc.other-ns.&lt;/code&gt; to upstream resolution, then made the search-path calls.&lt;/p&gt;

&lt;p&gt;That is why &lt;code&gt;ndots:1&lt;/code&gt; is not a free win. Run the application smoke tests for short names, partially qualified Service names, and external names after rollout. If the workload only uses full Service FQDNs and external hosts, its trade-off can differ from a namespace-heavy application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reduce work in the DNS layer
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/" rel="noopener noreferrer"&gt;NodeLocal DNSCache&lt;/a&gt; runs a caching agent as a DaemonSet on every node. Kubernetes says Pods normally reach the CoreDNS Service through kube-proxy, while the local-cache design lets Pods query the local agent, avoids DNAT and connection tracking, and provides per-node DNS visibility. It is a cluster change, so size and test it, including its memory limit. The documentation notes that cache entries and concurrent queries affect memory demand.&lt;/p&gt;

&lt;p&gt;The CoreDNS &lt;a href="https://coredns.io/plugins/cache/" rel="noopener noreferrer"&gt;&lt;code&gt;cache&lt;/code&gt; plugin&lt;/a&gt; caches successful and denial responses. Its documentation says a default cache is split into 256 shards with 9,984 entries in total, and that negative responses can be cached. Review TTLs, eviction behavior, and the names that are actually repeated before relying on it to mask an avoidable query multiplier.&lt;/p&gt;

&lt;p&gt;The CoreDNS &lt;a href="https://coredns.io/plugins/autopath/" rel="noopener noreferrer"&gt;&lt;code&gt;autopath&lt;/code&gt; plugin&lt;/a&gt; moves search-path completion to the server. In Kubernetes it needs &lt;code&gt;pods verified&lt;/code&gt; for the Kubernetes plugin and the client Pod IP to reach CoreDNS. CoreDNS documents that &lt;code&gt;pods verified&lt;/code&gt; maintains a watch on all Pods and uses substantially more memory than insecure mode. It also documents a Pod-IP reuse race and notes that a final negative answer still makes the client search manually. Treat it as a measured CoreDNS configuration change, not a default switch.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure before and after
&lt;/h2&gt;

&lt;p&gt;Start at the Pod, then the resolver. Capture &lt;code&gt;/etc/resolv.conf&lt;/code&gt; (the &lt;code&gt;getent&lt;/code&gt; calls below need an image that ships it; use a debug container otherwise), list the hostname patterns in configuration, and compare a known external name, an internal short name, a cross-namespace Service name, and a deliberately nonexistent name.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n app &lt;span class="nb"&gt;exec&lt;/span&gt; deploy/api -- cat /etc/resolv.conf
kubectl -n app &lt;span class="nb"&gt;exec&lt;/span&gt; deploy/api -- getent ahosts api.example.com
kubectl -n app &lt;span class="nb"&gt;exec&lt;/span&gt; deploy/api -- getent ahosts payments.finance
kubectl -n app &lt;span class="nb"&gt;exec&lt;/span&gt; deploy/api -- getent ahosts does-not-exist.example.invalid

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At the resolver, use the CoreDNS Prometheus plugin's documented counters. &lt;a href="https://coredns.io/plugins/metrics/" rel="noopener noreferrer"&gt;&lt;code&gt;coredns_dns_requests_total&lt;/code&gt;&lt;/a&gt; carries query type and transport labels; &lt;code&gt;coredns_dns_responses_total&lt;/code&gt; carries &lt;code&gt;rcode&lt;/code&gt;. The latter makes this a useful before-and-after query:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight prometheus"&gt;&lt;code&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;rate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;coredns_dns_requests_total&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5m&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="na"&gt;sum(rate(coredns_dns_responses_total{rcode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"NXDOMAIN"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5m&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep the query &lt;code&gt;type&lt;/code&gt;, &lt;code&gt;zone&lt;/code&gt;, &lt;code&gt;server&lt;/code&gt;, and response &lt;code&gt;rcode&lt;/code&gt; labels in your dashboard when narrowing the investigation. They identify the traffic class more quickly than a cluster-wide total. For a brief diagnostic window, you can also add the CoreDNS &lt;a href="https://coredns.io/plugins/log/" rel="noopener noreferrer"&gt;&lt;code&gt;log&lt;/code&gt; plugin&lt;/a&gt; to the relevant server block, collect a small sample, and remove it. CoreDNS warns that logging on busy servers has a performance cost. Query logs can also expose names, so treat them as incident-scoped instrumentation.&lt;/p&gt;

&lt;p&gt;If resolver counters say the problem is upstream of CoreDNS, capture packets from an approved privileged debug environment on the node, for example with &lt;a href="https://kubernetes.io/docs/tasks/debug/debug-cluster/kubectl-node-debug/" rel="noopener noreferrer"&gt;&lt;code&gt;kubectl debug node/NODE -it --image=IMAGE&lt;/code&gt;&lt;/a&gt; using an approved image that ships &lt;code&gt;tcpdump&lt;/code&gt;, and filter UDP and TCP port 53. Check your cluster's security policy before doing this. The goal is to distinguish client-to-CoreDNS repeats from CoreDNS-to-upstream forwarding, not to leave packet capture running.&lt;/p&gt;

&lt;p&gt;For a refresher on what happens after a resolver cache miss, the &lt;a href="https://devops-daily.com/games/dns-simulator" rel="noopener noreferrer"&gt;DNS Resolution Simulator&lt;/a&gt; lets you step through browser, OS, and resolver caches, then root, TLD, and authoritative servers, and rerun a lookup to see a cache hit. For the Kubernetes naming side, see &lt;a href="https://devops-daily.com/posts/how-to-manage-multiple-environments-in-kubernetes#h2-how-do-i-reach-a-service-in-another-namespace" rel="noopener noreferrer"&gt;How Do I Reach a Service in Another Namespace?&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The operating rule
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ndots:5&lt;/code&gt; exists so Kubernetes names work naturally. Its cost is that many normal external names look short enough to search first. Use your query mix to decide whether that cost matters, then change the narrowest layer that solves it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Add a trailing dot to a controlled external hostname if the full HTTP and TLS path accepts it.&lt;/li&gt;
&lt;li&gt;Test &lt;code&gt;ndots: "2"&lt;/code&gt; on one workload if its cross-namespace Service names still resolve as expected.&lt;/li&gt;
&lt;li&gt;Add caching or NodeLocal DNSCache when repeated lookup load or DNS-path reliability needs a cluster-level response.&lt;/li&gt;
&lt;li&gt;Watch CoreDNS request and NXDOMAIN rates before and after, not just application success rates.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The goal is not the smallest possible &lt;code&gt;ndots&lt;/code&gt; number. It is fewer accidental DNS queries without breaking the service-discovery names that made &lt;code&gt;ndots:5&lt;/code&gt; useful in the first place.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/kubernetes-ndots-5-dns-query-amplification" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>dns</category>
      <category>coredns</category>
      <category>networking</category>
    </item>
    <item>
      <title>Five Production Bugs You Can Reproduce in a Browser Tab</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 13 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/five-production-bugs-you-can-reproduce-in-a-browser-tab-5gj8</link>
      <guid>https://dev.to/devopsdaily/five-production-bugs-you-can-reproduce-in-a-browser-tab-5gj8</guid>
      <description>&lt;p&gt;Many production failures are easy to name after the fact and hard to see while you are building: a cache key contains too little or too much, a retry creates a second payment, or a request cache outlives the request. The details differ, but each failure starts with an assumption about what a protocol, intermediary, or library remembers.&lt;/p&gt;

&lt;p&gt;The five free DevOps Daily labs linked below make those assumptions visible. This is not a tour of the labs. Each section starts with the operational symptom, explains the protocol rule, gives you a check to run against your own system, then provides a compact reproduction. You can use the post without opening a lab.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;First thing to inspect&lt;/th&gt;
&lt;th&gt;Safe default&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A shared cache returns personal content&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Cache-Control&lt;/code&gt;, &lt;code&gt;Vary&lt;/code&gt;, and the cache key&lt;/td&gt;
&lt;td&gt;Mark personal responses &lt;code&gt;private&lt;/code&gt; or &lt;code&gt;no-store&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One lost packet delays a page&lt;/td&gt;
&lt;td&gt;negotiated HTTP version and loss-sensitive waterfalls&lt;/td&gt;
&lt;td&gt;Test HTTP/2 and HTTP/3 on the links your users have&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A legitimate message fails DMARC&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Authentication-Results&lt;/code&gt; and alignment&lt;/td&gt;
&lt;td&gt;Align either SPF or DKIM with visible &lt;code&gt;From&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A retry creates another operation&lt;/td&gt;
&lt;td&gt;idempotency key, request fingerprint, saved response&lt;/td&gt;
&lt;td&gt;One key per intended operation, reused only for retries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nested GraphQL fields make too many queries&lt;/td&gt;
&lt;td&gt;resolver SQL and DataLoader scope&lt;/td&gt;
&lt;td&gt;Batch per request, never across requests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A terminal with &lt;code&gt;curl&lt;/code&gt; and &lt;code&gt;dig&lt;/code&gt; (from &lt;code&gt;bind9-dnsutils&lt;/code&gt; or &lt;code&gt;bind-utils&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Read access to one of your own HTTP endpoints, one sending domain, and one API that accepts retries.&lt;/li&gt;
&lt;li&gt;A modern browser for the labs. They are free and run in the browser.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A shared cache serves a cart to the wrong person
&lt;/h2&gt;

&lt;p&gt;The frightening version of this bug is a user seeing somebody else's cart, profile, or account page. The quieter version is a cache hit ratio that collapses because every campaign link has a distinct tracking query string. Both are cache-key errors: the cache either treats two different representations as the same object, or one representation as too many objects.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9111" rel="noopener noreferrer"&gt;RFC 9111&lt;/a&gt; says a cache key normally includes the request method and target URI. A cache that stores a response selected by &lt;code&gt;Vary&lt;/code&gt; must also use the nominated request-field values to choose a matching response. The response controls matter just as much: &lt;code&gt;private&lt;/code&gt; prohibits storage in a shared cache, and &lt;code&gt;no-store&lt;/code&gt; says not to store it. &lt;a href="https://www.rfc-editor.org/rfc/rfc5861#section-3" rel="noopener noreferrer"&gt;RFC 5861&lt;/a&gt; defines &lt;code&gt;stale-while-revalidate&lt;/code&gt;, which lets a cache serve a stale response while it validates in the background.&lt;/p&gt;

&lt;p&gt;Start with a response that must be private and an asset that should be reusable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI https://app.example.com/api/cart
curl -sI https://app.example.com/assets/app.8d42c.js

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cart response should show &lt;code&gt;Cache-Control: private&lt;/code&gt; or &lt;code&gt;Cache-Control: no-store&lt;/code&gt;. A content-addressed asset can use a long &lt;code&gt;max-age&lt;/code&gt;. For a cacheable response, read &lt;code&gt;Age&lt;/code&gt;, &lt;code&gt;Vary&lt;/code&gt;, and &lt;code&gt;ETag&lt;/code&gt;; these are standardized response fields, though any vendor cache-status header is not. Then test validation with the actual entity tag you received:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;post_etag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'"replace-with-the-etag-value"'&lt;/span&gt;
curl -sSI \
  -H &lt;span class="s2"&gt;"If-None-Match: ${post_etag}"&lt;/span&gt; \
  https://app.example.com/index.html

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For an unchanged representation, the expected shape is &lt;code&gt;HTTP/... 304&lt;/code&gt; with an &lt;code&gt;ETag&lt;/code&gt;. A &lt;code&gt;304&lt;/code&gt; confirms conditional validation, not that every edge has the same object. Review whether your CDN intentionally removes tracking parameters from its cache key before deciding that &lt;code&gt;/pricing?utm_source=...&lt;/code&gt; and &lt;code&gt;/pricing&lt;/code&gt; are equivalent.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/cdn-edge-cache-lab" rel="noopener noreferrer"&gt;Reproduce it in the CDN Edge Cache Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;code&gt;/api/cart&lt;/code&gt; and press &lt;strong&gt;Send traffic&lt;/strong&gt;. Observe the &lt;code&gt;BYPASS&lt;/code&gt; results when its response is private or no-store.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;utm_ links&lt;/strong&gt; , send traffic, and inspect the multiple cached keys.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Ignore utm_ in cache key&lt;/strong&gt; , then send traffic again to see later visitors get &lt;code&gt;HIT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Select &lt;code&gt;/index.html&lt;/code&gt;, send traffic, choose &lt;strong&gt;Wait 61 s&lt;/strong&gt; , and send traffic again to expose a &lt;code&gt;304&lt;/code&gt; revalidation.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab models one origin and three CDN edges. Its latency values are illustrative estimates, and it deliberately does not apply heuristic freshness when an explicit lifetime is absent.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  One lost packet pauses every active HTTP/2 stream
&lt;/h2&gt;

&lt;p&gt;HTTP/2 usually removes the browser-side request queues that made HTTP/1.1 pages painful: it multiplexes streams on one connection. The tradeoff is lower in the stack. HTTP/2 runs over TCP, and TCP delivers bytes in order. If a TCP packet containing bytes for one stream is missing, later bytes for all active streams on that connection wait for recovery. On a lossy high-latency path, that can make HTTP/2 slower than several HTTP/1.1 connections carrying independent TCP flows.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc9113" rel="noopener noreferrer"&gt;RFC 9113&lt;/a&gt; defines HTTP/2 streams and multiplexed framing. &lt;a href="https://www.rfc-editor.org/rfc/rfc9114#section-1" rel="noopener noreferrer"&gt;RFC 9114&lt;/a&gt; describes HTTP/3 as HTTP semantics over QUIC and calls out avoiding TCP head-of-line blocking. QUIC streams have independent delivery and loss recovery under &lt;a href="https://www.rfc-editor.org/rfc/rfc9000#section-2.2" rel="noopener noreferrer"&gt;RFC 9000&lt;/a&gt;, so a lost packet holds up only the streams whose data it carried, while streams with data in other packets keep moving. That removes TCP's connection-wide head-of-line blocking, not every stall.&lt;/p&gt;

&lt;p&gt;Check what your endpoint actually negotiates before drawing conclusions from an application dashboard:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI --http2 -w &lt;span class="s1"&gt;'%{http_version}\n'&lt;/span&gt; -o /dev/null https://app.example.com
curl -sI --http3-only -w &lt;span class="s1"&gt;'%{http_version}\n'&lt;/span&gt; -o /dev/null https://app.example.com
curl -sI https://app.example.com &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s1"&gt;'^alt-svc:'&lt;/span&gt;
curl -V

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first two commands print the negotiated version, such as &lt;code&gt;2&lt;/code&gt; or &lt;code&gt;3&lt;/code&gt;. Use &lt;code&gt;--http3-only&lt;/code&gt; for the check: plain &lt;code&gt;--http3&lt;/code&gt; falls back to an earlier version when QUIC cannot be set up, which hides the answer you are looking for. The &lt;code&gt;Alt-Svc&lt;/code&gt; header can advertise an &lt;code&gt;h3&lt;/code&gt; alternative service. Before running either command, check &lt;code&gt;curl -V&lt;/code&gt;: its Features line must list &lt;code&gt;HTTP2&lt;/code&gt; for the first and &lt;code&gt;HTTP3&lt;/code&gt; for the second, and &lt;code&gt;--http3-only&lt;/code&gt; needs curl 7.88.0 or later. Test on representative networks as well as a clean office connection: protocol choice is a latency and loss question, not a permanent leaderboard.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One packet lost&lt;/strong&gt; during app.js&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recover loss&lt;/strong&gt; retransmission needed&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;HTTP/2 over TCP: active streams wait for in-order delivery&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;HTTP/3 over QUIC: streams in the lost packet wait, others progress&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/http-versions-lab" rel="noopener noreferrer"&gt;Reproduce it in the HTTP Versions Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;strong&gt;HTTP/2&lt;/strong&gt; , set &lt;strong&gt;250 ms RTT&lt;/strong&gt; , turn on &lt;strong&gt;Lose one packet&lt;/strong&gt; , and press &lt;strong&gt;Play all&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Read the waterfall and the loss explanation: multiple receiving streams stall behind the TCP retransmission.&lt;/li&gt;
&lt;li&gt;Switch to &lt;strong&gt;HTTP/3&lt;/strong&gt; , keep &lt;strong&gt;Lose one packet&lt;/strong&gt; on, and press &lt;strong&gt;Play all&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Compare the stalled stream count and the setup steps with &lt;strong&gt;Next&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is a deterministic page-load model, not a browser trace. It excludes DNS, server work, congestion control, priorities, caches, header compression, server push, and QUIC 0-RTT resumption.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A valid-looking email fails DMARC after forwarding
&lt;/h2&gt;

&lt;p&gt;An order confirmation can leave your sender with SPF, DKIM, and DMARC all passing, then fail after a forwarder or a mailing list touches it. The customer sees a rejected message or a spam-folder delivery, while the sending system sees a successful SMTP handoff. The common mistake is treating three authentication results as interchangeable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc7208" rel="noopener noreferrer"&gt;RFC 7208&lt;/a&gt; specifies SPF for the SMTP MAIL FROM identity and the connecting IP address. &lt;a href="https://www.rfc-editor.org/rfc/rfc6376" rel="noopener noreferrer"&gt;RFC 6376&lt;/a&gt; specifies DKIM signatures over selected headers and the message body, verified using a selector key in DNS. Current DMARC is &lt;a href="https://www.rfc-editor.org/rfc/rfc9989" rel="noopener noreferrer"&gt;RFC 9989&lt;/a&gt;, which requires SPF or DKIM to pass &lt;strong&gt;and&lt;/strong&gt; align with the visible &lt;code&gt;From&lt;/code&gt; domain. A forwarder changes the connecting IP, so SPF often fails; a mailing list that changes a signed body can invalidate DKIM. Rewriting &lt;code&gt;From&lt;/code&gt; is one way a list can preserve DMARC when it modifies a message.&lt;/p&gt;

&lt;p&gt;Inspect the records and one delivered message rather than guessing from a provider UI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig +short TXT example.com
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open the delivered message source and locate the &lt;code&gt;Authentication-Results&lt;/code&gt; header added by your receiving provider, not an untrusted copy from upstream (&lt;a href="https://www.rfc-editor.org/rfc/rfc8601" rel="noopener noreferrer"&gt;RFC 8601&lt;/a&gt; explains why the header needs a trust check). Its useful shape names the evaluated identities and results, for example &lt;code&gt;spf=pass smtp.mailfrom=...&lt;/code&gt;, &lt;code&gt;dkim=pass header.d=...&lt;/code&gt;, and &lt;code&gt;dmarc=pass header.from=...&lt;/code&gt;. Confirm the domain in &lt;code&gt;header.from&lt;/code&gt; aligns with either the SPF MAIL FROM domain or the DKIM &lt;code&gt;d=&lt;/code&gt; domain. The earlier &lt;a href="https://dev.to/devopsdaily/dmarcbis-is-here-what-changed-in-the-new-dmarc-and-what-to-do-to-your-records-413h"&gt;DMARCbis guide&lt;/a&gt; covers the record migration in more detail: DMARCbis replaced RFC 7489 and removed its &lt;code&gt;pct&lt;/code&gt; tag.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/email-auth-lab" rel="noopener noreferrer"&gt;Reproduce it in the Email Authentication Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Select &lt;code&gt;d=shop.example&lt;/code&gt; for &lt;strong&gt;DKIM signing domain&lt;/strong&gt; , publish the selector, and choose a DMARC policy.&lt;/li&gt;
&lt;li&gt;Send the &lt;strong&gt;Direct&lt;/strong&gt; path, then send the &lt;strong&gt;Forwarded&lt;/strong&gt; path (the forwarder does not use SRS) and compare the SPF, DKIM, and DMARC verdicts.&lt;/li&gt;
&lt;li&gt;Select the mailing-list path, turn on &lt;strong&gt;List rewrites From and sets Reply-To&lt;/strong&gt; , and send again.&lt;/li&gt;
&lt;li&gt;Change &lt;strong&gt;SPF alignment&lt;/strong&gt; or &lt;strong&gt;DKIM alignment&lt;/strong&gt; from relaxed to strict and read the receiver result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab checks the MAIL FROM identity, not HELO, and does not model SRS or ARC. Its percentage control deliberately demonstrates the retired RFC 7489 &lt;code&gt;pct&lt;/code&gt; behavior; do not use it as a model of current DMARCbis policy.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  An ambiguous retry creates a second payment
&lt;/h2&gt;

&lt;p&gt;The failure looks like a double charge, duplicate order, or two provisioned resources. Often the first request did succeed, but its response was lost after the server committed work. Retrying is the correct client action after an ambiguous failure. Retrying it as a new operation is not.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://docs.stripe.com/api/idempotent_requests" rel="noopener noreferrer"&gt;Stripe's idempotent requests documentation&lt;/a&gt; describes the pattern: the API saves the status code and body of the first request for a key and returns them for later requests with the same key, and it rejects a reuse of the key with different parameters. The IETF is standardizing the header itself in &lt;a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer"&gt;The Idempotency-Key HTTP Header Field&lt;/a&gt;, which is still an Internet-Draft, not an RFC. A key identifies one intended operation, not one customer or one retry attempt. Store a request fingerprint beside the key, reject a payload mismatch, and distinguish an in-progress duplicate from a completed replay.&lt;/p&gt;

&lt;p&gt;Here is the client side. It keeps one generated key for every retry and applies full jitter to retryable failures:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;randomUUID&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'node:crypto'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;idempotencyKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// One intended order, not one HTTP attempt.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;retryableStatuses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;409&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;502&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;503&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;504&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;createOrder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'https://api.example.com/orders'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'POST'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="c1"&gt;// Stripe documents a bare value; the IETF draft defines a quoted string.&lt;/span&gt;
        &lt;span class="c1"&gt;// Use the serialization your API documents.&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'application/json'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'Idempotency-Key'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;idempotencyKey&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;AbortSignal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// a stalled request must reach the retry path&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'pro-monthly'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'cus_123'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;retryableStatuses&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="c1"&gt;// A 409 is retryable here only because this API uses it for an in-progress key.&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Idempotent request did not complete after five attempts'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;ceilingMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;setTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;ceilingMs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the server, make the uniqueness boundary visible. Scope keys to the authenticated caller, so one client's key can never replay another client's saved response, and save the fingerprint and final response before a later retry can replay it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;idempotency_responses&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;account_id&lt;/span&gt; &lt;span class="kt"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- from authentication, never from the request body&lt;/span&gt;
  &lt;span class="n"&gt;idempotency_key&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;request_fingerprint&lt;/span&gt; &lt;span class="kt"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="kt"&gt;integer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;response_body&lt;/span&gt; &lt;span class="kt"&gt;jsonb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;completed_at&lt;/span&gt; &lt;span class="kt"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idempotency_key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every lookup, insert and replay filters on both columns. The &lt;a href="https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/" rel="noopener noreferrer"&gt;Idempotency-Key draft&lt;/a&gt; calls out data leaks from shared keys for the same reason.&lt;/p&gt;

&lt;p&gt;Use a transaction or equivalent concurrency control to insert the key, create the order, and save the response. A matching completed row returns &lt;code&gt;status_code&lt;/code&gt; and &lt;code&gt;response_body&lt;/code&gt;; a differing fingerprint returns a client error; a matching unfinished row should tell the client the first request is still processing. The same pattern matters for webhooks, where a delivery can be retried after your handler completes. See &lt;a href="https://dev.to/devopsdaily/how-stripe-avoids-double-charging-anyone-2me1"&gt;how Stripe avoids double charging&lt;/a&gt; and &lt;a href="https://dev.to/devopsdaily/what-it-actually-takes-to-deliver-a-webhook-in-production-4ghe"&gt;reliable webhook delivery&lt;/a&gt;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/idempotency-retry-lab" rel="noopener noreferrer"&gt;Reproduce it in the Idempotency and Retry Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Choose &lt;strong&gt;Run duplicate checkout&lt;/strong&gt; and inspect the ledger after the lost response.&lt;/li&gt;
&lt;li&gt;Choose &lt;strong&gt;Run keyed checkout&lt;/strong&gt; and find the saved result in &lt;strong&gt;Idempotency store&lt;/strong&gt; instead of another charge.&lt;/li&gt;
&lt;li&gt;Press &lt;strong&gt;Reuse key with new amount&lt;/strong&gt; , then &lt;strong&gt;Send two at once&lt;/strong&gt; , and compare the mismatch and in-progress outcomes.&lt;/li&gt;
&lt;li&gt;Run &lt;strong&gt;Run no backoff&lt;/strong&gt; and &lt;strong&gt;Run full jitter&lt;/strong&gt; , then use &lt;strong&gt;Run down API job&lt;/strong&gt; to inspect the dead letter.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The outage chart is a simplified deterministic model of client arrivals, not a production measurement. Its dead-letter jobs retain a key and final error and do not retry themselves.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A shared DataLoader exposes data from the previous request
&lt;/h2&gt;

&lt;p&gt;GraphQL makes N+1 look harmless. &lt;code&gt;posts { author { name } }&lt;/code&gt; reads like one query, but a resolver that fetches every author separately can produce one posts query plus one author query per post. Add nested comments and comment authors, and the work grows at each resolver level.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://github.com/graphql/dataloader#batching" rel="noopener noreferrer"&gt;DataLoader README&lt;/a&gt; describes batching loads that occur in the same execution frame and caching values in a loader instance. The cache is intentionally not a shared application cache. The README's &lt;a href="https://github.com/graphql/dataloader#caching-per-request" rel="noopener noreferrer"&gt;per-request caching guidance&lt;/a&gt; says to create new loader instances for each request because authorization or other request-specific context can differ.&lt;/p&gt;

&lt;p&gt;Create loaders in the GraphQL context, not at module scope. This Apollo Server pattern puts the loader beside the authenticated request context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;ApolloServer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@apollo/server'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;DataLoader&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'dataloader'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;startStandaloneServer&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@apollo/server/standalone'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ApolloServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;typeDefs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;resolvers&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;startStandaloneServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;viewer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;authenticate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;authorization&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;viewer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="c1"&gt;// A new loader per request: its cache dies with the request.&lt;/span&gt;
      &lt;span class="n"&gt;userById&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;DataLoader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Authorization lives in the data access: only rows this viewer may see.&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;loadUsersVisibleTo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;viewer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;loadUsersVisibleTo&lt;/code&gt; stands in for your own query, for example &lt;code&gt;SELECT id, name FROM users WHERE id = ANY($1::uuid[])&lt;/code&gt; plus the viewer's visibility rule. A per-request loader stops one request's cache from serving the next; it does not authorize reads inside a request, so the permission check has to be in the batch function.&lt;/p&gt;

&lt;p&gt;Without the loader, resolver logs often show repeated &lt;code&gt;SELECT ... WHERE id = $1&lt;/code&gt; statements. With it, same-tick loads become one &lt;code&gt;WHERE id IN (...)&lt;/code&gt; query or its parameterized equivalent. Deduplication reduces repeated keys in that batch, but it does not authorize the cached result for another request. Log query count and the request ID together so a shared cache is visible before it becomes a data exposure.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://devops-daily.com/games/graphql-dataloader-lab" rel="noopener noreferrer"&gt;Reproduce it in the GraphQL N+1 and DataLoader Lab&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Leave &lt;strong&gt;Post authors&lt;/strong&gt; on, keep &lt;strong&gt;Use User DataLoader&lt;/strong&gt; off, and press &lt;strong&gt;Run query&lt;/strong&gt;. Inspect the SQL panel.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Comments&lt;/strong&gt; and &lt;strong&gt;Comment authors&lt;/strong&gt; , then run again to see the nested resolver work.&lt;/li&gt;
&lt;li&gt;Turn on &lt;strong&gt;Use User DataLoader&lt;/strong&gt; and &lt;strong&gt;Use Comments DataLoader&lt;/strong&gt; to see the batched &lt;code&gt;IN (...)&lt;/code&gt; queries.&lt;/li&gt;
&lt;li&gt;Set &lt;strong&gt;Loader scope&lt;/strong&gt; to &lt;strong&gt;Shared by all requests&lt;/strong&gt; , run as Ada, then run as Bo. Set it back to &lt;strong&gt;Per request&lt;/strong&gt; and run Bo again.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The lab has a small deterministic data set. It models one event-loop tick per resolver level; &lt;code&gt;await&lt;/code&gt; calls between loads can split real DataLoader batches.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Keep the state boundary explicit
&lt;/h2&gt;

&lt;p&gt;These bugs look unrelated until you name the state boundary. A CDN decides which request fields identify a response. TCP decides which bytes must arrive before later bytes are delivered. DMARC decides which authenticated identifier may stand for &lt;code&gt;From&lt;/code&gt;. An idempotency store decides whether a retry is an earlier operation. DataLoader decides how long a loaded value remains valid.&lt;/p&gt;

&lt;p&gt;Use the checks before an incident, then use the labs to make the failure shape memorable.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bug&lt;/th&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;One-line fix&lt;/th&gt;
&lt;th&gt;Lab&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Personal content at the edge&lt;/td&gt;
&lt;td&gt;Shared cache key or directives allow storage&lt;/td&gt;
&lt;td&gt;Send &lt;code&gt;private&lt;/code&gt; or &lt;code&gt;no-store&lt;/code&gt; for personalized responses&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/cdn-edge-cache-lab" rel="noopener noreferrer"&gt;CDN Edge Cache Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP/2 page stalls on loss&lt;/td&gt;
&lt;td&gt;TCP in-order delivery blocks active streams&lt;/td&gt;
&lt;td&gt;Test QUIC and HTTP/3 where loss and RTT warrant it&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/http-versions-lab" rel="noopener noreferrer"&gt;HTTP Versions Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DMARC fails after an intermediary&lt;/td&gt;
&lt;td&gt;SPF or DKIM no longer passes and aligns&lt;/td&gt;
&lt;td&gt;Preserve an aligned identity through forwarding and lists&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/email-auth-lab" rel="noopener noreferrer"&gt;Email Authentication Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duplicate work after retry&lt;/td&gt;
&lt;td&gt;New key makes a retry a new operation&lt;/td&gt;
&lt;td&gt;Reuse one key and replay its saved response&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/idempotency-retry-lab" rel="noopener noreferrer"&gt;Idempotency and Retry Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GraphQL data leak or query storm&lt;/td&gt;
&lt;td&gt;Loader cache crosses requests, or loads are unbatched&lt;/td&gt;
&lt;td&gt;Instantiate loaders in request context&lt;/td&gt;
&lt;td&gt;&lt;a href="https://devops-daily.com/games/graphql-dataloader-lab" rel="noopener noreferrer"&gt;GraphQL N+1 and DataLoader Lab&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/five-production-bugs-you-can-reproduce-in-a-browser" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>http</category>
      <category>cdn</category>
      <category>email</category>
    </item>
    <item>
      <title>CKNE Is Live: What the Kubernetes Network Engineer Exam Tests and How to Practice</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/ckne-is-live-what-the-kubernetes-network-engineer-exam-tests-and-how-to-practice-276</link>
      <guid>https://dev.to/devopsdaily/ckne-is-live-what-the-kubernetes-network-engineer-exam-tests-and-how-to-practice-276</guid>
      <description>&lt;p&gt;Kubernetes networking failures rarely announce which layer is wrong. A Service can have no ready endpoints, a policy can block DNS, a route can point at the wrong interface, or a Gateway can accept a listener but reject the route. The new &lt;strong&gt;Certified Kubernetes Network Engineer (CKNE)&lt;/strong&gt; asks you to find and fix problems like these from a terminal under time pressure. This guide turns the published curriculum into drills you can repeat in a disposable cluster.&lt;/p&gt;

&lt;p&gt;This is not an account of taking the exam. It is a study guide built from the &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;published CKNE curriculum&lt;/a&gt; and the &lt;a href="https://www.cncf.io/announcements/2026/11/10/cncf-launches-certified-kubernetes-network-engineer-certification-ckne-to-advance-production-ready-networking-skills-for-ai-era/" rel="noopener noreferrer"&gt;CNCF launch announcement&lt;/a&gt;. Treat every lab as a loop: break one thing deliberately, prove where the packet stops, repair it, then retest from the client.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CKNE is a performance-based, two-hour exam in a Linux remote desktop. Its largest domains are Service Networking and DNS, and Network Security and Policy, at 25% each.&lt;/li&gt;
&lt;li&gt;Start with CKA-level cluster administration. There are no formal prerequisites, but both CNCF and Linux Foundation Education say the exam assumes that baseline.&lt;/li&gt;
&lt;li&gt;Build one lab with a real policy-enforcing CNI. Then practice resolving a name, examining EndpointSlices, reading node routes, applying a policy, and observing the resulting flow without reaching for a GUI first.&lt;/li&gt;
&lt;li&gt;As of 2026-11-11, the official exam page does not publish a passing score, Kubernetes version, or number of tasks. Check the &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;candidate handbook and FAQ&lt;/a&gt; before booking.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;CKA-level comfort with &lt;code&gt;kubectl&lt;/code&gt;: contexts, namespaces, Deployments, Services, logs, and &lt;code&gt;describe&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A workstation that can run a three-node &lt;a href="https://kind.sigs.k8s.io/" rel="noopener noreferrer"&gt;kind&lt;/a&gt; cluster in Docker or Podman.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;kubectl&lt;/code&gt;, &lt;code&gt;kind&lt;/code&gt;, the &lt;a href="https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/" rel="noopener noreferrer"&gt;Cilium CLI&lt;/a&gt;, and the &lt;a href="https://docs.cilium.io/en/stable/observability/hubble/setup/" rel="noopener noreferrer"&gt;Hubble CLI&lt;/a&gt; installed.&lt;/li&gt;
&lt;li&gt;Basic Linux networking: &lt;code&gt;ip&lt;/code&gt;, &lt;code&gt;ss&lt;/code&gt;, &lt;code&gt;tcpdump&lt;/code&gt;, and reading a route table.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What CKNE tests, and whether it fits you
&lt;/h2&gt;

&lt;p&gt;CKNE is for an administrator who already knows how to inspect workloads and cluster resources, and now needs to reason about the data path between them. It has no prerequisite certification, but doing CKA first is a sensible order because a networking drill should not also be a &lt;code&gt;kubectl&lt;/code&gt; syntax drill.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://training.linuxfoundation.org/certification/certified-kubernetes-network-engineer-ckne/" rel="noopener noreferrer"&gt;Linux Foundation certification page&lt;/a&gt; lists the exam-only price as $445, the exam plus THRIVE-ONE annual subscription as $625, and the exam plus Certification Pathway as $645. Eligibility lasts 12 months, one retake is included, and the certification is valid for two years. Those are purchase decisions, not study requirements. The useful practice resource bundled with the exam is the &lt;a href="https://killer.sh/ckne" rel="noopener noreferrer"&gt;Killer.sh CKNE simulator&lt;/a&gt;: two sessions, each open for 36 hours after activation. (The Linux Foundation page and Killer.sh currently give different question counts per session, 17 and 20, so do not plan around an exact number.) Use those sessions late in preparation to test navigation, pacing, and recovery after a bad assumption, not as your first exposure to a CNI or Gateway API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CKNE published domain weights&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Core Infrastructure and CNI&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service Networking and DNS&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Advanced Traffic Management&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network Security and Policy&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observability&lt;/td&gt;
&lt;td&gt;15%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Published by Linux Foundation Education and CNCF. Checked 2026-11-11.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Warm up concepts in the browser before you open the terminal. The &lt;a href="https://devops-daily.com/games/kubernetes-networking-cni-simulator" rel="noopener noreferrer"&gt;Kubernetes networking and CNI simulator&lt;/a&gt; traces pod, Service, policy, egress, and ingress paths across overlay, routed, and eBPF dataplanes. Use the &lt;a href="https://devops-daily.com/games/dns-simulator" rel="noopener noreferrer"&gt;DNS simulator&lt;/a&gt; for resolver and cache order, &lt;a href="https://devops-daily.com/games/packet-journey" rel="noopener noreferrer"&gt;Packet Journey&lt;/a&gt; for packet headers and hops, and &lt;a href="https://devops-daily.com/games/tcp-vs-udp" rel="noopener noreferrer"&gt;TCP vs UDP&lt;/a&gt; when you need to refresh transport behavior. The &lt;a href="https://devops-daily.com/games/load-balancer-simulator" rel="noopener noreferrer"&gt;load balancer simulator&lt;/a&gt; is useful for endpoint-selection intuition. For mTLS and policy, use the &lt;a href="https://devops-daily.com/games/service-mesh-simulator" rel="noopener noreferrer"&gt;service mesh simulator&lt;/a&gt; and &lt;a href="https://devops-daily.com/games/ssl-tls-handshake" rel="noopener noreferrer"&gt;SSL/TLS handshake simulator&lt;/a&gt;. They are warm-ups, not substitutes for command-line verification.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build a practice cluster that can fail interestingly
&lt;/h2&gt;

&lt;p&gt;You need a cluster where NetworkPolicy and encryption actually have an effect. &lt;code&gt;kind&lt;/code&gt; can start without its default CNI, and Cilium can then supply policy enforcement and WireGuard encryption. The &lt;a href="https://kind.sigs.k8s.io/docs/user/configuration/" rel="noopener noreferrer"&gt;kind configuration reference&lt;/a&gt; documents both &lt;code&gt;disableDefaultCNI&lt;/code&gt; and &lt;code&gt;podSubnet&lt;/code&gt;; Cilium documents the encryption settings below.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# ckne-kind.yaml&lt;/span&gt;
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: &lt;span class="kc"&gt;true&lt;/span&gt;
  podSubnet: &lt;span class="s1"&gt;'10.244.0.0/16'&lt;/span&gt;
nodes:
  - role: control-plane
  - role: worker
  - role: worker


kind create cluster --name ckne --config ckne-kind.yaml

&lt;span class="c"&gt;# Installs Cilium, turns on policy enforcement, and encrypts eligible&lt;/span&gt;
&lt;span class="c"&gt;# cross-node Cilium endpoint traffic with WireGuard.&lt;/span&gt;
cilium install \
  --set encryption.enabled=true \
  --set encryption.type=wireguard
cilium status --wait

&lt;span class="c"&gt;# Hubble flow visibility for the observability drills&lt;/span&gt;
cilium hubble &lt;span class="nb"&gt;enable&lt;/span&gt;
cilium hubble port-forward &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
hubble status

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;WireGuard encrypts Cilium-managed traffic between nodes, not traffic on the same node. Confirm the control plane before trusting the lab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n kube-system &lt;span class="nb"&gt;exec&lt;/span&gt; ds/cilium -- cilium-dbg status &lt;span class="o"&gt;|&lt;/span&gt; grep Encryption
kubectl get nodes -o wide
kubectl get pods -A

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a second interface exercise, install Multus and create a &lt;code&gt;NetworkAttachmentDefinition&lt;/code&gt;, whose API version is &lt;code&gt;k8s.cni.cncf.io/v1&lt;/code&gt;. A pod requests it with the &lt;code&gt;k8s.v1.cni.cncf.io/networks&lt;/code&gt; annotation. The &lt;a href="https://k8snetworkplumbingwg.github.io/multus-cni/docs/quickstart.html" rel="noopener noreferrer"&gt;Multus quickstart&lt;/a&gt; shows this attachment model and verifies the additional &lt;code&gt;net1&lt;/code&gt; interface. Do not memorize the YAML in isolation: check &lt;code&gt;ip addr&lt;/code&gt; and &lt;code&gt;ip route&lt;/code&gt; in the pod and ask which interface owns the default route.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;k8s.cni.cncf.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkAttachmentDefinition&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab-net&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'{"cniVersion":"1.0.0","type":"macvlan","master":"eth0","mode":"bridge","ipam":{"type":"host-local","subnet":"192.0.2.0/24"}}'&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Pod&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;dual-homed&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;k8s.v1.cni.cncf.io/networks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab-net&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;shell&lt;/span&gt;
      &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;busybox&lt;/span&gt;
      &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'sh'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'-c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'sleep 3600'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The NAD is only a definition. &lt;code&gt;master&lt;/code&gt; must name an interface that exists on every node that can schedule the Pod (kind nodes have &lt;code&gt;eth0&lt;/code&gt;; check yours in the node's host namespace first), and the &lt;code&gt;macvlan&lt;/code&gt; and &lt;code&gt;host-local&lt;/code&gt; plugin binaries must be present in the node's CNI directory. Use the Multus documentation for the plugin you choose. That distinction is a good troubleshooting habit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Practice every published domain as a terminal drill
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Core Infrastructure and CNI: trace before changing
&lt;/h3&gt;

&lt;p&gt;Know how pod CIDRs are allocated by your lab CNI, which node owns a destination CIDR, and whether traffic is routed or encapsulated. Under a clock, start with &lt;code&gt;kubectl get nodes -o wide&lt;/code&gt;, pod IPs, and routes before changing any YAML.&lt;/p&gt;

&lt;p&gt;Use an ephemeral node debug container to inspect the host network namespace. &lt;code&gt;ip route&lt;/code&gt; and &lt;code&gt;ip addr&lt;/code&gt; tell you what the node believes. A capture on port 53 separates a DNS transport problem from a bad record or bad search path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Open a shell in the node's network namespace (pick an image that ships tcpdump)&lt;/span&gt;
&lt;span class="nv"&gt;NODE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$(kubectl get nodes -o jsonpath='{.items[1].metadata.name}')"&lt;/span&gt;
kubectl debug &lt;span class="s2"&gt;"node/${NODE}"&lt;/span&gt; -it --image=nicolaka/netshoot --profile=sysadmin

&lt;span class="c"&gt;# What does the node believe about addresses and routes?&lt;/span&gt;
ip addr
ip route

&lt;span class="c"&gt;# Watch DNS on every interface while a pod resolves a name&lt;/span&gt;
tcpdump -ni any port 53

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Be able to distinguish kube-proxy modes from CNI service replacements. Inspect the actual cluster configuration before assuming &lt;code&gt;iptables&lt;/code&gt; rules explain a Service. Kubernetes supports &lt;code&gt;iptables&lt;/code&gt; and &lt;code&gt;nftables&lt;/code&gt; modes, while the &lt;a href="https://dev.to/devopsdaily/kubernetes-137-garhwal-what-shipped-and-what-slipped-1cmi"&gt;Kubernetes 1.37 release guide&lt;/a&gt; covers why &lt;code&gt;ipvs&lt;/code&gt; is deprecated and should not be the new thing you learn for this exam.&lt;/p&gt;

&lt;h3&gt;
  
  
  Service Networking and DNS: prove the backend set
&lt;/h3&gt;

&lt;p&gt;For every Service type, state what it exposes and how a client reaches it: &lt;code&gt;ClusterIP&lt;/code&gt; is cluster-internal virtual addressing, &lt;code&gt;NodePort&lt;/code&gt; exposes a port on node addresses, &lt;code&gt;LoadBalancer&lt;/code&gt; relies on an implementation to expose it externally, and a headless Service has no cluster IP for virtual-IP proxying. Then inspect the real backend set:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get svc payments -o yaml
kubectl get endpointslices \
  -l kubernetes.io/service-name=payments \
  -o yaml
kubectl get pods -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;payments

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;EndpointSlices are the source of Service backends that a proxy watches, and a Pod's readiness maps to the endpoint's readiness condition. Make one backend unready and watch the slice change. For external traffic, set &lt;code&gt;externalTrafficPolicy: Local&lt;/code&gt; on a NodePort or LoadBalancer test Service, send traffic to a node without a local endpoint, and explain the drop. &lt;code&gt;Local&lt;/code&gt; preserves source IP by forwarding only to node-local endpoints, so it exchanges reachability on every node for that property. The &lt;a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noopener noreferrer"&gt;Kubernetes Service documentation&lt;/a&gt; and &lt;a href="https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/" rel="noopener noreferrer"&gt;EndpointSlice documentation&lt;/a&gt; define those behaviors.&lt;/p&gt;

&lt;p&gt;DNS should be its own timed drill. From a pod, resolve a full Service name, inspect &lt;code&gt;/etc/resolv.conf&lt;/code&gt;, then try the short name. Kubernetes commonly configures search domains and &lt;code&gt;options ndots:5&lt;/code&gt;; read the file in the workload instead of assuming a particular setting.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# The dnsutils pod from the Kubernetes DNS debugging guide&lt;/span&gt;
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- cat /etc/resolv.conf
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- nslookup kubernetes.default.svc.cluster.local
kubectl &lt;span class="nb"&gt;exec&lt;/span&gt; -it dnsutils -- nslookup kubernetes.default

kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system logs -l k8s-app=kube-dns

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a configuration drill, edit the CoreDNS ConfigMap in a disposable cluster and add a name rewrite inside the existing server block. Its position in the Corefile does not matter: CoreDNS fixes the &lt;a href="https://coredns.io/manual/configuration/" rel="noopener noreferrer"&gt;plugin execution order&lt;/a&gt; when it is built. &lt;code&gt;answer auto&lt;/code&gt; rewrites the response back to the name the client asked for, which the &lt;a href="https://coredns.io/plugins/rewrite/" rel="noopener noreferrer"&gt;rewrite plugin&lt;/a&gt; requires for suffix rewrites to look right to the client.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight vcl"&gt;&lt;code&gt;rewrite stop &lt;span class="o"&gt;{&lt;/span&gt;
    name suffix .example.internal. .default.svc.cluster.local.
    answer auto
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That maps &lt;code&gt;payments.example.internal&lt;/code&gt; to &lt;code&gt;payments.default.svc.cluster.local&lt;/code&gt;. Restart the CoreDNS deployment if your distribution does not reload the Corefile, then test both names. Also find its metrics endpoint in the installed Corefile before deciding that a dashboard is a DNS diagnosis. Kubernetes has a useful &lt;a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="noopener noreferrer"&gt;DNS debugging sequence&lt;/a&gt; for this exact progression.&lt;/p&gt;

&lt;h3&gt;
  
  
  Advanced Traffic Management: make the controller part explicit
&lt;/h3&gt;

&lt;p&gt;Install the Gateway API Standard channel from its current release URL, then use the &lt;code&gt;GatewayClass&lt;/code&gt; supplied by the controller you installed. CRDs alone do not route traffic. The &lt;a href="https://gateway-api.sigs.k8s.io/guides/getting-started/introduction/" rel="noopener noreferrer"&gt;Gateway API installation guide&lt;/a&gt; publishes this current Standard-channel command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.7.0/standard-install.yaml

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Next, create a path match and a weighted split. Replace &lt;code&gt;&amp;lt;gateway-class&amp;gt;&lt;/code&gt; with an installed class and create &lt;code&gt;api-tls&lt;/code&gt; as a TLS Secret before applying the Gateway. The listener's Secret reference is the important TLS troubleshooting surface.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Gateway&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;public&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;gatewayClassName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;&amp;lt;gateway-class&amp;gt;&lt;/span&gt;
  &lt;span class="na"&gt;listeners&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;https&lt;/span&gt;
      &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPS&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;
      &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api.example.com&lt;/span&gt;
      &lt;span class="na"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Terminate&lt;/span&gt;
        &lt;span class="na"&gt;certificateRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Secret&lt;/span&gt;
            &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-tls&lt;/span&gt;
      &lt;span class="na"&gt;allowedRoutes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;namespaces&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Same&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gateway.networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;HTTPRoute&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;parentRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;public&lt;/span&gt;
  &lt;span class="na"&gt;hostnames&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'api.example.com'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;PathPrefix&lt;/span&gt;
            &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/v1&lt;/span&gt;
      &lt;span class="na"&gt;backendRefs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-v1&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;api-v2&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
          &lt;span class="na"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check &lt;code&gt;kubectl describe gateway public&lt;/code&gt; and &lt;code&gt;kubectl describe httproute api&lt;/code&gt;; accepted objects and programmed addresses are separate questions. If you are migrating from Ingress, see &lt;a href="https://devops-daily.com/posts/ingress-nginx-eol-gateway-api-migration" rel="noopener noreferrer"&gt;our Gateway API migration guide&lt;/a&gt; for a controller-aware cutover.&lt;/p&gt;

&lt;p&gt;LLM traffic belongs here, but do not reduce it to weighted HTTP backends. The &lt;a href="https://gateway-api-inference-extension.sigs.k8s.io/" rel="noopener noreferrer"&gt;Gateway API Inference Extension&lt;/a&gt; currently documents the &lt;code&gt;InferencePool&lt;/code&gt; API (v1) and an alpha &lt;code&gt;InferencePoolImport&lt;/code&gt; and describes endpoint selection using model-serving metrics and capabilities. Practice explaining why model-aware selection can consider a model name, cache state, or adapter availability. Treat its APIs as extension APIs and verify your controller's support and release status before using them in an answer.&lt;/p&gt;

&lt;p&gt;For cluster exit traffic, Cilium's implementation is &lt;a href="https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/" rel="noopener noreferrer"&gt;Egress Gateway&lt;/a&gt;. Enable it with &lt;code&gt;egressGateway.enabled=true&lt;/code&gt; (it also needs &lt;code&gt;bpf.masquerade=true&lt;/code&gt; and &lt;code&gt;kubeProxyReplacement=true&lt;/code&gt;, which the lab install above does not set), then write a &lt;code&gt;CiliumEgressGatewayPolicy&lt;/code&gt; that selects source pods and external destination CIDRs, plus a gateway node and egress IP or interface. Egress Gateway is not compatible with Cilium ClusterMesh. For cross-cluster service discovery, the &lt;a href="https://multicluster.sigs.k8s.io/concepts/multicluster-services-api/" rel="noopener noreferrer"&gt;Multi-Cluster Services API&lt;/a&gt; uses &lt;code&gt;ServiceExport&lt;/code&gt; to mark a Service for export and creates &lt;code&gt;ServiceImport&lt;/code&gt; plus EndpointSlices in consuming clusters. Cilium ClusterMesh is another implementation path; before working through it, read &lt;a href="https://devops-daily.com/posts/cilium-1-19-clustermesh-policy-flip" rel="noopener noreferrer"&gt;the policy-default change that can block cross-cluster traffic&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Network Security and Policy: deny, restore DNS, then prove it
&lt;/h3&gt;

&lt;p&gt;Start a namespace with both ingress and egress default-deny. Then permit only what the workload needs. The classic failure is to allow application traffic while forgetting DNS egress.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;default-deny&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;Ingress&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;Egress&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;networking.k8s.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;NetworkPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-dns&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="na"&gt;policyTypes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;Egress&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="na"&gt;egress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;namespaceSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;kubernetes.io/metadata.name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-system&lt;/span&gt;
          &lt;span class="na"&gt;podSelector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;k8s-app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;kube-dns&lt;/span&gt;
      &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;UDP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;TCP&lt;/span&gt;
          &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;53&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The label and namespace are cluster conventions, so inspect your DNS Service and Pods before applying this unchanged. Test from a labeled client pod, first expecting a timeout, then allowing the target Service on its actual port and running &lt;code&gt;curl -sv "http://${SERVICE}.${NAMESPACE}.svc.cluster.local:${PORT}/"&lt;/code&gt; with those three variables set to your target. Kubernetes defines NetworkPolicy as an API, but the chosen network plugin must enforce it. The &lt;a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="noopener noreferrer"&gt;NetworkPolicy documentation&lt;/a&gt; is the baseline for the API semantics.&lt;/p&gt;

&lt;p&gt;Encryption is not proof of authorization. In this Cilium lab, confirm WireGuard with &lt;code&gt;cilium-dbg status | grep Encryption&lt;/code&gt;, and capture on &lt;code&gt;cilium_wg0&lt;/code&gt; from a Cilium pod if you need to see the tunnel. Cilium documents the status check, tunnel device, and the fact that same-node traffic is not encrypted in its &lt;a href="https://docs.cilium.io/en/stable/security/network/encryption-wireguard/" rel="noopener noreferrer"&gt;WireGuard guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;For pod-level identity, use a service mesh. An Istio &lt;code&gt;AuthorizationPolicy&lt;/code&gt; can allow only a workload identity after mTLS identifies the caller. This is a compact policy drill, not a complete mesh install:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;security.istio.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AuthorizationPolicy&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;allow-frontend&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments&lt;/span&gt;
  &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ALLOW&lt;/span&gt;
  &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;principals&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;cluster.local/ns/lab/sa/frontend&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Verify the calling ServiceAccount and the mesh's mTLS mode before deciding a denial is an authorization error. Istio documents this defense-in-depth mTLS pattern in its &lt;a href="https://istio.io/latest/docs/ops/configuration/security/security-policy-examples/" rel="noopener noreferrer"&gt;security policy examples&lt;/a&gt;. For route behavior, retries, and traffic splits in a mesh, revisit &lt;a href="https://devops-daily.com/posts/istio-traffic-management-routing-retries-circuit-breaking" rel="noopener noreferrer"&gt;our Istio traffic-management guide&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability: observe the flow that proves the fix
&lt;/h3&gt;

&lt;p&gt;Your final action should produce evidence. For DNS, inspect CoreDNS logs and the Prometheus metrics configured by its &lt;code&gt;prometheus&lt;/code&gt; plugin. For Services, inspect EndpointSlices and, where relevant, kube-proxy metrics from the endpoint that your cluster exposes. For policy enforcement in Cilium, use Hubble to request the flows you care about:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;hubble observe --pod client --verdict DROPPED
hubble observe --namespace lab --protocol http
kubectl -n kube-system &lt;span class="nb"&gt;exec&lt;/span&gt; ds/cilium -- cilium-dbg status

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://docs.cilium.io/en/latest/observability/hubble/hubble-cli/" rel="noopener noreferrer"&gt;Hubble CLI documentation&lt;/a&gt; verifies the &lt;code&gt;--pod&lt;/code&gt; and &lt;code&gt;--verdict DROPPED&lt;/code&gt; form. Trace a request through a Gateway by sending a request ID such as &lt;code&gt;X-Request-ID&lt;/code&gt; from the client, then find that header in the gateway and application telemetry only if those components are configured to record it. A header alone does not create distributed tracing.&lt;/p&gt;

&lt;h2&gt;
  
  
  A six-week plan and exam-day routine
&lt;/h2&gt;

&lt;p&gt;Use the weights to bias your time, while returning to old drills often enough that command recall becomes automatic.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Week&lt;/th&gt;
&lt;th&gt;Focus&lt;/th&gt;
&lt;th&gt;Deliverable&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Core Infrastructure and CNI&lt;/td&gt;
&lt;td&gt;Explain one same-node and one cross-node packet path with routes and captures.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Service Networking and DNS&lt;/td&gt;
&lt;td&gt;Repair an empty EndpointSlice, a readiness issue, and a broken DNS lookup.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Network Security and Policy&lt;/td&gt;
&lt;td&gt;Build default-deny, restore DNS, allow one application path, and prove both allowed and denied flows.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Advanced Traffic Management&lt;/td&gt;
&lt;td&gt;Configure Gateway plus HTTPRoute, TLS Secret reference, weighted backends, and one egress or multi-cluster concept.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Observability plus weak spots&lt;/td&gt;
&lt;td&gt;Correlate CoreDNS, Service, Hubble, and Gateway evidence for one injected fault.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Timed mixed drills&lt;/td&gt;
&lt;td&gt;Use the two Killer.sh sessions as realistic pacing checks and catalogue every command you had to look up.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On exam day, the published format gives you 120 minutes in a Linux remote desktop with terminal and web interfaces. Spend the opening minutes inventorying contexts, namespaces, nodes, and the resources named by the task. Read status conditions before edits. Use a short loop for each task: observe, form one hypothesis, make the smallest change, and prove the result from the client side. Do not invent browser-tab or reference-material rules from hearsay. The official policies and candidate materials, not a study blog, decide those details.&lt;/p&gt;

&lt;p&gt;The goal is not to memorize every CNI's settings. It is to see a network problem as a chain of claims you can test: name resolves, Service has ready endpoints, dataplane has a route, policy permits it, encryption is configured where expected, and logs or flows show the result.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/ckne-kubernetes-network-engineer-exam-guide" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>networking</category>
      <category>certification</category>
      <category>ckne</category>
    </item>
    <item>
      <title>Who's On Call for the AI Agent? A Production Runbook</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 15:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/whos-on-call-for-the-ai-agent-a-production-runbook-715</link>
      <guid>https://dev.to/devopsdaily/whos-on-call-for-the-ai-agent-a-production-runbook-715</guid>
      <description>&lt;p&gt;An AI agent that opens pull requests, queries a database, restarts a service, or sends email is already a production service. At 3 a.m., the question is not whether its prompt was well written. It is who can stop it, which identity it used, and how to tell what it did. This runbook gives that service the same operational basics as any other: ownership, a page policy, controls that still work when the model does not, and an audit trail that answers an incident question quickly.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Put every production agent in the service catalog with an owning team, a primary on-call rotation, an escalation path, and named owners for its prompt, tools, and credentials.&lt;/li&gt;
&lt;li&gt;Page for harm that is happening now: an action loop, spend burning above budget, writes outside their window, permission-denied spikes, an aging approval queue, or failed tasks. Ticket slow trends such as evaluation, latency, and cost drift.&lt;/li&gt;
&lt;li&gt;Make the kill switch independent of the model. The tool runner checks it before every call, credentials can be revoked, the gateway applies a cap, and network egress is the final containment layer.&lt;/li&gt;
&lt;li&gt;Give the agent its own short-lived, read-scoped identity. A read-only database role helps, but does not limit query duration or call frequency.&lt;/li&gt;
&lt;li&gt;Record every tool action outside the agent's writable storage. During an incident, preserve that record before deciding whether to resume.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A service catalog and an on-call system with a primary and escalation target.&lt;/li&gt;
&lt;li&gt;A tool runner or gateway where you can enforce policy before an external action starts.&lt;/li&gt;
&lt;li&gt;Metrics, centralized logs, and a protected audit-log destination.&lt;/li&gt;
&lt;li&gt;An identity system that can issue a distinct identity for each agent.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Give the agent an owner before it gets a tool
&lt;/h2&gt;

&lt;p&gt;Treat an agent as a service, not as a feature hidden inside another service. Create a catalog entry that names its purpose, environments, repository, dashboards, runbook, primary rotation, and escalation path. A stale record is worse than a missing one when responders need to find the right team, which is why the catalog itself needs ownership and review. Our &lt;a href="https://dev.to/devopsdaily/a-service-catalog-nobody-updates-can-be-worse-than-none-we-checked-217-entries-against-github-1do7"&gt;service-catalog drift check&lt;/a&gt; shows how easily ownership records can stop matching reality.&lt;/p&gt;

&lt;p&gt;Split accountability by control surface. One team may own the agent's prompt, model choice, and configuration. The team that owns each target system owns the tool contract and allowed actions. The identity or platform team owns credential issuance, revocation, and the gateway policy. The catalog entry should name all three, but it must name one accountable service owner who receives the page and drives the incident.&lt;/p&gt;

&lt;p&gt;Write the escalation rule down with the service, not in a chat thread. The primary responder acknowledges and contains the agent. If the incident touches a target service, page that service's owner. Escalate to security when credentials, untrusted instructions, or data exposure are involved. The rotation and its escalation policy should have the same attention as any other production service. See the &lt;a href="https://devops-daily.com/posts/on-call-rotation-escalation-policy-guide" rel="noopener noreferrer"&gt;on-call rotation and escalation guide&lt;/a&gt; for the people and paging side of that setup.&lt;/p&gt;

&lt;h2&gt;
  
  
  Page on harm now, ticket slow drift
&lt;/h2&gt;

&lt;p&gt;The page rule is simple: wake someone only when waiting makes the outcome worse. A tool loop can create more actions in the next minute. A spend burn rate can exhaust a budget this hour. A week-over-week latency change needs investigation, but it does not usually need an immediate phone call.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Page when&lt;/th&gt;
&lt;th&gt;Ticket when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Tool actions&lt;/td&gt;
&lt;td&gt;Action rate is far above the service's baseline, or identical calls repeat&lt;/td&gt;
&lt;td&gt;A gradual change in the action mix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spend&lt;/td&gt;
&lt;td&gt;The hourly burn rate crosses the incident budget&lt;/td&gt;
&lt;td&gt;Cost trends upward across days or releases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Writes&lt;/td&gt;
&lt;td&gt;A successful write occurs outside an approved window&lt;/td&gt;
&lt;td&gt;The normal write volume slowly grows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Access&lt;/td&gt;
&lt;td&gt;Permission-denied errors spike, which can indicate drift or a prompt-injection attempt&lt;/td&gt;
&lt;td&gt;A single expected denial or a policy cleanup&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Human approval&lt;/td&gt;
&lt;td&gt;The oldest approval has exceeded the response promise&lt;/td&gt;
&lt;td&gt;Approval volume trends higher&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Task outcome&lt;/td&gt;
&lt;td&gt;Failure rate crosses the service's error budget&lt;/td&gt;
&lt;td&gt;Evaluation score or latency gradually declines&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The following names and thresholds are &lt;strong&gt;examples owned by this runbook&lt;/strong&gt; , not Prometheus, OpenTelemetry, or vendor metric names. Pick thresholds from your own normal traffic and give every rule an owner. The counter functions use a range vector, and each alert has a &lt;code&gt;for:&lt;/code&gt; duration. Note what &lt;code&gt;for:&lt;/code&gt; does: it requires the expression to stay true across evaluations, so it filters flapping rate rules, but an event rule such as a single forbidden write stays true for as long as the event sits inside its range window. That is intended for the write and spend rules: one forbidden write is worth a page.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;agent-production&lt;/span&gt;
    &lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentActionRateHigh&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tool_calls_total{outcome="success"}[5m])) &amp;gt; 20&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent action rate is above its configured baseline'&lt;/span&gt;

      &lt;span class="c1"&gt;# The runner increments this when the same tool + argument fingerprint
&lt;/span&gt;      &lt;span class="c1"&gt;# repeats inside its window. The fingerprint stays out of the label set.
&lt;/span&gt;      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentRepeatedToolCallLoop&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id, tool) (rate(agent_repeated_tool_calls_total[5m])) &amp;gt; 0.2&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Repeated tool-call rate indicates a loop'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentHourlySpendBudgetExceeded&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (increase(agent_spend_usd_total[1h])) &amp;gt; 25&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent spend exceeded the hourly incident budget'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentWriteOutsideWindow&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id, tool) (increase(agent_tool_calls_total{action="write",allowed_window="false",outcome="success"}[5m])) &amp;gt; 0&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent completed a write outside its allowed window'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentPermissionDeniedSpike&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tool_calls_total{outcome="permission_denied"}[5m])) &amp;gt; 1&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent permission denials are spiking'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentApprovalQueueOld&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;max by (agent_id) (agent_approval_queue_oldest_seconds) &amp;gt; 300&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'An agent approval has waited more than five minutes'&lt;/span&gt;

      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;AgentTaskFailureRateHigh&lt;/span&gt;
        &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sum by (agent_id) (rate(agent_tasks_total{outcome="failure"}[10m])) / sum by (agent_id) (rate(agent_tasks_total[10m])) &amp;gt; 0.20&lt;/span&gt;
        &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
        &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s"&gt;severity:&lt;/span&gt; &lt;span class="s"&gt;page&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Agent task failure rate is above 20 percent'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For the ticket queue, record evaluation score, latency percentiles, cost per completed task, and denied requests over a longer window. They tell you where to improve the service without training people to ignore their pager.&lt;/p&gt;

&lt;p&gt;For traces, OpenTelemetry's GenAI conventions define &lt;code&gt;invoke_agent&lt;/code&gt; and &lt;code&gt;execute_tool&lt;/code&gt; operations and tool-call attributes. They now live in their own &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai" rel="noopener noreferrer"&gt;semantic-conventions-genai&lt;/a&gt; repository, and the &lt;a href="https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/" rel="noopener noreferrer"&gt;registry page&lt;/a&gt; lists those operation values at &lt;strong&gt;Development&lt;/strong&gt; stability, so version your dashboards against them. The same registry notes that &lt;code&gt;gen_ai.tool.call.arguments&lt;/code&gt; may contain sensitive information. Keep raw arguments out of metric labels: a per-call value in a label is a cardinality problem as well as a leak.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build a kill switch that the model cannot bypass
&lt;/h2&gt;

&lt;p&gt;The model must not decide whether it is paused. Enforce a stop before every tool call in the runner that executes the call. Then give responders independent layers in case one control is unavailable.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Tool runner flag&lt;/strong&gt; Checked before every call&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Credential revoke&lt;/strong&gt; Identity can no longer authenticate&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gateway cap&lt;/strong&gt; Rate and budget stop&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Egress deny&lt;/strong&gt; Last network containment layer&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Any on-call engineer may pull the switch, with no approval. That rule prevents a debate from extending an incident. "Pause" means calls that have not yet passed the runner's check are refused. A call that passed the check a moment before the switch can still start or finish, which is why credential revocation and egress deny exist as the next layers for immediate containment at the target. Pausing does not mean that the runner discards the current run, conversation, pending approval, or audit records. Keep that state inspectable so responders can see what the agent was about to do.&lt;/p&gt;

&lt;p&gt;This short wrapper uses Redis as the control store. The switch is the existence of a key, so there is no value format to get wrong, and an unreachable or slow store stops the call instead of allowing it. Treat the store as a security control: connect over TLS, and give the runner an ACL user that can only read the pause keys, while responders use a separate user that can set and delete them (&lt;a href="https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/" rel="noopener noreferrer"&gt;Redis ACLs&lt;/a&gt;). Put the same check in every execution path, including retries and asynchronous workers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;

&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exceptions&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;RedisError&lt;/span&gt;

&lt;span class="c1"&gt;# rediss:// URL with the runner's read-only ACL user. Short connect and read&lt;/span&gt;
&lt;span class="c1"&gt;# timeouts: a hung control store must fail closed quickly.&lt;/span&gt;
&lt;span class="n"&gt;control&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"AGENT_CONTROL_REDIS_URL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;socket_timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;socket_connect_timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;call_tool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;paused&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;control&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"agent:{agent_id}:paused"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;RedisError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# A broken control plane must not permit an external action.&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"kill-switch state is unavailable"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;paused&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AgentPaused&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{agent_id} is paused; refusing {tool_name}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;call_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pulling the switch is then one command that any responder can run, as the incident-control ACL user. Deleting the key is the deliberate resume step, and it belongs in the incident record with who ran it and why:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Pause (any on-call engineer)&lt;/span&gt;
redis-cli --tls -h agent-control --user incident-control --askpass \
  SET agent:deploy-agent-prod:paused &lt;span class="s2"&gt;"paged-by-oncall $(date -u +%FT%TZ)"&lt;/span&gt;

&lt;span class="c"&gt;# Resume (service owner, after the review in step 7 of the runbook below)&lt;/span&gt;
redis-cli --tls -h agent-control --user incident-control --askpass \
  DEL agent:deploy-agent-prod:paused

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Test all four layers in a monthly game day. Start a harmless, repeatable test action, pause the agent while it is active, revoke its identity, exhaust a test gateway budget, and verify that egress policy blocks the target. Record time to containment and any action that escaped after the first stop signal. A kill switch that only exists in documentation is not a control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scope the identity, then bound the work
&lt;/h2&gt;

&lt;p&gt;Give each agent one identity. Do not hand an agent a human's long-lived token, and do not let several agents share a broad service identity. Use a read-scoped role for discovery work, short-lived credentials, and a separate path where a human approves a write before a runner obtains a write-capable identity.&lt;/p&gt;

&lt;p&gt;For AWS, the runner can obtain a 15-minute role session with &lt;code&gt;AssumeRole&lt;/code&gt;. The &lt;a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html" rel="noopener noreferrer"&gt;AWS STS API reference&lt;/a&gt; documents a &lt;code&gt;DurationSeconds&lt;/code&gt; minimum of 900 seconds and a maximum controlled by the role's maximum session duration, which may be from one to twelve hours. The command deliberately requests the 900-second minimum.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/AgentReadInventory \
  --role-session-name agent-inventory-run-20261112 \
  --duration-seconds 900 \
  --output json

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The role policy below allows only an example inventory task. The calling workload must separately be trusted to assume this role. A write action should not add write verbs here. Route it to an approval service that records an approver, checks the change, and issues a different, time-limited identity only after approval.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"Version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2012-10-17"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"Statement"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"Sid"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ReadInventoryOnly"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Effect"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Allow"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Action"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"ec2:DescribeInstances"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"cloudwatch:GetMetricData"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"Resource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"*"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a database, use a distinct read-only role, a connection limit, a &lt;code&gt;statement_timeout&lt;/code&gt;, and a gateway rate limit. Our &lt;a href="https://dev.to/devopsdaily/we-gave-an-ai-agent-a-readonly-neon-role-here-is-what-it-blocked-o07"&gt;read-only role probes&lt;/a&gt; showed why the role alone is not enough: a 15-second agent read on the primary made an &lt;code&gt;ALTER TABLE&lt;/code&gt; wait more than 11 seconds, and the app's inserts queued behind it, even though the agent never wrote anything. Sending agent reads to a read replica kept the primary clear in that test. The same probes found that nothing in the role bounds how often the agent repeats a legal read, and that the agent's session could run &lt;code&gt;SET statement_timeout = 0&lt;/code&gt;. So enforce the timeout and the rate cap in the tool runner or gateway, where the agent cannot change them.&lt;/p&gt;

&lt;p&gt;This is the practical answer to Excessive Agency, which the &lt;a href="https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/" rel="noopener noreferrer"&gt;2026 OWASP Top 10 for LLM Applications&lt;/a&gt; ranks third (&lt;a href="https://genai.owasp.org/2026/09/01/owasp-genai-security-project-unveils-2026-top-10-for-llm-applications-new-agent-control-standard-and-sponsors-as-community-tops-30000-members/" rel="noopener noreferrer"&gt;announcement&lt;/a&gt;). A model can be wrong or manipulated. The identity and policy boundary should still limit what reaches a real system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep an audit trail outside the agent's reach
&lt;/h2&gt;

&lt;p&gt;Log every attempted tool call as a structured event. Include the run ID, agent identity, credential or role, tool, redacted arguments, result, approver when one exists, timestamp, and trace ID. Store it in append-only storage with a retention period set by your security and compliance requirements. The agent's own workspace, database, or object-store prefix is not an acceptable sole copy because an agent with write access could alter the evidence.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"timestamp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-12T02:18:44Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"run_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"run_01JCG7"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"agent_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"deploy-agent-prod"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"credential"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:sts::123456789012:assumed-role/AgentReadInventory/agent-inventory-run-20261112"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ec2.describe_instances"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"eu-west-1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"filters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"[REDACTED]"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"success"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"approver"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"trace_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"4bf92f3577b34da6a3ce929d0e0e4736"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To answer "what did the agent do between 02:10 and 02:40?", query the protected JSONL archive, not the model transcript:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;jq -r '
  select(
    .agent_id == "deploy-agent-prod" and
    .timestamp &amp;gt;= "2026-11-12T02:10:00Z" and
    .timestamp &amp;lt; "2026-11-12T02:40:00Z"
  ) |
  [.timestamp, .run_id, .tool, .result, .trace_id] | @tsv
' agent-audit.jsonl

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Redact secrets before an event leaves the runner. In our &lt;a href="https://dev.to/devopsdaily/we-hid-96-instructions-in-the-logs-an-ops-agent-reads-here-is-what-stopped-them-3fec"&gt;prompt-injection test of an ops agent&lt;/a&gt;, a check outside the model stopped every forbidden action, but the agent still copied secrets into its own reports, so redaction belongs in code, not in the prompt. Store a stable argument fingerprint for loop detection, but do not put raw prompts, credentials, tokens, or user data into an alert label. Use the trace ID to join the audit event to the request and tool spans.&lt;/p&gt;

&lt;h2&gt;
  
  
  First 15 minutes: copy this runbook
&lt;/h2&gt;

&lt;p&gt;Put the following template beside the service catalog entry. Fill in the actual dashboard, credential-revocation command, audit location, and escalation targets before the agent goes live.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI AGENT INCIDENT: FIRST 15 MINUTES

Service: &amp;lt;agent service name&amp;gt;
Primary on-call: &amp;lt;rotation&amp;gt;
Escalation: &amp;lt;service owner&amp;gt;, &amp;lt;security contact&amp;gt;, &amp;lt;target-system owner&amp;gt;

1. Acknowledge the page and start an incident record. Record the run ID, alert,
   responder, and current time.
2. Pull the kill switch. Confirm the tool runner refuses a test call and note the
   last tool action accepted after the pause time.
3. Revoke or disable the agent credential. Apply the gateway rate or budget cap.
   Use egress deny if actions may still reach an external target.
4. Preserve the agent run state, approval queue, traces, and append-only audit
   events. Do not delete the run or ask the agent to explain itself.
5. Assess blast radius from the audit trail: actions, targets, credential used,
   successful writes, denied calls, spend, and affected data or services.
6. Communicate status, containment time, known impact, and next update time to
   the owning team and target-system owner. Escalate security when appropriate.
7. Resume only after the owner identifies the trigger, corrects the control gap,
   tests it in a safe environment, and explicitly records the decision.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;The best time to settle ownership and containment authority is before the first tool call. Once the agent is a cataloged service with a real pager, a tested stop path, scoped credentials, and a protected action record, responders can handle its failures as operational work instead of improvising around a model.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/who-is-on-call-for-the-ai-agent" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>aiagents</category>
      <category>oncall</category>
      <category>incidentresponse</category>
    </item>
    <item>
      <title>We Tested AWS Emulator: It Blocks More Than Its Docs Say, and Less Than AWS Does</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 09:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/we-tested-aws-emulator-it-blocks-more-than-its-docs-say-and-less-than-aws-does-bg6</link>
      <guid>https://dev.to/devopsdaily/we-tested-aws-emulator-it-blocks-more-than-its-docs-say-and-less-than-aws-does-bg6</guid>
      <description>&lt;p&gt;AWS released AWS Emulator on 2026-11-11: a free local copy of the AWS APIs that runs on your laptop or in CI, with no AWS account. We installed v1.0.322 on a throwaway droplet and ran it through 40 workflow checks across 9 services, 32 permission cases, the AWS CLI, the JavaScript SDK and Terraform. &lt;strong&gt;All 40 workflow checks passed, and the emulator refused two calls our own test code made with roles that had no permissions, as AWS would.&lt;/strong&gt; That is more permission checking than AWS's own security page describes. It is also less than AWS does: a user with no policies at all assumed an admin role and created an IAM user with it, and permissions boundaries and session policies changed nothing.&lt;/p&gt;

&lt;p&gt;Three other things would bite a real test suite: the JavaScript SDK's SQS client sent a request to the real &lt;code&gt;sqs.us-east-1.amazonaws.com&lt;/code&gt; by default, Terraform AWS provider 6.73.0 could not read back an S3 bucket, and the emulator refuses to hold more than 64 MiB in one bucket or 1,000 messages in one queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Install is one signed &lt;code&gt;.deb&lt;/code&gt; (163.0 MiB) that unpacks to one 925.7 MiB Rust binary. It answered its health check in a median 383.5 ms on repeated starts and 1,325 ms with the page cache dropped, using 172.5 MiB of anonymous memory at idle.&lt;/li&gt;
&lt;li&gt;Our scenario suite passed 40 of 40 checks: S3 presigned URLs, multipart and versioning; SQS visibility timeout, DLQ and redrive; DynamoDB conditional writes, transactions and streams; Lambda; EventBridge to SQS; SNS fan-out with a filter policy; KMS; Secrets Manager; Step Functions.&lt;/li&gt;
&lt;li&gt;The emulator enforces identity policies, bucket policies, KMS key policies and secret policies for the users and roles you create in it. That caught a missing &lt;code&gt;s3:PutObject&lt;/code&gt; and a missing &lt;code&gt;lambda:InvokeFunction&lt;/code&gt; in our own test code.&lt;/li&gt;
&lt;li&gt;It matched AWS's documented result in 27 of 32 permission cases. All 5 misses let through a request that AWS refuses, and they come from 4 causes: permissions boundaries, session policies, the caller's own &lt;code&gt;sts:AssumeRole&lt;/code&gt; check (a user with no policies assumed an admin role and used it), and &lt;code&gt;ListBuckets&lt;/code&gt; with no policy.&lt;/li&gt;
&lt;li&gt;The JS SDK v3 SQS client follows the AWS-style queue URL the emulator returns and calls real AWS. Set &lt;code&gt;useQueueUrlAsEndpoint: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Terraform AWS provider 6.73.0 fails on &lt;code&gt;aws_s3_bucket&lt;/code&gt; because it reads tags through S3 Control. Provider 5.100.0 applied and destroyed the same 25-resource stack, with one plan diff left.&lt;/li&gt;
&lt;li&gt;The repo has 29,684 commits, all by agent identities, and about 16.1 million lines of Rust. AWS's own fidelity scores for this release pass 56.5% of 42,937 scored checks.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux x86_64 machine you can throw away. We used a 4 vCPU / 8 GB DigitalOcean droplet with Ubuntu 24.04.&lt;/li&gt;
&lt;li&gt;AWS CLI v2 and an SDK (we used boto3 1.44.1 and the JavaScript SDK v3).&lt;/li&gt;
&lt;li&gt;Terraform if you want to test infrastructure code.&lt;/li&gt;
&lt;li&gt;No AWS account, and no real AWS credentials anywhere on the machine. The emulator writes its own keys per run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every script and raw result is in the companion repo:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/The-DevOps-Daily/aws-emulator-field-test" rel="noopener noreferrer"&gt;The-DevOps-Daily/aws-emulator-field-test on GitHub&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What AWS Emulator is, from the primary sources
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://aboutamazon.com/news/aws/how-autonomous-ai-agents-built-aws-emulator" rel="noopener noreferrer"&gt;launch post&lt;/a&gt; gives 438 AWS API models. Werner Vogels' &lt;a href="https://www.allthingsdistributed.com/2026/11/aws-emulator-and-the-engineers-job-when-agents-write-the-code.html" rel="noopener noreferrer"&gt;post&lt;/a&gt; says "more than 400 API models covering more than 200 AWS services" and that "all of AWS Emulator's code was written and tested by agents." The method in both posts: agents write fidelity tests from the public Smithy models, fix each test until it passes against real AWS, then build the emulator until it passes the same test. Every change goes through a merge gate that includes several security agents doing adversarial reviews.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.aws.amazon.com/emulator/latest/userguide/what-is-service.html" rel="noopener noreferrer"&gt;user guide&lt;/a&gt; adds the parts that matter in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It is for building and testing, not production. It "doesn't evaluate AWS Identity and Access Management (IAM) policies and resource-based policies the way AWS does."&lt;/li&gt;
&lt;li&gt;Behaviour is checked against AWS in &lt;code&gt;us-west-2&lt;/code&gt; only, and global services in &lt;code&gt;us-east-1&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;One account (&lt;code&gt;000000000000&lt;/code&gt;) and, in practice, one Region per emulator.&lt;/li&gt;
&lt;li&gt;It serves HTTPS on &lt;code&gt;127.0.0.1:9297&lt;/code&gt; with a new self-signed certificate and new access keys on every start.&lt;/li&gt;
&lt;li&gt;Telemetry is on by default. The &lt;a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer"&gt;README&lt;/a&gt; explains how to turn it off. We ran every test with &lt;code&gt;--telemetry off&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The code is CC0, except the &lt;code&gt;vendor/&lt;/code&gt; directory, which is Apache 2.0. The repository does not take pull requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Install and start
&lt;/h2&gt;

&lt;p&gt;On Linux it is a native package. The Debian package was 163.0 MiB and passed the GPG check from the install guide (&lt;code&gt;Good signature from "SSM Publishing &amp;lt;ssm-publishing@amazon.com&amp;gt;"&lt;/code&gt;). It installs two files: a license file and one stripped Rust binary of 970,692,256 bytes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -fsSLo aws-emulator.deb \
  https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest/aws-emulator_amd64.deb
sudo apt-get install -y ./aws-emulator.deb
aws-emulator --telemetry off

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We timed 10 repeated starts from &lt;code&gt;exec&lt;/code&gt; to the first HTTP 200 on &lt;code&gt;/__admin/health&lt;/code&gt;, each with a fresh home directory, then 5 more with the page cache dropped so the binary had to come off disk.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Measure&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;.deb&lt;/code&gt; download&lt;/td&gt;
&lt;td&gt;163.0 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Installed binary&lt;/td&gt;
&lt;td&gt;925.7 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to healthy, repeated starts (10 runs)&lt;/td&gt;
&lt;td&gt;338 to 470 ms, median 383.5 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to healthy, cold page cache (5 runs)&lt;/td&gt;
&lt;td&gt;1,260 to 1,399 ms, median 1,325 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RSS at ready&lt;/td&gt;
&lt;td&gt;about 282 MiB, flat over 60 s idle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anonymous memory at ready&lt;/td&gt;
&lt;td&gt;172.5 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The guide says the emulator "uses about 160 MiB when idle." Plain RSS reads higher because it counts pages of the memory-mapped binary. The anonymous part, 172.5 MiB, is close to the documented figure.&lt;/p&gt;

&lt;p&gt;A clean &lt;code&gt;SIGTERM&lt;/code&gt; deleted the credentials, config and certificate files from &lt;code&gt;~/.aws/emulator/9297/&lt;/code&gt;, as the guide says, and left the empty directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  The AWS CLI session
&lt;/h2&gt;

&lt;p&gt;The setup is eight environment variables from the &lt;a href="https://docs.aws.amazon.com/emulator/latest/userguide/point-sdk-to-emulator.html" rel="noopener noreferrer"&gt;SDK guide&lt;/a&gt;: the endpoint, the per-run credentials and config files, the CA bundle, two Region variables, and two switches that stop the SDK from asking the instance metadata service for credentials or adding prefixes to host names. Here is part of our recorded CLI session against a fresh emulator:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;aws cli against AWS Emulator v1.0.322&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ aws sts get-caller-identity --output text
000000000000    arn:aws:iam::000000000000:user/emulator AIDAH5I7GEMO7ADXNFAZU
$ aws s3 mb s3://cli-demo
make_bucket: cli-demo
$ aws s3 cp /tmp/notes.txt s3://cli-demo/notes.txt
upload: ../tmp/notes.txt to s3://cli-demo/notes.txt
$ curl --cacert &lt;span class="nv"&gt;$AWS_CA_BUNDLE&lt;/span&gt; &lt;span class="s2"&gt;"$(aws s3 presign s3://cli-demo/notes.txt --expires-in 60)"&lt;/span&gt;
release notes
$ aws sqs create-queue --queue-name cli-demo --query QueueUrl --output text
https://sqs.us-east-1.amazonaws.com/000000000000/cli-demo
&lt;span class="c"&gt;# the same validation error as AWS&lt;/span&gt;
$ aws dynamodb create-table --table-name ab ...
aws: &lt;span class="o"&gt;[&lt;/span&gt;ERROR&lt;span class="o"&gt;]&lt;/span&gt;: An error occurred &lt;span class="o"&gt;(&lt;/span&gt;ValidationException&lt;span class="o"&gt;)&lt;/span&gt; when calling the CreateTable operation: 1 validation error detected: Value &lt;span class="s1"&gt;'ab'&lt;/span&gt; at &lt;span class="s1"&gt;'tableName'&lt;/span&gt; failed to satisfy constraint: Member must have length greater than or equal to 3
&lt;span class="c"&gt;# S3 Control puts the account ID in the host name, so it never arrives&lt;/span&gt;
$ aws s3control list-access-points --account-id 000000000000
aws: &lt;span class="o"&gt;[&lt;/span&gt;ERROR&lt;span class="o"&gt;]&lt;/span&gt;: Could not connect to the endpoint URL: &lt;span class="s2"&gt;"https://000000000000.127.0.0.1:9297/v20180820/accesspoint"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the queue URL. It looks exactly like a real one. That detail comes back later.&lt;/p&gt;

&lt;h2&gt;
  
  
  40 workflow checks, 40 passes
&lt;/h2&gt;

&lt;p&gt;We did not want to test single API calls. We wrote 9 workflows of the kind that integration tests use, with 40 checks between them. Each check records the exact error text when it fails. Many checks test the error path, because a fake that accepts everything passes the happy path too.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workflow&lt;/th&gt;
&lt;th&gt;Checks&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;th&gt;What it covered&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;S3&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;9/9&lt;/td&gt;
&lt;td&gt;presigned GET and PUT, an expired URL refused with 403, 3-part multipart, completing with a 1 MiB first part refused with &lt;code&gt;EntityTooSmall&lt;/code&gt;, versioning, delete markers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SQS&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;5/5&lt;/td&gt;
&lt;td&gt;visibility timeout, the third receive moving a message to the DLQ, &lt;code&gt;StartMessageMoveTask&lt;/code&gt; back, FIFO deduplication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DynamoDB&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6/6&lt;/td&gt;
&lt;td&gt;insert-if-absent, a stale optimistic-lock update, a cancelled transaction with per-item reasons, streams&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4/4&lt;/td&gt;
&lt;td&gt;create and invoke a python3.12 zip, handler exception as &lt;code&gt;FunctionError&lt;/code&gt;, boto3 called from inside the function&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;EventBridge&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2/2&lt;/td&gt;
&lt;td&gt;a numeric pattern (&lt;code&gt;total &amp;gt; 100&lt;/code&gt;) delivering only the matching event to SQS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SNS&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2/2&lt;/td&gt;
&lt;td&gt;fan-out to two queues, one with raw delivery and one with a filter policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KMS&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6/6&lt;/td&gt;
&lt;td&gt;round trip by alias, wrong encryption context, tampered ciphertext, data keys, a disabled key&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets Manager&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3/3&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;AWSCURRENT&lt;/code&gt; and &lt;code&gt;AWSPREVIOUS&lt;/code&gt;, scheduled deletion blocking reads, restore&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Step Functions&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3/3&lt;/td&gt;
&lt;td&gt;Choice, Wait, a Lambda task, a Fail state&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The whole suite ran in 38.0 s against a fresh emulator. The emulator answered small calls fast: under load, a 100 KiB &lt;code&gt;PutObject&lt;/code&gt; took a median 14.66 ms and a 25-item &lt;code&gt;BatchWriteItem&lt;/code&gt; 13.64 ms.&lt;/p&gt;

&lt;p&gt;Getting to 40 of 40 took four runs. Every failure in between was a mistake in our test, and the repo keeps each of those runs. That story is the most useful part of this test.&lt;/p&gt;

&lt;h3&gt;
  
  
  The emulator caught our missing permissions
&lt;/h3&gt;

&lt;p&gt;On the third run, the Lambda function's call to S3 failed. We reproduced that run on purpose as a named variant (&lt;code&gt;FT_NO_ROLE_POLICIES=1&lt;/code&gt;), and this is its recorded error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AccessDenied: User: arn:aws:sts::000000000000:assumed-role/ft-fn-8820f7df/ft-fn-8820f7df is not authorized
to perform: s3:PutObject on resource: "arn:aws:s3:::ft-lambda-8820f7df/from-lambda.txt" because no
identity-based policy allows the s3:PutObject action

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our test created the Lambda execution role with a trust policy and nothing else. On AWS that function fails the same way. The Step Functions test failed for the same reason: &lt;code&gt;ft-sfn-8820f7df is not authorized to perform: lambda:InvokeFunction&lt;/code&gt;. We added the two missing policies and both checks passed. That variant is the best argument for the emulator we found.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Lambda sandbox
&lt;/h3&gt;

&lt;p&gt;The first two runs failed for a different reason: our handler probed the machine it ran on. The emulator refused every probe that reached for the host:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Probe from inside the function&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sys.version&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;3.14.0.alpha&lt;/code&gt;, for a &lt;code&gt;python3.12&lt;/code&gt; function&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;open("/etc/hostname")&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator cannot open a file of the host&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;os.getuid()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator has no host filesystem or process to reach&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;socket.gethostname()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OSError: a Lambda function in this emulator has no host network to reach&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;import subprocess&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ModuleNotFoundError: No module named '_signal'&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That is good isolation for zip-packaged Python, and better than the security page suggests ("it doesn't isolate that code from your machine"). We only tested Python zip functions, so do not assume the same for other runtimes. Two practical effects: a handler that imports &lt;code&gt;subprocess&lt;/code&gt; at module level fails to load with &lt;code&gt;Runtime.ImportModuleError&lt;/code&gt;, and your code runs on a different Python than the one you deploy to. Four invokes in a row took 369, 335, 354 and 355 ms, with no consistent warm-start speedup.&lt;/p&gt;

&lt;h2&gt;
  
  
  The permission test: 27 of 32
&lt;/h2&gt;

&lt;p&gt;The security page is direct about permissions:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Some services in the emulator evaluate IAM policies for IAM users and roles that you create in the emulator, but not the same way as AWS. The emulator never denies a request signed with its own key. It doesn't check AWS KMS key policies or Amazon S3 bucket policies.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The Lambda result suggested more checking than that, so we wrote 32 cases. For each one we wrote down what AWS documents. &lt;strong&gt;These cases were not run on real AWS&lt;/strong&gt; : the expected column comes from the AWS documentation for policy evaluation, bucket policies, KMS key policies, role trust policies and SQS queue policies. Then we ran them against the emulator, and against Moto and Floci for context.&lt;/p&gt;

&lt;p&gt;Bold marks a result that differs from what AWS documents.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;AWS documents&lt;/th&gt;
&lt;th&gt;AWS Emulator&lt;/th&gt;
&lt;th&gt;Floci&lt;/th&gt;
&lt;th&gt;Moto&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sts:GetCallerIdentity&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:ListAllMyBuckets&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sqs:SendMessage&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: dynamodb:PutItem&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: kms:Encrypt&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: secretsmanager:GetSecretValue&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: sns:Publish&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: events:PutEvents&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: iam:CreateUser&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: admin user: s3:PutObject (baseline)&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow * plus explicit Deny s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow * but permissions boundary allows only reads: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow PutObject only with SSE-KMS: put without SSE&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow PutObject only with SSE-KMS: put with SSE-KMS&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: bucket policy denies this admin user: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: bucket policy denies everyone over HTTPS: emulator's own key PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: KMS key policy denies this admin user: kms:Encrypt&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: secret resource policy denies this admin user: GetSecretValue&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;resource policies: no identity policy, bucket policy allows this user: s3:PutObject&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: PutObject allowed on bucket A only: put to bucket A&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: PutObject allowed on bucket A only: put to bucket B&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow NotAction s3:*: sqs:SendMessage&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;td&gt;allow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;identity policies: Allow NotAction s3:*: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:ListBucket (ListObjectsV2)&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;user with no policies: s3:GetObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: admin user assumes a role that trusts only lambda.amazonaws.com&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: no-policy user assumes a role that trusts the account (needs sts:AssumeRole itself)&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: no-policy user: assume the account-trusted admin role, then s3:PutObject + iam:CreateUser&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;roles and sessions: admin role assumed with a GetObject-only session policy: s3:PutObject&lt;/td&gt;
&lt;td&gt;deny&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;allow&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;service-to-service: EventBridge rule -&amp;gt; SQS queue with no queue policy&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;service-to-service: SNS topic -&amp;gt; SQS queue with no queue policy&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;not delivered&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;delivered&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On v1.0.322, the "never denies its own key" sentence is out of date. A bucket policy that denies &lt;code&gt;s3:PutObject&lt;/code&gt; to everyone over HTTPS stopped the emulator's own key. A KMS key policy deny, a secret resource policy deny and a bucket policy deny all stopped an admin user. Explicit denies, resource scoping, &lt;code&gt;NotAction&lt;/code&gt; and condition keys such as &lt;code&gt;s3:x-amz-server-side-encryption&lt;/code&gt; all worked. EventBridge and SNS did not deliver to a queue without a queue policy, which is what AWS does.&lt;/p&gt;

&lt;p&gt;The 5 misses come from 4 causes, and they have one thing in common. Each is a control that &lt;strong&gt;narrows&lt;/strong&gt; access, and the emulator skipped it and let the request through:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Permissions boundaries.&lt;/strong&gt; A user with &lt;code&gt;Allow *&lt;/code&gt; and a read-only boundary could still &lt;code&gt;PutObject&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session policies.&lt;/strong&gt; An admin role assumed with a &lt;code&gt;GetObject&lt;/code&gt;-only session policy could still &lt;code&gt;PutObject&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The caller's own &lt;code&gt;sts:AssumeRole&lt;/code&gt; permission.&lt;/strong&gt; A role that trusts the account (&lt;code&gt;arn:aws:iam::000000000000:root&lt;/code&gt;) means "any principal in this account that IAM allows to assume it." Our user had no policies at all, so AWS refuses. The emulator returned credentials, and with them the user wrote to S3 and created a new IAM user.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ListBuckets&lt;/code&gt; with no policy&lt;/strong&gt; returned the bucket list.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The repo's &lt;code&gt;issues/things-that-do-not-work/&lt;/code&gt; folder has open files about session-policy scoping and permissions boundaries, so the agents know about at least some of this. We did not match each of our 4 causes to one issue.&lt;/p&gt;

&lt;p&gt;One fix to our own harness changed the comparison numbers. Moto refuses a bucket-policy deny with a bare &lt;code&gt;403: Forbidden&lt;/code&gt; where AWS returns &lt;code&gt;AccessDenied&lt;/code&gt;, and our first classifier counted that as an error, not a refusal. We now count every HTTP 403 as a refusal, which adds those two cases to Moto's total (8 of 32). The saved results still show the bare &lt;code&gt;403&lt;/code&gt; code. The fix did not change the AWS Emulator or Floci results.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this matters for your tests
&lt;/h3&gt;

&lt;p&gt;Boundaries and session policies are the controls teams use for least privilege: a CI role that may create roles only inside a boundary, a broker that hands out narrowed credentials, a platform team that delegates IAM. Locally, both directions go wrong. A job whose boundary blocks a write it needs passes against the emulator and fails on AWS. A test that asserts "the boundary stops this write" fails against the emulator even when the boundary on AWS is correct.&lt;/p&gt;

&lt;p&gt;What to do about it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Create the real roles in the emulator.&lt;/strong&gt; Do not run everything with the emulator's admin key. Create the roles and policies you deploy, and run each component with its own role. That is how the emulator caught our two missing permissions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep negative permission tests for real AWS.&lt;/strong&gt; Run a small set against a sandbox account in CI, with short-lived credentials. Boundaries, session policies and role trust belong there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check policies without calling the emulator.&lt;/strong&gt; IAM Access Analyzer policy validation and custom policy checks catch many mistakes before anything runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a local &lt;code&gt;AccessDenied&lt;/code&gt; as a bug, and a local success as unproven.&lt;/strong&gt; The emulator's denies matched AWS in every case we tried. Its allows did not.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Three more ways a green local run can lie
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The JavaScript SDK sent SQS calls to real AWS
&lt;/h3&gt;

&lt;p&gt;Our JS SDK v3 smoke test passed for S3 and DynamoDB. SQS failed with &lt;code&gt;InvalidClientTokenId: The security token included in the request is invalid.&lt;/code&gt; The emulator would not say that about its own key. The SDK printed the reason:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;QueueUrl=https://sqs.us-east-1.amazonaws.com/000000000000/js-demo differs from SQSClient resolved
endpoint=https://127.0.0.1:9297/, using QueueUrl host as endpoint.
Set [endpoint=string] or [useQueueUrlAsEndpoint=false] on the SQSClient.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The emulator returns queue URLs in the AWS format. &lt;code&gt;@aws-sdk/client-sqs&lt;/code&gt; 3.1167.0 uses the queue URL's host by default, so the request went to &lt;code&gt;sqs.us-east-1.amazonaws.com&lt;/code&gt;, and AWS rejected the emulator's key. To prove it, we pointed that host name at &lt;code&gt;0.0.0.0&lt;/code&gt; in &lt;code&gt;/etc/hosts&lt;/code&gt;. The error changed to &lt;code&gt;connect ECONNREFUSED 0.0.0.0:443&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Keep SQS calls on the emulator&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sqs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;SQSClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;useQueueUrlAsEndpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With that option, the same test passed. The SDK guide lists host names that SDKs change (host prefixes, S3, S3 Control) but not this case. Our machine had no real credentials, so AWS rejected the emulator's key and nothing else happened. But environment variables take precedence over the emulator's credentials file. In a CI job with real &lt;code&gt;AWS_ACCESS_KEY_ID&lt;/code&gt; and &lt;code&gt;AWS_SECRET_ACCESS_KEY&lt;/code&gt; set, the same test would have sent a request to AWS signed with real keys. The guide's advice to unset real credentials in the shell that runs your tests is not optional.&lt;/p&gt;

&lt;h3&gt;
  
  
  Terraform: provider 6.73.0 could not read back an S3 bucket
&lt;/h3&gt;

&lt;p&gt;We applied a small stack with 25 resources: a versioned bucket with a lifecycle rule, queues with a DLQ, a DynamoDB table with a stream, a GSI and TTL, SNS, KMS, Secrets Manager, SSM, IAM roles, a Lambda function with an SQS event source, an EventBridge rule and a Step Functions state machine.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;AWS provider 6.73.0&lt;/th&gt;
&lt;th&gt;AWS provider 5.100.0&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;terraform apply&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;failed, 23 entries in state&lt;/td&gt;
&lt;td&gt;passed, 26 entries in state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Error&lt;/td&gt;
&lt;td&gt;&lt;code&gt;S3 Control: ListTagsForResource ... lookup 000000000000.127.0.0.1: no such host&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Second &lt;code&gt;plan&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;failed on the same S3 call&lt;/td&gt;
&lt;td&gt;1 change: DynamoDB &lt;code&gt;on_demand_throughput&lt;/code&gt; (-1 to null)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;terraform destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;failed on the same S3 call&lt;/td&gt;
&lt;td&gt;passed in 39.3 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In our test, provider 6.73.0 read bucket tags through S3 Control, and S3 Control puts the account ID in the host name. The SDK guide says S3 Control requests cannot reach the emulator. We tried a hosts-file entry for &lt;code&gt;000000000000.127.0.0.1&lt;/code&gt;, but &lt;code&gt;getent&lt;/code&gt; did not resolve it. When we forced the address with &lt;code&gt;curl --resolve&lt;/code&gt;, the TLS handshake failed (curl exit code 35) even with certificate checks off. In this test, provider 5.100.0 worked where 6.73.0 failed, with one plan diff left. Until that changes, stacks with S3 buckets need an older provider or a separate test for the bucket.&lt;/p&gt;

&lt;p&gt;Apply took about 85 s with both providers. The three SQS resources took 25 s each. In our load test the emulator answered SQS batch sends in a median 13.08 ms, so slow SQS responses do not explain those 25 s. We did not trace what the provider waits for.&lt;/p&gt;

&lt;h3&gt;
  
  
  Caps that AWS does not have
&lt;/h3&gt;

&lt;p&gt;Our first memory test tried to put 1,000 objects of 100 KiB in one bucket and 10,000 messages in one queue. Both stopped early, so we wrote a probe that fills one bucket and one queue until the emulator refuses. The bucket took 654 objects and the queue took 1,000 messages:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ValidationException: storing "obj-00654" would leave the bucket "cap-probe" holding 67131605 bytes of
objects, past the 67108864 bytes this emulator keeps per bucket

OverLimit: This emulator holds at most 1000 messages per queue; queue cap-probe cannot take 1 more.
Delete messages from it first.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is 64 MiB per bucket and 1,000 messages per queue. Real S3 buckets and SQS queues have no such limits on stored data or backlog. The errors are clear, and the source has more caps of the same kind, with messages such as "this emulator holds at most {} files per repository tree" and "this emulator holds tables for at most {}". A test that uploads a large fixture or builds a queue backlog has to stay under them.&lt;/p&gt;

&lt;p&gt;With 58.6 MiB in S3, 20,000 DynamoDB items and 900 queued messages, anonymous memory grew from 172.5 MiB to 277.2 MiB. Deleting everything and waiting 15 s did not give memory back (278.8 MiB). Per the docs, restart the emulator between suites, which also gives you clean state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AWS Emulator memory while a test suite fills it&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;idle&lt;/th&gt;
&lt;th&gt;+58.6 MiB in S3&lt;/th&gt;
&lt;th&gt;+20,000 DynamoDB items&lt;/th&gt;
&lt;th&gt;+900 SQS messages&lt;/th&gt;
&lt;th&gt;all deleted, +15 s&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RSS&lt;/td&gt;
&lt;td&gt;281.7 MiB&lt;/td&gt;
&lt;td&gt;367.2 MiB&lt;/td&gt;
&lt;td&gt;411.5 MiB&lt;/td&gt;
&lt;td&gt;415.4 MiB&lt;/td&gt;
&lt;td&gt;418.9 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anonymous memory&lt;/td&gt;
&lt;td&gt;172.5 MiB&lt;/td&gt;
&lt;td&gt;236.6 MiB&lt;/td&gt;
&lt;td&gt;276.4 MiB&lt;/td&gt;
&lt;td&gt;277.2 MiB&lt;/td&gt;
&lt;td&gt;278.8 MiB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;v1.0.322, one fresh process, 4 vCPU / 8 GB droplet. Source: results/loaded-memory.json in the companion repo.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How it compares to Moto and Floci
&lt;/h2&gt;

&lt;p&gt;LocalStack was the usual choice for this. Since March 2026, &lt;code&gt;localstack/localstack&lt;/code&gt; is a single image that needs an auth token, and the free plan is meant for non-commercial use (&lt;a href="https://blog.localstack.cloud/the-road-ahead-for-localstack/" rel="noopener noreferrer"&gt;LocalStack's announcement&lt;/a&gt;). We did not test it, because that needs an account. Moto (Apache 2.0) and Floci (MIT) run with no account, so we ran the same two suites against both, &lt;strong&gt;in their default configurations&lt;/strong&gt;. Both have opt-in authorization checks that we did not turn on, so their permission rows show the defaults, not their best case. Start times are not measured the same way: AWS Emulator is timed to its health check, Moto to its first HTTP response, and Floci from &lt;code&gt;docker run&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;AWS Emulator 1.0.322&lt;/th&gt;
&lt;th&gt;Floci 2.4.0&lt;/th&gt;
&lt;th&gt;Moto 5.2.5 (server mode)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;License&lt;/td&gt;
&lt;td&gt;CC0 (&lt;code&gt;vendor/&lt;/code&gt; is Apache 2.0)&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;Apache 2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Account or token needed&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Install&lt;/td&gt;
&lt;td&gt;163.0 MiB &lt;code&gt;.deb&lt;/code&gt;, 925.7 MiB binary&lt;/td&gt;
&lt;td&gt;90.2 MiB image (&lt;code&gt;docker image inspect&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Python package&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Start to first response&lt;/td&gt;
&lt;td&gt;median 383.5 ms (10 runs)&lt;/td&gt;
&lt;td&gt;992 ms including &lt;code&gt;docker run&lt;/code&gt; (1 run)&lt;/td&gt;
&lt;td&gt;1,174 ms (1 run)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Idle memory&lt;/td&gt;
&lt;td&gt;172.5 MiB anonymous, about 282 MiB RSS&lt;/td&gt;
&lt;td&gt;25.4 MiB (&lt;code&gt;docker stats&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;51.1 MiB RSS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Workflow checks passed&lt;/td&gt;
&lt;td&gt;40 of 40&lt;/td&gt;
&lt;td&gt;38 of 40&lt;/td&gt;
&lt;td&gt;32 of 40 (1 miss is our harness)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Permission cases matching AWS docs&lt;/td&gt;
&lt;td&gt;27 of 32&lt;/td&gt;
&lt;td&gt;6 of 32&lt;/td&gt;
&lt;td&gt;8 of 32&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Share of checks that matched AWS, default configurations&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AWS Emulator, workflows&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Floci, workflows&lt;/td&gt;
&lt;td&gt;95%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moto, workflows&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;td&gt;Workflow checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS Emulator, permissions&lt;/td&gt;
&lt;td&gt;84.4%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moto, permissions&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Floci, permissions&lt;/td&gt;
&lt;td&gt;18.8%&lt;/td&gt;
&lt;td&gt;Permission cases&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Workflow checks: 40 per target; one of Moto's misses is a networking problem in our harness. Permission cases: 32 per target, expected results from AWS documentation, not run on real AWS. Source: results/&lt;/em&gt;-scenarios.json and results/&lt;em&gt;-iam-gap.json in the companion repo.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Some notes so the table is fair:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Moto's Lambda-to-S3 check failed because of our setup: Moto listened on &lt;code&gt;127.0.0.1&lt;/code&gt;, and its Lambda container could not reach &lt;code&gt;host.docker.internal:5000&lt;/code&gt;. We count it as a fail, but it is a network problem in our harness, not proof that Moto cannot do it.&lt;/li&gt;
&lt;li&gt;Moto's other misses: an expired presigned URL still worked, a presigned PUT stored an empty body, &lt;code&gt;StartMessageMoveTask&lt;/code&gt; returned HTTP 500, a handler exception came back as &lt;code&gt;Handled&lt;/code&gt; instead of &lt;code&gt;Unhandled&lt;/code&gt;, a disabled KMS key still encrypted, and both Step Functions executions were still &lt;code&gt;RUNNING&lt;/code&gt; after 30 s.&lt;/li&gt;
&lt;li&gt;Floci's misses: an expired presigned URL still worked, and the EventBridge rule with a numeric pattern delivered nothing.&lt;/li&gt;
&lt;li&gt;In the permission matrix, both alternatives allowed every request in their default modes, with one exception: Moto refused both bucket-policy denies.&lt;/li&gt;
&lt;li&gt;On size and memory, the others are much smaller. Floci's image is 90.2 MiB and used 25.4 MiB idle. Moto used 51.1 MiB idle. AWS Emulator is a 925.7 MiB binary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On these suites, AWS Emulator matched AWS more often than either alternative, and by a wide margin on permissions.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the repo shows about "built by agents"
&lt;/h2&gt;

&lt;p&gt;We cloned &lt;a href="https://github.com/aws/aws-emulator" rel="noopener noreferrer"&gt;aws/aws-emulator&lt;/a&gt; and the &lt;a href="https://github.com/aws/aws-emulator-evaluation-suite" rel="noopener noreferrer"&gt;evaluation suite&lt;/a&gt; and counted.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Goal: Each release publishes its own pass rate: 56.5% for v1.0.322&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fidelity tests&lt;/strong&gt; 29,676 boto3 checks&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IssueFinderAgent&lt;/strong&gt; 16,664 commits&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;issues/ folder&lt;/strong&gt; 6,544 open, 12,708 fixed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;IssueFixerAgent&lt;/strong&gt; 12,981 commits&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Release&lt;/strong&gt; with scores.json&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;em&gt;every change passes a merge gate (not visible in the repo): failing fidelity tests become new issues, then back to step 1.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every commit is by an agent identity.&lt;/strong&gt; 29,684 commits: IssueFinderAgent 16,664, IssueFixerAgent 12,981, Scaffolder 29, TraceIdMigration 6, ReleaseManagerAgent 4. No commit carries a person's name. The launch post says: "Our team hasn't written or reviewed any of the code."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The public history is three weeks long.&lt;/strong&gt; It starts on 2026-10-21 with a commit called "Initial emulator scaffold", and 3,734 commits landed that first day. All of it falls between 2026-10-21 and 2026-11-10.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is big.&lt;/strong&gt; 436 service crates, 437 Smithy model files, 40,130 operation files, 14,099 test files, and 16,142,920 lines of Rust outside &lt;code&gt;vendor/&lt;/code&gt;. The EC2 crate alone is 709,186 lines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The work is issue-driven, in the open.&lt;/strong&gt; Commits are "file issues" and "Fix issue" with &lt;code&gt;Producer:&lt;/code&gt; trailers, and each issue is a Markdown file with a severity, the failing fidelity test and the commit it was found against. 6,544 are open and 12,708 are fixed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The merge gate cannot be audited from the repo.&lt;/strong&gt; There are no CI workflows in &lt;code&gt;.github/&lt;/code&gt;, and commits carry no review trailers. The adversarial security review may well happen, but you cannot see it here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The fidelity tests are public, their history is not.&lt;/strong&gt; The evaluation suite has 29,676 tests in &lt;code&gt;manifest.jsonl&lt;/code&gt;, covering 1,206 service directories (859 of them cross-service), in a repository with 2 commits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AWS publishes its own score, and it is honest.&lt;/strong&gt; The &lt;code&gt;scores.json.gz&lt;/code&gt; asset on the v1.0.322 release has 42,937 scored checks: 24,243 pass, 18,614 fail, 80 skipped. That is 56.5%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;AWS's own fidelity pass rate for v1.0.322, by service&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;EventBridge&lt;/td&gt;
&lt;td&gt;71.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KMS&lt;/td&gt;
&lt;td&gt;71.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SNS&lt;/td&gt;
&lt;td&gt;69.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DynamoDB&lt;/td&gt;
&lt;td&gt;68.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets Manager&lt;/td&gt;
&lt;td&gt;68%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SQS&lt;/td&gt;
&lt;td&gt;67.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S3&lt;/td&gt;
&lt;td&gt;62.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IAM&lt;/td&gt;
&lt;td&gt;58%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lambda&lt;/td&gt;
&lt;td&gt;51.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;STS&lt;/td&gt;
&lt;td&gt;47.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;From scores.json.gz on the v1.0.322 release (42,937 scored checks, 56.5% overall). Our own 40 workflow checks passed 100% on these services. Source: results/scores-summary.json.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Those numbers do not contradict our 40 of 40. The evaluation suite is written to find gaps, and the issue-finder agents keep adding tests where the emulator fails. Our suite tests the common paths that a typical app uses. Both are true: the common paths work, and the edges are still moving. The releases page listed 12 releases between 2026-11-07 and 2026-11-11, so pin a version in CI.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we could not conclude
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;We did not run the permission cases on real AWS.&lt;/strong&gt; The expected results come from AWS documentation. We are confident about each of them, but "AWS documents X" is not the same as "we saw AWS do X."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One machine, one version.&lt;/strong&gt; Everything ran on v1.0.322 on one 4 vCPU droplet. With 12 releases in 5 days, some of these results may already be different.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Default configurations only&lt;/strong&gt; for Moto and Floci. Both can do more permission checking if you turn it on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Python zip functions only&lt;/strong&gt; for the Lambda sandbox. Container images and other runtimes may behave differently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;We did not verify the "200+ services" claim&lt;/strong&gt; one by one. We counted 436 service crates and 437 model files, and we exercised at least 11 services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;We could not observe the merge gate&lt;/strong&gt; or the security agents. We can only say that their output is not in the public repo.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Using it safely
&lt;/h2&gt;

&lt;p&gt;If you adopt it for local tests or CI, this is the checklist we would use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Start it with &lt;code&gt;--telemetry off&lt;/code&gt; if your policy needs that, and pin the version.&lt;/li&gt;
&lt;li&gt;Unset &lt;code&gt;AWS_ACCESS_KEY_ID&lt;/code&gt;, &lt;code&gt;AWS_SECRET_ACCESS_KEY&lt;/code&gt;, &lt;code&gt;AWS_SESSION_TOKEN&lt;/code&gt; and &lt;code&gt;AWS_PROFILE&lt;/code&gt; in the same script that runs the tests, and set &lt;code&gt;AWS_EC2_METADATA_DISABLED=true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;In the JS SDK, create SQS clients with &lt;code&gt;useQueueUrlAsEndpoint: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Test your Terraform provider version first. In our run, 6.73.0 failed on S3 buckets and 5.100.0 worked; otherwise keep buckets out of the emulated stack.&lt;/li&gt;
&lt;li&gt;Keep fixtures under 64 MiB per bucket and backlogs under 1,000 messages per queue.&lt;/li&gt;
&lt;li&gt;Create real roles with real policies and run each component as its role. Let the emulator's &lt;code&gt;AccessDenied&lt;/code&gt; catch missing permissions.&lt;/li&gt;
&lt;li&gt;Test boundaries, session policies and role trust on a real sandbox account.&lt;/li&gt;
&lt;li&gt;Restart the emulator between suites for clean state and memory.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you still run LocalStack from an old pinned image, our &lt;a href="https://devops-daily.com/posts/experimenting-locally-with-terraform" rel="noopener noreferrer"&gt;local Terraform experiments post&lt;/a&gt; shows the older setup.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Of the three local emulators we ran, AWS Emulator matched AWS most often. Our everyday workflows passed, the error codes matched, and it found real permission bugs in our test code. The part to watch is not the warning AWS gives: it checks more policies than its docs say. The risk is the controls it skips. A green run against the emulator proves that your code works with the permissions it has. It does not prove that your permission limits hold. Keep that small set of tests on real AWS, and use the emulator for everything else.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/aws-emulator-hands-on-test" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aws</category>
      <category>iam</category>
      <category>testing</category>
      <category>terraform</category>
    </item>
    <item>
      <title>Disk Full but du Disagrees: Finding Deleted Files That Still Use Space</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Thu, 12 Nov 2026 07:00:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/disk-full-but-du-disagrees-finding-deleted-files-that-still-use-space-24h</link>
      <guid>https://dev.to/devopsdaily/disk-full-but-du-disagrees-finding-deleted-files-that-still-use-space-24h</guid>
      <description>&lt;p&gt;&lt;code&gt;df&lt;/code&gt; says &lt;code&gt;/var&lt;/code&gt; is full. &lt;code&gt;du -sh /var&lt;/code&gt; is hundreds of gigabytes short. Someone has already removed the large log, so searching the directory tree turns up nothing. Do not start deleting packages yet. A process can keep an unlinked file, and every block behind it, alive through an open file descriptor. This is the fastest way to prove that is the problem, release the space without an unplanned restart, and keep log rotation from creating it again.&lt;/p&gt;

&lt;h2&gt;
  
  
  TLDR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;rm&lt;/code&gt; removes a directory entry. If it was the last link but a process still has the file open, Linux releases the blocks only when the final descriptor closes. &lt;a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer"&gt;unlink(2)&lt;/a&gt; describes that lifetime.&lt;/li&gt;
&lt;li&gt;Start with &lt;code&gt;sudo lsof -nP -a +L1 &amp;lt;mountpoint&amp;gt;&lt;/code&gt;. The &lt;code&gt;-a&lt;/code&gt; matters: lsof normally ORs selections, while &lt;code&gt;-a&lt;/code&gt; makes the deleted-file selection and mountpoint selection intersect.&lt;/li&gt;
&lt;li&gt;Prefer restarting the owner or making it reopen logs. Truncating &lt;code&gt;/proc/PID/fd/N&lt;/code&gt; is an emergency action that discards the data, but it can return space immediately.&lt;/li&gt;
&lt;li&gt;If there is no deleted open file, check ext4 reserved blocks, files hidden below a mount point, and inode exhaustion with &lt;code&gt;df -i&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Root or equivalent access to the affected Linux node.&lt;/li&gt;
&lt;li&gt;The filesystem mount point that &lt;code&gt;df&lt;/code&gt; reports as full, not just a directory that happens to be beneath it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsof&lt;/code&gt; if installed. The &lt;code&gt;/proc&lt;/code&gt; fallback below works without it.&lt;/li&gt;
&lt;li&gt;A decision from the service owner before restarting, reloading, or truncating a live writer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why the deleted log still consumes space
&lt;/h2&gt;

&lt;p&gt;On Linux, a pathname is a directory entry that refers to a file's inode. &lt;code&gt;rm&lt;/code&gt; calls &lt;code&gt;unlink(2)&lt;/code&gt;: it removes that name. When that was the last link and no process has the file open, the filesystem can reuse the space. If a process still has it open, the inode and its blocks remain until the last file descriptor closes. That is why &lt;code&gt;du&lt;/code&gt;, which walks named directory entries, cannot count these bytes while &lt;code&gt;df&lt;/code&gt;, which reports filesystem allocation, still can. &lt;a href="https://man7.org/linux/man-pages/man2/unlink.2.html" rel="noopener noreferrer"&gt;The &lt;code&gt;unlink(2)&lt;/code&gt; manual page&lt;/a&gt; states both conditions explicitly.&lt;/p&gt;

&lt;p&gt;The descriptor can outlive the name because it refers through an open file description to the inode, rather than by resolving the old pathname again. Closing the final descriptor frees an unlinked file's occupied space. &lt;a href="https://man7.org/linux/man-pages/man3/close.3p.html" rel="noopener noreferrer"&gt;POSIX &lt;code&gt;close(3p)&lt;/code&gt;&lt;/a&gt; documents that result.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;app.log&lt;/strong&gt; named file, 200 MiB&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;rm&lt;/strong&gt; name removed&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FD 3 still open&lt;/strong&gt; link count is zero&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;close or truncate&lt;/strong&gt; blocks become reusable&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Reproduce the mismatch, then find the holder
&lt;/h2&gt;

&lt;p&gt;Here is a real run on Linux 6.12 with lsof 4.95.0 as packaged by Debian 12. A Python process opened a log for append, wrote 200 MiB on &lt;code&gt;/dev/shm&lt;/code&gt;, and kept descriptor 3 open. The lsof warnings from an unprivileged invocation are trimmed below. The numbers are not a capacity benchmark, &lt;code&gt;/dev/shm&lt;/code&gt; was chosen so this test could not fill a real filesystem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A deleted 200 MiB log remains allocated&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 201M 1.7G 11% /dev/shm
$ du -sh /dev/shm/ddemo
200M    /dev/shm/ddemo
$ rm /dev/shm/ddemo/app.log
$ lsof -nP +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; wc -l
15
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; wc -l
2
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 201M 1.7G 11% /dev/shm
$ du -sh /dev/shm/ddemo
0   /dev/shm/ddemo
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'COMMAND|app.log'&lt;/span&gt;
COMMAND PID &lt;span class="k"&gt;USER&lt;/span&gt; FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1586127 nobody 3w REG 0,24 209715200 0 139344 /dev/shm/ddemo/app.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;
$ stat -L -c &lt;span class="s1"&gt;'%s bytes'&lt;/span&gt; /proc/1586127/fd/3
209715200 bytes
$ : &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /proc/1586127/fd/3
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 140K 1.9G 1% /dev/shm
$ lsof -nP -a +L1 /dev/shm &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'COMMAND|app.log'&lt;/span&gt;
COMMAND PID &lt;span class="k"&gt;USER&lt;/span&gt; FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1586127 nobody 3w REG 0,24 0 0 139344 /dev/shm/ddemo/app.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key line has &lt;code&gt;NLINK&lt;/code&gt; equal to &lt;code&gt;0&lt;/code&gt;, which is what &lt;code&gt;+L1&lt;/code&gt; selects. &lt;code&gt;SIZE/OFF&lt;/code&gt; is 209715200 bytes before the truncate and zero after it. With &lt;code&gt;+L&lt;/code&gt;, lsof exposes the link-count column; &lt;code&gt;+L1&lt;/code&gt; selects open files with a link count below one. &lt;a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer"&gt;Its manual page&lt;/a&gt; documents both behaviours.&lt;/p&gt;

&lt;p&gt;Use this incident command against the mount point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo lsof -nP -a +L1 /var

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-nP&lt;/code&gt; avoids DNS and service-name lookups during an incident. &lt;code&gt;+L1&lt;/code&gt; selects the unlinked open files. &lt;code&gt;/var&lt;/code&gt; is a selection too, which produces an important lsof trap: selection options are normally ORed. &lt;code&gt;lsof -nP +L1 /var&lt;/code&gt; therefore includes every deleted open file anywhere and every open file on &lt;code&gt;/var&lt;/code&gt;. Adding &lt;code&gt;-a&lt;/code&gt; ANDs all selection options, so it returns the intersection. &lt;a href="https://lsof.readthedocs.io/en/stable/manpage/" rel="noopener noreferrer"&gt;lsof documents its default OR behaviour and &lt;code&gt;-a&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Pass a mount point. For a non-mount directory, lsof selects that directory rather than the entire filesystem. If you need a tree search, use lsof's &lt;code&gt;+D&lt;/code&gt; or &lt;code&gt;+d&lt;/code&gt; options instead. The &lt;a href="https://lsof.readthedocs.io/en/stable/tutorial/" rel="noopener noreferrer"&gt;lsof tutorial&lt;/a&gt; shows the mountpoint form, &lt;code&gt;lsof -a +L1 /home&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  When lsof is unavailable
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;/proc/PID/fd&lt;/code&gt; contains one symbolic link for every open descriptor. Search it as root, because access to another process's descriptor links is permission checked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo find /proc/&lt;span class="o"&gt;[&lt;/span&gt;0-9&lt;span class="o"&gt;]&lt;/span&gt;*/fd -lname &lt;span class="s1"&gt;'*(deleted)'&lt;/span&gt; -printf &lt;span class="s1"&gt;'%p -&amp;gt; %l\n'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null

&lt;span class="c"&gt;# Inspect an interesting result. -L follows the /proc descriptor link.&lt;/span&gt;
sudo stat -L -c &lt;span class="s1"&gt;'%s bytes'&lt;/span&gt; /proc/1586127/fd/3

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;a href="https://man7.org/linux/man-pages/man5/proc_pid_fd.5.html" rel="noopener noreferrer"&gt;proc_pid_fd(5) manual page&lt;/a&gt; describes these entries as links to the files a process has open and notes the access check. Expect races: a process can close a descriptor between &lt;code&gt;find&lt;/code&gt; and &lt;code&gt;stat&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Do not treat every deleted-looking target as disk usage. &lt;code&gt;/memfd:... (deleted)&lt;/code&gt; is an anonymous in-memory file, not a file consuming your ext4 or XFS blocks. &lt;a href="https://man7.org/linux/man-pages/man2/memfd_create.2.html" rel="noopener noreferrer"&gt;memfd_create(2)&lt;/a&gt; says it lives in RAM. A &lt;code&gt;(deleted)&lt;/code&gt; executable or shared library after an upgrade is different again: it can be an old program image still mapped by a running process. That is normally a restart-planning signal, not the explanation for a full data filesystem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Release the space without making the incident worse
&lt;/h2&gt;

&lt;p&gt;The clean fix is for the holder to close the file. Restart the affected service, or use its documented reopen operation. If a systemd unit has a reload action, use the service's runbook, for example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo systemctl reload nginx

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For nginx log rotation, rename the log first and then signal the master process to reopen logs. nginx documents &lt;code&gt;USR1&lt;/code&gt; for this operation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Confirm the configured PID file before running this.&lt;/span&gt;
sudo &lt;span class="nb"&gt;kill&lt;/span&gt; -USR1 &lt;span class="s2"&gt;"$(cat /run/nginx.pid)"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After a successful reopen, nginx closes the old logs and workers open the new ones. &lt;a href="https://nginx.org/en/docs/control.html" rel="noopener noreferrer"&gt;nginx's log-rotation documentation&lt;/a&gt; describes that sequence. Use the actual PID file for the installation, not the sample path blindly.&lt;/p&gt;

&lt;p&gt;If the filesystem is full and a controlled restart or reopen is unavailable, truncate the specific deleted descriptor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Last resort: verify PID, FD, path, owner and size first.&lt;/span&gt;
sudo sh -c &lt;span class="s1"&gt;': &amp;gt; /proc/1586127/fd/3'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is what released the 200 MiB in the demo while Python kept running and descriptor 3 remained open. It also destroys the unrecovered log content. Copy it out first if it might be needed, to a different filesystem with confirmed free space. &lt;code&gt;/var/tmp&lt;/code&gt; usually sits on the same full &lt;code&gt;/var&lt;/code&gt;, so it is the wrong target:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# /mnt/recovery: a separately mounted volume with room for the file&lt;/span&gt;
df -h /mnt/recovery
sudo cp -- /proc/1586127/fd/3 /mnt/recovery/recovered-app.log

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Truncation changes the file size but not the open file description's seek offset. &lt;a href="https://man7.org/linux/man-pages/man3/ftruncate.3p.html" rel="noopener noreferrer"&gt;ftruncate(3p)&lt;/a&gt; says the seek pointer is not modified. A writer that does not use &lt;code&gt;O_APPEND&lt;/code&gt; can therefore write again at its old offset, extending the file and leaving a zero-filled hole before the new data. That is a reason to treat truncation as an emergency measure, then repair the application's logging setup.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Do not truncate a descriptor because its name ends in &lt;code&gt;(deleted)&lt;/code&gt; alone. Check the mount point, type, &lt;code&gt;SIZE/OFF&lt;/code&gt;, process identity, and whether its data is needed. A wrong &lt;code&gt;/proc/PID/fd/N&lt;/code&gt; target can discard live application data.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Stop the mismatch from returning
&lt;/h2&gt;

&lt;p&gt;The durable fix is to make every writer close and reopen its logs during rotation. A logrotate rule commonly creates the new file and runs a service-specific post-rotation reload:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;myapp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;daily&lt;/span&gt;
    &lt;span class="n"&gt;rotate&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;
    &lt;span class="n"&gt;compress&lt;/span&gt;
    &lt;span class="n"&gt;delaycompress&lt;/span&gt;
    &lt;span class="n"&gt;missingok&lt;/span&gt;
    &lt;span class="n"&gt;notifempty&lt;/span&gt;
    &lt;span class="n"&gt;create&lt;/span&gt; &lt;span class="mi"&gt;0640&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt;
    &lt;span class="n"&gt;postrotate&lt;/span&gt;
        &lt;span class="n"&gt;systemctl&lt;/span&gt; &lt;span class="n"&gt;reload&lt;/span&gt; &lt;span class="n"&gt;myapp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;
    &lt;span class="n"&gt;endscript&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;delaycompress&lt;/code&gt; leaves the most recently rotated file uncompressed until the next rotation, for programs that keep writing to the previous file for a while before they reopen. &lt;a href="https://man7.org/linux/man-pages/man5/logrotate.conf.5.html" rel="noopener noreferrer"&gt;logrotate's manual page&lt;/a&gt; documents that directive.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;copytruncate&lt;/code&gt; is sometimes needed for a process that cannot reopen a file, but it is a tradeoff. It copies the old file and truncates it in place. The logrotate manual page warns that there is a very small interval between those operations in which log data can be lost. Prefer a reopen signal whenever the software supports one.&lt;/p&gt;

&lt;p&gt;Container logs need caps too. Docker's &lt;code&gt;json-file&lt;/code&gt; driver accepts &lt;code&gt;max-size&lt;/code&gt; and &lt;code&gt;max-file&lt;/code&gt;; its defaults are unlimited size and one file, and changing the daemon setting affects only newly created containers. &lt;a href="https://docs.docker.com/engine/logging/drivers/json-file/" rel="noopener noreferrer"&gt;Docker's &lt;code&gt;json-file&lt;/code&gt; documentation&lt;/a&gt; covers those settings. This example limits a new container to three 10 MiB log files:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  my-image:current

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Docker Engine on servers still defaults to &lt;code&gt;json-file&lt;/code&gt; with no rotation; we measured how that compares with the &lt;code&gt;local&lt;/code&gt; driver that Docker Desktop now uses in &lt;a href="https://dev.to/devopsdaily/docker-desktop-switched-new-containers-to-the-local-log-driver-we-measured-what-that-changes-49m0"&gt;Docker Desktop's switch to the local log driver&lt;/a&gt;. On a host that uses the systemd journal, set a budget instead of relying on disk pressure:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/journald.conf.d/size.conf
&lt;/span&gt;&lt;span class="k"&gt;[Journal]&lt;/span&gt;
&lt;span class="na"&gt;SystemMaxUse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1G&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;SystemMaxUse=&lt;/code&gt; limits persistent journals under &lt;code&gt;/var/log/journal&lt;/code&gt;. The current systemd documentation says the default maximum is 10% of the filesystem, capped at 4G, and only archived files are removed to reduce usage. &lt;code&gt;journalctl --vacuum-size=500M&lt;/code&gt; removes oldest archived journal files until their usage falls below that target. &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html" rel="noopener noreferrer"&gt;journald.conf(5)&lt;/a&gt; and &lt;a href="https://www.freedesktop.org/software/systemd/man/latest/journalctl" rel="noopener noreferrer"&gt;journalctl(1)&lt;/a&gt; document those limits and the active-file caveat.&lt;/p&gt;

&lt;p&gt;Alert before a page becomes a disk-full outage. node_exporter exposes &lt;code&gt;node_filesystem_avail_bytes&lt;/code&gt; as space available to non-root users, and its node-mixin uses &lt;code&gt;predict_linear&lt;/code&gt; for a run-out forecast. &lt;a href="https://github.com/prometheus/node_exporter/blob/master/collector/filesystem_common.go" rel="noopener noreferrer"&gt;The collector source&lt;/a&gt; and &lt;a href="https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/alerts/alerts.libsonnet" rel="noopener noreferrer"&gt;the maintained alert rule&lt;/a&gt; show both names.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;FilesystemPredictedFullSoon&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;predict_linear(&lt;/span&gt;
      &lt;span class="s"&gt;node_filesystem_avail_bytes{mountpoint="/var",fstype!~"tmpfs|overlay"}[6h],&lt;/span&gt;
      &lt;span class="s"&gt;24 * 60 * 60&lt;/span&gt;
    &lt;span class="s"&gt;) &amp;lt; 0&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/var is predicted to run out of space within 24 hours&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Host log rotation limits local damage. For the separate question of how log volume reaches an observability bill, see &lt;a href="https://dev.to/devopsdaily/your-logs-cost-more-than-the-service-that-emits-them-3ij3"&gt;Your Logs Cost More Than the Service That Emits Them&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  If no process is holding a deleted file
&lt;/h2&gt;

&lt;p&gt;Two other mismatches are common.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reserved ext4 blocks.&lt;/strong&gt; ext2, ext3 and ext4 reserve blocks for privileged processes, so &lt;code&gt;df&lt;/code&gt; availability for a normal user is not the same as raw free blocks. &lt;code&gt;mke2fs&lt;/code&gt; defaults that reservation to 5 percent unless configured otherwise. &lt;code&gt;tune2fs -l&lt;/code&gt; only reads the superblock, so it is safe to run against the mounted device:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo tune2fs -l /dev/mapper/vg0-var &lt;span class="o"&gt;|&lt;/span&gt; grep -E &lt;span class="s1"&gt;'Block count|Reserved block count|Block size'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://man7.org/linux/man-pages/man8/mke2fs.8.html" rel="noopener noreferrer"&gt;&lt;code&gt;mke2fs(8)&lt;/code&gt;&lt;/a&gt; documents the 5 percent default, while &lt;a href="https://man7.org/linux/man-pages/man8/tune2fs.8.html" rel="noopener noreferrer"&gt;&lt;code&gt;tune2fs(8)&lt;/code&gt;&lt;/a&gt; documents &lt;code&gt;-l&lt;/code&gt; and adjustment of the reserved-block percentage. Do not lower the reservation on a system filesystem as a reflex. It exists so root-owned services can still write when unprivileged writers cannot.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Files hidden under a mount point.&lt;/strong&gt; If a process writes &lt;code&gt;/var/log/old.log&lt;/code&gt; before a separate volume mounts on &lt;code&gt;/var/log&lt;/code&gt;, that file remains on the underlying root filesystem. After the mount, the old directory contents are invisible by that pathname. &lt;a href="https://man7.org/linux/man-pages/man8/mount.8.html" rel="noopener noreferrer"&gt;mount(8)&lt;/a&gt; says a mount makes the previous contents of the directory invisible while it is mounted.&lt;/p&gt;

&lt;p&gt;Inspect the underlying root through a temporary bind mount. A plain bind of &lt;code&gt;/&lt;/code&gt; does not bring the child mount at &lt;code&gt;/var/log&lt;/code&gt; along, so the root filesystem's hidden &lt;code&gt;/var/log&lt;/code&gt; becomes visible at the alternate path. &lt;a href="https://man7.org/linux/man-pages/man2/mount.2.html" rel="noopener noreferrer"&gt;mount(2)&lt;/a&gt; specifies that a directory bind mount does not include submounts unless it is recursive:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;root_view&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;mktemp -d&lt;span class="o"&gt;)&lt;/span&gt;
sudo mount --bind / &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;
sudo du -xsh &lt;span class="s2"&gt;"$root_view/var/log"&lt;/span&gt;
sudo umount &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;
rmdir &lt;span class="s2"&gt;"$root_view"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Finally, &lt;code&gt;No space left on device&lt;/code&gt; can mean &lt;strong&gt;inodes&lt;/strong&gt; , not blocks. &lt;code&gt;df -i /var&lt;/code&gt; reports inode information instead of block usage, as documented by &lt;a href="https://man7.org/linux/man-pages/man1/df.1.html" rel="noopener noreferrer"&gt;df(1)&lt;/a&gt;. Millions of tiny files can exhaust the inode table while &lt;code&gt;df -h&lt;/code&gt; still reports available bytes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Containers and Kubernetes
&lt;/h2&gt;

&lt;p&gt;A deleted file held open by a container process still consumes storage on the node: the container's writable layer lives under the runtime's directory on the node's disk, and a volume lives wherever it is mounted from. Investigate from the node, not from inside the container, which often has neither &lt;code&gt;lsof&lt;/code&gt; nor permission to see other processes. Find the container's host PID, then read its descriptors through &lt;code&gt;/proc&lt;/code&gt;, which works across namespaces:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# containerd/CRI-O nodes: host PID of the container's main process&lt;/span&gt;
&lt;span class="nv"&gt;container_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'replace-with-container-id'&lt;/span&gt;
&lt;span class="nv"&gt;pid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;sudo crictl inspect &lt;span class="s2"&gt;"$container_id"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; jq -r &lt;span class="s1"&gt;'.info.pid'&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Link targets are shown as paths inside the container&lt;/span&gt;
sudo find &lt;span class="s2"&gt;"/proc/$pid/fd"&lt;/span&gt; -lname &lt;span class="s1"&gt;'*(deleted)'&lt;/span&gt; -printf &lt;span class="s1"&gt;'%p -&amp;gt; %l\n'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The paths in the link targets are relative to the container's mount namespace, so &lt;code&gt;/app/logs/app.log (deleted)&lt;/code&gt; is the container's view. &lt;a href="https://man7.org/linux/man-pages/man7/namespaces.7.html" rel="noopener noreferrer"&gt;Linux namespace documentation&lt;/a&gt; explains why the same file has different paths from the node and from the container.&lt;/p&gt;

&lt;p&gt;Kubernetes has an extra blind spot. When kubelet manages local ephemeral storage, it measures container logs, writable container layers and non-memory &lt;code&gt;emptyDir&lt;/code&gt; volumes. Its directory-scan mode does not track deleted files that remain open, so a container can retain space that the kubelet does not categorize as in use. The same &lt;a href="https://kubernetes.io/docs/concepts/storage/ephemeral-storage/" rel="noopener noreferrer"&gt;Kubernetes local ephemeral storage documentation&lt;/a&gt; says a container can be evicted when its writable layer and log usage exceed its ephemeral-storage limit. Node-level disk pressure is still a reason to inspect the host.&lt;/p&gt;

&lt;p&gt;For the lifecycle rules that apply when the data belongs on a PersistentVolume rather than a node's ephemeral storage, see &lt;a href="https://dev.to/devopsdaily/the-anatomy-of-kubernetes-persistent-storage-pv-pvc-and-the-parts-that-bite-3b6e"&gt;The Anatomy of Kubernetes Persistent Storage&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Incident checklist
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Confirm the affected mount with &lt;code&gt;df -h &amp;lt;mountpoint&amp;gt;&lt;/code&gt; and compare a scoped &lt;code&gt;du&lt;/code&gt; result.&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;sudo lsof -nP -a +L1 &amp;lt;mountpoint&amp;gt;&lt;/code&gt;, and record PID, FD, size, link count and owner before changing anything.&lt;/li&gt;
&lt;li&gt;If lsof is absent, search &lt;code&gt;/proc/[0-9]*/fd&lt;/code&gt; for &lt;code&gt;*(deleted)&lt;/code&gt; as root and inspect candidate sizes with &lt;code&gt;stat -L&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Recover required data to a different filesystem, then restart the service or use its documented log-reopen action.&lt;/li&gt;
&lt;li&gt;Use &lt;code&gt;: &amp;gt; /proc/PID/fd/N&lt;/code&gt; only for a verified emergency target, then confirm &lt;code&gt;df&lt;/code&gt; changes and fix rotation or log limits.&lt;/li&gt;
&lt;li&gt;If no holder exists, check &lt;code&gt;df -i&lt;/code&gt;, ext4 reserved blocks, hidden files below mounts, and node-level container storage.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The important distinction is simple: deleting a pathname is not the same thing as closing the file. Once the incident response follows that distinction, the &lt;code&gt;df&lt;/code&gt; and &lt;code&gt;du&lt;/code&gt; disagreement becomes evidence instead of a mystery.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/disk-full-du-disagrees-deleted-open-files" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>linux</category>
      <category>troubleshooting</category>
      <category>diskspace</category>
      <category>lsof</category>
    </item>
    <item>
      <title>Node.js November 2026 Security Release: How We Tracked and Patched It</title>
      <dc:creator>DevOps Daily</dc:creator>
      <pubDate>Wed, 11 Nov 2026 17:50:00 +0000</pubDate>
      <link>https://dev.to/devopsdaily/nodejs-november-2026-security-release-how-we-tracked-and-patched-it-of</link>
      <guid>https://dev.to/devopsdaily/nodejs-november-2026-security-release-how-we-tracked-and-patched-it-of</guid>
      <description>&lt;p&gt;On 4 November 2026 the Node.js project posted a pre-alert: new releases of the 26.x, 24.x and 22.x lines "on or shortly after, Wednesday, November 11, 2026", and "The highest severity issue fixed in this release is HIGH". No CVE numbers, no components, nothing to triage against. You know a fix is coming, you know it matters, and you do not know when it lands or what it touches.&lt;/p&gt;

&lt;p&gt;That gap is where most patch delays come from. The release ships in the afternoon, nobody is watching, and the upgrade happens days later. This time we wrote a small watcher for the release, saw Node.js 24.22.3 appear at 16:58 UTC on release day, and had production on the new version with smoke tests passing 14 minutes later.&lt;/p&gt;

&lt;p&gt;This post covers what the release fixes and who should move first, the watcher script, the exact nvm, pm2 and systemd steps we used, and the checklist we will reuse for the next one.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fixed versions:&lt;/strong&gt; Node.js &lt;strong&gt;22.23.5&lt;/strong&gt; , &lt;strong&gt;24.22.3&lt;/strong&gt; and &lt;strong&gt;26.13.1&lt;/strong&gt;. &lt;code&gt;https://nodejs.org/dist/index.json&lt;/code&gt; marks all three as security releases.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What was fixed:&lt;/strong&gt; 16 CVEs: 7 high, 6 medium, 3 low. All seven high-severity ones are in HTTP/2 and TLS handling: remote crashes, server memory sent to the peer, and a client-certificate bypass. llhttp, undici, sqlite and brotli were updated too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Move first:&lt;/strong&gt; any Node process that terminates TLS or serves HTTP/2 to untrusted clients. Everyone on 22, 24 or 26 should upgrade. End-of-life lines get no fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch for it:&lt;/strong&gt; poll &lt;code&gt;dist/index.json&lt;/code&gt; every 15 minutes, or subscribe to the &lt;code&gt;nodejs-sec&lt;/code&gt; list and the vulnerability RSS feed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Upgrade order under nvm and pm2:&lt;/strong&gt; install, reinstall pm2, back up &lt;code&gt;dump.pm2&lt;/code&gt;, move apps one at a time with a health check, then move the pm2 daemon by restarting its systemd unit. Do not use &lt;code&gt;pm2 update&lt;/code&gt; for that last step.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A Linux host running Node.js apps installed with &lt;strong&gt;nvm&lt;/strong&gt; and supervised by &lt;strong&gt;pm2&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;A systemd unit for the pm2 daemon (the one &lt;code&gt;pm2 startup&lt;/code&gt; generates)&lt;/li&gt;
&lt;li&gt;Shell access as the app user, plus root for the unit&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;curl&lt;/code&gt; and &lt;code&gt;python3&lt;/code&gt; for the watcher and the checks&lt;/li&gt;
&lt;li&gt;A health endpoint per app and a smoke test you trust&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The announcement and the release
&lt;/h2&gt;

&lt;p&gt;Node.js security releases follow a fixed pattern. About a week ahead, a pre-alert goes up at &lt;a href="https://nodejs.org/en/blog/vulnerability/november-2026-security-releases" rel="noopener noreferrer"&gt;nodejs.org/en/blog/vulnerability&lt;/a&gt; with the target date, the affected lines, and the highest severity. The details stay private until the fixed builds are out, so nobody can work out the bugs from the patch before you can install it.&lt;/p&gt;

&lt;p&gt;For this release:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;4 November:&lt;/strong&gt; pre-alert published. 26.x, 24.x and 22.x, highest severity HIGH.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;11 November, afternoon UTC:&lt;/strong&gt; the builds appeared in &lt;code&gt;dist/index.json&lt;/code&gt; with &lt;code&gt;"security": true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;11 November, 17:11 UTC:&lt;/strong&gt; the advisory page was updated with the full CVE list (the time comes from the nodejs.org repository history).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the binaries were downloadable before the write-up existed. If you wait for the blog post, you start later than you need to.&lt;/p&gt;

&lt;p&gt;You can confirm the fixed versions yourself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://nodejs.org/dist/index.json &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'
import sys, json
for r in json.load(sys.stdin):
    if r["version"] in ("v26.13.1", "v24.22.3", "v22.23.5"):
        print(r["version"], r["date"], "security:", r["security"])'&lt;/span&gt;


v26.13.1 2026-11-10 security: True
v24.22.3 2026-11-10 security: True
v22.23.5 2026-11-11 security: True

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;date&lt;/code&gt; field is the build date, which is why two of them show the day before release.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was fixed
&lt;/h2&gt;

&lt;p&gt;The advisory lists 16 CVEs. Severity and affected lines below are as the advisory states them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;CVE&lt;/th&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Severity&lt;/th&gt;
&lt;th&gt;Lines&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123863&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Any unauthenticated client can abort the server through an assertion in session teardown, over TLS or cleartext&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123864&lt;/td&gt;
&lt;td&gt;HTTPS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A zero-length TLS write re-enters the HTTP parser; memory from other connections, including TLS private keys, can be copied into a request body&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123865&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A protocol error during shutdown leaves the session behind; memory and file descriptors leak with no further cost to the attacker&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123866&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Stream teardown during a backpressured write sends freed memory to the peer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-126051&lt;/td&gt;
&lt;td&gt;TLS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Remote crash in TLS servers that use &lt;code&gt;SNICallback&lt;/code&gt;, &lt;code&gt;OCSPRequest&lt;/code&gt;, &lt;code&gt;newSession&lt;/code&gt;, &lt;code&gt;resumeSession&lt;/code&gt; or &lt;code&gt;server.addContext()&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123872&lt;/td&gt;
&lt;td&gt;TLS&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;TLS 1.2 renegotiation can change the servername and carry client-certificate auth to a stricter host&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123874&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Use-after-free when a &lt;code&gt;respondWithFile()&lt;/code&gt; or &lt;code&gt;respondWithFD()&lt;/code&gt; stream is reset during a pending write&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123828&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;One-byte heap overflow in &lt;code&gt;session.altsvc()&lt;/code&gt; with a long origin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123867&lt;/td&gt;
&lt;td&gt;TLS (mTLS)&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A CA-only SNI context falls back to the default client CA, bypassing per-host mTLS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123868&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;24, 26&lt;/td&gt;
&lt;td&gt;Queued inbound DATA is not charged to &lt;code&gt;maxSessionMemory&lt;/code&gt;, so one stream can hold hundreds of megabytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123871&lt;/td&gt;
&lt;td&gt;HTTP proxy&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Proxy credentials appear in &lt;code&gt;ERR_PROXY_INVALID_CONFIG&lt;/code&gt; error messages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123877&lt;/td&gt;
&lt;td&gt;https.Agent&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A TLS session from the default trust store can be reused by a request with a stricter &lt;code&gt;ca&lt;/code&gt; policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123875&lt;/td&gt;
&lt;td&gt;HTTP/2&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Cancelling a &lt;code&gt;respondWithFile()&lt;/code&gt; download crashes the server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123829&lt;/td&gt;
&lt;td&gt;Permission Model&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;Recursive &lt;code&gt;fs.watch&lt;/code&gt; leaks names of denied files (macOS and Windows)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-123862&lt;/td&gt;
&lt;td&gt;zlib&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;params()&lt;/code&gt; plus &lt;code&gt;destroy()&lt;/code&gt; throws an uncatchable assertion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVE-2026-106071&lt;/td&gt;
&lt;td&gt;Permission Model&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;22, 24, 26&lt;/td&gt;
&lt;td&gt;A wildcard filesystem grant can match paths it should not&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The release notes also update &lt;strong&gt;llhttp&lt;/strong&gt; to 9.4.4 on all three lines, &lt;strong&gt;sqlite&lt;/strong&gt; to 3.54.1, &lt;strong&gt;undici&lt;/strong&gt; to 7.30.2 on 24.x and 8.12.4 on 26.x, and &lt;strong&gt;brotli&lt;/strong&gt; to 1.2.1 on 24.x and 26.x.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The advisory and the release notes do not match exactly on the low-severity Permission Model items. The release notes list CVE-2026-123873 ("permission: check heap snapshot destination") and do not mention CVE-2026-106071. If you run with &lt;code&gt;--permission&lt;/code&gt;, read both. The action is the same either way: upgrade.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Who should upgrade first
&lt;/h2&gt;

&lt;p&gt;Everyone on 22.x, 24.x or 26.x should upgrade. End-of-life lines such as 20.x and 25.x get no fixed release, and the advisory notes that end-of-life versions are always affected. Use the following order to decide what moves first.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Node terminates TLS or speaks HTTP/2 to the internet.&lt;/strong&gt; &lt;code&gt;https.createServer&lt;/code&gt;, &lt;code&gt;http2.createSecureServer&lt;/code&gt;, &lt;code&gt;http2.createServer&lt;/code&gt;, &lt;code&gt;tls.createServer&lt;/code&gt;, or a framework option that enables them. Most of the high-severity bugs sit here, and several need nothing more than a default server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;mTLS with per-host policies.&lt;/strong&gt; CVE-2026-123872 and CVE-2026-123867 are authorization bypasses, not just crashes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Everything else.&lt;/strong&gt; If a reverse proxy terminates TLS and talks plain HTTP/1.1 to Node, the HTTP/2 and TLS server bugs are not reachable from clients. The client-side fixes (&lt;code&gt;https.Agent&lt;/code&gt; session reuse, proxy credentials in error messages, the undici update) still apply.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Two quick checks tell you where you stand. First, which Node processes listen, and on which address:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Run as root so ss can see every process&lt;/span&gt;
ss -ltnpH &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; -r _ _ _ addr _ users&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;pid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$users"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -o &lt;span class="s1"&gt;'pid=[0-9]*'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; head -1 &lt;span class="o"&gt;|&lt;/span&gt; cut -d= -f2&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt;-n &lt;span class="s2"&gt;"$pid"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;
  &lt;span class="nv"&gt;exe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;readlink &lt;span class="s2"&gt;"/proc/$pid/exe"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; */node&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$addr $exe"&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt; &lt;span class="c"&gt;# match the binary, not the process name&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything bound to &lt;code&gt;0.0.0.0&lt;/code&gt;, &lt;code&gt;[::]&lt;/code&gt; or &lt;code&gt;*&lt;/code&gt; is reachable from outside the host unless a firewall says otherwise. Second, which code paths you use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rnE &lt;span class="s1"&gt;'https\.createServer|http2\.create(Secure)?Server|tls\.createServer|respondWith(File|FD)|SNICallback|addContext|altsvc|flushHeaders'&lt;/span&gt; \
  --include=&lt;span class="s1"&gt;'*.js'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.mjs'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.cjs'&lt;/span&gt; --include=&lt;span class="s1"&gt;'*.ts'&lt;/span&gt; src/

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No hits does not make you safe from the client-side fixes, but it does tell you how urgent the rest is.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we watched for the release
&lt;/h2&gt;

&lt;p&gt;The pre-alert gives a date, not a time. We did not want someone refreshing a browser tab all day, and we did not want a cron entry to remember to delete afterwards. So we started one background script on release morning that polls &lt;code&gt;dist/index.json&lt;/code&gt; every 15 minutes, stops after about 16 hours, and exits as soon as a new 24.x shows up.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c"&gt;# node-release-watch.sh: poll nodejs.org until a Node 24 release newer than&lt;/span&gt;
&lt;span class="c"&gt;# the current newest 24.x appears. Every 15 minutes, gives up after ~16 h.&lt;/span&gt;
&lt;span class="nv"&gt;LATEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;v24.22.2 &lt;span class="c"&gt;# the newest 24.x when you start the watcher&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;seq 1 64&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="c"&gt;# index.json is sorted newest first, so the first v24.* entry is the latest 24.x&lt;/span&gt;
  &lt;span class="nv"&gt;v&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;curl -s -m 20 https://nodejs.org/dist/index.json &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'import sys,json; d=json.load(sys.stdin); print(next(x["version"] for x in d if x["version"].startswith("v24.")))'&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;-n &lt;span class="s2"&gt;"$v"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"$v"&lt;/span&gt; != &lt;span class="s2"&gt;"$LATEST"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"NEW NODE 24: $v ($(date -u +%H:%M) UTC)"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 0&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi&lt;/span&gt;
  sleep 900
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"no new Node 24 release after 16 h"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;release day&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# start it in the background and forget about it&lt;/span&gt;
$ chmod +x node-release-watch.sh
$ nohup ./node-release-watch.sh &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; node-watch.log 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;
&lt;span class="c"&gt;# hours later&lt;/span&gt;
$ cat node-watch.log
NEW NODE 24: v24.22.3 &lt;span class="o"&gt;(&lt;/span&gt;16:58 UTC&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few notes if you adapt it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A failed &lt;code&gt;curl&lt;/code&gt; or a bad JSON response leaves &lt;code&gt;v&lt;/code&gt; empty, so a network blip does not trigger a false alert.&lt;/li&gt;
&lt;li&gt;Swap the &lt;code&gt;echo&lt;/code&gt; for a call to your chat webhook if you want a ping instead of a log line.&lt;/li&gt;
&lt;li&gt;To wake only for security releases, add &lt;code&gt;and x["security"]&lt;/code&gt; to the filter and set &lt;code&gt;LATEST&lt;/code&gt; to the newest 24.x security release.&lt;/li&gt;
&lt;li&gt;For the long term, subscribe to the low-volume &lt;code&gt;nodejs-sec&lt;/code&gt; mailing list and the &lt;a href="https://nodejs.org/en/feed/vulnerability.xml" rel="noopener noreferrer"&gt;vulnerability RSS feed&lt;/a&gt;. They tell you a release is coming. The watcher tells you the moment it is downloadable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we upgraded
&lt;/h2&gt;

&lt;p&gt;Our Node apps run on a Linux VM. Node comes from nvm, pm2 supervises the apps, and the pm2 daemon starts at boot from the systemd unit that &lt;code&gt;pm2 startup&lt;/code&gt; generated. The timeline on 11 November, in UTC:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;16:58&lt;/strong&gt; watcher sees v24.22.3&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:00&lt;/strong&gt; nvm install, pm2, dump backup&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:02&lt;/strong&gt; apps moved one by one&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:03&lt;/strong&gt; move the pm2 daemon&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;17:12&lt;/strong&gt; all on 24.22.3, smoke tests pass&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Step 1: install the new version and pm2
&lt;/h3&gt;

&lt;p&gt;As the app user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nvm install 24.22.3
nvm &lt;span class="nb"&gt;alias&lt;/span&gt; default 24.22.3
nvm use 24.22.3
npm install -g pm2@6.0.13 &lt;span class="c"&gt;# same pm2 version you run now; globals are per Node version under nvm&lt;/span&gt;
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.bak-&lt;span class="nv"&gt;$(&lt;/span&gt;date -u +%Y%m%dT%H%M&lt;span class="o"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under nvm, global packages live inside each Node version, so the new Node has no pm2 until you install it. The new &lt;code&gt;pm2&lt;/code&gt; CLI then talks to the old daemon, which is fine for now.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: find out how each entry picks its Node
&lt;/h3&gt;

&lt;p&gt;This is the part that decides whether a restart moves an app. pm2 stores an interpreter per entry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pm2 jlist &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'
import sys, json
for p in json.load(sys.stdin):
    e = p["pm2_env"]
    print(p["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("node_version"), sep="\t")'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bare &lt;code&gt;node&lt;/code&gt;, fork mode:&lt;/strong&gt; resolved through &lt;code&gt;PATH&lt;/code&gt;. A restart with &lt;code&gt;--update-env&lt;/code&gt; picks up the new version.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An absolute path&lt;/strong&gt; such as &lt;code&gt;/home/app/.nvm/versions/node/v24.22.2/bin/node&lt;/code&gt;: written in when the app was started with &lt;code&gt;--interpreter&lt;/code&gt;, or copied from the daemon's own Node. A restart reuses that path, so the entry has to be deleted and started again. If the ecosystem file hard-codes the path, fix the file first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cluster mode&lt;/strong&gt; (&lt;code&gt;exec_mode&lt;/code&gt; is &lt;code&gt;cluster_mode&lt;/code&gt;): the daemon forks the workers itself, so they run on the daemon's Node whatever the entry says. They move when the daemon moves in step 4.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 3: move apps one at a time
&lt;/h3&gt;

&lt;p&gt;One app, one health check, then the next. If a check fails, you stop with one app affected and the rest still serving. Run it as a script, so &lt;code&gt;exit&lt;/code&gt; stops it at the first failure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;health&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="c"&gt;# health &amp;lt;name&amp;gt; &amp;lt;port&amp;gt; [path]: retry up to 30 times until a 200&lt;/span&gt;
  &lt;span class="k"&gt;for&lt;/span&gt; i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nv"&gt;$(&lt;/span&gt;seq 1 30&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"$(curl -s -o /dev/null -w '%{http_code}' -m 5 "&lt;/span&gt;http://127.0.0.1:&lt;span class="nv"&gt;$2${3:-/}&lt;/span&gt;&lt;span class="s2"&gt;")"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; 200&lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 ok"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 0&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
    sleep 2
  &lt;span class="k"&gt;done&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$1 NOT healthy on :$2"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;return&lt;/span&gt; 1
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# Bare "node" interpreter: --update-env passes the new PATH&lt;/span&gt;
pm2 restart web --update-env &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; health web 3000 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1

&lt;span class="c"&gt;# Absolute interpreter path: recreate the entry from the ecosystem file&lt;/span&gt;
pm2 delete api
pm2 start ecosystem.config.cjs --only api &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; health api 3001 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1

pm2 save

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then check what each process actually runs. A 200 only tells you the app is up, not which Node it is on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pm2 jlist &lt;span class="o"&gt;|&lt;/span&gt; python3 -c &lt;span class="s1"&gt;'import sys,json; [print(p["name"], p["pm2_env"].get("node_version"), p["pm2_env"]["status"], sep="\t") for p in json.load(sys.stdin)]'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two minutes in, every app was on 24.22.3 except one. That entry had no interpreter set, and in that case pm2 starts the script with the &lt;strong&gt;daemon's&lt;/strong&gt; Node. Cluster-mode apps behave the same way, since the daemon forks them itself. Restarting it changed nothing, because the daemon was still on the old version. So the daemon had to move too.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: move the pm2 daemon through systemd
&lt;/h3&gt;

&lt;p&gt;The unit that &lt;code&gt;pm2 startup&lt;/code&gt; writes has the Node version in it twice: in &lt;code&gt;Environment=PATH=&lt;/code&gt; and in the path to the pm2 binary. Point both at the new version, as root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;UNIT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/etc/systemd/system/pm2-app.service
cp &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT.bak"&lt;/span&gt;
sed -i &lt;span class="s1"&gt;'s#/node/v[0-9][0-9.]*/#/node/v24.22.3/#g'&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt;
grep -n &lt;span class="s1"&gt;'/node/v'&lt;/span&gt; &lt;span class="s2"&gt;"$UNIT"&lt;/span&gt; &lt;span class="c"&gt;# every match should now say v24.22.3&lt;/span&gt;
systemctl daemon-reload

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then restart the daemon through the unit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# As the app user, with the new pm2 on PATH&lt;/span&gt;
pm2 save
cp ~/.pm2/dump.pm2 ~/.pm2/dump.pm2.pre-daemon
pm2 &lt;span class="nb"&gt;kill&lt;/span&gt; &lt;span class="c"&gt;# no daemon left for resurrect to attach to&lt;/span&gt;

&lt;span class="c"&gt;# As root&lt;/span&gt;
systemctl restart pm2-app &lt;span class="c"&gt;# ExecStart runs "pm2 resurrect" on the new Node&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pm2 kill&lt;/code&gt; stops every app, and &lt;code&gt;pm2 resurrect&lt;/code&gt; starts them all again from the dump, so plan for a short gap. Give it 30 to 60 seconds before you check.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't use &lt;code&gt;pm2 update&lt;/code&gt; for this. Restart the systemd unit instead.&lt;/strong&gt; &lt;code&gt;pm2 update&lt;/code&gt; relaunches the daemon from your shell, on the Node that runs the pm2 CLI there. From a login shell or &lt;code&gt;sudo&lt;/code&gt;, that can be an older version still installed on the box, and the new daemon then runs outside the unit that starts it at boot. The unit has the version written into it and is the same path a reboot takes, so restarting it gives the same result every time. If the daemon ever comes back without your apps: &lt;code&gt;pm2 kill&lt;/code&gt;, copy &lt;code&gt;dump.pm2.pre-daemon&lt;/code&gt; back to &lt;code&gt;~/.pm2/dump.pm2&lt;/code&gt;, and restart the unit.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  Step 5: verify from /proc, then smoke test
&lt;/h3&gt;

&lt;p&gt;pm2's view is a good start. The kernel's view is the one that counts. As root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# The real binary of every Node process on the host, grouped&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; p &lt;span class="k"&gt;in&lt;/span&gt; /proc/&lt;span class="o"&gt;[&lt;/span&gt;0-9&lt;span class="o"&gt;]&lt;/span&gt;*&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nv"&gt;exe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;readlink &lt;span class="s2"&gt;"$p/exe"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;continue&lt;/span&gt;
  &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; */node&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$exe"&lt;/span&gt; &lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; sort &lt;span class="o"&gt;|&lt;/span&gt; uniq -c

&lt;span class="c"&gt;# The daemon itself&lt;/span&gt;
readlink &lt;span class="s2"&gt;"/proc/$(cat /home/app/.pm2/pm2.pid)/exe"&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every line should end in &lt;code&gt;v24.22.3/bin/node&lt;/code&gt;. Then run the external smoke tests against the public URLs, not localhost. Ours passed at 17:12 UTC, 14 minutes after the watcher fired.&lt;/p&gt;

&lt;p&gt;Once the new version has run cleanly for a day, remove the old ones with &lt;code&gt;nvm uninstall&lt;/code&gt;. An old Node left on the box is something a shell, a script or a daemon can fall back to without telling you. Until you remove it, check &lt;code&gt;which node&lt;/code&gt; in every shell you work from, including root's.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rest of the fleet
&lt;/h2&gt;

&lt;p&gt;The production host is the urgent part. The rest follows on its own schedule:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CI:&lt;/strong&gt; &lt;code&gt;actions/setup-node&lt;/code&gt; with &lt;code&gt;node-version: 24&lt;/code&gt; picks up 24.22.3 without workflow changes once the runner's tool cache has it. If you want it today rather than when the runner image refreshes, pin &lt;code&gt;node-version: '24.22.3'&lt;/code&gt;. &lt;code&gt;check-latest: true&lt;/code&gt; helps too, but the version manifest it reads can lag nodejs.org.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Container images:&lt;/strong&gt; they need a rebuild. At the time of writing, Docker Hub had no &lt;code&gt;24.22.3-slim&lt;/code&gt; tag yet and &lt;code&gt;node:24-slim&lt;/code&gt; was last updated on 5 November, so a rebuild right now would still get the old Node. Check before you rebuild, then use &lt;code&gt;docker build --pull&lt;/code&gt; and confirm with &lt;code&gt;docker run --rm --pull always node:24-slim node -v&lt;/code&gt;.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://hub.docker.com/v2/repositories/library/node/tags/24.22.3-slim &lt;span class="o"&gt;|&lt;/span&gt;
  python3 -c &lt;span class="s1"&gt;'import sys,json; print(json.load(sys.stdin).get("last_updated", "not published yet"))'&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dev machines:&lt;/strong&gt; last, since they do not serve untrusted traffic. &lt;code&gt;nvm install 24.22.3 &amp;amp;&amp;amp; nvm alias default 24.22.3&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a full audit of every place Node hides on a host (cron jobs, helper scripts, child processes, systemd units with old paths), see our &lt;a href="https://dev.to/devopsdaily/nodejs-october-2026-security-release-finding-every-node-you-actually-run-1ag4"&gt;October 2026 post on finding every Node you run&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for the next Node.js security release
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;When the pre-alert lands&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Note the date, the affected lines and the highest severity.&lt;/li&gt;
&lt;li&gt;List every host, image and pipeline that runs those lines.&lt;/li&gt;
&lt;li&gt;Run the exposure checks above to decide the order.&lt;/li&gt;
&lt;li&gt;Check each pm2 entry's interpreter, and fix ecosystem files that hard-code a Node path.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;On release day&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Start the watcher in the morning, and confirm the fixed versions and &lt;code&gt;"security": true&lt;/code&gt; in &lt;code&gt;dist/index.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;nvm install&lt;/code&gt; the new version, &lt;code&gt;nvm alias default&lt;/code&gt;, reinstall pm2 under it.&lt;/li&gt;
&lt;li&gt;Back up &lt;code&gt;~/.pm2/dump.pm2&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Move apps one at a time: &lt;code&gt;restart --update-env&lt;/code&gt; for bare &lt;code&gt;node&lt;/code&gt;, delete and start for absolute paths. Health check each one, then &lt;code&gt;pm2 save&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;node_version&lt;/code&gt; in &lt;code&gt;pm2 jlist&lt;/code&gt; for every app.&lt;/li&gt;
&lt;li&gt;Point the systemd unit at the new version, &lt;code&gt;daemon-reload&lt;/code&gt;, &lt;code&gt;pm2 kill&lt;/code&gt;, &lt;code&gt;systemctl restart&lt;/code&gt;. Never &lt;code&gt;pm2 update&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Verify from &lt;code&gt;/proc&lt;/code&gt;, then run external smoke tests.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;After&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Rebuild container images once the new tag is on Docker Hub, with &lt;code&gt;--pull&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Make sure CI runs the new version (pin the exact version on release day).&lt;/li&gt;
&lt;li&gt;Update dev machines.&lt;/li&gt;
&lt;li&gt;After a day on the new version, &lt;code&gt;nvm uninstall&lt;/code&gt; the old ones so nothing can fall back to them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The watcher is what made this fast: it cost nothing to run, and we started within minutes of the release instead of hours. The per-app health checks made every step safe to stop. Keep the checklist somewhere you will find it next time.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://devops-daily.com/posts/node-november-2026-security-release-track-and-patch" rel="noopener noreferrer"&gt;devops-daily.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>node</category>
      <category>cve</category>
      <category>pm2</category>
    </item>
  </channel>
</rss>
