<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Span Observer</title>
    <description>The latest articles on DEV Community by Span Observer (@spanobserver).</description>
    <link>https://dev.to/spanobserver</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4230759%2Fec482140-e2ca-44fb-b097-403595d07ccf.png</url>
      <title>DEV Community: Span Observer</title>
      <link>https://dev.to/spanobserver</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/spanobserver"/>
    <language>en</language>
    <item>
      <title>Model Context Protocol (MCP) explained: host, client, server, tools and transports</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Thu, 19 Nov 2026 07:30:30 +0000</pubDate>
      <link>https://dev.to/spanobserver/model-context-protocol-mcp-explained-host-client-server-tools-and-transports-1egc</link>
      <guid>https://dev.to/spanobserver/model-context-protocol-mcp-explained-host-client-server-tools-and-transports-1egc</guid>
      <description>&lt;p&gt;The Model Context Protocol (MCP) is an open standard that defines how an AI application connects to external tools and data sources. Instead of writing a specific integration for every «&amp;nbsp;assistant × system&amp;nbsp;» pair, you expose your capabilities once behind an MCP server, and any compatible client can use them: a coding assistant, a conversational agent or an internal tool.&lt;/p&gt;

&lt;p&gt;Published by Anthropic in late 2024 and then adopted by many vendors and open source projects, MCP quickly became the gateway between language models and the rest of the IT estate. For an operations team, this is a very concrete question: tomorrow, agents will read your metrics, open your runbooks and suggest actions. Better to understand the protocol before plugging it into production.&lt;/p&gt;

&lt;p&gt;This article is the entry point to our MCP series. It explains the architecture, the three primitives a server exposes and the transports, then what all this changes for an SRE or a platform engineer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is the Model Context Protocol, in practice?
&lt;/h2&gt;

&lt;p&gt;MCP is a message exchange protocol based on JSON-RPC 2.0. It says nothing about the model used or how it reasons: it only standardises the conversation between the application hosting the model and the services that provide it with context or the ability to act. It is often compared to a «&amp;nbsp;USB-C port for AI&amp;nbsp;»: a single connector, interchangeable peripherals.&lt;/p&gt;

&lt;p&gt;The specification is versioned by date and published at &lt;a href="https://modelcontextprotocol.io/" rel="noopener noreferrer"&gt;modelcontextprotocol.io&lt;/a&gt;, with official SDKs in several languages. It evolves regularly: always check which revision your clients and servers support.&lt;/p&gt;

&lt;h2&gt;
  
  
  Host, client, server: who does what?
&lt;/h2&gt;

&lt;p&gt;MCP vocabulary distinguishes three roles, and mixing them up is the first source of design mistakes.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;The host&lt;/strong&gt; is the application the user interacts with: a code editor, a chat app, an in-house agent. It holds the model, manages user consent and decides which servers are allowed.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;The client&lt;/strong&gt; is a component of the host. It maintains a connection with &lt;em&gt;one&lt;/em&gt; given server. A host connected to three servers therefore runs three clients.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;The server&lt;/strong&gt; is a program, often lightweight, that exposes capabilities: querying an observability platform, reading a knowledge base, creating a ticket.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This separation has an important consequence: the server never sees the full conversation. It receives specific requests and returns results. It is the host that assembles everything and presents it to the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three primitives of an MCP server
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Tools
&lt;/h3&gt;

&lt;p&gt;A tool is a function the model can decide to call: it has a name, a natural-language description and a JSON schema for its arguments. The client discovers them with &lt;code&gt;tools/list&lt;/code&gt; and invokes them with &lt;code&gt;tools/call&lt;/code&gt;. Tools are what give an agent its ability to act, and therefore most of the risk.&lt;/p&gt;

&lt;h3&gt;
  
  
  Resources
&lt;/h3&gt;

&lt;p&gt;A resource is a piece of data identified by a URI: a file, a runbook, a database schema, a documentation extract. It is mainly driven by the application, which chooses to add it to the context. The corresponding methods are &lt;code&gt;resources/list&lt;/code&gt; and &lt;code&gt;resources/read&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Prompts
&lt;/h3&gt;

&lt;p&gt;A prompt is a reusable message template, usually triggered explicitly by the user, for example an «&amp;nbsp;analyse the current incident&amp;nbsp;» command that prepares the right questions. It is discovered with &lt;code&gt;prompts/list&lt;/code&gt; and retrieved with &lt;code&gt;prompts/get&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The specification also describes client-side capabilities, such as sampling (the server asks the host’s model for a generation) or requesting information from the user. Support for them varies between hosts, and some change from one revision to the next: check their status in the revision you are targeting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Transports: stdio or HTTP?
&lt;/h2&gt;

&lt;p&gt;MCP defines two standard transports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;stdio&lt;/strong&gt;: the host launches the server as a local process and exchanges messages over standard input and output. It is simple, network-free and ideal on a developer workstation. The trade-off is that standard output is reserved for the protocol: any logging must go to standard error.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;HTTP («&amp;nbsp;Streamable HTTP&amp;nbsp;»)&lt;/strong&gt;: the server is a remote service reachable over HTTP, able to stream responses. It replaces the older HTTP + SSE transport from the first versions. It is the natural choice for a server shared by a team, with authentication that the specification bases on OAuth.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In practice, stdio suits individual use and HTTP suits shared servers, deployed and monitored like any other microservice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why MCP matters to ops teams
&lt;/h2&gt;

&lt;p&gt;For a platform team, MCP brings three tangible benefits. First, a &lt;strong&gt;single integration&lt;/strong&gt;: one server for your internal tool serves every assistant in the company. Second, a &lt;strong&gt;control point&lt;/strong&gt;: every tool call goes through a declared interface that you can restrict, log and trace. Finally, &lt;strong&gt;real production context&lt;/strong&gt;: a coding assistant that queries your observability platform reasons on facts, not assumptions, as we saw with the &lt;a href="https://span.observer/en/dynatrace-cursor-plugin-mcp/" rel="noopener noreferrer"&gt;Dynatrace plugin for Cursor&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The flip side is symmetrical: a badly configured MCP server gives a model, and indirectly anyone who can influence it, access to your systems. Security and observability are not optional.&lt;/p&gt;

&lt;h2&gt;
  
  
  In practice
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1 — Choose a read-only first server
&lt;/h3&gt;

&lt;p&gt;Start with an existing server that only reads, for example the one for your observability platform. The &lt;a href="https://span.observer/en/dynatrace-mcp-server-symptom-fix/" rel="noopener noreferrer"&gt;Dynatrace MCP server in a coding assistant&lt;/a&gt; is a good textbook case: you go from a symptom to a fix without changing anything in production.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2 — Declare the server in the host
&lt;/h3&gt;

&lt;p&gt;Most hosts accept a JSON configuration that describes how to launch or reach each server. The exact format depends on the host; it often looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"runbooks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"python"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"args"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"serveur_runbooks.py"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"env"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"RUNBOOKS_DIR"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"/opt/runbooks"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 3 — Observe the exchange
&lt;/h3&gt;

&lt;p&gt;To understand what flows between them, inspect the messages. A tool call is a simple JSON-RPC request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"jsonrpc"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"method"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"tools/call"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"params"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"get_runbook"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"arguments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"service"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"paiement"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The project provides a debugging tool, MCP Inspector, for exploring a server and replaying its tools: check the official documentation for how to launch it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4 — Write your own server
&lt;/h3&gt;

&lt;p&gt;Once you understand the protocol, exposing your own procedures becomes straightforward. Our guide to &lt;a href="https://span.observer/en/build-mcp-server-python/" rel="noopener noreferrer"&gt;building an MCP server in Python&lt;/a&gt; starts from a concrete case: making operations runbooks searchable by an assistant.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5 — Secure and trace before widening access
&lt;/h3&gt;

&lt;p&gt;Before opening access to other teams, apply the rules described in &lt;a href="https://span.observer/en/secure-mcp-server/" rel="noopener noreferrer"&gt;securing an MCP server&lt;/a&gt; and instrument the calls as explained in &lt;a href="https://span.observer/en/monitor-mcp-agents-opentelemetry/" rel="noopener noreferrer"&gt;observing MCP agents with OpenTelemetry&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Start read-only and add write tools only with explicit human approval.&lt;/li&gt;
&lt;li&gt;  Write precise tool descriptions: the model chooses its tools based on them.&lt;/li&gt;
&lt;li&gt;  Limit the number of tools per server; too much choice degrades selection.&lt;/li&gt;
&lt;li&gt;  Use stdio locally, authenticated HTTP for any shared server.&lt;/li&gt;
&lt;li&gt;  Pin the versions of third-party servers and review their updates.&lt;/li&gt;
&lt;li&gt;  Log every tool call with its caller, arguments and result.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is MCP tied to a particular model or vendor?
&lt;/h3&gt;

&lt;p&gt;No. The protocol is open and model-independent. It was started by Anthropic, but many hosts, vendors and open source projects implement it.&lt;/p&gt;

&lt;h3&gt;
  
  
  What is the difference between MCP and a classic REST API?
&lt;/h3&gt;

&lt;p&gt;MCP does not replace your APIs: an MCP server often wraps them. It adds dynamic discovery of capabilities and descriptions designed so that a model knows when and how to use them.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I prefer stdio or HTTP?
&lt;/h3&gt;

&lt;p&gt;stdio for local, personal use; HTTP for a shared server that needs to be authenticated, scaled and monitored.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The Model Context Protocol provides a common language between AI assistants and your systems. For ops teams, it opens up an interesting prospect, agents that work on real data, provided you treat every server as a production component: narrow scope, controlled access, traced calls. The satellite articles in this series detail each of these steps.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC or the projects mentioned. The trademarks mentioned belong to their respective owners. Tools evolve quickly: check commands and options in their official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/model-context-protocol-explained/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/model-context-protocol-mcp-explique/" rel="noopener noreferrer"&gt;https://span.observer/fr/model-context-protocol-mcp-explique/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>aiagents</category>
      <category>api</category>
      <category>dynatracemcp</category>
    </item>
    <item>
      <title>Dynatrace log ingestion: detecting silence, coverage gaps and excess volume</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 21:30:33 +0000</pubDate>
      <link>https://dev.to/spanobserver/dynatrace-log-ingestion-detecting-silence-coverage-gaps-and-excess-volume-4352</link>
      <guid>https://dev.to/spanobserver/dynatrace-log-ingestion-detecting-silence-coverage-gaps-and-excess-volume-4352</guid>
      <description>&lt;p&gt;A log ingestion pipeline almost never fails noisily. It goes quiet. An API token expires, a file changes format after an update, an overly broad ingest rule suddenly sucks in production debug logs. In all these cases nothing flashes: you discover the gap when you need the data, in the middle of an investigation, or at the end of the month when you read the consumption bill. Dynatrace log ingestion deserves the same monitoring as any other production service.&lt;/p&gt;

&lt;p&gt;Dynatrace SaaS has improved visibility into this pipeline: an enriched ready-made dashboard, self-monitoring events emitted by the OneAgent log module, log module configuration grouped in the Settings app, and more complete sizing guides for the Environment ActiveGate. Taken separately, these are incremental improvements. Together, they finally let you treat log ingestion as a service in its own right, with its own monitoring.&lt;/p&gt;

&lt;p&gt;Rather than reviewing the new features one by one, this article starts from the three most common failures and shows how to detect them, then suggests a checking routine to put in place.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three ways to lose logs without knowing it
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Silence
&lt;/h3&gt;

&lt;p&gt;A source stops sending: unsupported encoding, a file compressed or renamed by rotation, a misconfigured time zone, a missing timestamp pattern. Historically, you had to inspect source by source to understand. The OneAgent log module now emits self-monitoring events (SFM) that describe source status and health signals. They are written to Grail, in &lt;code&gt;dt.system.events&lt;/code&gt;, and each event type is documented with its meaning and a suggested fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  Partial coverage
&lt;/h3&gt;

&lt;p&gt;Here nothing has stopped: the source was simply never captured. A new host group, a recently added Kubernetes cluster, an application deployed without anyone creating the matching ingest rule. The difficulty is that this absence produces no signal. Only a coverage view makes it visible.&lt;/p&gt;

&lt;h3&gt;
  
  
  Excess
&lt;/h3&gt;

&lt;p&gt;The opposite problem: a rule whose matcher is too permissive, a log level left on debug, an error loop flooding a file. Volume climbs, consumption follows, and the noise makes the useful logs harder to read.&lt;/p&gt;

&lt;h2&gt;
  
  
  The tools to monitor Dynatrace log ingestion
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The Log ingest overview dashboard
&lt;/h3&gt;

&lt;p&gt;This ready-made dashboard is the entry point. It now shows a health view of the ingestion components, for both the OneAgent log modules and the Environment ActiveGate, as well as a view of the top log producers. It highlights the components that need attention, which saves you building a query from scratch at the worst possible moment.&lt;/p&gt;

&lt;h3&gt;
  
  
  Log module configuration in Settings
&lt;/h3&gt;

&lt;p&gt;Ingest rules, custom sources, masking and timestamp patterns are now brought together under &lt;em&gt;Settings &amp;gt; Collect and capture &amp;gt; Log monitoring &amp;gt; Configure log module&lt;/em&gt;. The Sources tab opens on an autodiscovery summary: number of host groups, Kubernetes clusters, hosts and sources detected, with the percentage covered by a rule. Another welcome addition for anyone who has inherited a complex configuration: active rules and rules inherited from a higher level (environment or host group) are shown together, with a direct link to their definition. You can also ask Dynatrace Assist about coverage in natural language.&lt;/p&gt;

&lt;h3&gt;
  
  
  ActiveGate sizing guides
&lt;/h3&gt;

&lt;p&gt;For logs sent by API (JSON or OTLP) to the Environment ActiveGate, the official documentation provides sizing guides covering containerised and host-based deployments, on Linux and Windows, with figures extended to large environments.&lt;/p&gt;

&lt;h2&gt;
  
  
  In practice
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1 — Make the dashboard a ritual
&lt;/h3&gt;

&lt;p&gt;Open Log ingest overview in the Dashboards app and add it to the platform team’s favourites. A weekly glance at the component health section and the top producers is enough to spot most drifts before they become incidents.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2 — Explore the self-monitoring events
&lt;/h3&gt;

&lt;p&gt;When a component appears degraded, drill down to the events. An exploratory query in a &lt;a href="https://span.observer/en/dynatrace-notebooks-dashboards-dql/" rel="noopener noreferrer"&gt;Notebook&lt;/a&gt; gives you an overview; then refine the filters based on the fields you see in your environment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fetch dt.system.events, from:now()-24h
| filter event.provider == "Log Module"
| summarize nb = count(), by:{event.type}
| sort nb desc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then open a representative event: it shows the probable cause, for example an invalid time zone setting on a Kubernetes log module. The documentation page for the event type gives the fix to apply.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3 — Measure coverage
&lt;/h3&gt;

&lt;p&gt;In the Sources tab of the log module configuration, sort by coverage rate. A poorly covered host group is almost always the explanation for «&amp;nbsp;we’re missing logs&amp;nbsp;». From the same row, examine the entity configuration and create the missing rule without leaving the page. While you are there, check the inherited rules: an exclusion defined at environment level can cancel out a local inclusion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4 — Track down excess volume
&lt;/h3&gt;

&lt;p&gt;When the top producers view reveals an unexpected source, open it in the Logs app (&lt;a href="https://span.observer/en/log-pattern-analysis-dql/" rel="noopener noreferrer"&gt;log pattern analysis&lt;/a&gt; helps you understand what it produces) and look at the breakdown by severity level. A query like this gives the same information in a reproducible way:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fetch logs, from:now()-24h
| summarize volume = count(), by:{log.source, loglevel}
| sort volume desc
| limit 15
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a production source is dominated by debug messages, look at recent changes to the ingest rules: a matcher meant for a development environment that also covers production is a classic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5 — Automate detection
&lt;/h3&gt;

&lt;p&gt;Once the cause is fixed, create a log metric on the critical sources and attach a Dynatrace Intelligence alert on volume drops and spikes. Since self-monitoring events can also trigger &lt;a href="https://span.observer/en/dynatrace-workflows-automation/" rel="noopener noreferrer"&gt;Workflows&lt;/a&gt;, you can automatically open a ticket or notify the owning team as soon as a source goes into error.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6 — Size the ActiveGate before a migration
&lt;/h3&gt;

&lt;p&gt;If you are bringing logs into Dynatrace from another platform via OTLP, the calculation has two stages: first establish the base ActiveGate configuration for routing OneAgent traffic according to cluster size, then add the replicas needed for the expected OTLP log throughput, based on the per-replica capacity given in the documentation. Measure the source’s actual throughput before the cut-over rather than relying on an estimate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Monitor the monitoring&lt;/strong&gt;: treat ingestion health as a platform SLO, with an owner and alerts.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Narrow the matchers&lt;/strong&gt;: every ingest rule should explicitly target its scope (environment, namespace, host group).&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Review coverage after every infrastructure addition&lt;/strong&gt;: a new cluster or a new host group means a new check of the Sources tab.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Plan for token rotation&lt;/strong&gt;: silent expiry is one of the most mundane causes of losing logs sent by API.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Document inherited rules&lt;/strong&gt;: the unified view helps, but an explicit name and description remain essential.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Availability
&lt;/h2&gt;

&lt;p&gt;The Log ingest overview dashboard and the OneAgent log module self-monitoring events are available to Dynatrace SaaS customers using Grail and the latest version of the platform, with no extra configuration. Customers still on Log Monitoring Classic need to enable these events explicitly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Why does a log source stop arriving without any alert?
&lt;/h3&gt;

&lt;p&gt;The causes are often mundane: an expired API token, a file renamed or compressed by rotation, an unsupported encoding, a misconfigured time zone or timestamp pattern. The self-monitoring events from the OneAgent log module, stored in &lt;code&gt;dt.system.events&lt;/code&gt;, describe these situations and point to the fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you check that every source is covered?
&lt;/h3&gt;

&lt;p&gt;In the Sources tab of the log module configuration, sort by coverage rate: a poorly covered host group almost always explains missing logs. Also check inherited rules, which can cancel out a local inclusion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Are these improvements available in Log Monitoring Classic?
&lt;/h3&gt;

&lt;p&gt;The dashboard and self-monitoring events are available with no configuration for SaaS customers on Grail and the latest platform version. In Log Monitoring Classic, the events must be enabled explicitly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Most log losses are not spectacular outages but quiet drifts. By combining the ingestion dashboard, the self-monitoring events stored in Grail, the log module coverage view and volume alerts, a platform team can move from a reactive mode, where gaps are discovered mid-crisis, to one where ingestion is monitored and fixed like any other service.&lt;/p&gt;

&lt;p&gt;Further reading: &lt;a href="https://www.dynatrace.com/news/blog/modern-log-management-resilience-and-self-service-upgrades-for-log-ingestion/" rel="noopener noreferrer"&gt;Modern log management just got simpler: Resilience and self-service upgrades for log ingestion&lt;/a&gt; (Dynatrace blog).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/dynatrace-log-ingestion-monitoring/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/ingestion-logs-resilience-self-service/" rel="noopener noreferrer"&gt;https://span.observer/fr/ingestion-logs-resilience-self-service/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>activegate</category>
      <category>dashboards</category>
      <category>dql</category>
      <category>dynatracesaas</category>
    </item>
    <item>
      <title>Linnix Linux observability: eBPF and AI for incidents</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 19:30:27 +0000</pubDate>
      <link>https://dev.to/spanobserver/linnix-linux-observability-ebpf-and-ai-for-incidents-2lm0</link>
      <guid>https://dev.to/spanobserver/linnix-linux-observability-ebpf-and-ai-for-incidents-2lm0</guid>
      <description>&lt;p&gt;A Linux node slows down, &lt;code&gt;top&lt;/code&gt; shows 80% CPU, and nobody knows which process is getting in the way of the others. That is where Linnix starts, and it is why Linnix Linux observability deserves a look: the project tries to find who causes the stalls, not just who consumes the most. It relies on eBPF, on the kernel’s pressure counters (PSI) and, optionally, on a language model that explains incidents. The repository is &lt;a href="https://github.com/linnix-os/linnix" rel="noopener noreferrer"&gt;&lt;code&gt;linnix-os/linnix&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I start from four operating decisions rather than a tour of features: which signal to trust, what you accept to automate, which licence you sign up to, and how to watch the agent itself.&lt;/p&gt;

&lt;p&gt;A note on honesty: span.observer has not installed or run Linnix. The facts come from the README, the documentation and the files in the repository, read on 11 November 2026. I could not read the star count at the source, so I do not quote it. The latest commit I read is dated 19 September 2026, and &lt;code&gt;cognitod/Cargo.toml&lt;/code&gt; shows version 0.3.0. This is a young project: treat it as a candidate to test, not as a reference.&lt;/p&gt;

&lt;h2&gt;
  
  
  Linnix Linux observability: from CPU use to lost time
&lt;/h2&gt;

&lt;p&gt;A container burning 100% CPU may lose nothing. Another one at 40% may wait for its turn half the time. The README builds on that idea: pressure (PSI) measures real waiting, while usage only measures activity. Its example: a pod at 40% CPU and 60% pressure is worse off than a pod at 100% CPU and 5% pressure.&lt;/p&gt;

&lt;p&gt;The project describes three layers. The eBPF collector watches &lt;code&gt;fork&lt;/code&gt;, &lt;code&gt;exec&lt;/code&gt;, &lt;code&gt;exit&lt;/code&gt; and scheduler events. The &lt;code&gt;cognitod&lt;/code&gt; daemon aggregates those signals with the process tree. An optional analysis layer writes the explanation. The diagram below is the one in the repository, under &lt;code&gt;docs/images&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbtw4aqgojniaxmab7vkc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbtw4aqgojniaxmab7vkc.png" alt="Linnix architecture diagram: eBPF probes in the kernel, the cognitod daemon in user space with a rules engine and an HTTP API, then a CLI, an AI Reasoner and Prometheus Grafana" width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/linnix-os/linnix/blob/main/docs/images/architecture.png" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One thing to notice: the diagram says «&amp;nbsp;Apache 2.0 license&amp;nbsp;» at the bottom, while the rest of the repository talks about the AGPL. I come back to that mismatch in the licence section.&lt;/p&gt;

&lt;h2&gt;
  
  
  First decision: which signal do you trust when a pod slows down?
&lt;/h2&gt;

&lt;p&gt;The headline feature is attribution: who took time from whom. According to the README, Linnix detects noisy neighbours, fork storms, PSI saturation and memory leaks. Its detection table gives thresholds: a «&amp;nbsp;circuit breaker&amp;nbsp;» when pressure exceeds 40% with more than 90% CPU, and a fork storm above 10 forks per second for 2 seconds.&lt;/p&gt;

&lt;p&gt;Attribution comes out as Prometheus metrics. Two matter to an SRE:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;code&gt;linnix_pod_psi_pressure_total&lt;/code&gt;: the pressure suffered, in microseconds, per victim pod;&lt;/li&gt;
&lt;li&gt;  &lt;code&gt;linnix_stall_induced_seconds_total&lt;/code&gt;: the stall time blamed on an offender pod, labelled by offender and victim pair.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The README also ships ready-made PromQL, for instance the query that ranks the offenders of one victim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sum by (offender_pod) (rate(linnix_stall_induced_seconds_total{victim_pod="payment-api"}[5m]))
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the fact, the &lt;code&gt;linnix-cli investigate&lt;/code&gt; command summarises that data. The example below comes from the README: it was not run here.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv475uu4qwa7z4ajn47ek.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv475uu4qwa7z4ajn47ek.png" alt="Sample output of linnix-cli investigate from the README: the payment-api pod loses 2.6 s, 76% attributed to media/image-resizer and 24% to batch/etl-runner" width="800" height="386"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/linnix-os/linnix/blob/main/README.md" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The project stresses a useful nuance: this is attribution, not proven causality. If nothing contended, the tool says so instead of naming a suspect. To confirm, change one thing and see whether the stall drops. That caution matches a wider habit of sorting signals; see &lt;a href="https://span.observer/en/ai-agent-guardrails-production/" rel="noopener noreferrer"&gt;https://span.observer/en/ai-agent-guardrails-production/&lt;/a&gt; for the same logic applied to AI agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Second decision: what do you let the AI do?
&lt;/h2&gt;

&lt;p&gt;This is the part the title promises. In the sample &lt;code&gt;configs/linnix.toml&lt;/code&gt;, the &lt;code&gt;[reasoner]&lt;/code&gt; section is enabled and points to a local OpenAI-compatible server, with the &lt;code&gt;linnix-3b-distilled&lt;/code&gt; model. Yet the README says the analysis model is optional and off by default in the quickstart; &lt;code&gt;./quickstart.sh --with-ai&lt;/code&gt; downloads a 2.1 GB model.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1vwk9wb4hrj2x2trmzmq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1vwk9wb4hrj2x2trmzmq.png" alt="Excerpt of configs/linnix.toml: API on 127.0.0.1:3000, reasoner section enabled towards a local endpoint, Prometheus metrics on 127.0.0.1:9464" width="800" height="470"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/linnix-os/linnix/blob/main/configs/linnix.toml" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two safeguards stand out in the documentation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;code&gt;SAFETY.md&lt;/code&gt; announces a «&amp;nbsp;monitor-first&amp;nbsp;» mode: the project detects and proposes, but kills or throttles nothing without explicit configuration;&lt;/li&gt;
&lt;li&gt;  according to the FAQ, you can bring your own model through an OpenAI-compatible endpoint, or empty the endpoint to keep only rules and raw events.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The CHANGELOG adds a reassuring detail: the analyser must cite numbered facts, and a hypothesis that cites a fact it was never given is discarded. The model’s stated confidence is stored as its own estimate, not a calibrated figure. My reading: start without the AI, check that attribution holds, then add the plain-language summary. Important decisions stay with rules and humans.&lt;/p&gt;

&lt;p&gt;The repository also offers an MCP server, &lt;code&gt;linnix-cli mcp serve&lt;/code&gt;, so an assistant can query what the daemon observed. For isolating such a server, see &lt;a href="https://span.observer/en/mcp-stdio-server-command-execution/" rel="noopener noreferrer"&gt;https://span.observer/en/mcp-stdio-server-command-execution/&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Third decision: can you use it at work?
&lt;/h2&gt;

&lt;p&gt;The LICENSE file is the AGPL-3.0 text, and &lt;code&gt;Cargo.toml&lt;/code&gt; declares &lt;code&gt;AGPL-3.0-or-later&lt;/code&gt; for the agent. The README splits the components: the &lt;code&gt;cognitod&lt;/code&gt; agent under AGPL-3.0, the eBPF programs under «&amp;nbsp;GPL-2.0 or MIT&amp;nbsp;», because the kernel requires a GPL-compatible licence at load time. Yet the &lt;code&gt;Cargo.toml&lt;/code&gt; files of those programs say &lt;code&gt;Apache-2.0&lt;/code&gt;, and so does the diagram. Three sources, three answers: ask the project to settle it before you embed that code.&lt;/p&gt;

&lt;p&gt;What does the AGPL change for you? The project’s licence FAQ says that installing and running Linnix internally, unmodified, is not a problem. The duty to share source targets whoever distributes the software, or offers a modified service to third parties. The project also sells a commercial licence for teams whose legal department rejects the AGPL.&lt;/p&gt;

&lt;p&gt;This is not legal advice. Your counsel should read the text, especially if you modify the code, embed it in a product or expose it to customers. One practical point: if you patch &lt;code&gt;cognitod&lt;/code&gt; for internal use, keep the patches in a repository whose content you could publish.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting started and checking it works
&lt;/h2&gt;

&lt;p&gt;The README offers two routes. On Kubernetes, a DaemonSet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;kubectl create secret generic linnix-api-token \
  --namespace default \
  --from-literal=token="$(openssl rand -base64 32)"
kubectl apply -f k8s/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Locally, with Docker:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;git clone https://github.com/linnix-os/linnix.git &amp;amp;&amp;amp; cd linnix
./quickstart.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt; Check the kernel version: the FAQ sets the minimum at Linux 5.12 on x86_64 and 5.18 on arm64.&lt;/li&gt;
&lt;li&gt; Check that &lt;code&gt;/sys/kernel/btf/vmlinux&lt;/code&gt; exists: without BTF, telemetry is degraded.&lt;/li&gt;
&lt;li&gt; Query &lt;code&gt;/readyz&lt;/code&gt;: it returns 503 if the eBPF probes are not attached.&lt;/li&gt;
&lt;li&gt; Import &lt;code&gt;k8s/grafana/linnix-noisy-neighbor.json&lt;/code&gt; into Grafana for the «&amp;nbsp;who blocks whom&amp;nbsp;» view.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Be careful: the README says the DaemonSet currently runs in privileged mode «&amp;nbsp;for simplicity&amp;nbsp;». &lt;code&gt;SAFETY.md&lt;/code&gt;, for its part, talks about &lt;code&gt;CAP_BPF&lt;/code&gt; and &lt;code&gt;CAP_PERFMON&lt;/code&gt; without &lt;code&gt;privileged: true&lt;/code&gt;. The two texts do not line up: check what your security policy accepts. The sample configuration file requires an API token (the &lt;code&gt;LINNIX_API_TOKEN&lt;/code&gt; variable) when the API listens beyond loopback. Never store it in a repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching the watcher
&lt;/h2&gt;

&lt;p&gt;An agent that sees everything must be watched too. The FAQ explains that if eBPF cannot attach, &lt;code&gt;cognitod&lt;/code&gt; keeps serving the pressure read from &lt;code&gt;/proc/pressure&lt;/code&gt;: you keep the symptom but lose the culprit. To avoid finding that out during an incident, alert on &lt;code&gt;linnix_kernel_instrumentation_active&lt;/code&gt; (1 when probes are attached, 0 otherwise), exposed on &lt;code&gt;/metrics/prometheus&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;On cost, two project figures coexist. The README claims under 1% CPU for the eBPF collector. &lt;code&gt;OVERHEAD.md&lt;/code&gt; measures the userspace daemon, with BPF disabled, between 3.63% and 4.10% of one core depending on the scenario. The link to &lt;code&gt;performance-proof.md&lt;/code&gt;, which should back the kernel-side figure, points to a file missing from the cloned repository. So I do not repeat «&amp;nbsp;under 1%&amp;nbsp;» as a measured fact: test on your own nodes. To compare with other eBPF approaches, see &lt;a href="https://span.observer/en/ebpf-kubernetes-cilium-hubble-network-flows/" rel="noopener noreferrer"&gt;https://span.observer/en/ebpf-kubernetes-cilium-hubble-network-flows/&lt;/a&gt; and &lt;a href="https://span.observer/en/ebpf-without-sidecar-service-mesh/" rel="noopener noreferrer"&gt;https://span.observer/en/ebpf-without-sidecar-service-mesh/&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits to know
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Recent project: version 0.3.0 in the code, no releases page read, latest commit read dated September 2026.&lt;/li&gt;
&lt;li&gt;  Documentation sometimes contradicts itself (licence, privileges), as seen above.&lt;/li&gt;
&lt;li&gt;  The DaemonSet is privileged; the agent reads the activity of every process on the node.&lt;/li&gt;
&lt;li&gt;  The README says analysis stays local unless you configure Slack. I did not audit the code to confirm it.&lt;/li&gt;
&lt;li&gt;  No independent performance figure: all of them come from the project.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Good practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Test on a non-production node, with a recent kernel and BTF.&lt;/li&gt;
&lt;li&gt;  Stay in monitor mode: no automatic action before weeks of observation.&lt;/li&gt;
&lt;li&gt;  Alert on the instrumentation state, not only on incidents.&lt;/li&gt;
&lt;li&gt;  Have your lawyer read the licence before deployment.&lt;/li&gt;
&lt;li&gt;  Complement attribution with your traces and logs: it points at a neighbour, not at a proven cause.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is Linnix open source?
&lt;/h3&gt;

&lt;p&gt;The agent is under AGPL-3.0, an open source licence with network copyleft. The eBPF programs are described as GPL-2.0 or MIT, with Apache-2.0 mentions elsewhere: check before reusing that code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do you need a language model to use it?
&lt;/h3&gt;

&lt;p&gt;No, according to the README and the FAQ. Stall attribution works without a model; the AI adds plain-language summaries.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does Linnix replace Prometheus or an APM tool?
&lt;/h3&gt;

&lt;p&gt;No. It exports its metrics to Prometheus and Grafana. It answers «&amp;nbsp;who gets in whose way on this node&amp;nbsp;», not «&amp;nbsp;which request is slow&amp;nbsp;».&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Linnix asks a good question: who causes the stall, not who consumes. Test it without the AI first, measure its cost on your nodes, settle the licence, then decide. The promise is attractive and the project is young. For another project in the «&amp;nbsp;Top repos&amp;nbsp;» series, see &lt;a href="https://span.observer/en/gatus-status-page-yaml-alerts/" rel="noopener noreferrer"&gt;https://span.observer/en/gatus-status-page-yaml-alerts/&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This blog is independent and not affiliated with Dynatrace LLC. Linnix is a third-party project, not published by Dynatrace.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/linnix-linux-observability-ebpf/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/linnix-observabilite-linux-ebpf/" rel="noopener noreferrer"&gt;https://span.observer/fr/linnix-observabilite-linux-ebpf/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>ebpf</category>
      <category>kubernetes</category>
      <category>linnix</category>
    </item>
    <item>
      <title>OpenSRE SRE agent: AI wired into your observability tools</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 17:30:23 +0000</pubDate>
      <link>https://dev.to/spanobserver/opensre-sre-agent-ai-wired-into-your-observability-tools-54kl</link>
      <guid>https://dev.to/spanobserver/opensre-sre-agent-ai-wired-into-your-observability-tools-54kl</guid>
      <description>&lt;p&gt;Should an AI agent be allowed to read your production logs and push a fix? That is the real question behind the &lt;strong&gt;OpenSRE SRE agent&lt;/strong&gt;: the &lt;code&gt;Tracer-Cloud/opensre&lt;/code&gt; repository, licensed Apache 2.0, showed about 12,800 stars on 11 November 2026, and its latest commit was dated the same day. The project is clearly active, yet it calls itself a «&amp;nbsp;public alpha&amp;nbsp;».&lt;/p&gt;

&lt;p&gt;This article starts from the operating decision rather than the feature list: what do you grant the agent, what may it read, and how do you monitor the agent in turn? The facts come from the README, the LICENSE file and the repository documentation, read on 11 November 2026.&lt;/p&gt;

&lt;p&gt;A note on honesty: span.observer has not installed OpenSRE. The screenshots below come from the project’s repository (Apache 2.0). For the wider picture of what SRE agents promise, see &lt;a href="https://span.observer/en/sre-agent-aiops-platform-claims/" rel="noopener noreferrer"&gt;our look at AIOps claims&lt;/a&gt;; for the Dynatrace side, &lt;a href="https://span.observer/en/autonomous-sre-agents-dynatrace-announced-vs-delivered/" rel="noopener noreferrer"&gt;the article on autonomous SRE agents&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The OpenSRE SRE agent: what the repository says
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Positioning&lt;/strong&gt;: the README title calls it an «&amp;nbsp;open-source agent for CI/CD reliability&amp;nbsp;». It reads a failed run, traces the cause and pushes a fix for you to review. The same README also describes alerts and plain-language questions about production.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Licence&lt;/strong&gt;: Apache 2.0 (the repository’s LICENSE file).&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Maturity&lt;/strong&gt;: a banner says «&amp;nbsp;Public Alpha&amp;nbsp;», with core workflows «&amp;nbsp;usable for early exploration, though not yet fully stable&amp;nbsp;». No release was displayed when I read the page.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Models&lt;/strong&gt;: self-hosted, with your own models (Anthropic, OpenAI, Ollama, Gemini, OpenRouter, Bedrock and others in the README list). A hosted model tied to an OpenSRE account is also offered at first launch.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Integrations&lt;/strong&gt;: «&amp;nbsp;60+ tools&amp;nbsp;», including Grafana, Datadog, CloudWatch, Sentry, Splunk, Kubernetes, PagerDuty, Slack and the MCP protocol.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I found no mention of Dynatrace in the repository. A connection might go through MCP, but I did not verify it, so do not assume it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The operating decision: what scope to grant
&lt;/h2&gt;

&lt;p&gt;Before installing anything, three choices set the risk.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Where the model runs.&lt;/strong&gt; Production logs go to an external provider, or stay in-house with Ollama. The README mentions optional masking of sensitive identifiers before model calls: turn it on, then check it against a set of fake logs.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;What it reads.&lt;/strong&gt; One read-only token per source. The Grafana documentation asks for a service account token «&amp;nbsp;with read access&amp;nbsp;».&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;What it writes.&lt;/strong&gt; Pushing to a pull request branch is an action a human reviews. Running remediation actions is «&amp;nbsp;optional&amp;nbsp;» according to the README: leave it off at the start.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It is the same reasoning as for &lt;a href="https://span.observer/en/devin-oncall-dynatrace-guardrails/" rel="noopener noreferrer"&gt;an on-call agent wired to Dynatrace&lt;/a&gt;: negotiate the scope before the first demo, not after an incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick tour and installation
&lt;/h2&gt;

&lt;p&gt;The README gives a one-line install that needs no sudo and fetches the latest build of the &lt;code&gt;main&lt;/code&gt; branch. On macOS or Linux:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -fsSL https://install.opensre.com | bash -s -- -gh
opensre
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Windows the README provides a PowerShell equivalent. Piping a remote script into bash calls for care: read &lt;code&gt;install.sh&lt;/code&gt; in the repository before running it, and try it on a disposable machine.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Febwmkx600h57grqy1mit.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Febwmkx600h57grqy1mit.png" alt="Screenshot from the OpenSRE README: the CLI welcome screen with the version number, the Skills and CI/CD fixes counters, and the Sign in or create account prompt" width="799" height="327"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/Tracer-Cloud/opensre" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The screenshot shows the welcome screen: on first run, the interactive shell asks you to sign in, create an account, or stay signed out. The README states that the shell only opens for an active account.&lt;/p&gt;

&lt;h2&gt;
  
  
  Connecting an observability source
&lt;/h2&gt;

&lt;p&gt;Integrations are set up with &lt;code&gt;opensre integrations setup&lt;/code&gt;. For Grafana, the documentation offers four routes: the interactive wizard, environment variables, a persistent store or, later, the hosted web app, which is marked «&amp;nbsp;coming soon&amp;nbsp;».&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fstp0y3mwdyv8eu1tb7ri.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fstp0y3mwdyv8eu1tb7ri.png" alt="Shortened excerpt from the OpenSRE Grafana documentation: the GRAFANA\_INSTANCE\_URL, GRAFANA\_READ\_TOKEN, GRAFANA\_VERIFY\_SSL and GRAFANA\_CA\_BUNDLE variables with example values" width="799" height="197"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/Tracer-Cloud/opensre" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two details matter for security. First, &lt;code&gt;GRAFANA_VERIFY_SSL=false&lt;/code&gt; is for labs only; for an internal instance the documentation recommends supplying a certificate authority bundle. Second, the token in the screenshot is a placeholder: keep yours in a secrets manager, never in a repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Asking the agent from a script
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;opensre ask&lt;/code&gt; command runs a single agent turn, prints the answer and exits. Each run is saved as a resumable session unless you pass &lt;code&gt;--ephemeral&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frne2qa4b9v278cwk4379.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frne2qa4b9v278cwk4379.png" alt="Excerpt from the OpenSRE headless documentation: the opensre ask command, Session and Continue lines with an example identifier, then the --ephemeral variant" width="800" height="219"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Source : &lt;a href="https://github.com/Tracer-Cloud/opensre" rel="noopener noreferrer"&gt;github.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If the agent needs a decision, it exits with the status &lt;code&gt;needs_input&lt;/code&gt; and offers numbered options. In a script or a CI job that is a useful stopping point: a human stays in the loop. JSON output and stdin input are described on the «&amp;nbsp;Headless CLI&amp;nbsp;» page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Monitoring the agent itself
&lt;/h2&gt;

&lt;p&gt;An agent that reads your production becomes a production component. According to the documentation, four things can be measured or configured:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Project telemetry&lt;/strong&gt;: product analytics and Sentry error reports are on by default (opt-out). &lt;code&gt;export OPENSRE_NO_TELEMETRY=1&lt;/code&gt; turns them off.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Prompt log&lt;/strong&gt;: &lt;code&gt;OPENSRE_PROMPT_LOG_DISABLED=1&lt;/code&gt; disables it, including for ephemeral requests.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Cost and sessions&lt;/strong&gt;: the shell offers &lt;code&gt;/cost&lt;/code&gt; and &lt;code&gt;/sessions&lt;/code&gt; commands. I did not check what they display.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Silent failures&lt;/strong&gt;: a confident but wrong answer raises no alert. &lt;a href="https://span.observer/en/ai-agent-silent-failures-detect/" rel="noopener noreferrer"&gt;Detecting silent AI agent failures&lt;/a&gt; covers what to measure.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The repository also contains a triage pilot for the OpenTelemetry demo payment service: OpenSRE reads metrics, traces and logs stored in ClickHouse, not the Collector. The distinction is useful: the agent consults a history, and replaces neither collection nor storage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits and open questions
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Alpha&lt;/strong&gt;: APIs and integrations may change, says the README.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Main target&lt;/strong&gt;: the current positioning is CI/CD reliability; production incident investigation is there, but I have not measured its quality.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;No quoted performance figures&lt;/strong&gt;: the project mentions end-to-end tests in &lt;code&gt;tests/e2e&lt;/code&gt;, not a reproducible comparison I can report.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Data&lt;/strong&gt;: without masking, your logs travel to the model you chose.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;An active repository is a moving one&lt;/strong&gt;: the README notes the removal of older «&amp;nbsp;graph and chain&amp;nbsp;» layers. Documentation read today may age fast.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Start read-only, on a test environment and a single source.&lt;/li&gt;
&lt;li&gt;  Keep automatic remediation off until you have reviewed ten investigations.&lt;/li&gt;
&lt;li&gt;  Compare each conclusion from the agent with your own dashboards, as with &lt;a href="https://span.observer/en/twenty-crm-open-source-self-hosting/" rel="noopener noreferrer"&gt;any young open source project&lt;/a&gt;: the repository is not a guarantee.&lt;/li&gt;
&lt;li&gt;  Decide who receives the Slack or PagerDuty summaries and who answers them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Is OpenSRE free and open source?
&lt;/h3&gt;

&lt;p&gt;The code is under the Apache 2.0 licence. The models you call (paid APIs) and the optional hosted model of an OpenSRE account have their own terms, which I did not read.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can it replace a human on-call rota?
&lt;/h3&gt;

&lt;p&gt;Nothing in the README claims so: the agent offers an evidence-backed answer and a fix to review. The decision stays human, and the alpha status calls for caution.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does it work with Dynatrace?
&lt;/h3&gt;

&lt;p&gt;Dynatrace is not in the README’s integration list. MCP support leaves a door open that I did not verify. To compare, see &lt;a href="https://span.observer/en/autonomous-sre-agents-dynatrace-announced-vs-delivered/" rel="noopener noreferrer"&gt;the announcement of Dynatrace’s SRE agents&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;OpenSRE deserves a supervised trial if you want a self-hosted agent wired to observability tools you already run. Set the scope first, read-only access and a model of your choice, then measure the agent as you would any service. Repository and documentation: &lt;a href="https://github.com/Tracer-Cloud/opensre" rel="noopener noreferrer"&gt;Tracer-Cloud/opensre&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC or Tracer. Trademarks belong to their owners. Tools change quickly: check commands and options against the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/opensre-sre-agent-open-source/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/opensre-agent-sre-open-source/" rel="noopener noreferrer"&gt;https://span.observer/fr/opensre-agent-sre-open-source/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aiagents</category>
      <category>datadog</category>
      <category>grafana</category>
      <category>incident</category>
    </item>
    <item>
      <title>Secure Amazon Bedrock access: who calls which model, under which identity?</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 15:30:34 +0000</pubDate>
      <link>https://dev.to/spanobserver/secure-amazon-bedrock-access-who-calls-which-model-under-which-identity-j5m</link>
      <guid>https://dev.to/spanobserver/secure-amazon-bedrock-access-who-calls-which-model-under-which-identity-j5m</guid>
      <description>&lt;p&gt;On a Thursday the Bedrock bill doubles and nobody can say which team called which model. The AWS logs show one shared role, not a person or an application. That scenario is the simplest test of secure Amazon Bedrock access: can you answer, after the fact, «&amp;nbsp;who called what, under which identity, and at what cost&amp;nbsp;»?&lt;/p&gt;

&lt;p&gt;In November 2025 AWS and Dynatrace published a reference pattern for this: a gateway placed in front of Bedrock. Dynatrace’s post explains the idea; the technical detail lives in an AWS code sample under the MIT-0 licence. Dynatrace presents the pattern as coming from its own generative AI workloads (&lt;a href="https://www.dynatrace.com/news/blog/aws-publishes-dynatrace-developed-blueprint-for-secure-amazon-bedrock-access-at-scale/" rel="noopener noreferrer"&gt;vendor post&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;This article does not tour the product. It follows an operations decision: what should your team check before ten teams call Bedrock through the same door? It is separate from our piece on &lt;a href="https://span.observer/en/vulnerable-ai-services-dynatrace/" rel="noopener noreferrer"&gt;vulnerable AI services&lt;/a&gt;, which is about software components, not identity.&lt;/p&gt;

&lt;p&gt;A note on honesty: span.observer has not deployed this pattern. The facts come from the Dynatrace post, the AWS architecture post (only partly readable) and the repository’s CloudFormation file, read as published. The identity model and the Dynatrace-side tracking remain for you to verify.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why secure Amazon Bedrock access goes through a single door
&lt;/h2&gt;

&lt;p&gt;Without an intermediary, each application signs its own requests with AWS credentials. Three problems follow, which the Dynatrace post sums up: models that anyone holding credentials can invoke, audit evidence that is hard to produce, and developers who handle request signing themselves.&lt;/p&gt;

&lt;p&gt;The gateway moves the question. The client presents a token from your identity system; the gateway checks it, signs the request on its behalf, then forwards it to Bedrock. The client keeps the usual AWS SDK (Boto3 in the sample). According to the AWS sample, you disable client-side signing and add an &lt;code&gt;aws-endpoint-prefix&lt;/code&gt; header, plus an optional bearer token.&lt;/p&gt;

&lt;p&gt;The post also cites quotas and throttling, per-tenant usage tracking, and no extra code when Bedrock adds a feature. The repository does not describe how those are implemented: usage plans appear only as suggested enhancements. Treat them as goals, not shipped features.&lt;/p&gt;

&lt;h2&gt;
  
  
  The path of a call, from client to model
&lt;/h2&gt;

&lt;p&gt;The path has four stages. Each is a decision point and a place to leave a trace.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Amazon API Gateway&lt;/strong&gt; receives the call. Authorisation, throttling and release management apply here.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;A Lambda authorizer&lt;/strong&gt; decides whether the token is valid. Dynatrace validates JWTs there; you can swap in your own logic or an Amazon Cognito user pool.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;An integration Lambda&lt;/strong&gt; copies the request, signs it with AWS Signature Version 4 and sends it to the right Bedrock endpoint.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Amazon Bedrock&lt;/strong&gt; runs the model.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;An optional Route 53 piece gives the gateway its own domain name. The integration Lambda does not know Bedrock’s APIs one by one: it passes requests through. That avoids rewriting the gateway for every new feature, but it also leaves the gateway blind to content.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the AWS template does not do for you yet
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;bedrock-llm-gateway.yaml&lt;/code&gt; file is a starting point, not a finished solution. Three details found in the file (read on 11 November 2026) deserve a decision before production.&lt;/p&gt;

&lt;p&gt;First, one parameter opens or closes the door.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffjg7cvnef72rqgjwtu70.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffjg7cvnef72rqgjwtu70.png" alt="Excerpt of the AWS sample's CloudFormation file: EndpointType and EnableAuthorizer parameters, and the rule requiring an authorizer for a REGIONAL endpoint" width="800" height="637"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The default is a &lt;code&gt;PRIVATE&lt;/code&gt; endpoint (reachable from a VPC) with the authorizer on. A rule rejects a &lt;code&gt;REGIONAL&lt;/code&gt; deployment without an authorizer. However, &lt;code&gt;EnableAuthorizer: false&lt;/code&gt; remains possible on a private endpoint: the repository reserves it for testing, but nothing stops someone forgetting it.&lt;/p&gt;

&lt;p&gt;Second, identity is not supplied. The shipped authorizer is a stub that denies everything.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyfagr7nktxxxec5ckrd4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyfagr7nktxxxec5ckrd4.png" alt="Excerpt of the AWS sample's CloudFormation file: authorizer definition with a 300-second cache, and a placeholder Lambda function that denies every request with a fixed principalId" width="800" height="532"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two lines matter for traceability. The authorizer result is cached for 300 seconds, so a revoked token can still pass for five minutes. And the stub hard-codes the principal as &lt;code&gt;'user'&lt;/code&gt;: if you keep it and only replace the token test, all your callers will look like one. Return the real identity (team, application or user) in &lt;code&gt;principalId&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Third, the integration Lambda’s permissions are broad.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F21x3bztdxajosotan12n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F21x3bztdxajosotan12n.png" alt="Excerpt of the AWS sample's CloudFormation file: IAM role of the integration Lambda with the managed policies AWSLambdaBasicExecutionRole and AmazonBedrockFullAccess" width="800" height="407"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The role gets the managed policy &lt;code&gt;AmazonBedrockFullAccess&lt;/code&gt;. The repository itself recommends least privilege. Since every call goes through this role, Bedrock sees the role, not the caller: restrict the allowed models, and do not rely on Bedrock’s own logs alone to tell your teams apart.&lt;/p&gt;

&lt;p&gt;One last point: the file we read configures neither access logs nor tracing on the API stage. Tracking is yours to build.&lt;/p&gt;

&lt;h2&gt;
  
  
  Connecting the gateway to observability
&lt;/h2&gt;

&lt;p&gt;Dynatrace’s post says that observability of Bedrock workloads gives continuous insight into performance, reliability and cost, and helps confirm that controls stay effective. It does not say how to wire it, and we have not tested it. What follows is the author’s proposal, not from the post.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Enable access logs&lt;/strong&gt; on the &lt;code&gt;v1&lt;/code&gt; stage and record the authorizer principal, the request ID and the status code.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Route those logs&lt;/strong&gt; to Dynatrace with the AWS integration of your choice (see &lt;a href="https://span.observer/en/dynatrace-aws-integration-cloudwatch/" rel="noopener noreferrer"&gt;CloudWatch metrics and logs in Dynatrace&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Set an alert&lt;/strong&gt; on two simple signals: the authorizer’s denial rate and the call volume per principal.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Tie tokens to cost&lt;/strong&gt; per application, as in &lt;a href="https://span.observer/en/ai-agent-cost-tokens-opentelemetry/" rel="noopener noreferrer"&gt;the cost of an AI agent with OpenTelemetry&lt;/a&gt;; the role of a general-purpose LLM gateway is compared in &lt;a href="https://span.observer/en/llm-gateway-cost-tracking-litellm-helicone/" rel="noopener noreferrer"&gt;our article on LiteLLM and Helicone&lt;/a&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;From an SRE viewpoint, a gateway adds a link: give it its own availability and latency SLO, separate from Bedrock’s, otherwise an authorizer outage will look like a model outage. The 300-second timeout on the integration Lambda, read in the file, suggests long responses fit; the AWS post also mentions streaming, which we did not verify end to end.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Replace the authorizer before any other setting, and test an expired token and a token from another team.&lt;/li&gt;
&lt;li&gt;  Keep &lt;code&gt;EndpointType: PRIVATE&lt;/code&gt; as long as no client outside the VPC needs access.&lt;/li&gt;
&lt;li&gt;  Replace &lt;code&gt;AmazonBedrockFullAccess&lt;/code&gt; with a policy that lists the models and regions you need.&lt;/li&gt;
&lt;li&gt;  Never put a token or key in an access log; log an identifier, not the secret.&lt;/li&gt;
&lt;li&gt;  Shorten the authorizer cache if fast revocation matters to you.&lt;/li&gt;
&lt;li&gt;  For encryption with a key you manage, see &lt;a href="https://span.observer/en/byok-aws-dynatrace-customer-managed-key/" rel="noopener noreferrer"&gt;BYOK encryption between AWS and Dynatrace&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do you need a gateway to use Bedrock in a company?
&lt;/h3&gt;

&lt;p&gt;No, but it becomes useful when several teams share one account and you must tell callers apart, cap usage and prove who did what.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is the AWS template ready for production?
&lt;/h3&gt;

&lt;p&gt;Not as it stands. The repository says so itself: the authorizer denies everything by default and least privilege must be applied. It is a sample, with little activity on the repository when we read it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does this pattern apply to Dynatrace Managed or SaaS?
&lt;/h3&gt;

&lt;p&gt;The gateway is an AWS component, independent of the Dynatrace edition. The post mentions Dynatrace observability without distinguishing Managed from SaaS; check your edition’s documentation for how to ingest AWS logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Secure Amazon Bedrock access is proven by its logs: first put a real identity in the authorizer, cut the role’s permissions, then log the stage. The &lt;a href="https://github.com/aws-samples/sample-ai-gateway-for-amazon-bedrock" rel="noopener noreferrer"&gt;sample-ai-gateway-for-amazon-bedrock&lt;/a&gt; repository gives you the skeleton; the rest is your operating decision.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC; Amazon Bedrock is an Amazon Web Services service. Details change: check the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/secure-amazon-bedrock-access/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/acces-amazon-bedrock-securise/" rel="noopener noreferrer"&gt;https://span.observer/fr/acces-amazon-bedrock-securise/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>aiobservability</category>
      <category>amazonbedrock</category>
      <category>apigateway</category>
      <category>aws</category>
    </item>
    <item>
      <title>Dynatrace App Toolkit: build your first app step by step</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 13:30:19 +0000</pubDate>
      <link>https://dev.to/spanobserver/dynatrace-app-toolkit-build-your-first-app-step-by-step-1m9d</link>
      <guid>https://dev.to/spanobserver/dynatrace-app-toolkit-build-your-first-app-step-by-step-1m9d</guid>
      <description>&lt;p&gt;Your team needs a view that no standard screen provides: one that joins teams, services and a home-grown DQL query. The operational decision is easy to state. Either you keep a dashboard and screenshots pasted into a wiki, or you build a small app that lives inside your environment. The &lt;strong&gt;Dynatrace App Toolkit&lt;/strong&gt; is the command-line tool that makes the second option affordable. This article takes it the opposite way round from the documentation: first what to decide before writing code, then the minimal loop that gets an app running.&lt;/p&gt;

&lt;p&gt;It applies to &lt;strong&gt;Dynatrace SaaS&lt;/strong&gt; only (addresses ending in &lt;code&gt;apps.dynatrace.com&lt;/code&gt;); Managed is not covered. span.observer did not deploy an app to write this piece: commands and options come from the developer documentation and the vendor’s post, linked below, and no output is made up.&lt;/p&gt;

&lt;p&gt;The vendor’s post presents App Toolkit 1.0 as a single toolchain: create, develop and deploy, add app functions and update packages, all from one binary, &lt;code&gt;dt-app&lt;/code&gt;. Keep that idea in mind; the rest follows from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dynatrace App Toolkit: do you really need an app?
&lt;/h2&gt;

&lt;p&gt;Before opening a terminal, put three questions to the team. They save you from maintaining code for nothing.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Would a dashboard do?&lt;/strong&gt; If the view is a set of DQL tiles, stay with a dashboard: no code, no release cycle.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Who maintains the app?&lt;/strong&gt; An app has a version, dependencies and an owner. Name that person before the first line of code, as for any service.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;What data will it read?&lt;/strong&gt; The app’s &lt;code&gt;scopes&lt;/code&gt; are declared in its configuration: the fewer you ask for, the shorter the security review.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the answer leans towards an app (custom interface, navigation, presentation logic), move on to the setup. For the queries behind your future pages, the article on &lt;a href="https://span.observer/en/advanced-dql-lookup-join-parse/" rel="noopener noreferrer"&gt;advanced DQL techniques&lt;/a&gt; is a good base.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisites to check before running the command
&lt;/h2&gt;

&lt;p&gt;The App Toolkit page lists precise requirements. Here they are as read in the documentation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Node.js 24&lt;/strong&gt; installed locally (check with &lt;code&gt;node --version&lt;/code&gt;);&lt;/li&gt;
&lt;li&gt;  your environment address, in the form &lt;code&gt;https://{environment-id}.apps.dynatrace.com/&lt;/code&gt;, which you can copy from the browser address bar;&lt;/li&gt;
&lt;li&gt;  network access to &lt;code&gt;dt-cdn.net&lt;/code&gt;, &lt;code&gt;registry.npmjs.org&lt;/code&gt;, your environment and &lt;code&gt;github.com&lt;/code&gt; (for templates);&lt;/li&gt;
&lt;li&gt;  free ports: 3000 to 3005 for the dev server, 30000 for local execution of app functions, 5343 for SSO authentication (the toolkit tries the next port if it is busy).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On a corporate workstation the network line is the one that fails first: ask for the four hosts to be opened before losing an hour on an installation error.&lt;/p&gt;

&lt;h2&gt;
  
  
  Creating the app: the minimal loop
&lt;/h2&gt;

&lt;p&gt;The «&amp;nbsp;First app in 5 minutes&amp;nbsp;» tutorial gives this path. These are the steps in the order I would run them.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; In your editor’s terminal, run &lt;code&gt;npx dt-app@latest create&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt; Name the app, for example &lt;code&gt;my-first-app&lt;/code&gt;, then enter the environment address (the command may prefill it if you are already signed in).&lt;/li&gt;
&lt;li&gt; Move into the folder: &lt;code&gt;cd my-first-app&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt; Start the local server: &lt;code&gt;npx dt-app dev&lt;/code&gt;. The app opens in the browser.&lt;/li&gt;
&lt;li&gt; Open &lt;code&gt;ui/app/pages/Home.tsx&lt;/code&gt;, change the text of the &lt;code&gt;Heading&lt;/code&gt; element to «&amp;nbsp;My first app&amp;nbsp;» and save: the browser refreshes on its own.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcqobj50sqzeyr0y1npnz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcqobj50sqzeyr0y1npnz.png" alt="Terminal listing the dt-app commands: node --version, create, cd, dev, deploy --dry-run and deploy" width="800" height="239"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This image is an assembly of commands taken from the documentation, not a recorded session. Mind the browser detail: the documentation says Chrome 142 or later asks for permission to connect to devices on your local network; if the page stays blank, look for that prompt first.&lt;/p&gt;

&lt;p&gt;What the vendor’s post stresses is the speed of the loop. It reports rebuilds dropping from about 2.7 seconds to 0.05 seconds thanks to change detection (only modified components are rebuilt and checked), and incremental type checking that only revisits changed files and their dependencies. Those figures are the vendor’s; span.observer has not measured them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading the generated folder and the configuration
&lt;/h2&gt;

&lt;p&gt;Version 1.x of the toolkit separates the interface from app functions. The workspace described in the documentation looks like this:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp86bhzoureayluc75kro.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp86bhzoureayluc75kro.png" alt="Layout of an App Toolkit 1.x workspace: ui folder with main.tsx and tsconfig.json, documents folder, app.config.json file and .dt-app/logs folder" width="799" height="260"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The file to know is &lt;code&gt;app.config.json&lt;/code&gt; (or &lt;code&gt;app.config.ts&lt;/code&gt;). At the top level, &lt;code&gt;environmentUrl&lt;/code&gt; is required. Under &lt;code&gt;app&lt;/code&gt;, the documentation requires &lt;code&gt;id&lt;/code&gt; (lowercase, dot-separated namespace), &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;version&lt;/code&gt;, &lt;code&gt;description&lt;/code&gt; and &lt;code&gt;scopes&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzbw3zowg8yhq0vgsu7zc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzbw3zowg8yhq0vgsu7zc.png" alt="Example app.config.json with environmentUrl, then app holding id, name, version, description and empty scopes" width="800" height="323"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The example above is built from the documented fields with placeholder values: the environment ID &lt;code&gt;abc12345&lt;/code&gt; does not exist and your &lt;code&gt;scopes&lt;/code&gt; depend on what the app will read. The &lt;code&gt;version&lt;/code&gt; field deserves a team rule: bump it on every deployment, it is your rollback reference.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deploying without surprises: dry run, then deploy
&lt;/h2&gt;

&lt;p&gt;The dev server shares nothing. For colleagues to see the app, you must deploy it. &lt;code&gt;npx dt-app deploy&lt;/code&gt; builds and deploys to the configured environment. Two useful options, according to the documentation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;code&gt;--dry-run&lt;/code&gt; creates a distributable &lt;code&gt;.zip&lt;/code&gt; archive in &lt;code&gt;out/&lt;/code&gt; without deploying: this is the check to make before a first production release;&lt;/li&gt;
&lt;li&gt;  &lt;code&gt;--skip-build&lt;/code&gt; reuses an existing build, handy when a pipeline has already built the app.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To remove the app, &lt;code&gt;dt-app uninstall&lt;/code&gt; exists too, with its own &lt;code&gt;--dry-run&lt;/code&gt;. What the documentation I read does not detail is the exact interactive sign-in flow and what appears in the environment after deployment. I therefore do not describe it; follow the «&amp;nbsp;deploy your app&amp;nbsp;» guide on the developer site.&lt;/p&gt;

&lt;h2&gt;
  
  
  Automating: the part that matters for on-call
&lt;/h2&gt;

&lt;p&gt;The post stresses a point useful to SRE teams: every command can run in scripts or pipelines, with arguments and options on the command line instead of interactive prompts. The documentation gives the levers: the &lt;code&gt;--non-interactive&lt;/code&gt; option, and environment variables. &lt;code&gt;DT_APP_PLATFORM_TOKEN&lt;/code&gt; skips the default authentication flow; &lt;code&gt;DT_APP_OAUTH_CLIENT_ID&lt;/code&gt; and &lt;code&gt;DT_APP_OAUTH_CLIENT_SECRET&lt;/code&gt; automate SSO token retrieval, which helps in continuous integration. These secrets must never enter the repository: keep them in your CI tool’s vault.&lt;/p&gt;

&lt;p&gt;Two more maintenance commands: &lt;code&gt;dt-app update&lt;/code&gt; updates packages, and &lt;code&gt;dt-app migration execute&lt;/code&gt; helps move between versions (options &lt;code&gt;--from&lt;/code&gt; and &lt;code&gt;--to&lt;/code&gt;). Schedule an update each quarter rather than doing it in an emergency. If you already manage your platform as code, the logic matches &lt;a href="https://span.observer/en/dynatrace-terraform-provider-guide/" rel="noopener noreferrer"&gt;Terraform for alerts and dashboards&lt;/a&gt;: the app becomes a versioned artefact like the others. And to understand how the data behind your pages is identified, see the article on the &lt;a href="https://span.observer/en/find-dynatrace-entity-id/" rel="noopener noreferrer"&gt;entity ID&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  One named owner and one repository per app, with code review as for any service.&lt;/li&gt;
&lt;li&gt;  Minimal &lt;code&gt;scopes&lt;/code&gt;, reviewed with every new page.&lt;/li&gt;
&lt;li&gt;  A &lt;code&gt;deploy --dry-run&lt;/code&gt; before any production release, and the &lt;code&gt;version&lt;/code&gt; bumped.&lt;/li&gt;
&lt;li&gt;  CI secrets kept out of the repository; &lt;code&gt;DT_APP_DISABLE_TELEMETRY=1&lt;/code&gt; exists if your policy forbids the toolkit’s telemetry.&lt;/li&gt;
&lt;li&gt;  Start with one simple page that reads a single DQL query, then extend.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Which Node.js version does the Dynatrace App Toolkit need?
&lt;/h3&gt;

&lt;p&gt;The documentation states Node.js 24. Check it with &lt;code&gt;node --version&lt;/code&gt; before running &lt;code&gt;npx dt-app@latest create&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does the toolkit work with Dynatrace Managed?
&lt;/h3&gt;

&lt;p&gt;The sources I read refer to &lt;code&gt;apps.dynatrace.com&lt;/code&gt; environments, so SaaS. Nothing was verified for Managed: this article does not claim it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can the commands run in CI without interaction?
&lt;/h3&gt;

&lt;p&gt;Yes, according to the documentation: the &lt;code&gt;--non-interactive&lt;/code&gt; option and the &lt;code&gt;DT_APP_OAUTH_CLIENT_ID&lt;/code&gt; and &lt;code&gt;DT_APP_OAUTH_CLIENT_SECRET&lt;/code&gt; variables are meant for this. Test with &lt;code&gt;--dry-run&lt;/code&gt; first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Building a Dynatrace app is a maintenance decision before it is a technical one. If the answer is yes, the &lt;code&gt;create&lt;/code&gt;, &lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;deploy --dry-run&lt;/code&gt;, &lt;code&gt;deploy&lt;/code&gt; loop fits in five commands. For the official detail, read the &lt;a href="https://www.dynatrace.com/news/blog/build-dynatrace-apps-faster-simpler-and-smarter-with-the-new-app-toolkit/" rel="noopener noreferrer"&gt;vendor’s post on App Toolkit 1.0&lt;/a&gt; and the &lt;a href="https://developer.dynatrace.com/quickstart/app-toolkit/" rel="noopener noreferrer"&gt;developer documentation&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Trademarks belong to their owners. Tools change fast: check commands and options against the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/dynatrace-app-toolkit-first-app-step-by-step/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/dynatrace-app-toolkit-demarrer-pas-a-pas/" rel="noopener noreferrer"&gt;https://span.observer/fr/dynatrace-app-toolkit-demarrer-pas-a-pas/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>apptoolkit</category>
      <category>automation</category>
      <category>cicd</category>
      <category>dtapp</category>
    </item>
    <item>
      <title>Dynatrace ownership: route every alert to the right team</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 11:30:22 +0000</pubDate>
      <link>https://dev.to/spanobserver/dynatrace-ownership-route-every-alert-to-the-right-team-4ej</link>
      <guid>https://dev.to/spanobserver/dynatrace-ownership-route-every-alert-to-the-right-team-4ej</guid>
      <description>&lt;p&gt;It is 3 a.m., a problem opens on a host, and the on-call engineer has no idea who owns it. They dig through a wiki, ping three channels and lose twenty minutes. &lt;strong&gt;Dynatrace ownership&lt;/strong&gt; exists to remove that search: every monitored entity carries a tag naming a team, and the team record carries the ways to reach it.&lt;/p&gt;

&lt;p&gt;This article starts from an operations decision: where should the truth about «&amp;nbsp;who owns what&amp;nbsp;» live? I compare three options, then show how to place the tags, feed the team records and route an alert with a workflow. It applies to Dynatrace SaaS; Managed documentation exists too, with similar content, but I have not tested it (see below).&lt;/p&gt;

&lt;p&gt;A note of honesty: span.observer did not build this setup on a tenant for this article. The facts come from the official documentation and a vendor post; the examples are templates to adapt, not measured results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dynatrace ownership: two layers not to confuse
&lt;/h2&gt;

&lt;p&gt;The documentation describes ownership as key-value pairs placed on Smartscape nodes. In practice, two separate objects work together.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;The tag on the entity&lt;/strong&gt;: a recognised key (by default &lt;code&gt;owner&lt;/code&gt; or &lt;code&gt;dt.owner&lt;/code&gt;) and a value, the team identifier.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;The team record&lt;/strong&gt;: a name, an identifier, responsibilities, links and contact channels (Slack, Microsoft Teams, Jira, email).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The identifier is the join. If the tag says &lt;code&gt;team-payments&lt;/code&gt; and a record carries that identifier, Dynatrace knows whom to notify. With no record, you have a name but no way to reach anyone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide first: the source of truth
&lt;/h2&gt;

&lt;p&gt;Before tagging anything, choose who is authoritative. Three options come up.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Manual entry&lt;/strong&gt; in the UI. Fast for ten teams, unsustainable for a hundred: responsibilities shift with every deployment.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;An existing directory&lt;/strong&gt; (Microsoft Entra ID or ServiceNow) synced regularly. The most durable choice if your teams are already kept current there.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Configuration as code&lt;/strong&gt; (Settings API, Monaco or Terraform). You get Git history, but someone must be willing to review JSON files.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;My view: pick one master and let Dynatrace hold a copy. Two diverging sources recreate exactly the doubt you wanted to remove. For the as-code route, &lt;a href="https://span.observer/en/dynatrace-terraform-provider-guide/" rel="noopener noreferrer"&gt;the Terraform article&lt;/a&gt; gives the general frame.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choose the tag keys
&lt;/h2&gt;

&lt;p&gt;The documentation lists two default keys, &lt;code&gt;owner&lt;/code&gt; and &lt;code&gt;dt.owner&lt;/code&gt;, and up to three custom keys, set under Settings &amp;gt; Environment segmentation &amp;gt; Ownership &amp;gt; Keys. One point surprises people: keys are matched as &lt;strong&gt;prefixes&lt;/strong&gt;, not exact keys.&lt;/p&gt;

&lt;p&gt;In practice, if you declare &lt;code&gt;team.&lt;/code&gt;, then &lt;code&gt;team.frontend&lt;/code&gt;, &lt;code&gt;team.backend&lt;/code&gt; and &lt;code&gt;team.support&lt;/code&gt; all count. That helps you reuse an existing convention, but a prefix that is too broad also catches unrelated tags. Keep a prefix that looks like none of your other tags.&lt;/p&gt;

&lt;p&gt;The value is an identifier, optionally followed by a supplementary identifier. The identifier is mandatory, unique and &lt;strong&gt;cannot be changed later&lt;/strong&gt;; the supplementary identifier stays editable. Choose stable identifiers (&lt;code&gt;team-payments&lt;/code&gt;) rather than names that will change at the next reorganisation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Place the tags on entities
&lt;/h2&gt;

&lt;p&gt;The documentation recommends three routes: Kubernetes labels and annotations, host properties set with &lt;code&gt;oneagentctl&lt;/code&gt;, and process group environment variables. On a cluster, an annotation in the manifest is the cleanest, because it travels with the deployment. On a classic host, the command is enough.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2ubkss3he8tgf08gbj98.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2ubkss3he8tgf08gbj98.png" alt="Excerpt of a Kubernetes manifest with the annotation dt.owner: team-payments and the oneagentctl command --set-host-property owner-1=team-payments" width="799" height="340"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I did not verify whether the annotation must be allowed in Dynatrace’s Kubernetes settings before it is read, so test on a single deployment before rolling out. Likewise, the documentation does not detail propagation along the topology; do not assume it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Feed the team records
&lt;/h2&gt;

&lt;p&gt;The workflow action &lt;strong&gt;Import teams&lt;/strong&gt; reads three sources: Entra ID groups, ServiceNow groups, or JSON in the Ownership schema. It also asks for two preferences: skip or overwrite existing teams, and fail everything or continue while logging errors. For a first sync, «&amp;nbsp;continue and log&amp;nbsp;» stops one bad row from blocking the rest.&lt;/p&gt;

&lt;p&gt;The output gives a status (&lt;code&gt;SUCCESS&lt;/code&gt;, &lt;code&gt;PARTIAL&lt;/code&gt; or &lt;code&gt;FAILED&lt;/code&gt;), the imported identifiers, duplicates and failures. Watch the duplicates: an identifier already taken will not be redefined.&lt;/p&gt;

&lt;p&gt;Here is a record in the schema’s JSON format, with sample values. The five responsibilities are required, as are the contact and link lists.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd15ka6s4mis8uz7czfjc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd15ka6s4mis8uz7czfjc.png" alt="Sample team record in the Ownership schema JSON format with identifier team-payments, responsibilities, Slack channel, email and a runbook link" width="800" height="532"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;To create teams by hand or by API you need the permission to manage monitoring settings (UI), or &lt;code&gt;settings.write&lt;/code&gt; or an IAM policy on the &lt;code&gt;builtin:ownership.teams&lt;/code&gt; schema (API). The team-creation page also mentions &lt;code&gt;builtin:ownership.config&lt;/code&gt; without reconciling the two: check the exact schema ID in your environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Route an alert with the Get owners action
&lt;/h2&gt;

&lt;p&gt;The routing itself goes through a workflow. The &lt;strong&gt;Get owners&lt;/strong&gt; action takes entity IDs (for example &lt;code&gt;{ { event()["dt.entity.host"] } }&lt;/code&gt; for a problem event), a team filter and a responsibility filter. It returns &lt;code&gt;owners&lt;/code&gt; and, per team, the Jira, Teams, Slack and email contact details.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8aw59zty2y6ew1z6u4tn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8aw59zty2y6ew1z6u4tn.png" alt="Fields of the Get owners workflow action: entity\_ids, team, filter\_by\_responsibility and the outputs owners, jira, msTeams, slackChannels, email" width="800" height="239"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The next task only has to read those outputs to send the message or open the ticket. Filtering on the Operations responsibility avoids waking the security team for a full disk. For ticket creation, see &lt;a href="https://span.observer/en/dynatrace-workflow-jira-servicenow/" rel="noopener noreferrer"&gt;the Jira and ServiceNow workflow&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The action pays off more once noise is under control: route problems, not hundreds of raw alerts. The article on &lt;a href="https://span.observer/en/reduce-alert-noise-davis/" rel="noopener noreferrer"&gt;reducing Davis alert noise&lt;/a&gt; prepares the ground.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure coverage, do not just install it
&lt;/h2&gt;

&lt;p&gt;Ownership decays silently: a team is renamed, a service migrates, the tag stays. The vendor post suggests visualising coverage and blind spots in a notebook or dashboard, with an example dashboard to adapt. It also notes that DQL access to ownership data was still in development: check its state in the documentation before building a query on it.&lt;/p&gt;

&lt;p&gt;A simple check beats a big project: each month, list critical entities with no tag and assign them. That is toil reduction in the SRE sense: you pay once so nobody searches at 3 a.m.&lt;/p&gt;

&lt;h2&gt;
  
  
  What about security context?
&lt;/h2&gt;

&lt;p&gt;Do not confuse ownership with access control. Ownership says whom to notify; data access belongs to permissions and, on Grail, other mechanisms covered in &lt;a href="https://span.observer/en/dynatrace-grail-vs-classic-permissions-tags/" rel="noopener noreferrer"&gt;the Grail versus Classic article&lt;/a&gt;. I did not re-read the security context page for this piece: treat it as a separate topic, to read before mixing the two. Tagging for management zones remains covered in &lt;a href="https://span.observer/en/dynatrace-management-zones-auto-tagging/" rel="noopener noreferrer"&gt;the Management zones article&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Best practices
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;  One master for teams, synced on a fixed schedule.&lt;/li&gt;
&lt;li&gt;  Stable identifiers, never a team name that may change.&lt;/li&gt;
&lt;li&gt;  A key prefix distinct from your other tags.&lt;/li&gt;
&lt;li&gt;  A first trial on a single deployment, then a monthly coverage audit.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does Dynatrace ownership work on Managed?
&lt;/h3&gt;

&lt;p&gt;A Managed documentation page describes ownership (updated on 7 November 2023, with no version number). The SaaS page does not state the platform. I did not compare them feature by feature: check each action on your version before assuming it is available.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I change a team identifier?
&lt;/h3&gt;

&lt;p&gt;No, according to the assignment page: the identifier is unique and permanent, only the supplementary identifier is editable. Plan neutral identifiers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should I use dt.owner or my own key?
&lt;/h3&gt;

&lt;p&gt;Both work. If you already have a convention (for example &lt;code&gt;team.&lt;/code&gt;), add it as a custom key rather than retagging everything, and watch the prefix effect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Settle the source of truth first, put &lt;code&gt;dt.owner&lt;/code&gt; on one deployment, import one team, then test &lt;strong&gt;Get owners&lt;/strong&gt; on a real problem. The vendor post, &lt;a href="https://www.dynatrace.com/news/blog/the-right-person-at-the-right-time-makes-all-the-difference-best-practices-for-ownership-information/" rel="noopener noreferrer"&gt;The right person at the right time&lt;/a&gt;, and the &lt;a href="https://docs.dynatrace.com/docs/deliver/ownership/" rel="noopener noreferrer"&gt;Ownership documentation&lt;/a&gt; give the official detail.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Trademarks belong to their owners. Tools change quickly: check commands and options in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/dynatrace-ownership-route-alerts-to-team/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/ownership-dynatrace-router-alertes-equipe/" rel="noopener noreferrer"&gt;https://span.observer/fr/ownership-dynatrace-router-alertes-equipe/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>alerting</category>
      <category>dynatrace</category>
      <category>dynatracesaas</category>
      <category>kubernetes</category>
    </item>
    <item>
      <title>Batch job monitoring in Dynatrace: detect, alert, remediate</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 09:30:30 +0000</pubDate>
      <link>https://dev.to/spanobserver/batch-job-monitoring-in-dynatrace-detect-alert-remediate-3blo</link>
      <guid>https://dev.to/spanobserver/batch-job-monitoring-in-dynatrace-detect-alert-remediate-3blo</guid>
      <description>&lt;p&gt;At 02:15 the nightly job starts, and by 02:40 the broker service takes six seconds to answer. Nobody is paged and no static threshold fires, yet customers complain in the morning. That is the real problem behind batch job monitoring in Dynatrace: a batch is neither a service nor a host, it simply eats CPU, memory and I/O at the worst moment. This article proposes a way to decide what to alert on, what to leave to Davis and what to automate. The vendor describes a similar scenario in &lt;a href="https://www.dynatrace.com/news/blog/next-level-batch-job-monitoring-and-alerting-part-2-using-ai-to-automatically-identify-issues-and-workflows-to-remediate-them/" rel="noopener noreferrer"&gt;its two-part post&lt;/a&gt;; I start from the operational decision, not from its outline.&lt;/p&gt;

&lt;p&gt;Method: I re-read the post and the cited documentation pages on 11 November 2026, but I did not replay this scenario on a live environment. Anything I could not confirm is marked «&amp;nbsp;not verified&amp;nbsp;». The reasoning applies to Dynatrace SaaS; I did not check a Managed equivalent, so I promise none.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why batch job monitoring in Dynatrace defeats classic alerts
&lt;/h2&gt;

&lt;p&gt;A batch raises three difficulties. It is short-lived: the pod exists for an hour, then vanishes. It is periodic: its load is normal at 02:15 and abnormal at 14:00. Above all it is a &lt;em&gt;cause&lt;/em&gt;, rarely the &lt;em&gt;victim&lt;/em&gt;: the neighbouring service suffers. An alert on the batch’s own CPU therefore says little; you must tie the batch to the symptoms of the service it hurts.&lt;/p&gt;

&lt;p&gt;The first operational question is what you want to know. Three answers are enough. Did the batch fail? Did it run unusually long? Did it degrade something else? The first two call for a periodic check, the third for Davis. The vendor’s post illustrates the first part with a workflow that reads job status every 15 minutes; it gives no threshold for «&amp;nbsp;too long&amp;nbsp;», so you must set it yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Give Davis context with a start and an end event
&lt;/h2&gt;

&lt;p&gt;Davis sees metrics and traces, not your job calendar. To let it connect a slowdown to a batch, announce the batch with an event. The event ingestion API (&lt;code&gt;POST /api/v2/events/ingest&lt;/code&gt;) only requires &lt;code&gt;eventType&lt;/code&gt; and &lt;code&gt;title&lt;/code&gt;; the &lt;code&gt;entitySelector&lt;/code&gt; attaches the event to the relevant service, otherwise it sticks to the environment. The API token needs the &lt;code&gt;events.ingest&lt;/code&gt; scope.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff7p3q6ecnaod5k6rv0jm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff7p3q6ecnaod5k6rv0jm.png" alt="JSON file of a CUSTOM\_INFO event announcing the start of the update-account batch, with an entitySelector on the broker-service service and four properties" width="800" height="386"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Three details matter in this payload. The &lt;code&gt;CUSTOM_INFO&lt;/code&gt; type is informational: it does not open a problem by itself. The &lt;code&gt;timeout&lt;/code&gt; (in minutes, 15 by default, 360 at most) bounds the event’s lifetime; set it to the usual batch duration. Finally, non-&lt;code&gt;dt.*&lt;/code&gt; properties are free, up to 100 entries: put the job name, run identifier, workload and namespace there. Send a second event at the end with the same identifier so you can measure durations.&lt;/p&gt;

&lt;h2&gt;
  
  
  Let Davis tie the slowdown to the batch
&lt;/h2&gt;

&lt;p&gt;Once the event is in place, detection is no longer your job. In the vendor’s scenario, Davis spots the rise in latency and errors on the service before any ticket exists, and blames a batch that saturated CPU, memory and I/O next to it. My operational takeaway: do not set thresholds on the batch, tune detection on the &lt;em&gt;business service&lt;/em&gt; and let the event supply the explanation. That cuts noise, a topic already covered in &lt;a href="https://span.observer/en/reduce-alert-noise-davis/" rel="noopener noreferrer"&gt;our article on Davis alert noise&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;One limit: the event enriches context, it does not guarantee Davis will name the batch as root cause. Confirm this on a real incident or a load test before building automation on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide what a workflow may do on its own
&lt;/h2&gt;

&lt;p&gt;This is the central decision, and it comes before any configuration. A workflow can start on a Davis problem, open a ticket, test a condition, then act. Use a simple grid:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Harmless&lt;/strong&gt;: open or comment a ticket, notify a channel. Automate straight away; see &lt;a href="https://span.observer/en/dynatrace-workflow-jira-servicenow/" rel="noopener noreferrer"&gt;the Jira and ServiceNow workflow&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Reversible&lt;/strong&gt;: suspend a CronJob, postpone it. Automate if a condition checks the job is not critical to the accounting close.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Irreversible&lt;/strong&gt;: delete a running job, restart a database. Keep that for a human.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a batch, suspending a CronJob has one advantage: it only affects the &lt;em&gt;next&lt;/em&gt; schedule. A job already started carries on, so stopping the current run is a second gesture that you must decide separately. Two safeguards exist in the manifest before Dynatrace is involved: &lt;code&gt;concurrencyPolicy: Forbid&lt;/code&gt; prevents overlaps and &lt;code&gt;activeDeadlineSeconds&lt;/code&gt; caps the duration.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8sj4fdjofvu50plvc82c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8sj4fdjofvu50plvc82c.png" alt="YAML manifest of an account-updater CronJob with concurrencyPolicy Forbid, activeDeadlineSeconds 5400 and the suspend field set to false, commented as the remediation lever" width="800" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Reach the private cluster without opening a port
&lt;/h2&gt;

&lt;p&gt;A workflow runs on the Dynatrace side; the Kubernetes API of a private cluster must not be exposed. EdgeConnect solves this: it opens an outbound WebSocket connection to Dynatrace AppEngine and needs no inbound connection. HTTP requests whose host matches a configured pattern are then executed inside your network. According to the documentation, that pattern is declared under &lt;strong&gt;Settings &amp;gt; General &amp;gt; External Requests &amp;gt; EdgeConnect&lt;/strong&gt;, not in the YAML file.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foudyg1ertsljq61aio6r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foudyg1ertsljq61aio6r.png" alt="Sample edgeConnect.yaml file with name, api\_endpoint\_host, an oauth block and restrict\_hosts\_to, placeholder values" width="799" height="302"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The service account used against the Kubernetes API should have the bare minimum: the right to patch CronJobs in a single namespace. I did not read the page describing the exact action that calls the Kubernetes API from a workflow: that detail is not verified, so check it in the workflow editor.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to check that it works
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt; Send the start event by hand against a test service and check it appears on its timeline. The 201 response lists a &lt;code&gt;status&lt;/code&gt; per event: look for &lt;code&gt;OK&lt;/code&gt;, not &lt;code&gt;INVALID_ENTITY_TYPE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt; Generate real load on a pre-production environment, not in production.&lt;/li&gt;
&lt;li&gt; Confirm the Davis problem names the batch and the workflow starts.&lt;/li&gt;
&lt;li&gt; Check that the CronJob &lt;code&gt;suspend&lt;/code&gt; field flips to &lt;code&gt;true&lt;/code&gt;, and that the ticket is commented rather than duplicated.&lt;/li&gt;
&lt;li&gt; Replay in reverse: a problem unrelated to the batch must trigger no suspension.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  One unique run identifier, present in the start event, the end event and the ticket.&lt;/li&gt;
&lt;li&gt;  A «&amp;nbsp;dry run&amp;nbsp;» mode: for the first two weeks the workflow writes in the ticket what it &lt;em&gt;would&lt;/em&gt; do.&lt;/li&gt;
&lt;li&gt;  An explicit exemption for closing batches, never suspended automatically.&lt;/li&gt;
&lt;li&gt;  Workflows permissions (&lt;code&gt;automation:workflows:read&lt;/code&gt; and &lt;code&gt;admin&lt;/code&gt;) limited to the platform team; to get started, see &lt;a href="https://span.observer/en/dynatrace-workflows-automation/" rel="noopener noreferrer"&gt;the Workflows module of the SaaS course&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;  The EdgeConnect OAuth secret kept in a vault, never in the repository.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do I need an event for every batch?
&lt;/h3&gt;

&lt;p&gt;Yes if you want Davis to be able to explain it. Without an event, the slowdown stays visible but its cause is not tied to the scheduled job.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can Dynatrace stop a job by itself?
&lt;/h3&gt;

&lt;p&gt;It can trigger a workflow that calls the Kubernetes API through EdgeConnect, provided you built and authorised it. Nothing stops without that chain.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does this work on Managed?
&lt;/h3&gt;

&lt;p&gt;The pages I read describe Workflows and EdgeConnect on the SaaS side. I did not verify a Managed equivalent, so treat the scenario as SaaS only.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Announce your batches with an event, watch the business service rather than the job, then automate only reversible gestures. Start with the dry run before letting a workflow suspend anything in production.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Brands mentioned belong to their owners. Tools change quickly: check commands and options in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/batch-job-monitoring-dynatrace-davis-workflows/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/traitements-batch-dynatrace-davis-workflows/" rel="noopener noreferrer"&gt;https://span.observer/fr/traitements-batch-dynatrace-davis-workflows/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>alerting</category>
      <category>automation</category>
      <category>davisai</category>
      <category>dynatracesaas</category>
    </item>
    <item>
      <title>Histogram metrics at ingest: distribution-based latency SLOs with OpenPipeline</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Wed, 18 Nov 2026 07:30:38 +0000</pubDate>
      <link>https://dev.to/spanobserver/histogram-metrics-at-ingest-distribution-based-latency-slos-with-openpipeline-4ch</link>
      <guid>https://dev.to/spanobserver/histogram-metrics-at-ingest-distribution-based-latency-slos-with-openpipeline-4ch</guid>
      <description>&lt;p&gt;Performance dashboards still very often show an average. It is reassuring, readable and almost always misleading. Take a service where 90% of calls respond in 50 ms and 10% in 1.5 seconds: the average comes out at 195 ms. That figure describes neither the satisfied users nor the ones who are waiting. It hides precisely the population that will open a ticket or abandon their basket. That is exactly what the histogram metrics described here correct.&lt;/p&gt;

&lt;p&gt;SRE practice settled the question long ago: a service level objective is expressed as a distribution («&amp;nbsp;99% of requests under 800 ms&amp;nbsp;»), not as an average. You still need data that preserves that distribution. That is the job of histogram metrics, and Dynatrace SaaS now lets you produce them directly at ingest with OpenPipeline, from the spans and logs you already collect.&lt;/p&gt;

&lt;p&gt;This article explains what this type of metric brings, when it is better to use it rather than a query on raw data, and how to fit it into an SLO approach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Average, summary, histogram: three ways to lose (or keep) information
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The single value
&lt;/h3&gt;

&lt;p&gt;Average, minimum, maximum: these aggregates reduce thousands of observations to one number. Once stored on their own, there is no going back. You will never know what share of requests exceeded a given threshold.&lt;/p&gt;

&lt;h3&gt;
  
  
  The precomputed summary
&lt;/h3&gt;

&lt;p&gt;Some instrumentation libraries compute quantiles on the client side, then export them. The problem appears as soon as you have several instances: the p99 of instance A and the p99 of instance B do not let you calculate the overall p99. Quantiles do not add up.&lt;/p&gt;

&lt;h3&gt;
  
  
  The histogram
&lt;/h3&gt;

&lt;p&gt;A histogram places each observation in an interval (a «&amp;nbsp;bucket&amp;nbsp;») and counts occurrences per interval. The shape of the distribution is preserved: the bulk of fast requests, the slow tail and everything in between. Because they are counters, histograms from several instances can be merged, and percentiles can be calculated on the server side across the whole fleet.&lt;/p&gt;

&lt;p&gt;On the format side, the OpenTelemetry specification defines two variants: the explicit-bucket histogram and the exponential histogram, whose boundaries adapt to the range of observed values. Support differs by ingest path: through OTLP ingest, Dynatrace stores the buckets of explicit histograms, but only min, max, sum and count for exponential ones. Before building an SLO, check the Dynatrace histogram documentation to see which format is kept for your source.&lt;/p&gt;

&lt;h2&gt;
  
  
  Query the raw data or extract histogram metrics?
&lt;/h2&gt;

&lt;p&gt;Nothing stops you from calculating a percentile directly on the spans stored in Grail:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fetch spans
| makeTimeseries percentile(duration, 95), by:{endpoint.name}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This approach remains perfect for one-off exploration. However, each run rereads all the spans for the period. On a dashboard tile refreshed every minute, over a thirty-day window or in a continuously evaluated SLO, that reprocessing costs time and resources.&lt;/p&gt;

&lt;p&gt;With a histogram metric extraction rule in OpenPipeline, the distribution is calculated once, at ingest. The same analysis becomes a simple time series query:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;timeseries percentile(span.api.duration, 95), by:{endpoint.name}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Important point: no re-instrumentation is needed. Neither the SDKs, nor the &lt;a href="https://span.observer/en/opentelemetry-end-to-end-dynatrace/" rel="noopener noreferrer"&gt;OpenTelemetry collectors&lt;/a&gt;, nor the application code change. OpenPipeline works on the span and log streams that already arrive in Dynatrace. The resulting metrics are stored in Grail and can be used in DQL, Dashboards, SLO definitions and anomaly detection, with the same retention and access controls as any other metric.&lt;/p&gt;

&lt;p&gt;A simple rule for choosing: if a distribution question comes up regularly (permanent tile, SLO, alert), it deserves a histogram metric; if it is asked once during an investigation, querying the raw data is enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  In practice
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1 — Identify the distributions that matter
&lt;/h3&gt;

&lt;p&gt;List the measurements your users or your SLOs are really sensitive to: public API call duration, job processing time, machine learning model inference time, or a numeric value present in logs (message size, queue delay). Start with two or three candidates, not the whole catalogue.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2 — Check the source data
&lt;/h3&gt;

&lt;p&gt;Before creating the rule, make sure the field exists and is numeric across the target scope. For spans, an exploratory query like this lets you validate the scope and the useful dimensions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fetch spans, from:now()-1h
| filter isNotNull(endpoint.name)
| summarize appels = count(), p95 = percentile(duration, 95), by:{endpoint.name}
| sort appels desc
| limit 20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a value extracted from logs, the field must first be available in numeric form, for example thanks to a &lt;a href="https://span.observer/en/log-pattern-analysis-dql/" rel="noopener noreferrer"&gt;parsing processor&lt;/a&gt; earlier in the same pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3 — Create the extraction rule in OpenPipeline
&lt;/h3&gt;

&lt;p&gt;In the &lt;a href="https://span.observer/en/openpipeline-logs-grail-ingestion/" rel="noopener noreferrer"&gt;OpenPipeline configuration&lt;/a&gt; for the relevant stream (spans or logs), add a histogram-type metric extraction. You define a metric key (here &lt;code&gt;span.api.duration&lt;/code&gt;, replace it with your own), select the source field, and OpenPipeline records each value as a histogram data point. For spans, the Sampling Aware Histogram Metric variant accounts for sampling. Choose the dimensions you keep with care: each extra dimension multiplies the number of series. The OpenPipeline documentation details the available options.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4 — Query the metric
&lt;/h3&gt;

&lt;p&gt;Once the data has arrived, several percentiles can be calculated from the same metric, without rereading the raw data. The choice between p50, p90, p95 or p99 is made at query time rather than at instrumentation time. To plot the median and the tail on the same chart, combine several aggregations in the &lt;code&gt;timeseries&lt;/code&gt; command.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5 — Base an SLO on the distribution
&lt;/h3&gt;

&lt;p&gt;The classic SLO question is the inverse of the percentile: not «&amp;nbsp;what is the p99 latency?&amp;nbsp;», but «&amp;nbsp;what proportion of requests completes under 500 ms?&amp;nbsp;». The new &lt;code&gt;percentRank()&lt;/code&gt; aggregation function answers that question: given a threshold, it returns the estimated fraction of observations below it. It is the natural basis for a distribution-based latency SLO. Check the DQL documentation for its exact syntax.&lt;/p&gt;

&lt;h2&gt;
  
  
  What else the distribution reveals
&lt;/h2&gt;

&lt;p&gt;Beyond SLOs, keeping the shape of the data changes how you read incidents. A bimodal distribution often points to two distinct code paths, for example a cache that does or does not respond. A tail that lengthens after a deployment signals a regression that the median does not show yet. An alert set on a high percentile of the histogram metric can therefore spot a widening tail, where a threshold on the average would stay silent; the anomaly detection documentation lists the available models.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Ban the average from latency SLOs&lt;/strong&gt;: express them as a proportion under a threshold or as a percentile.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Limit dimensions&lt;/strong&gt;: endpoint and service are often enough; avoid high-cardinality identifiers.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Keep the raw data for exploration&lt;/strong&gt;: the metric answers recurring questions; spans and logs remain the reference for detailed analysis.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Compare before switching&lt;/strong&gt;: over a test period, compare the percentile from the metric with the one calculated on raw data to validate the rule.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Document metric keys&lt;/strong&gt;: a clear naming convention avoids duplicates between teams.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do you need to change your instrumentation to get histogram metrics?
&lt;/h3&gt;

&lt;p&gt;No. OpenPipeline works on the spans and logs that already arrive in Dynatrace: neither the SDKs, nor the OpenTelemetry collectors, nor the application code change.&lt;/p&gt;

&lt;h3&gt;
  
  
  When is it better to query the raw data?
&lt;/h3&gt;

&lt;p&gt;For a question asked only once, during an investigation, a query on spans or logs is enough. A histogram metric is worthwhile as soon as the question keeps coming back: a permanent tile, an SLO or an alert.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you express a latency SLO from a histogram?
&lt;/h3&gt;

&lt;p&gt;As a proportion below a threshold rather than as an average. The &lt;code&gt;percentRank()&lt;/code&gt; aggregation function returns the estimated fraction of observations below a given threshold; the DQL documentation gives its exact syntax.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Histogram metrics reconcile two needs that were long at odds: the precision of analysis on raw data and the speed of a pre-aggregated metric. By producing them at ingest, OpenPipeline makes them available without touching existing instrumentation. For a team that steers its reliability through SLOs, it is a chance to finally replace reassuring averages with figures that describe the experience of every user, including the worst served. To get to grips with the syntax before touching your own environment, the Dynatrace Playground offers sample queries on histogram metrics.&lt;/p&gt;

&lt;p&gt;Further reading: &lt;a href="https://www.dynatrace.com/news/blog/create-histogram-metrics-at-ingest-with-openpipeline/" rel="noopener noreferrer"&gt;Create histogram metrics at ingest with OpenPipeline&lt;/a&gt; (Dynatrace blog).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/histogram-metrics-openpipeline-slo/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/metriques-histogrammes-openpipeline/" rel="noopener noreferrer"&gt;https://span.observer/fr/metriques-histogrammes-openpipeline/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dql</category>
      <category>dynatracesaas</category>
      <category>grail</category>
      <category>histograms</category>
    </item>
    <item>
      <title>Dynatrace Microsoft Teams integration: a Copilot Studio agent connected to the MCP server</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Tue, 17 Nov 2026 21:30:30 +0000</pubDate>
      <link>https://dev.to/spanobserver/dynatrace-microsoft-teams-integration-a-copilot-studio-agent-connected-to-the-mcp-server-32n9</link>
      <guid>https://dev.to/spanobserver/dynatrace-microsoft-teams-integration-a-copilot-studio-agent-connected-to-the-mcp-server-32n9</guid>
      <description>&lt;p&gt;In many organisations, the observability tool is restricted territory: SREs and developers use it every day, while the rest of the company goes through them for the smallest answer. A product owner wondering whether the latest release degraded the sign-up journey posts in a Teams channel; someone on call drops what they were doing, opens three apps, assembles an answer and pastes it into the conversation. Multiply that scenario by the number of business teams and you get a workload that is invisible but very real. A Dynatrace / Microsoft Teams agent is aimed squarely at that workload.&lt;/p&gt;

&lt;p&gt;The Dynatrace MCP server offers another way. The &lt;a href="https://span.observer/en/model-context-protocol-explained/" rel="noopener noreferrer"&gt;Model Context Protocol&lt;/a&gt; is an open standard that lets an AI agent call external tools in a structured way. Connected to Microsoft Copilot Studio, it lets you publish a conversational agent in Teams that can query your Dynatrace SaaS environment live, instead of answering from general knowledge.&lt;/p&gt;

&lt;p&gt;This article does not just describe the setup. It focuses on the questions a platform team will ask before opening this channel to hundreds of users: who sees what, how to frame the agent, and where to start so that the experience is useful rather than a gimmick.&lt;/p&gt;

&lt;h2&gt;
  
  
  What an MCP-connected Dynatrace Microsoft Teams agent changes
&lt;/h2&gt;

&lt;p&gt;A general-purpose assistant can explain what 95th-percentile latency is, but it knows nothing about your payment service. An agent connected to the Dynatrace MCP server has tools that let it retrieve the real context: open problems, service health, data stored in Grail. Its answer is therefore based on your data rather than on statistical plausibility.&lt;/p&gt;

&lt;p&gt;The chain has four links, each with a clearly defined role:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Teams&lt;/strong&gt; is the interface: this is where the question is asked, on desktop or mobile;&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;the Copilot Studio agent&lt;/strong&gt; interprets the request and chooses which tools to call;&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;the Dynatrace MCP server&lt;/strong&gt; exposes those tools and returns the requested context;&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;the Dynatrace platform&lt;/strong&gt; remains the source of truth for observability and business observability data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The end user never touches Dynatrace directly. That is both the point of the approach and the reason governance needs care.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you start: the question of permissions
&lt;/h2&gt;

&lt;p&gt;Access to data goes through the permissions and policies defined on the Dynatrace side and on the Microsoft side. In practice, the connection between Copilot Studio and the MCP server uses a Dynatrace platform token. Anything that token can read, the agent can potentially return to anyone who has access to the agent in Teams.&lt;/p&gt;

&lt;p&gt;So you need to think of it as a shared service account:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  grant only the read permissions needed for the target use case, nothing more;&lt;/li&gt;
&lt;li&gt;  ask whether some data (customer identifiers, application log content) should stay out of reach, and adjust the policies accordingly;&lt;/li&gt;
&lt;li&gt;  limit the agent’s rollout in Teams to a population consistent with those permissions, at least at first;&lt;/li&gt;
&lt;li&gt;  plan token rotation and name a clearly identified owner.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is also the time to check internal rules on using AI assistants with production data. Better to settle these points before the demo than after. Our guide to &lt;a href="https://span.observer/en/secure-mcp-server/" rel="noopener noreferrer"&gt;securing an MCP server&lt;/a&gt; covers these principles in more detail (token scope, least privilege, logging).&lt;/p&gt;

&lt;h2&gt;
  
  
  In practice
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1 — Create the agent in Copilot Studio
&lt;/h3&gt;

&lt;p&gt;Create a new agent and give it an explicit name that shows where its answers come from (for example «&amp;nbsp;Observability assistant&amp;nbsp;»). Then write its instructions: this is where much of the perceived quality is decided. Describe its role, its scope and the expected tone. A possible starting point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Role: you help product and support teams understand
the state of applications using Dynatrace data.
Scope: service health, ongoing problems, impact
of recent deployments, user journeys.
Rules:
- back every statement with data retrieved through the tools;
- always state the period analysed;
- if the data does not allow a conclusion, say so;
- suggest a concrete next investigation step.
Context: our main applications are "boutique-web",
"api-commandes" and "paiement".
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The more you describe your environment (application names, business processes tracked, internal vocabulary), the better the agent will be able to link a question phrased in business language to the right entities.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2 — Enrich the agent with Dynatrace knowledge
&lt;/h3&gt;

&lt;p&gt;The Dynatrace for AI project provides skills, prompts and instructions that guide an agent in using the platform. Including them saves every team from reinventing its own way of querying Dynatrace and makes answers more consistent from one user to the next.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3 — Configure the MCP connection
&lt;/h3&gt;

&lt;p&gt;Add a connection to the Dynatrace MCP server in Copilot Studio. Three things are needed: access to your Dynatrace environment, a platform token with the permissions chosen in the governance step, and the endpoint of the corresponding MCP server. Refer to the MCP server documentation for the current list of permissions required by each tool.&lt;/p&gt;

&lt;p&gt;To check what the agent is supposed to get, it helps to run, yourself, in a &lt;a href="https://span.observer/en/dynatrace-notebooks-dashboards-dql/" rel="noopener noreferrer"&gt;Notebook&lt;/a&gt;, a query equivalent to what it should retrieve. For example, for a question about ongoing problems:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;fetch dt.davis.problems
| filter event.status == "ACTIVE"
| fields display_id, event.name, event.start
| sort event.start desc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the agent’s answer differs markedly from this result, the issue lies in the instructions or the permissions, not in the data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4 — Test with real questions
&lt;/h3&gt;

&lt;p&gt;Pull from your Teams channel history the questions actually asked of technical teams over the last few weeks and replay them. A few typical examples: «&amp;nbsp;did yesterday’s deployment change the conversion rate?&amp;nbsp;», «&amp;nbsp;which business processes are affected by the current incident?&amp;nbsp;», «&amp;nbsp;how many orders failed during the outage?&amp;nbsp;». For each one, note whether the answer is correct, sourced and actionable. Failures reveal either a gap in the instructions or context missing from Dynatrace.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5 — Publish to Teams
&lt;/h3&gt;

&lt;p&gt;Once the results are satisfactory, publish the agent to the Teams channel. Start with a small pilot group, collect feedback, then widen the rollout.&lt;/p&gt;

&lt;h2&gt;
  
  
  Choosing a first use case
&lt;/h2&gt;

&lt;p&gt;It is tempting to promise an agent that answers everything. That is the best way to disappoint. Choose a single high-value, measurable scenario and make it excellent:&lt;/p&gt;

&lt;h3&gt;
  
  
  Incident status updates
&lt;/h3&gt;

&lt;p&gt;During an incident, non-technical stakeholders want to know what is affected and since when. An agent that answers these questions frees the incident manager from repetitive requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  Validating a release
&lt;/h3&gt;

&lt;p&gt;After a deployment, the product owner wants to know whether their business indicators have moved. This is a case where Dynatrace business observability brings real value, provided business events are already being collected.&lt;/p&gt;

&lt;h3&gt;
  
  
  Service health questions
&lt;/h3&gt;

&lt;p&gt;Support teams can check a service’s status before escalating a customer ticket, without asking for full access to the platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Least privilege&lt;/strong&gt;: the agent’s token should read only what all its users are allowed to see.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Dated, sourced answers&lt;/strong&gt;: require in the instructions that the agent mentions the period and the nature of the data used.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;A reference question set&lt;/strong&gt;: keep your test questions and replay them after every change to the instructions or permissions.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;A named owner&lt;/strong&gt;: the agent is an internal product, with an owner, a feedback loop and a roadmap.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Do not replace the tool&lt;/strong&gt;: for an in-depth investigation, the agent should point to Notebooks, Dashboards or the relevant app rather than summarise everything.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do Teams users need a Dynatrace account?
&lt;/h3&gt;

&lt;p&gt;No: they never touch Dynatrace directly. The agent queries the MCP server with a platform token, and anything that token can read can be passed on to anyone with access to the agent. Hence the importance of limiting its permissions and the agent’s rollout.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which permissions should the agent’s token have?
&lt;/h3&gt;

&lt;p&gt;Only the read permissions needed for the target use case, leaving out any data that must stay out of reach. The MCP server documentation gives the up-to-date list of permissions each tool requires.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do you check that the agent answers correctly?
&lt;/h3&gt;

&lt;p&gt;Run the DQL query equivalent to the question in a Notebook and compare. Also keep a set of real questions taken from your Teams channels and replay it after every change to the instructions or permissions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Connecting Dynatrace to Teams through MCP and Copilot Studio takes only a few configuration steps. Most of the work lies elsewhere: defining permissions, writing precise instructions and choosing a first use case that speaks to business teams. Well framed, this kind of agent cuts the back-and-forth between teams and brings observability data closer to the people who make product decisions, without turning every Teams user into a Dynatrace administrator.&lt;/p&gt;

&lt;p&gt;Further reading: &lt;a href="https://www.dynatrace.com/news/blog/how-to-connect-dynatrace-mcp-to-microsoft-teams-using-copilot-studio/" rel="noopener noreferrer"&gt;How to connect Dynatrace MCP to Microsoft Teams using Copilot Studio&lt;/a&gt; (Dynatrace blog).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. SaaS features evolve quickly: check their availability in your environment and in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/dynatrace-microsoft-teams-copilot-studio/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/dynatrace-mcp-microsoft-teams-copilot-studio/" rel="noopener noreferrer"&gt;https://span.observer/fr/dynatrace-mcp-microsoft-teams-copilot-studio/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>aiagents</category>
      <category>businessobservability</category>
      <category>copilotstudio</category>
    </item>
    <item>
      <title>Alibaba Cloud Dynatrace: monitor ECS, SLB and databases in multi-cloud</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Tue, 17 Nov 2026 19:30:32 +0000</pubDate>
      <link>https://dev.to/spanobserver/alibaba-cloud-dynatrace-monitor-ecs-slb-and-databases-in-multi-cloud-4hal</link>
      <guid>https://dev.to/spanobserver/alibaba-cloud-dynatrace-monitor-ecs-slb-and-databases-in-multi-cloud-4hal</guid>
      <description>&lt;p&gt;Your application runs on AWS, but the product catalogue, or the Chinese region, lives on Alibaba Cloud. When a user complains, the first question is whether the fault is on your side or the other provider’s. To answer without opening a second console, the Alibaba Cloud Dynatrace extension brings that cloud’s metrics into the same tool. The &lt;a href="https://www.dynatrace.com/news/blog/achieving-enhanced-observability-for-alibaba-cloud-in-multi-cloud-environments-with-dynatrace/" rel="noopener noreferrer"&gt;vendor’s blog post&lt;/a&gt; sets out the ambition; what follows is an operator’s reading, checked against the documentation page.&lt;/p&gt;

&lt;p&gt;The starting point is a decision: what do you want to see from that cloud, and at what cost? The extension is free, but every ingested data point is billed. Choosing your services before activation avoids a nasty surprise.&lt;/p&gt;

&lt;p&gt;A note on method: span.observer has not connected this extension to an Alibaba account. The facts come from the documentation and the post, read on 11 November 2026. Anything that could not be read is marked «&amp;nbsp;not verified&amp;nbsp;». The scope is Dynatrace SaaS; I read nothing about Managed for this topic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Alibaba Cloud Dynatrace: what the extension collects
&lt;/h2&gt;

&lt;p&gt;The extension runs on a remote ActiveGate. The ActiveGate queries Alibaba’s Cloud Monitor APIs and sends the values to Dynatrace. No agent is installed on the Alibaba resources. The post says a partner, Phenisys, developed it and that Dynatrace supports it.&lt;/p&gt;

&lt;p&gt;The documentation lists six core services: ECS (virtual machines), SLB (load balancers), NAS (file storage), NAT Gateway, CEN (cross-region networking) and VPN. Kafka and ECI appear in its feature-set tables without being in the service list, so I make no promise about their coverage. Four database engines are optional: RDS for MySQL, PostgreSQL and SQL Server, plus MongoDB.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fok1o6ppe458m1extcv11.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fok1o6ppe458m1extcv11.png" alt="YAML list of the Alibaba Cloud extension feature sets: ecs, slb, natGateway, nas, cen, vpn, then the databases rds\_mysql, rds\_postgresql, rds\_sqlserver and mongodb" width="800" height="386"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Metrics carry the &lt;code&gt;cloud.alicloud&lt;/code&gt; prefix. For example, &lt;code&gt;cloud.alicloud.ecs.cpu_utilization&lt;/code&gt; for the CPU of an ECS instance, or &lt;code&gt;cloud.alicloud.slb.unhealthyservercount&lt;/code&gt; for servers judged unhealthy behind a load balancer. The page flags one limit: the &lt;code&gt;cloud.alicloud.ecs.status&lt;/code&gt; metric has no Alibaba metric name listed, so I would not rely on it without a test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prepare access before you activate
&lt;/h2&gt;

&lt;p&gt;The prerequisites come down to four items. You need a provisioned ActiveGate with network access to the Alibaba endpoints. You need an access key (ID and secret), entered directly or referenced from the Dynatrace vault. You need the region ID for each region. Finally, an API token with the &lt;code&gt;entities.read&lt;/code&gt; and &lt;code&gt;entities.write&lt;/code&gt; scopes is needed only to import tags.&lt;/p&gt;

&lt;p&gt;On the Alibaba side, the page asks you to authorise read-only capabilities for RDS, MongoDB and Cloud Monitor, according to your security baseline. It does not list the RAM actions: those names change and must be read from Alibaba. Create a dedicated, read-only key. No minimum ActiveGate or Dynatrace version is stated: not verified.&lt;/p&gt;

&lt;p&gt;Two limits matter for a real rollout: proxies are not yet supported, and CDN monitoring does not exist. If your ActiveGate goes out through a corporate proxy, settle that first. For the ActiveGate’s role, see &lt;a href="https://span.observer/en/environment-vs-cluster-activegate/" rel="noopener noreferrer"&gt;the difference between environment and cluster ActiveGates&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Activate the extension, region by region
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt; In Dynatrace Hub, activate the Alibaba Cloud extension.&lt;/li&gt;
&lt;li&gt; Create one monitoring configuration per region, with its region ID and the key (or the vault reference).&lt;/li&gt;
&lt;li&gt; Choose the feature sets. Leave the databases for later if you hesitate.&lt;/li&gt;
&lt;li&gt; If you want Alibaba tags, add the token with the two scopes.&lt;/li&gt;
&lt;li&gt; Check that at least one metric is collected: the page requires it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A useful architecture point: feature sets can match network segments. You can pair a feature set with an ActiveGate group that reaches that segment. One configuration can cover several instances; creating several is optional.&lt;/p&gt;

&lt;h2&gt;
  
  
  Estimate the cost before enabling everything
&lt;/h2&gt;

&lt;p&gt;The extension itself costs nothing. You pay for the ingested data. In the classic model, each point costs 0.001 DDU. In the platform model, consumption is data points per minute multiplied by 60, 24 and 365. The documentation gives the points per minute for each resource: 5 for an ECS instance, 6 for an SLB, 4 for a NAT Gateway, 12 for RDS MySQL (databases are counted per region).&lt;/p&gt;

&lt;p&gt;Take an example of my own making: 10 ECS instances, 2 SLBs, 1 NAT Gateway and 2 RDS MySQL databases. The calculation gives 90 points per minute, or about 47.3 million points a year.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe0wn1e7txmceql2vx7jr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe0wn1e7txmceql2vx7jr.png" alt="Output of a Python script computing the estimate: 90 points per minute, 47,304,000 points per year, 47,304 DDU per year in the classic model" width="799" height="197"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That figure is an estimate built from the page’s formula, not a measurement. Events and logs triggered by the extension are billed separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to check that it works
&lt;/h2&gt;

&lt;p&gt;After a few minutes, query a base metric with DQL, for example ECS CPU. The query below has not been run on a tenant: it is a starting check.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdhooiu8jaqvu75xfckcp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdhooiu8jaqvu75xfckcp.png" alt="DQL timeseries query averaging cloud.alicloud.ecs.cpu\_utilization over the last two hours" width="798" height="155"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; If the series is empty, check the region ID and the ActiveGate’s network path to Alibaba.&lt;/li&gt;
&lt;li&gt; Compare one value with the Cloud Monitor console for the same minute.&lt;/li&gt;
&lt;li&gt; Check that tags appear, if you supplied the token.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Metrics do not replace correlation with your applications. The page says retrieved tags are matched with OneAgent entities, so you can filter by team. For the same exercise on Azure, see &lt;a href="https://span.observer/en/dynatrace-azure-integration-monitor-logs/" rel="noopener noreferrer"&gt;Azure Monitor logs&lt;/a&gt;, and on AWS see &lt;a href="https://span.observer/en/dynatrace-aws-integration-cloudwatch/" rel="noopener noreferrer"&gt;sending CloudWatch to Dynatrace&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Enable ECS, SLB and networking first; add databases afterwards, one at a time.&lt;/li&gt;
&lt;li&gt;  Use a dedicated read-only Alibaba key stored in the vault.&lt;/li&gt;
&lt;li&gt;  Estimate your points before activation, not after the first bill.&lt;/li&gt;
&lt;li&gt;  Alert on unhealthy SLB servers rather than on every metric.&lt;/li&gt;
&lt;li&gt;  Document the Alibaba services that are not covered: the page states that not everything is supported.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does Alibaba Cloud Dynatrace work on Managed?
&lt;/h3&gt;

&lt;p&gt;I read nothing specific to Managed on the page I consulted. Treat the topic as not verified on Managed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do I need OneAgent on the Alibaba resources?
&lt;/h3&gt;

&lt;p&gt;Not for metrics: they come from the Cloud Monitor API through an ActiveGate. OneAgent remains useful for application-level monitoring, and tags let you link the two.&lt;/p&gt;

&lt;h3&gt;
  
  
  Is the extension free?
&lt;/h3&gt;

&lt;p&gt;Activation costs nothing, but ingested metrics are billed. The calculation is based on data points.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Before activating the extension, decide which services to follow, check that the ActiveGate works without a proxy, and count the points. Start with one test region, compare against the Alibaba console, then extend. Official documentation: &lt;a href="https://docs.dynatrace.com/docs/observe/infrastructure-observability/extensions/alibaba-cloud" rel="noopener noreferrer"&gt;Alibaba Cloud extension&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC (or with Alibaba Cloud). Trademarks belong to their owners. Tools change quickly: check options in the official documentation.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/alibaba-cloud-dynatrace-multi-cloud-observability/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/alibaba-cloud-dynatrace-observabilite-multi-cloud/" rel="noopener noreferrer"&gt;https://span.observer/fr/alibaba-cloud-dynatrace-observabilite-multi-cloud/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>activegate</category>
      <category>alibabacloud</category>
      <category>dynatrace</category>
      <category>dynatracesaas</category>
    </item>
    <item>
      <title>Chaos engineering with Dynatrace: measure before you inject a failure</title>
      <dc:creator>Span Observer</dc:creator>
      <pubDate>Tue, 17 Nov 2026 17:30:19 +0000</pubDate>
      <link>https://dev.to/spanobserver/chaos-engineering-with-dynatrace-measure-before-you-inject-a-failure-3348</link>
      <guid>https://dev.to/spanobserver/chaos-engineering-with-dynatrace-measure-before-you-inject-a-failure-3348</guid>
      <description>&lt;p&gt;Before scheduling a &lt;strong&gt;chaos engineering with Dynatrace&lt;/strong&gt; experiment in production, one question matters: «&amp;nbsp;if I slow down or cut this component, how will I know within ten minutes that the experiment worked, or that it must stop?&amp;nbsp;» Without a numeric answer, a deliberate failure is just a failure.&lt;/p&gt;

&lt;p&gt;This article starts from that operations decision. It does not describe a fault-injection tool: in the documentation and the vendor post I read, Dynatrace is used to &lt;em&gt;measure&lt;/em&gt; (starting state, effects, cause), not to cause the failure. Injection stays with your usual tool. The original post (&lt;a href="https://www.dynatrace.com/news/blog/build-systems-more-reliably-with-dynatrace-chaos-engineering/" rel="noopener noreferrer"&gt;Build systems more reliably with Dynatrace: Chaos Engineering&lt;/a&gt;, August 2024, updated January 2025) defends the same general idea; the outline below is my own.&lt;/p&gt;

&lt;p&gt;Scope: the Service-Level Objectives, Site Reliability Guardian (SRG) and Workflows apps of Dynatrace SaaS, built on Grail and DQL. I did not verify their availability on Managed, so assume nothing for that case.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write the stop condition before the injection
&lt;/h2&gt;

&lt;p&gt;A chaos experiment is a hypothesis test. Write it as a three-line contract: what you break, what you expect, and the threshold that triggers an immediate stop. If you cannot put numbers on the second and third lines, you are not ready.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq3gboox7rgahag8jfz11.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq3gboox7rgahag8jfz11.png" alt="YAML chaos experiment hypothesis file: target, injected failure, blast radius, expected outcome and stop thresholds on failure rate and response time" width="800" height="344"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Three choices sit in this file. The &lt;strong&gt;blast radius&lt;/strong&gt; is deliberately small (one zone, a share of traffic). The &lt;strong&gt;expected outcome&lt;/strong&gt; is expressed as a service level objective. The &lt;strong&gt;stop thresholds&lt;/strong&gt; are stricter than the objective, so you can interrupt before the error budget is gone. To pick good indicators, see &lt;a href="https://span.observer/en/how-to-choose-slis-dql/" rel="noopener noreferrer"&gt;how to choose SLIs in DQL&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure the starting state with DQL
&lt;/h2&gt;

&lt;p&gt;The classic weak spot is a hypothesis built on a feeling («&amp;nbsp;this service is solid&amp;nbsp;»). First record a baseline over a normal period, for example seven days, at a traffic level comparable to the experiment.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjsms5j6dh18rmj0ln12v.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjsms5j6dh18rmj0ln12v.png" alt="DQL timeseries query on service request metrics computing the failure rate as a percentage, per service, over seven days" width="800" height="219"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The query is an example to adapt: check metric keys and dimensions in your environment before using it; I have not run it on a tenant. Keep the method: a failure rate and a response time per service, over a comparable period, recorded &lt;em&gt;before&lt;/em&gt; any injection. Add the number of open problems and their usual duration, to know whether your system is already fragile that day. If the baseline is already bad, postpone the experiment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turn the hypothesis into checkable objectives
&lt;/h2&gt;

&lt;p&gt;Two platform building blocks do this, and they answer different questions.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;An SLO&lt;/strong&gt;: the indicator (SLI) is a series normalised between 0 and 100% computed by a DQL query, compared with a target over an evaluation period. The error budget is the gap between the status and the target; the burn rate tells you how fast you are spending it. During the experiment, the burn rate says whether the test is costing too much. See &lt;a href="https://span.observer/en/slo-burn-rate-alerts-dynatrace/" rel="noopener noreferrer"&gt;burn-rate alerts&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;An SRG guardian&lt;/strong&gt;: a group of objectives, each fed by a DQL query and judged against warning and failure thresholds. The overall result is the most severe of the individual results.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For an experiment, the guardian is the referee. One objective per line of your hypothesis file: failure rate, response time, and a capacity indicator if you stress the CPU. Auto-adaptive thresholds need at least five past validations and return «&amp;nbsp;info&amp;nbsp;» until then: for a first experiment, set static thresholds taken from your baseline.&lt;/p&gt;

&lt;h2&gt;
  
  
  Automate the verdict with a workflow
&lt;/h2&gt;

&lt;p&gt;A workflow can call the guardian action after the injection window, with an event trigger or an on-demand call. The result exposes a validation status (&lt;code&gt;info&lt;/code&gt;, &lt;code&gt;pass&lt;/code&gt;, &lt;code&gt;warning&lt;/code&gt;, &lt;code&gt;fail&lt;/code&gt; or &lt;code&gt;error&lt;/code&gt;), an identifier and a link to the validation. You can derive a clear decision: &lt;code&gt;pass&lt;/code&gt;, widen the blast radius; &lt;code&gt;warning&lt;/code&gt;, repeat at the same scope; &lt;code&gt;fail&lt;/code&gt;, stop and open the ticket. It is the same logic as a release gate, described in &lt;a href="https://span.observer/en/site-reliability-guardian-tutorial/" rel="noopener noreferrer"&gt;the article on SRG and release SLOs&lt;/a&gt;, applied here to a deliberate failure. For workflow basics, see &lt;a href="https://span.observer/en/dynatrace-workflows-automation/" rel="noopener noreferrer"&gt;module 7 of the SaaS course&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;A word on vocabulary: the SRG documentation calls this step «&amp;nbsp;Site Reliability Guardian&amp;nbsp;» in the action picker, «&amp;nbsp;guardian validation action&amp;nbsp;» elsewhere, and «&amp;nbsp;run validation&amp;nbsp;» in generated workflows. Look for the match, not the exact label.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chaos engineering in six steps
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt; Record the baseline and note the start time.&lt;/li&gt;
&lt;li&gt; Create the guardian and the workflow; test them first without any fault, the result must be &lt;code&gt;pass&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt; Announce the window to on-call and name the person who can abort.&lt;/li&gt;
&lt;li&gt; Inject the failure on the smallest scope, with your injection tool.&lt;/li&gt;
&lt;li&gt; Let the validation run and watch the burn rate; if the stop threshold is crossed, cut without debate.&lt;/li&gt;
&lt;li&gt; Analyse: if a problem opened, Davis root cause analysis shows whether it was attributed to the fault or to something else; fix, then widen by one notch.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The vendor post mentions, on its demo, request slowdowns and CPU stress followed by a root cause analysis. I did not replay that demo: treat it as a usage example, not a result measured on your side.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best practices
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;  Start in pre-production, then production on a tiny share of traffic.&lt;/li&gt;
&lt;li&gt;  One change at a time: no experiment during a deployment.&lt;/li&gt;
&lt;li&gt;  Keep stop thresholds stricter than the SLO, otherwise you stop too late.&lt;/li&gt;
&lt;li&gt;  Keep a record of each experiment: hypothesis, baseline, guardian result, fix.&lt;/li&gt;
&lt;li&gt;  Blameless tone: the experiment tests the system, not the team.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Frequently asked questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Does Dynatrace inject the failures?
&lt;/h3&gt;

&lt;p&gt;In the documentation pages and the post I read, no: they describe measurement, analysis and automating the decision. Injection belongs to another tool. Check your version’s release notes if you are looking for a native feature.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do I need an SLO or a guardian?
&lt;/h3&gt;

&lt;p&gt;Both help. The SLO follows the service over time and its error budget; the guardian gives a one-off verdict on a precise window, which suits an experiment.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can I test directly in production?
&lt;/h3&gt;

&lt;p&gt;Only with a reduced blast radius, a solid baseline and an automatic stop criterion. When in doubt, start in pre-production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Chaos engineering with Dynatrace starts with measurement: a baseline, stop thresholds and an automated verdict. The failure comes after. To go further, error-budget logic is detailed in &lt;a href="https://span.observer/en/error-budget-release-gate-dynatrace/" rel="noopener noreferrer"&gt;the article on the error budget as a release gate&lt;/a&gt;. Official sources: &lt;a href="https://docs.dynatrace.com/docs/deliver/site-reliability-guardian" rel="noopener noreferrer"&gt;Site Reliability Guardian&lt;/a&gt; and &lt;a href="https://docs.dynatrace.com/docs/deliver/service-level-objectives" rel="noopener noreferrer"&gt;Service-Level Objectives&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Features change: check them in the official documentation for your environment.&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://span.observer/en/chaos-engineering-dynatrace-measure-first/" rel="noopener noreferrer"&gt;span.observer&lt;/a&gt;. French version: &lt;a href="https://span.observer/fr/chaos-engineering-dynatrace-mesurer-avant/" rel="noopener noreferrer"&gt;https://span.observer/fr/chaos-engineering-dynatrace-mesurer-avant/&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>chaosengineering</category>
      <category>dql</category>
      <category>dynatracesaas</category>
      <category>errorbudget</category>
    </item>
  </channel>
</rss>
