Tim Falzone’s Post

Check out this excellent paper on the evolution of SRE at Google through the adoption of Safety Design principles (aka STPA and CAST).

Back in the days at university we had a pretty wide range of fundamental disciplines related to control theory, analog and digital ways of solving various problems. I remember the feeling it brings when you analyze a system with transfer and step functions, and see how it would behave under certain conditions, without even building it. In other courses there were more practical ways to look at systems holistically and try to understand their dynamics, including statistical analysis. This article underscores a widespread look at software and ways to understand how it works. Logging is largely considered a “just enough” way to collect data, sometimes not even using structured logging and correlation. I’ve also been to similar conversations like - oh, we have disabled tracing in production, because it costs. Then how would it be possible to find the root cause of an issue or simply trace the stages of a process? In my opinion, mostly scientific and mathematical methods are not used, or considered sophisticated or irrelevant, because general software is not that complex and/or because it is build with “we will figure it out later” mentality. I think developers should help themselves to be ready to resolve complex production issues by reusing the engineering techniques known for long time.

If you are involved in #SRE and #Observability this is a worthwhile read. Vendors in this space should also digest this and think carefully about how their products might support these practices https://lnkd.in/eS-U8ZUY

This article and its insights are 🔥

Looking for "hazard states" reminds me of the "impending doom" style alerts, and all the arguments back-and-forth about cause-based vs symptom-based alerting. Even the most hard-core "symptom-based" people admit that sometimes other "cause-based" alerts are useful, but they are often hard to identify and tune alert thresholds for. These cases always (nearly always? is there any other kind?) represent "hazard states", and the STPA approach gives you a systematic way of trying to find them. The next tricky part is figuring out how to tune your alerting for them so that they get appropriately addressed without overwhelming people with too many alerts. The people dealing with these hazards are also part of the system, and pager-fatigue is a very real failure mode. For tuning alert thresholds I recommend a risk analysis style approach to translate it into your clearly defined SLO's. These "hazard states" represent a risk of an outage; what is the probability in average duration till an outage happens from this state, and how much of your SLO error budget will the outage cost? The cost/duration is the effective SLO error budget burn rate of being in that state, so set your alert threshold and priority to reflect that.

I’ve been working with Tim and his SRE team for many years at this point. They are true partners helping to ensure that we prevent outages and minimize risks in our Geo products. I can also attest that there is real power in STPA as a method for understanding control hazards in complex systems. There can be some high costs to the initial set up, but there are real upsides to actively preventing the worst scenarios by identifying areas of system decisions with inadequate controls or visibility. It is an empowering game changer for engineering teams to step away from post-mortem analysis as the primary driver for systemic changes.

Interesting paper. This is the evolution in thinking I was anticipating when I wrote my essay “Observations on Observability”: less emphasis on the discreet and linear using inductive reasoning, more emphasis on the gestalt using more systemic approaches. My essay from 2019: https://lnkd.in/gwU48j6

#Cybernetics and #ControlTheory are good (only?) tools for technical systems reliability. Still quite surprised to see Kalman and Ashby direct references in Google SRE practices in a article by Tim Falzone and Benjamin Treynor Sloss. Next to see how Law Of Requisite Variety meets complexity of modern distributed technical systems

"What if I told you breaking your system could make it stronger? Looks 🤪 Here we go ✍️ Chaos Engineering: Building Resilient Systems 🔧 What is Chaos Engineering? A proactive practice to test system resilience by intentionally introducing failures in a controlled manner. 🎯 Why It Matters 💡 Discover weaknesses before they become outages. 🚀 Build confidence in system reliability. 🛡️ Ensure a seamless user experience during disruptions. 📋 Core Steps 1️⃣ Define Steady State: What does “normal” look like? 2️⃣ Create Hypotheses: Predict system behavior under failure. 3️⃣ Run Experiments: Simulate issues like: 🔥 Service crashes 🌐 Network delays 🧮 High CPU usage 4️⃣ Analyze & Learn: Use observability tools (e.g., Grafana, Datadog) to improve. ⚙️ Popular Tools Chaos Monkey 🐵: Random instance failures. Gremlin 🛠️: Controlled fault injection. LitmusChaos ☸️: Kubernetes-specific experiments. 🏆 Real-World Wins Netflix: Stream without interruptions. Amazon: Handle Black Friday spikes. Google: Avoid cascading failures. 🔍 Key Takeaway: Break it intentionally, fix it intelligently "Liked this post? Follow me for more insights on engineering practices, cloud solutions, and building resilient systems!" Thanks Venkatesh Sarivisetty

Fun facts: 1) Ben Treynor talked to me about STPA and that conversation was one of the reasons I decided to come to Google. (I didn’t join Google - I joined Ben. Had I known that I would also get to work with Tim, that would have been another reason to join!) 2) We’ve been working with John Thomas to apply/tweak/apply STPA to complicated software systems. He his fantastic! 3) STPA is NOT a ‘read the book and give it a shot’ sort of thing. If you want to get STPA going in your org, you should connect with John to figure out how.