That problem became my onboarding project, a system that automatically switches payment vendors when one of them starts failing, and that now quietly handles real customer payments every day.

Here’s the short version of what I learned: the switching decision itself is simple. The hard part and what makes it trustworthy is everything around it: choosing the right signals, validating in shadow mode, making every decision observable, and being conservative about what counts as “the same bank”.

The rest of this post is the long version.

Some context

Wise lets customers pay directly from their bank account. For EUR pay-ins, we integrate with external vendors that act as the bridge between Wise and a large number of European banks.

Many banks are reachable through more than one vendor. That overlap is the key to everything that follows.

The problem nobody wanted to keep solving manually

When one of our vendors has an issue, even a subtle one that never makes it onto a status page, payments start failing. A customer goes through the whole authorisation flow at their bank, comes back to Wise, and the payment doesn’t go through.

Until now, responding to this needed a human: someone had to notice the failures, work out which vendor and bank were affected, and intervene by hand. That’s slow, it puts pressure on whoever is on call, and customers could feel the problem before we do.

The goal of my project was to let the system detect and respond to these situations on its own.

• Primary goal: automatically switch EUR pay-in traffic from one vendor to another, per bank, when one underperforms.
• Secondary goal: give the team visibility into vendor degradations that were previously harder to spot.
• Out of scope: redesigning our existing downtime detection platform, or extending this to recurring payments.

Step 1: Understand what already exists

Before writing any code, I spent time understanding the existing system. Wise already had downtime detection for GBP Open Banking payments, which identifies degraded banks and disables them. That gave me a mental model and patterns to build on.

For EUR, nothing equivalent existed. No automated detection, no health classification, no automated response.

I also needed to understand vendor coverage: which banks each vendor supports, and where they overlap. A bank reachable through only one vendor can’t be switched. A bank reachable through several is a candidate.

Step 2: Choose signals that reflect customer reality

The design question was: what should trigger a switch?

The obvious answer is “the vendor’s status page”. But vendors don’t always declare partial degradations. A vendor can show “all systems operational” while a meaningful share of payments for one specific bank is failing. Status pages reflect declared incidents, not necessarily what customers are experiencing.

So I designed two independent signals.

Signal 1 : Vendor status (declared outages). One vendor sends us webhooks when a bank’s status changes. The other exposes a health API that we poll on a schedule. Both paths write to the same availability store. This signal is fast and authoritative, but only for problems the vendor acknowledges.

Signal 2 : Conversion rate (actual customer outcomes). For every vendor × bank pair, we track what fraction of initiated payments reach a successful end state within a rolling time window, using a sorted set in Redis. Each bank has its own expected baseline, because banks differ in what “normal” looks like. If the observed rate drops far enough below that baseline, the pair is classified as degraded.

The second signal is what made the system genuinely useful. It reacts to what customers are experiencing, not to what a dashboard says.

The two signals cover each other’s blind spots: Signal 1 is fast but misses soft failures; Signal 2 takes longer to build up evidence, but catches exactly what Signal 1 misses.

Two details that matter more than they look

A minimum sample size. Without a floor on the number of payments in the window, one failed payment during a quiet hour can tank the conversion rate and trigger a switch. Below the floor, Signal 2 simply stays silent.

Failing safe. If anything goes wrong while calculating the conversion rate, Redis is unreachable, there’s no baseline configured, not enough data, the tracker answers “not degraded”. When the signal itself is unreliable, the system does nothing rather than something.

Step 3: Make the decision on every payment

Both signals feed into a three-state health classification: UNAVAILABLE, DEGRADED, or HEALTHY. A declared outage takes priority, if the vendor says a bank is down, we don’t need to wait for conversion data to confirm it.

The check runs on every payment request, not on a background schedule. When a customer confirms a payment, we evaluate the health of their bank’s vendor at that moment. If it’s unhealthy and a healthy alternative exists, we switch before the payment is created.

What happens when there’s no healthy alternative depends on how bad things are:

• If the bank is UNAVAILABLE, the customer sees a clear error rather than a payment we know will fail.
• If the bank is only DEGRADED, the payment still goes through the primary vendor. A degraded vendor still succeeds for most customers, and blocking the bank would make things worse, not better.

That principle , never make the customer’s situation worse than doing nothing , shaped most of the design.

Being conservative about “the same bank”

We only switch to an alternative vendor explicitly configured for the same bank the customer selected. The customer’s selected bank remains unchanged. An alternative also has to pass every check the original would have: it must be healthy, supported for that payment amount and account type, and available to that customer.

An incorrect switch, sending a customer to a different institution than the one they chose is far worse than leaving them on a degraded vendor. So the mapping is small, explicit and deliberately cautious.

Returning traffic to the primary vendor

Once traffic moves to the alternative, the primary vendor stops receiving payments for that bank. Over time its window empties out, drops below the minimum sample size, and Signal 2 goes quiet. The bank is classified as healthy again and traffic returns to the primary. If the problem is still there, the signal builds up again and traffic moves back.

There’s no separate “recovery” logic. The sliding window works as a natural, gradual probe.

Step 4: Shadow mode first

This is the part I’d emphasise most to anyone building something similar.

Before we switched a single real payment, the full decision logic ran in monitoring-only mode. Every payment request was evaluated, and the system recorded what it would have done, without acting on it.

That surfaced things no test suite would have found:

• Banks whose conversion rate naturally fluctuates and would have caused false positives
• Alternative mappings that were less equivalent than we had assumed
• Edge cases in the decision logic that only real production traffic could produce

Shadow mode is what gave us the confidence to turn switching on.

Step 5: Make every decision observable

Every decision, including the decision not to switch, emits a metric tagged with the bank, the primary vendor, the outcome and the reason. Actual switches are also written to a persistent log, tied to the payment, so any routing decision can be explained afterwards.

Every knob is a runtime option: switching on or off, monitoring-only mode, the window length, the drop threshold, the minimum sample size, and the per-bank baselines and alternatives. That means whoever is on call can:

• turn off switching instantly if something looks wrong
• mark a bank unavailable before the signals catch up
• tune thresholds for a bank with unusual traffic
• fall back to monitoring-only mode to investigate

All of this without a deployment.

This matters for two reasons. Practically, you can’t tune what you can’t see. Culturally, when a system makes autonomous decisions about real customer money, engineers need to trust it. You earn that trust by making the system auditable, not by promising it’s correct.

Rollout

We shipped in three phases, each behind runtime flags with a clear rollback path.

What I’d do differently

Start collecting conversion data earlier. Baselines need real traffic to mean anything. I added the instrumentation during development, so we had less history than I’d have liked when it came to setting thresholds. Shipping the tracking first, on its own, would have given us confidence sooner.

Take the bank mapping seriously from day one. I assumed it was a simple lookup table. It isn’t. “Same bank, different vendor” involves questions about account type coverage, market-specific behaviour, and differences in what each vendor supports. I underestimated this in the initial design and paid for it later.

What’s next

• Adaptive baselines: derive each bank’s expected conversion rate from its own history, instead of configuring it by hand, so baselines stay accurate as bank behaviour changes.
• Broader coverage: grow the set of banks with alternatives, and apply the same pattern to other payment flows where similar redundancy exists.

Closing

What stayed with me from this project is how much of the interesting engineering happens around the core logic. The “if unhealthy, switch” part fits on a napkin. Choosing signals that reflect customer reality, failing safe, validating in shadow mode, and making every decision auditable is what turns it into something you can trust.

Vendor degradations that previously required manual intervention are now handled automatically, helping prevent a negative customer experience.

And it went live before I’d finished my onboarding, which says as much about how Wise brings new engineers in as it does about the project.

Interested in problems like this? We’re always looking for engineers who want to work on real-money systems at scale.

Explore open roles at Wise

From Onboarding to Production: How I Built Dynamic Vendor Switching at Wise was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

How five changes cut our monthly observability spend by 40%. Co-authored by Andrew Marwood

Opening

Wise’s mission is to build the best way to move and manage the world’s money with — minimum fees, maximum ease, full speed. That promise reaches around 19 million people and businesses across 160+ countries and processes around 4.7 million transactions a day. Wise has saved their customers £2.4 billion in hidden fees compared to banks over the past year [FY26].

Behind every transfer, balance check, and currency conversion is a foundational platform running over 1,000 microservices 24/7. That platform costs money to run, and those costs feed into the fees customers pay. To maintain service reliability, engineers depend on comprehensive observability for the components they own, making our telemetry stack one of the platform’s largest infrastructure consumers.

Platform efficiency is therefore a first-class KPI at Wise, alongside stability, productivity, risk and enablement. The Observability team’s mission is to empower teams to confidently, conveniently and efficiently visualise and act on system behaviours, ensuring the delivery of reliable products to our customers. A slow log query, a growing storage bill or a bottleneck in the logging pipeline may seem far removed from a customer’s transfer, but each affects the platform that Wise teams rely on to keep services dependable.

This article follows five investigations driven by a simple principle: efficiency is an ongoing engineering discipline, not a project with an end date. From changing the log compression algorithm and tuning query splits to upgrading the metrics collector and tracing a Fluent Bit connection limit, this article shows how removing waste and fine tuning configuration can support a reliable and responsive product. Some gains came from experiments, others from structural changes and deep forensic investigations. Together, they improved storage, network, and compute efficiency across our observability stack, resulting in a 40% reduction in our monthly spend.

The Measurement Backbone

The Cloud Economics team at Wise builds and maintains cost attribution tooling, providing dashboards with enough granularity to surface per-service, per-day spend, which enables teams to test hypotheses against actual spend. Without that visibility, most of what follows would have been invisible savings, or invisible waste. You can only optimise what you can measure. That principle applies equally to customer-facing features and to the platform beneath them.

Storage: Challenging Our Compression Defaults

Logs are often the first audit trail engineers reach for when investigating issues. When combined with other telemetry like metrics, traces, and profiles, they give teams a clear forensic view of a problem. The faster you can query these signals, the faster you can find the root cause; and the cheaper you can store them, the longer you can keep them.

At Wise, our observability team maintains an in-house Grafana Loki cluster that gives engineers access to real-time logs and historical logs. These logs were persisted in our production Loki S3 bucket at a scale of roughly 50TB per day and that number was constantly growing. Loki compresses chunks before storing them in S3. We had used the Snappy compression algorithm since the cluster’s early days. Grafana recommends Snappy as the default¹ for a reason: it’s fast, well-understood, strikes a solid balance between resource usage and compression, and enjoys broad support. For many workloads, it’s a sensible choice.

Given our rapid growth, it simply wasn’t the right fit anymore as was evident from the chart above, which maps out our Loki S3 bucket storage growth from Q4 2025 to Q1 2026. This prompted us to explore various optimisation opportunities across the Loki stack and the first question that sprang to mind was whether we were using the most efficient compression for the data persisted in the S3 bucket by the ingesters.

We looked at alternatives that might compress our logs more efficiently. The lz4 algorithm appeared promising, but we needed to make sure its storage savings would hold up at our scale without compromising performance. An open GitHub issue² from someone running Loki 2.9.2 on EKS caught our eye: their querier pods were OOMing during large range queries, with pprof traces pointing squarely at the lz4 block allocator (and they had to switch back to Snappy to fix it). Though that issue was from late 2023 and we were running a much newer Loki version (3.5.x), instead of making assumptions we wanted to validate whether we would run into the same issue. What made the experiment worth trying despite that warning was another detail buried in that same thread: lz4 was showing 2 to 3x better compression ratios than Snappy for similar workloads.

We tested in a low-volume environment first, then staging, then production, with a clear rollback path at each step and the results were very promising. The irony of the whole optimisation was how little it actually took: unlocking these massive gains didn’t require an architectural overhaul, just a single line of configuration.

In theory, trading compression speed for better space savings comes at a cost: ingesters have to burn more CPU cycles packaging chunks before flushing to S3, while queriers spend more compute decompressing them during log searches. However, in the first few days after our production rollout the metrics confirmed that lz4 delivered the 40–50% storage drop with negligible impact on CPU/Mem across both the read and write path components.

Better yet, despite the warnings in that GitHub issue, we didn’t run into any negative performance impacts or query slowdowns, and our querier resource usage remained completely stable. Over the weeks that followed, roughly 900TB worth of Snappy compressed chunks were removed from the production Loki bucket as the retention policies kicked in. This resulted in a sustained ~50% reduction in overall S3 storage footprint.

Cutting that storage footprint didn’t just shrink our AWS bill; it bought us operational runway. Cheaper storage means the team can justify longer retention, broader log coverage, or simply let the platform grow with Wise’s customer base without watching the bill scale linearly alongside it.

This single-line change not only delivered a massive boost in storage efficiency, but also unlocked unexpected savings across our cross-AZ (Availability Zone) footprint. Smaller chunks mean less data moving across zone boundaries on every write and read. Because lz4 shrank chunk sizes by 40–50%, it cut the volume of cross-AZ data traffic by that exact same proportion—translating to a significant direct daily cost reduction in cross-AZ network costs for our observability cluster. One config change, two cost levers pulled simultaneously.

Changing the compression algorithm used in OpenTelemetry collector

The compression question extended to our tracing stack as well. The OpenTelemetry Collector is deployed into each of our Kubernetes clusters, and typically each collector in a deployment can process tens of thousands of spans per second. This gave us another opportunity to look at making a small change that would have a large impact at the scale we operate.

We noticed that gzip was enabled by default on our path from the OpenTelemetry Collector to Grafana Tempo, and realised there were optimisation opportunities based on what was inferred from the comparison of different compression algorithms detailed here³.

Switching the algorithm from gzip to zstd took us just eight hours to go from hypothesis to production. Because the pattern was already established, our measurement tooling was in place, and the team had the confidence to deploy quickly, we were able to validate the results immediately. We saw a roughly 40% reduction in data received by our Tempo deployments.

The compounding effect that we have described here comes from the combination of everything: the cumulative savings from each change, the insights we pick up along the way, and the velocity it gives us to apply those lessons to our next round of efficiency experiments. What we truly learned from these exercises is that defaults or recommendations are just a starting point, not a rule, and it should never stop one from testing what’s possible. At Wise we believe curiosity, a healthy dose of skepticism, and a willingness to experiment are what turn good engineering into great engineering.

Protocols: The Cost of Delayed Upgrades

Metrics tell Wise engineers whether their services are healthy in real time, allowing us to spot potential issues before our customers notice a problem. In our Kubernetes clusters we use Prometheus, managed via the kube-prometheus-stack, to scrape metrics and remote write them to Grafana Mimir which provides our long term storage and query layer. Every Prometheus instance was shipping metrics using Remote Write v1. It was reliable but verbose, due to the repetition of metric labels across payloads, and inefficiencies in payload encoding. Remote Write v2, finalised in 2024⁴, uses more efficient protobuf encoding and cuts the wire payload roughly in half.

The upgrade path to enable Remote Write v2 was not trivial. It required Prometheus Operator v0.76 to be upgraded to v0.78+, and kube-prometheus-stack to be upgraded to v66+, so the Prometheus Operator Custom Resource Definitions (CRDs) would support the messageVersion: V2.0 field for the remoteWrite configurations on our prometheusSpec definitions. These changes needed careful coordination to avoid potential compatibility issues.

Mimir needed no changes. It has supported Remote Write v2 since v2.17, and Wise runs v3.x. The automation pattern for cluster-wide rollouts was already established, which made a careful deployment across multiple Kubernetes clusters possible.

While Remote Write v1 worked, we avoided the complexity of the upgrade, but it turned out that avoiding this upgrade was resulting in much higher data transfer and as a result higher costs. That cost was real and recurring. We only saw it clearly once we measured it.

Comparing the baseline metrics, traffic hitting Mimir distributors before and after the rollout across all clusters, the impact was clear: daily network transfers fell from ~16TB per day to ~5.5TB per day, roughly a 66% reduction. The size of this reduction significantly improved costs for cross xAZ traffic, load balancers, and TransitGateway traffic. The more efficient metrics pipeline also scales better. This lets the observability platform grow with Wise without the cost of visibility growing at the same rate.

Compression as a first-class design decision

“The art of programming is the art of organising complexity.” — Edsger W. Dijkstra

Wise processes petabytes of telemetry data every month. At this scale, compression isn’t just a configuration detail; it’s a critical lever for performance and cost. While default settings served us well when our stacks were first deployed, the possible savings became evident as the volume of telemetry data grew. Thinking about optimisation should not be triggered when a large bill arrives, but rather should be a continual practice as a platform grows and evolves.

The wins in logs (lz4 for Loki chunks), metrics (protobuf encoding in Prometheus Remote Write v2), and traces (zstd for the OpenTelemetry Collector) each followed the same pattern: identify a high-volume data path, challenge the inherited default, validate against your own workload, deploy carefully. Three different stacks, three different algorithms, one discipline. Every high-volume data path in an observability platform at scale is a candidate for this question. With the right measurement tooling, validating the answer takes hours, not quarters.

Transit: Sending and processing less data

Compression reduces bytes per payload. We also found opportunities to avoid moving or processing the same data unnecessarily.

Choosing a Loki query split interval

During incidents or other general debugging, a sluggish log query experience can slow an investigation which extends resolution times and deepens customer impact. Even though we had significantly scaled up the Loki chunks cache component with the intention of improving the user experience, it still fell short. To tackle the issue head-on, we needed to get a clearer picture of where the actual bottlenecks were.

Loki stores log data in compressed chunks, each covering a time span of ingested logs per stream. The time span covered by a chunk depends on ingestion rate — a high-volume stream fills a chunk in minutes, a low-volume stream may take hours. We found that our Loki query-splitting configuration was worth revisiting:split_queries_by_interval was set to 10 minutes, while production p90 chunk-bound duration (measures the time span covered by timestamps within a chunk) was around 50–60 minutes.

With split_queries_by_interval set to 10 minutes, , a query covering an hour is divided into six sub-queries. A chunk whose timestamps span, say, 10:03–10:58 overlaps all six windows. Each sub-query may need to consider the chunk and filter for its own time range, creating repeated work since the same chunk is retrieved and queried multiple times. Whether that means repeated storage reads depends on caching and the query path.

We set the split interval to one hour, close to the observed p90 chunk timestamp span of about 57 minutes and observed fewer redundant reads, lower read-path network utilisation, and a change visible in the xAZ cost dashboards.

Queries now do fewer redundant chunk reads, which improves query performance and keeps costs from rising with query volume.

Query splitting is one lever, but not the only one. We can also tune query parallelism and querier autoscaling to balance response time, resource use and cost. The observability team is also currently working on a data-quality initiative to standardise log fields around OpenTelemetry semantic conventions. Consistent fields should make queries easier to write and results easier to interpret, especially as more log investigations become agent-assisted. A shared schema gives agents clearer, more consistent signals to query, which can reduce ambiguity and help ground their answers. Combined with a leaner network traffic footprint, that should help teams retrieve useful results faster and shorten debugging and incident-resolution time.

An AI powered Investigation: When the Debugger Has the Full Picture

Our logging pipeline involves Fluent Bit acting as an aggregator for writes to Loki. In our production Kubernetes cluster we contended with a bottleneck for some time without really understanding what the underlying issue could be.

This issue presented itself as TCP connection exhaustion in Fluent Bit which was expressed in logs as no upstream connections available. The only other symptom we had to work with was from the metric, node_netstat_Tcp_CurrEstab which increased according to the number of Fluent Bit pods running on the same node, but would plateau at around 2,100. Adding more pods to the same node resulted in more connection errors and ultimately dropped logs. To address the symptom we added topology constraints to prevent scheduling too many Fluent Bit pods on one node so that we didn’t hit the 2,100 supposed limit expressed by the node_netstat_Tcp_CurrEstab metric. This ensured a reliable flow of logs, but resulted in unnecessary waste in the Kubernetes cluster as more nodes were being scheduled to accommodate the topology spread constraints of Fluent Bit.

We looked at this from various different angles, looking at TCP connection utilisation efficiency, EC2 instance network limits, and upstream and downstream constraints but were unable to pinpoint the exact cause until we brought in AI to help with the investigation.

The agent helped to structure the investigation with experiments at each stage to validate if the root cause was determined or not. At each stage the agent could validate the results through our agent integrations to our telemetry stack. The agents were also able to expand the investigation to look more broadly at other components and metrics that weren’t part of the initial scope.

AI wasn’t a miracle cure for the issue, a human still needed to guide the investigation and dig into the detail where required. But where the agent excelled in this instance was in its ability to collect and maintain more context about the issue, and to correlate disparate sources of information to build a full picture of the issue we faced.

We use Cilium for network security in our Kubernetes clusters, and ultimately it transpired that a Cilium Network Policy (CNP) was causing network traffic from Fluent Bit to follow a network path that we didn’t fully comprehend and we were hitting a limit within the embedded Envoy component of Cilium that was acting as a bottleneck on TCP connections.

With the issue understood we could adjust the CNP to remove the bottleneck, and from there we could remove the topology spread constraints for Fluent Bit which allowed our Kubernetes cluster to be scheduled more efficiently.

Ultimately this reduced our scheduled node count by around 15 r8g.16xlarge instances in our production observability Kubernetes cluster.

This is what efficient observability infrastructure enables at its best: the humans, empowered with the tools to ask the right question, quickly, until the answer is clear enough to act on. Engineers debugging faster means customers experiencing fewer delays, fewer failed transactions, and fewer support interactions.

Closing

Together, lz4 for Loki storage, smarter query splitting, zstd for trace ingestion, node consolidation, and Remote Write v2 saved significant sums in annual network costs and compute plus hundreds of terabytes of storage. We achieved this without adding infrastructure or compromising reliability. The improvements stemmed from revisiting past challenges and scaling our solutions to match Wise’s growth.

The customer measure is on Wise’s mission page: over £2 billion saved each year in hidden fees. Platform efficiency directly contributes to that figure — the more affordable the platform, the more we can reinvest in shaping the product our customers truly want. Every dollar we avoid spending on redundant network traffic, half-empty nodes, or oversized storage can support the mission of enabling customers to move their money in a cheaper, faster and more transparent way.

These changes also give product engineering teams a leaner observability stack. They can rely on it to debug faster, understand their services more deeply, and build with confidence. The Platform mission is to build foundations that help teams ship confidently and efficiently. The Observability team’s role is to help those teams see clearly and act quickly.

Compression algorithms, query intervals, scheduler configurations, and protocol versions are ordinary technical choices, but at scale they affect how Wise keeps its promise. As AI-assisted debugging becomes more common, these improvements give the agents a more reliable, responsive observability stack to work with, helping engineers get useful results faster and respond to incidents sooner.

Efficiency, at Wise, is not a constraint. It’s a feature.

Want to know more? Read our engineering stories on Medium, or check out our open roles to help shape the future of our products.

1. https://grafana.com/docs/loki/latest/configure/bp-configure/#consider-the-snappy-compression-algorithm
2. https://github.com/grafana/loki/issues/11247
3. https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#compression-comparison
4. https://prometheus.io/docs/specs/prw/remote_write_spec_2_0/

Observability At Scale — Every Byte Counts was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is Data Inventory?

A data inventory is an extensive catalog of the Wise’s data assets. It helps us understand where the data is located and what kind of security / privacy controls are in place. This is also a requirement for ISO 27001:2022.

What are the constraints?

Any system has to work well with our controls and way-of-working. This led to following constraints:

• It shall be built in a way that satisfies our security controls such as network segmentation, least privilege access, secure secret management, etc.
• It shall be easy to integrate with other in-house platforms to provide cohesive experience.
• It shall be built to connect with heterogeneous and fragmented infrastructure. Many data systems are built through off-the-shelf or managed service offering but others are built & operated in-house.
• The system shall be integrated by default with technology provided by Data Platform. On another hand, it shall be extensible enough to integrate with data systems not maintained by Data Platform.

How did we build our inventory?

At high level, we split this into 4 main outcomes:

1. Register assets at correct granularity.
2. Identify the correct owner and make them accountable for the life cycle.
3. Extract the schema of the data asset.
4. Classify the asset with correct sensitivity.

This is how design looks at high level:

To achieve the above design and outcome, we follow the following steps:

Step 0: Discover data systems network metadata

Scanners need to be aware of the network metadata before doing scanning. For example:

• What is the host or connection string?
• What is the technology of the data system?
• What is the name of the data system?

Let us take an example of how we solve it in our operational databases.

We follow a database-per-service pattern. Operational databases are provisioned by the Database Platform team (DBP) in a semi-automated manner i.e. 1/ product teams request the database through a self-service user interface 2/ DBP apply the terraform plan and run provisioning script to create machine users.

We built a system to discover the databases as and when they are created or deleted.

This enabled data inventory to have all the metadata for scanning the operational databases.

Step 1: Register assets at the correct granularity

Once we have discovered the data systems then the next step is to enumerate the assets and register them at the central repository.

Registering the asset in the inventory has to follow 3 main properties:

1. asset_id shall be self explanatory or human readable.
2. asset_id shall have the right granularity to capture sufficient details.
3. asset_id shall be unique to validate the security and compliance controls.

Format of the asset identifiers

data_asset://{asset_type}/{asset_sub_type}/{top_level_identifier}/{sub_level_identifier}

where:
    asset_type = {MariaDB, PostgreSQL, S3, Kafka}
    top_level_identifier = {database_name, topic_name, bucket_name}
    sub_level_identifier = {table_name}

Example:
    1/ data_asset://mariadb/payout/anomaly
    2/ data_asset://s3/data_inventory
    3/ data_asset://postgresql/db_bank/public/bank_account

Let us take an example of payout database with following asset_id:

asset_id = "data_asset://mariadb/payout/anomaly"
asset_id = 
  data_asset://{asset_type}/{top_level_identifier}/{sub_level_identifier}

where
    asset_type = mariadb
    top_level_identifier = payout
    sub_level_identifier = anomaly (table_name)

Granularity of the asset identifiers

• Lesser the sensitivity, higher the granularity (capture less detail)
• Higher the sensitivity, lower the granularity (capture more detail)

Step 2: Identify the correct owner of data and make them accountable

Data ownership is important for the following reasons:

• Data Owners are aware of the content of the data system and surrounding business context. Hence they can classify the data based on sensitivity and criticality.
• They determine the authorised use, govern access controls, and ensure compliance with applicable laws, regulations, and company policies.
• Owners are the point of contact for cost, management, and lifecycle.

In 90% of the use cases, we can do the ownership attribution through additional metadata provided by product teams. For remaining cases, we came up with rough algorithm:

• Producer: Any service writing the data gets ownership by default.
• Consumer: Any service reading the data gets ownership next (this case can be triggered if producer has stopped writing)
• No consumer and producer: Last producer or consumer needs to drive the deletion of the data system.

Even though the above process has served us well so far, there are still unicorns and we try to keep them under 1% of the population.

Step 3: Extract the schema of the data assets

Schema of the data systems is important to build more granular controls like understanding the data, what columns might contain sensitive data, what data may be the candidate for data deletion, etc.

Extraction of schema for relational data systems is easy. We used information schema for MariaDB and system catalogs for PostgreSQL. We still need to solve the schema understanding for non-relational data systems in a secure manner.

Step 4: Classify the asset with correct sensitivity

As a first iteration, we built a simple user interface for teams to classify their data.

Benefit of the user interface approach:
• We can build one product for all different kinds of data systems and it scales well in comparison to engineering development cost.

Drawback of the approach:
• It does not provide good developer experience as the team has to step out from their development flow.

We have plans to improve the data classification approach which is less disruptive for developers.

Step 5: Notify teams to classify their data assets

In Wise, we have multiple mechanisms to notify engineers about mandatory campaigns or programs. We leveraged 3 layers of notifications: 1/ Soft nudges 2/ Intrusive notifications in Pull Requests 3/ Direct call to action.

We still need to go a long way to notify the teams in an optimal way which does not lead to bad developer experience.

What are the learnings?

• Qualitative feedback is much more economical in platform products than consumer products. Leverage direct feedback or closer proximity with your users.
• When the problem is ambiguous, it is better to solve the problem for 80% of the cases with a two-way door decision and make some practical progress than get stuck in an analysis-paralysis loop.
• Platform products are complex, hence can take a couple of quarters. Leverage stepping stones that deliver concrete value and illuminate “unknown unknowns”.
• We could not use other data catalogs like DataHub, OpenMetadata or BigID as they require lax security controls which does not work with our operating model.

We will be expanding the team to build many interesting products. Join us on this exciting adventure, and be part of an innovative and impact driven team. Check out our open Engineering roles here if you wish to join our team!

Data Inventory was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

As of the financial year 2025, Wise supports 15.6 million active customers, moving approximately £36 billion across borders each quarter. Approximately 65% of our transfers arrive instantly, and our Wise Platform facilitates payments for banks, financial institutions and enterprises globally. This success is driven by our technology-first approach, robust architecture, and dedicated engineering teams.

How we work at Wise

Wise has more than 850 engineers working across key global locations, organised into independent squads and tribes. These teams are empowered to innovate and make decisions independently, fostering transparency, trust, and collaboration.

This article follows our Tech Stack 2022 to cover the most recent improvements in Wise’s tech stack that enable us to achieve our mission of money without borders — instant, convenient, transparent and eventually free.

Moving money with Wise

Our web and mobile applications

Our web applications are built using CRAB (a Wise-specific abstraction on top of the popular Next.js framework) and comprises of 40 distinct apps, each handling specific product functions making deployments safer and more manageable.

One of the biggest changes has been in our testing methodology. We’ve adopted Storybook for visualising individual React components during development. Storybook pairs really well with Chromatic which captures snapshots after each change and can highlight the visual differences in the component. These snapshots are very effective for catching visual regressions during code changes which helps us prevent bugs from reaching our customers.

Wise mobile app: faster, smarter, and more efficient

Our iOS engineers have upgraded our infrastructure by migrating 250+ Xcode modules from Xcodegen to Tuist and switching from Cocoapods to Swift Package Manager (SPM), unlocking improvements in caching for builds. The team also improved flexibility, reducing zero-change build times from 28s to 2s. Development is smoother than ever now with the advanced build caching.

Our Android engineers remain focused on developing apps at scale. Our primary Android repository contains over 300 Gradle modules and roughly 1 million lines of code, comprising 2 production apps, 6 sample apps, 17 JVM modules, 221 Android modules, and 65 multiplatform modules. Our efforts to improve Android development velocity focus on these key themes:

• Using more BFFs to share code logic between Android, iOS, and web teams.
• Development of code generation tools built on KSP.
• Exploring applications of Kotlin Multiplatform.

On the user interface side, we’ve completely moved to Compose — first for our design system, and now for entire screens and navigation. We stay current with technology, quickly adopting Kotlin 2.0 and 2.1 when they were released. For handling asynchronous tasks we use Coroutines and Flows, while our architecture adheres to the standard MVVM patterns and is supported by Google’s Jetpack libraries.

Backend services

Wise runs on over 1000 services in total. In the backend we are primarily using Java and Kotlin. Since our last update, we have focused on automation and efficiency by developing in-house tools that improve development velocity and provide standard libraries to use across different services.

Building great applications faster

Since our last update, we have been focusing on enabling engineering at scale with automated code updates and scalable dependency management solutions. For this, we have:

• Introduced an in-house microservice chassis framework, built on the principle of minimal configuration and shipped as an artifact, allowing us to build standard micro services faster. It configures common capabilities that services use with a recommended defaults: security, observability, database communication, working with Kafka, and more, allowing teams to focus on business logic.
• Improved standardisation of build pipelines through an in-house collection of Gradle plugins. A notable example is our plugin that standardises GitHub Actions workflows. This enables organisation-wide workflow changes through simple plugin version updates, making initiatives like SLSA rollout effortless across our 700+ Java repositories.
• Introduced a language-agnostic automation service that enables us to make complex changes to the codebase at scale and create pull requests for the owning team to review. Using this service, we took our centralised Java dependency management platform a step further by automating dependency upgrades for Java services.

Directly integrating with local payment schemes

We went live with InstaPay, the instant payments system in the Philippines and were granted access to join Zengin, Japan’s instant payments system. We also received access to PIX in Brazil.

At Wise we put substantial effort into creating the most consistent architecture possible, with networking centralised using AWS Transit Gateways. There is substantial variance between the details of the integrations with physical data centres required in the UK, Hungary and Australia. Our Australian data centres were one of the first deployments of AWS Outpost Servers, allowing us to maintain consistent AWS tooling across as much of our infrastructure as possible.

Allowing businesses to use our API

Our public API allows businesses to integrate Wise’s cross-border payment services directly, using secure REST APIs backed by OAuth authentication. This provides businesses with functionality for transfers, currency exchange, and account management, along with comprehensive documentation and developer tools to streamline the integration process.

Wise Platform supports over 40 currencies and multiple payment routes, delivering seamless, globally connected solutions. The platform includes built-in compliance features, allowing for seamless cross-border operations while tapping into Wise’s extensive global infrastructure.

Scaling Wise’s infrastructure platform

To accommodate rapid growth, we’ve focused on rebuilding our infrastructure to ensure efficiency and flexibility while reducing operational burdens on our teams.

Introducing our new Kubernetes-backed Compute Runtime Platform

The Compute Runtime Platform (CRP) is our new, scalable platform leveraging Kubernetes, allowing engineering teams to host applications effortlessly without managing complex infrastructure setups.

Evolving our Kubernetes stack

Since 2018, Wise has relied on Kubernetes built with Terraform, JSONNET, and ConcourseCI to support service-mesh controls (Envoy), PCI-DSS compliance, and frictionless deployments. While this model served us well, we needed a more scalable & standardised approach. This is why we introduced CRP:

• Terraform still provisions infrastructure, but we rewrote our codebase from scratch for flexibility and maintainability.
• RKE2 handles cluster bootstrapping, with Rancher managing overall cluster state.
• Helm replaces JSONNET for better maintainability and upstream compatibility.
• ArgoCD with custom plugins ensures fully automated provisioning & consistency.
• Our Envoy-powered service proxy now includes seamless integration & discovery between services, boosting flexibility, resilience, and oversight across our platform.

As a result, we’ve grown from 6 Kubernetes clusters to more than 20 while keeping maintenance manageable and efficient.

Smarter autoscaling & cost optimisation

Alongside the ability to better provision and maintain our infrastructure, we have also introduced efficiency improvements with CRP:

• We’re building a flexible, opt-in autoscaling solution to reduce cloud costs & cognitive load for teams.
• Automated container CPU rightsizing (via Vertical Pod Autoscaler) is now live in non-production and rolling out to production for non-critical workloads.
• Fully managed sidecar containers (like Envoy proxy) now simplify deployments for product teams.
• Expanding horizontal scaling with KEDA, optimising workloads based on daily & weekly traffic patterns.

The focus on cost optimisation is helping Wise move closer to Mission Zero.

Building a scalable, reliable, and intelligent data infrastructure

A lot of what we do at Wise comes down to moving and making sense of data. Whether it’s transferring funds, updating real-time dashboards, or powering machine learning models behind the scenes, our systems are constantly processing and distributing huge volumes of information. As our global footprint grows, so does our need for faster, more secure, and more flexible ways of handling data. Below is a quick look at how we’re evolving our data tech stack to keep delivering reliable, convenient, and efficient experiences for our customers.

Powering Our Data Backbone

At Wise, our databases are one of the foundations of everything we do, so we’ve invested a lot in making them both robust and easy to manage. Behind the scenes, our database engineers are tackling fascinating technical challenges that push the boundaries of what’s possible in financial data management.

• We’ve worked hard to migrate most of our MariaDB and Postgres workloads off EC2 and into Amazon RDS. This shift has cut down on maintenance tasks, reduced operational overhead, and offered more robust security features.
• Likewise, we’re moving from self-hosted MongoDB to MongoDB Atlas, which frees us up to focus on building new features rather than wrestling with scaling.
• Redis continues to power our in-memory workloads.
• We’re also currently exploring distributed databases for greater relational scalability.

Smarter workflow orchestration & observability

• We have adopted a workflow engine called Temporal to automate critical tasks like switchovers and recovery tests. This helps us keep downtime to a minimum and stay compliant with strict regulations on resilience.
• Tools like RDS Performance Insights and Percona Monitoring and Management (PMM) give us a clear view of how our databases are doing, so we can tackle issues early.
• We’re also experimenting with using direct cloud SDKs to manage our infrastructure — moving away from Terraform Enterprise to simplify our provisioning processes.

Keeping Data in Motion

• Kafka underpins most of our real-time data movement — whether it’s asynchronous messaging between services, log collection, or streaming updates for analytics.
• Our Kafka clusters have grown significantly in capacity, and we’ve introduced features like rack-aware stand-by replicas to boost fault tolerance.
• Our in-house data movement service helps funnel information from Kafka or databases into destinations like Snowflake, S3 Parquet, Iceberg, or other targets.
• Automated checks in the configuration process reduce human error, and its growing usage shows teams are finding it simpler and faster to set up new pipelines.
• Another in-house service, Data Archives, now archives more than 100 billion records across multiple databases. This not only saves on costs but also makes our databases easier to back up and recover.

Turning Data into Insights

Teams across Wise use our Business Intelligence tools to make strategic, data-driven decisions that enhance the customer experience — from fraud detection to personalised marketing and predictive analytics.

• Although we still rely on Snowflake as a core component of our analytics, we’ve been laying the foundations of a Data Lake on Amazon S3 using Apache Iceberg. Thanks to its robust open table format, Apache Iceberg enables us to store huge amounts of data on S3 more efficiently. It lets us modify our table structures without needing to rewrite all the data, which helps our queries run faster and keeps storage costs in check. Plus, its active open source community continuously drives improvements that benefit our long-term scalability.
• Sitting between our data sources and business intelligence tools is Trino, which lets us query Iceberg tables, Snowflake, or even Kafka streams in one place.
• A new Trino gateway handles workload separation and fault-tolerant queries, while complex workflows continue to be managed by Airflow and dbt-core. For an in-depth look at this topic, watch our data engineers’ recent conference presentation.
• Reporting and dashboards are built with Looker or Superset — teams choose whichever toolset fits best.

Driving Intelligent Solutions

Our machine learning architecture is designed to support both exploration and production, seamlessly integrating ML features into products to improve customer onboarding and fraud prevention, and leveraging responsible AI tech.

• Our data scientists work in Amazon SageMaker Studio, choosing either JupyterLab or VSCode to build experiments and explore data.
• Large-scale processing happens on Spark in EMR, while Airflow orchestrates data collection, cleaning, model training, and periodic re-training to keep every step on schedule.
• We use SageMaker Feature Store to keep hundreds of features in sync for both training and inference, and MLflow tracks experiments, metrics, and model versions. This setup simplifies comparing model variants or rolling back if needed.
• When a model is ready for production, we deploy it through an in-house prediction service based on Ray Serve.
• Thanks to MLflow plugins, our data scientists can roll out models with minimal friction — speeding up inference times for fraud detection, KYC, or other use cases where every millisecond counts.
• Automated checks help catch data drift or feature inconsistencies before they turn into serious issues.

Unlocking New AI Capabilities

We’ve created a secure gateway that connects us to multiple large language model providers, including Anthropic (Claude), AWS (Bedrock), Google (Gemini), and OpenAI (gpt and o families). This approach lets us experiment with different models without juggling separate credentials or complex compliance checks. A Python library, inspired by LangChain, wraps these APIs to speed up prototyping.

For cases where we need to reference internal documents, knowledge bases, or user data, we offer a custom Retrieval-Augmented Generation (RAG) service. It pulls the latest information from various data stores before generating responses — a handy feature for summarising complex documents or automating parts of customer service workflows.

Smart Data Management

Our data architecture is both vast and complex, so we’ve built a comprehensive inventory system and a dedicated governance portal to show where data is stored and how it’s classified.

We have automated data discovery across our data estate to know what data is created; who has created it; and what is the category of the data. We are leveraging our inventory of data in data deletion, data compliance and data discovery initiatives. This setup not only supports compliance efforts for audits and regulations but also boosts developer productivity.

With more engineers joining the governance effort, we’re able to roll out stricter policies, enhanced privacy checks, and automated data lifecycle management across the board.

Developer Enablement — Evolving CI/CD at Wise

To strengthen our delivery pipeline and developer experience, we’re continuously evolving our CI/CD platform to empower developers to ship features to customers faster and more reliably than ever before.

CI Improvements: speed and security

The migration from CircleCI to Github Actions brought new possibilities for optimisation. By implementing detailed metrics tracking, we uncovered crucial insights into build performance. For example, by pre-populating caches for frequently used containers, we slashed build times by 15%. At our scale of 500K monthly builds, this translates to over 1,000 hours saved each month.

We’ve been methodically implementing the SLSA framework across our build processes, strengthening our supply-chain security one language at a time.

CD Transformation from Octopus to Spinnaker

Following up on our earlier post about the state of our CI/CD pipeline, our deployment strategy has shifted with the transition from Octopus, our in-house tool to Spinnaker. This wasn’t just a tool swap — it represented a paradigm shift from viewing deployments as simple transactions to seeing them as orchestrated sequences of events.

This transformation allowed us to reduce engineering time spent on deployment management, and to minimise the risk of defects reaching customers. This has increased developer velocity to deliver to our customers much faster, without sacrificing quality and stability.

Advanced canary testing

Spinnaker’s Automated Canary Analysis has become a cornerstone of our deployment pipeline. The process is elegant in its simplicity yet powerful in execution:

• Only 5% of traffic routes to new service versions during testing
• Comprehensive 30-minute analysis of technical and business metrics
• Automatic rollback triggers for significant anomalies

As a result, in 2024 alone this system automatically prevented hundreds of potentially incident-causing deployments and saved thousands of engineering hours.

With over half of Wise’s services now running on Spinnaker and full migration expected by mid-2025, we’re positioned to take the next step: implementing managed delivery to orchestrate the entire SDLC, including testing and data management.

LGTM stack for observability

We have improved our observability ecosystem to better monitor, understand, and optimise the Wise product. The reliability engineers are focused on building a more powerful, efficient, and insightful observability platform that addresses critical challenges in our rapidly scaling environment.

Dedicated observability infrastructure

We have implemented dedicated observability CRP clusters. This provides out-of-the-box observability for services running across different environments. As a result, we have simplified the monitoring setup and reduced manual configuration overhead.

Unified metrics and monitoring stack

To address scalability we have moved over from Thanos to Grafana Mimir. This means that we are now running fully on the LGTM stack: Loki for logs, Grafana for dashboards and visualisation, Tempo for traces and Mimir for metrics. As part of our continuous improvement in observability, we’re pilot testing Grafana Pyroscope for profiling select services, exploring new dimensions of performance insight and optimisation.

Our metrics stack is ingesting ~6 million metric samples per second and processing 150 million active series for our largest metric tenant.

By unifying our stack, we have:

• Standardised observability across our entire technology ecosystem.
• Enhanced correlation between logs, metrics, traces, and dashboards.
• Improved performance and scalability of our monitoring infrastructure.

Cost optimsation and efficiency in observability

Lastly we have continued to invest in optimising our observability stack. We have been able to reduce operational costs, improve resource utilisation and ultimately have a more sustainable long-term observability strategy. Check out our previous article that details some of the work that we did on these initiatives.

This strategic evolution empowers our engineering teams with deeper, more actionable insights while ensuring our observability infrastructure remains both powerful and cost-effective.

Conclusion

In wrapping up, our 2025 tech stack is a testament to how we’re steering the ship at Wise to provide the fastest, most reliable, and cost-effective way to move money for our 15.6 million active customers worldwide. A big focus on standardisation and integration means our systems are built to scale efficiently while ensuring robust risk and compliance management.

Our engineering teams continue to refine our infrastructure across all areas, from mobile and web applications to backend services and machine learning. These efforts simplify and accelerate how money moves across borders, ensuring we’re ready for both current demands and future growth.

We are committed to long-term investments in building the best infrastructure to move and manage the world’s money. With each technical enhancement and new direct connection to payment systems, we’re steadily progressing towards our vision of achieving Money Without Borders.

If you are curious in what we do, check our tech blog on Medium, and check out our open Engineering roles here if you wish to join our team!

I’d like to thank my colleagues who contributed to both the work described in this blog and the writing process itself. Special thanks to my team members and team lead Marta Lima, for the encouragement and for their editorial support; Sten Raudmets for the help with Frontend section; Forrest Pangborn and Dmitry Serov for mobile; Pavel Dionisev for his help on the Backend section; Ed Hargin for sharing details about the Platform Integrations; Adrian Lopez for the content and editorial support on the Data & AI/ML section; Ritesh Modi, Geno Racklin Asher and Telmo Oliveira for the help with the Data Governance & Analytics content; James Bach-Nutman for his content contribution on the Compute platform and Kubernetes; Ervin Lumberg and Doron Solomon for the updates on the CI/CD stack; Massimo Pacher for the overall encouragement and guidance; Zsofia Kiss for the assistance on employer branding and every other person who reviewed the draft and shared their feedback. Your insights and time made this blog post possible!

Wise Tech Stack (2025 update) was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This took us on a journey to understand the technical landscape and challenges around creating a seamless multi-platform solution for interactive 3D animations.

The new illustration set in Wise

Our new illustration set took inspiration from icons, allowing us to communicate with a global audience. They’re unique, bold and dynamic, even when still.

They give us opportunities to share moments of delight with our customers and help our brand stand out from billboard to button.

What sets illustrations at Wise apart is that they’re not static. Each one is paired with a crafted 3D model, featuring detailed UV maps, precise shading, and carefully designed lighting.

This raises an exciting question: What if we rendered these illustrations dynamically in real time, making them interactive on mobile apps?

Native or web view?

To render illustrations in real time, we explored several technical solutions:

• Native
• Filament
• three.js

We opted to create 3D scenes using three.js, one of the most popular open-source web projects. Some of the reasons behind this include:

• Managing native solutions for iOS and Android platforms would demand extra engineering effort.
• The support and documentation available for three.js surpass others.
• The web view solution delivers native-like performance by using WebGL on both platforms. As a result, we consistently achieve 60/120 FPS across all scenes on both devices.

The plan was simple. Build simple HTML files that render 3D scenes and load these files locally in mobile apps 🚀

But we had a “huge” problem…

The 3D models were specifically designed for static illustrations, resulting in a high polygon count that contributes to larger file sizes. For instance, the glTF file for the plane model is approximately 19.3 MB. In total, the original file size for all the 3D scenes we support was around 324 MB.

To make these files suitable for mobile apps, we had to optimise and compress them extensively.

Optimisation #1 — Remove duplicate images

The first optimisation we have to remove the duplicate textures in glTF files.

Both padlock and globe scenes use the common blue texture. If we loaded glTF files directly we would duplicate the texture in mobile app bundle.

To address this issue, we followed these steps:

1. Extract the embedded images from the glTF file.
2. Eliminate any duplicate images.
3. Recreate each glTF file, ensuring that it references the image files correctly.

As a result, we successfully reduced the overall asset size from 324 MB to 98.4 MB. While this was a significant improvement, we were confident that we could achieve even better results!

Optimisation #2 — Compress textures

The original textures extracted from the glTF files were huge. To enhance performance and to reduce the bundle size on mobile apps, we prioritised optimising textures over maintaining high-quality versions. Therefore, we were able to significantly compress the textures.

As a result of compressing textures, we managed to further reduce the overall asset size from 98.4 MB to 5.2 MB. While this was impressive progress, we knew we could do even better!

Optimisation #3 — Compress geometry data

The last step was to compress geometry data and to reduce the glTF file size. We compressed each model using an open source library Draco from Google.

Now with this final optimisation, we successfully reduced the overall asset size from 5.2 MB to 3.7 MB.

324 MB > 3.7 MB

After implementing all optimisations, the final mobile bundle size was reduced from 324 MB to 3.7 MB!

This new size was a significant achievement, given the mobile bundle consists of glTF model files, textures, JS libraries (three.js and draco.js) and HTML files.

All of these optimisations are fully automated through a dedicated internal repository that houses our entire optimisation pipeline. This setup enables us to add new 3D scenes more efficiently and quickly.

Displaying 3D scenes on mobile apps

Each 3D scene is contained within a standalone HTML file, which loads JavaScript libraries and assets locally. These scenes are then displayed in mobile apps through web views.

By rendering scenes on WebGL using three.js, we achieve a smooth 60 to 120 FPS, ensuring excellent performance on modern mobile devices without any issues.

One of the biggest challenges on both platforms was the delay in loading scenes. Since we heavily compress models using the Draco library, decompression takes time. To address this, on both platforms, we pre-load the scenes, store them in memory while rendering is off. This approach ensures instant loading times for a seamless user experience.

Both iOS and Android platforms have abstracted the integration of these 3D scenes. When a mobile engineer wants to use a 3D scene, they simply specify the desired 3D scene, and our UI components handle the rest automatically.

Interactivity on mobile apps

HTML files featuring 3D scenes also support interactivity, offering APIs to control various aspects such as starting or stopping rendering, managing animations, and adjusting rotation.

Mobile apps gather rotation data using their internal gyroscope natively and send this information to the HTML files, enabling the animation of rotation in the 3D scene.

And that’s it! This is how we turned a simple idea into an actual feature and made our illustrations 3D interactive!

A big thank you to Chao Quan Choo and Ness Grixti for their hard work and dedication in pushing this project forward and making it happen!

A big shout out to the folks at Ragged Edge for their amazing work in designing the illustrations!

Onwards 🚀

How did we build 3D animations in mobile apps? was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Intro

This technical blog article dives deep into a compaction bottleneck that was silently impacting our metrics infrastructure after we migrated from Thanos to Grafana Mimir. Follow along as we detail the investigation, solution, and the remarkable performance and cost improvements that transformed our metrics pipeline.

To give readers some context on the scale of our operations at Wise, our production Mimir stack is currently ingesting approximately a total of 6 million metric samples per second across different tenants at the time of writing — a testament to the critical role it plays in our observability infrastructure.

In the world of observability, managing large-scale metric storage systems can be a challenging endeavour. Recently, Wise made a significant shift from Thanos to Grafana Mimir, a decision driven by our growing need for better scalability, feature richness, and operational simplicity. The switch has already yielded several benefits, but a critical issue with Mimir compaction brought new challenges to light. Once resolved, this fix not only improved system reliability but also delivered substantial cost benefits in terms of S3 storage savings, cross-AZ network transfer reductions, and query latency improvements.

Here’s an overview of our journey — from the switch to Mimir to resolving compaction issues and reaping the rewards.

Why We Switched from Thanos to Mimir

Thanos had served us well as a metrics storage and query system, but as our use case evolved, we began to encounter limitations:

1. Scaling Challenges: Managing Thanos components at scale — especially with the growth in the volume of metrics data — became operationally complex.
2. Retention and Compaction Flexibility: Thanos’s approach to compaction and retention felt rigid for our growing requirements.
3. Feature Set: Mimir provided robust multi-tenancy capabilities, advanced query performance optimisations, and seamless integration with our existing workflows.

Benefits of Switching to Mimir

Since adopting Mimir, we’ve realized several advantages:

• Operational Simplicity: Mimir’s architecture reduced the number of different components we had to manage, making operations more straightforward.
• Improved Query Performance: Features like sharding and efficient query path optimizations resulted in faster query response times — 75% reduction in average query latency during peak usage.
• Seamless Multi-Tenancy: Native support for multiple tenants aligned perfectly with our requirements for isolating workloads across teams.
• Advanced Retention Policies: Mimir’s retention mechanisms allowed us to fine-tune data retention based on tenant-specific needs, reducing unnecessary storage.

Despite these benefits, the shift to Mimir wasn’t without its challenges. One major issue arose around compaction, which became a priority to address.

Problem & Symptoms

After transitioning to Mimir, we noticed the following critical operational inefficiencies that needed attention:

1. Query Performance: Increased error rate(5XXs) and latencies for long time-range queries, even though we have extensive caching in place.
2. Resource Usage: Mimir read path components (store-gateway, querier etc) demanding more and more compute resources.
3. Cross-AZ Cloud Network Costs: Elevated data transfer between availability zones hinted at inefficiencies in data processing.
4. S3 Storage Costs: Storage utilisation was increasing disproportionately, with redundant data lingering in object storage.

These symptoms did not initially suggest that Mimir compaction was the bottleneck, nor did it cross our minds as the likely cause. However, during troubleshooting, we came across a comment in one of the GitHub issues¹ flagged for Mimir. This comment provided a key insight into what the potential issue could be and became a turning point in identifying the root cause.

Detection

The first signs of the problem emerged through Mimir monitoring metrics, prompting us to dig deeper. We visualised the volume of non-compacted data being queried via the store gateways, and the numbers were staggering — our stack was handling around 250k blocks during the period of the issue.

Additionally, we noticed a clear trend: the number of blocks grew proportionally with the increasing volume of metrics data and the scaling efforts we made to improve the throughput of the ingester component on the write path.

For context, it is the compactor’s responsibility to merge and de-duplicate the blocks shipped by the ingester replicas. This insight pointed directly to compactor performance issues as the root cause of the growing inefficiencies.

Root Cause

We rely heavily on the mixin dashboards, alerts and runbooks³ provided by Grafana for monitoring our Grafana components. These tools have consistently served us well, offering robust insights and timely alerts. However, in this specific case, we did not receive any alerts indicating that the compactor was underperforming, nor did the dashboards provide clear clues about the situation.

This lack of direct signals meant we had to dig deeper to understand why the compactor was not performing as expected.

The Grafana Mimir official docs and runbooks² proved invaluable during this process. It provided detailed explanations of how Mimir compaction works³ and offered guidance on how to tune the compactor³ for better throughput and performance. This deeper understanding was crucial in helping us identify the root cause and implement the necessary fixes.

“A tenant should not have a steadily increasing number of blocks. A pattern of growth followed by compaction is normal. Total block counts can also be examined but these depend on the age of the tenants in the cluster and sharding settings. Values from <1200 blocks upward could be normal. 50K blocks would generally not be normal.”²

Our situation mirrored the challenges described above. In our case, compaction was functioning well for all the smaller tenants, but it struggled significantly with our largest tenant. Over time, this led to a backlog of un-compacted blocks spanning over two months.

As a result, the number of blocks in object storage kept growing steadily, exacerbating the performance and cost issues we were facing. This imbalance between smaller tenants and the large tenant made it evident that the compactor’s performance was not scaling adequately with the increased workload.

Underperforming — why ?

As we were still learning the intricacies of the new tool, the compactor component was somewhat overlooked during our efforts to stabilise the stack for writes and reads. Our focus had been on ensuring reliable ingestion and query performance, inadvertently leaving the compactor’s optimisation lower on the priority list.

This oversight became evident as the system scaled and the compactor struggled to keep up with the demands of our largest tenant, highlighting the importance of a holistic approach when adopting new tools.

The late detection of the issue was largely due to a lack of relevant alerts. While our monitoring stack provided visibility into many aspects of Mimir’s performance, there were no specific alerts configured to signal compactor underperformance or a growing backlog of un-compacted blocks.

This gap in alerting delayed our ability to identify and address the problem early, emphasising the need to refine monitoring and alerting configurations to cover all critical components comprehensively.

The compactor metrics and logs provided a clear indication of the issue: the synchronisation of block metadata was taking significantly longer to complete for the large tenant. This delay was a direct result of the accumulation of an excessive number of blocks in the object storage.

Whenever a compactor instance starts, it performs a metadata synchronisation by scanning all blocks in the object storage. Based on the synchronised metadata, it determines the compaction jobs to execute and then proceeds to run them.

In our case, the huge backlog of blocks for the large tenant caused the metadata synchronisation operation to take approximately 9 hours to complete. This prolonged synchronisation process significantly delayed compaction tasks for the tenant, compounding the issue over time and further increasing the backlog.

We also realised that the default settings for the compactor were not sufficient for our use case, particularly for the large tenant. The configuration was not tuned to handle the high volume of blocks efficiently, and as a result, some compaction jobs were being skipped.

The Fix

Once we detected the root cause the solution to the problem was more or less straight forward. To resolve these issues, we implemented several targeted solutions:

1. Resource Optimisation: Allocated additional CPU and memory to the compactor, ensuring it could process workloads efficiently.
2. Configuration Adjustments: Next step was to tune the compaction config parameters³ to help clear the backlog and optimise for our setup⁴.
3. Trial and Error: Experimentations in staging (where we had the same issue) & promote whatever worked to production environment.
4. Monitor & Repeat: Test, monitor, get feedback and repeat.

After several days of careful adjustments and patient monitoring, we finally began to see some positive results. The most notable sign of improvement was the significant decrease in the total number of blocks stored in the object storage.

This reduction indicated that the compaction process was finally catching up, processing and merging the accumulated blocks effectively. The backlog was starting to clear, and the system’s performance and storage utilisation began to stabilise, validating the changes we had made.

A few days and a hundreds of compactions later, we began to see a clear improvement. The overall block count in the object storage started to plateau at a stable number, indicating that the backlog was finally under control and the compaction process was functioning as expected 😌.

This stability was a major milestone, confirming that the adjustments made to the compactor settings and the system’s configuration had successfully addressed the underlying issues. The system was now in a steady state, and the efficiency gains were becoming evident.

Impact & Gains

A summary of the impact that was observed and the key gains are summarised below.

Cost Savings

• S3 Storage: Efficient compaction and retention reduced storage usage by 70%, directly lowering costs.

• Cross-AZ Transfers: Optimisations led to a 40% reduction in network traffic between availability zones, significantly cutting cloud networking expenses.

• Compute Efficiency: More efficiency in metrics stack, by enabling us to significantly scale down the memory requirements for some of the Mimir read path components especially store-gateway and queriers

Performance Improvements

• Query Latency: Query response times improved by an average of 90%, providing a smoother experience for end-users.

• System Reliability: With compaction running smoothly and consistently, unexpected failures and latency spikes were eliminated.

Operational Simplicity

• Fewer anomalies and clearer metrics made it easier to monitor, maintain, and scale the system. This experience also brought valuable learnings and a deeper confidence in operating Mimir at scale. We gained a better understanding of how to fine-tune the compactor and other components to handle large datasets more effectively.

Key Takeaways

Before ending this blog post we would like to highlight some of the key learnings that we had from this exercise and also the actions that we are taking to make our metrics stack more robust, resilient and reliable.

• Learnings:
  - The importance of monitoring block growth and compaction behaviour.
  - We should do a periodic review of optimised resource allocation for all the components.
• Ongoing efforts:
  - Prevent recurrence of such issues by implementing relevant alerts for early detection and proactive monitoring.
  - Extend Grafana Mimir runbooks for our setup.
• Future Actions:
  - Check-box exercise against Grafana Mimir production⁵ tips recommendations.
  - Apply the key learnings from this experience to other Grafana products with similar architectures that we maintain, such as Loki, Tempo, and others.

Conclusion

Our transition from Thanos to Grafana Mimir was motivated by a need for better scalability, flexibility, and operational simplicity. While the journey wasn’t without challenges, addressing the compaction inefficiencies reaffirmed the value of the switch. The resolution brought measurable cost savings, improved reliability, and enhanced performance, reinforcing Mimir as the right choice for our growing observability needs.

If your organisation is considering a similar migration or facing similar engineering challenges with metric storage systems, feel free to share your experiences or reach out for a discussion.

If you are interested in learning more what we do, check our tech blog on Medium, and check out our open Engineering roles here if you wish to become a part of our team!

[1] Gaps when displaying metrics in Grafana · grafana mimir · Discussion #4696

[2] Grafana Mimir runbooks & mixins
Grafana Mimir runbooks | Grafana Mimir documentation
https://github.com/grafana/mimir/tree/main/operations/mimir-mixin

[3] How Grafana Mimir compaction works.
Grafana Mimir compactor | Grafana Mimir documentation
How Grafana Mimir’s split-and-merge compactor enables scaling metrics to 1 billion active series | Grafana Labs

[4] Scaling / Configuration recommendations for compactor · grafana mimir · Discussion #3369

[5] Grafana Mimir production tips | Grafana Mimir documentation

Grafana Mimir Compaction: From Bottleneck to Savings was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Wise we’re building the best way to move and manage the world’s money — making it faster, cheaper, convenient and transparent. Product uptime and availability is contributing to three parts of our mission:

• An honest and meaningful uptime report is key to drive transparency of our systems.
• High availability is one of the greatest factors in making our product convenient. It is very hard to be convenient if the product is not working as designed.
• High availability means less retries and less waiting, driving us closer to instant too.

In this post we will talk about the reporting aspect of uptime and availability, focusing on transparency. We would like to share our journey and various technical considerations towards measuring meaningful availability/uptime for Wise.

Wise has a wide range of product offerings, condensing the availability or uptime of our business flows is not trivial. These flows consist of multiple business operations with various branches depending on user intent or group.

Providing a single number as business flow availability or uptime is always going to be an approximation, but we can and should be transparent about how we aggregate and what that represents.

Our goal is to have a meaningful measurement we can get behind as representative of user-experienced uptime/availability.

Definitions

First things first, let’s define some terms we’ll use:

• Business flow: A high level, complex use-case of the Wise product, e.g. “send money” (we aim to measure the availability of this, as this is the most meaningful for the end-users)
• Business operation: an individual operation a user can perform as part of one use case or business flow. A high level piece of functionality e.g. get a quote, set up a transfer, pay in for the transfer
• API endpoint: a technical term. Which HTTP(S) API to invoke in order to perform a Business operation. Not a service or a series of calls, a single HTTP(S) endpoint to perform a (part or whole) of Business operation.
• Availability: A ratio of successful operations (i.e not server errors) to total occurrences for a given time-frame. E.g. 9943/10000 = 0.9943
• Downtime: Number of seconds the application produces errors, behaves erroneously, or is not reachable.
• Uptime: Ratio of number of seconds the application did not produce errors, or behaved erroneously to the total number of seconds of the observed period.

Uptime vs. Availability

People often use these two terms interchangeably. However, they mean different things. Let’s build from the definition:

Availability and uptime are both a ratio (%) of “good to total”.

However it is very important to recognize that their viewpoint is totally different:

Availability is representative of the share of successful operations (regardless of their distribution in time).

Uptime is representative of the share of time where we could successfully complete operations (regardless of how many operations were actually performed).

Naturally if we have a perfectly even traffic pattern (e.g. exactly one operation attempted every minute all the time forever) uptime and availability will be the same nominal number.

In real-life application traffic is almost never perfectly even. There are daily/weekly/monthly (and so on) seasonalities, and spikes and valleys in traffic. If we measure the real traffic of a real application, its uptime and availability will never be the same nominal number! They will represent the same reality, but from different perspectives.

In general, if outages happen during high-traffic times then uptime will be a higher number, whereas if they happen during low-traffic times availability will be higher (for the same outage).

For example given the following operations/errors distribution:

• Downtime: 100 seconds
• Total time: 420 seconds
• Successful operations: 30
• Total operations: 44

Gives availability of 30/44 = 68.2% and uptime of (420–100)/420 = 76.2%

But if the same downtime happened over a different traffic distribution:

• Downtime: 100 seconds
• Total time: 420 seconds
• Successful operations: 33
• Total operations: 40

Gives availability of 33/40 = 82.5% and uptime of (420–100)/420 = 76.2%

There is no right or wrong choice, but given that people often turn ratios into “downtime seconds” in their heads, we believe it is better to report uptime to support this subconscious translation.

Simplifications

Rigorously pairing all operations to users performing them, ordering them and painting the various user journeys (which are different for each user), and deciding success or failure of these journeys is a complex and resource intensive endeavour — especially for a large user base, and a large set of product offerings.

Instead, approximating this value based on general user behaviour is much more feasible and can still produce meaningful results. In general, this is what the industry does. Not everyone is going for this much detail, but we believe in transparency and honesty even if it means we report “less nines”. We measure key business operations performed by our full user-base and use that information as a proxy to our overall business flow availability.

Ways to measure business flow uptime

Synthetic tests

A common practice to measure a flow is to set up an automation that behaves as a real production user, and performs a series of predefined set of business operations (the user journey) and log the outcome of the journey as a whole.

This approach is very easy to set up, and the frequency of these test runs are completely in our control. We can set up a perfectly even test schedule (e.g. once a minute) so it can report both uptime and availability (as with the perfectly even traffic they are the same nominal number). All in all easy to set up and report. Wise has been using such black-box approach for years.

The black box test served us quite well when we had a simpler product offering (one basic journey of sending money).

However convenient it was, the black box testing approach has shortcomings:

1. Does not scale well for multiple business flows (or multiple branches of a flow), as if we ramp up variations (and leave frequency intact) we’d generate so much load on our system for just the “test” that our actual users become negligible load. This is one of the reasons why some other companies (e.g. Google) are also moving away from synthetic monitoring and gravitate towards real user monitoring.
2. When the product improves, APIs change. So the test needs immediate upgrade. The test can also break when APIs are retired or changed around. This can be manageable so long as there are only a few of these blackbox tests (ties back to scaling issues).
3. When the tester tool is down we cannot calculate or backfill data for the lost time period (that we need for accurate reporting).

This has happened a few times with Wise. Some internal rules of some internal API had changed, and suddenly we needed an extra step in the user journey to fulfil strong customer authentication. The test itself was down. Real users did not experience downtime, but our tooling suggested so. In these cases we had to choose from the following options:

• Assume that a few minutes (or hours) never happened as we have no reliable data
• Assume we were “up” based on user contacts and logs of real user interactions
• Report outrageously low numbers even though the system was up, because the measurement tool was down.

Usually we opted for the first option and had a small asterisk in the report warning about resolution issues. Then we would run our incident process to make sure we catch these earlier next time without user impact. Then, next time, something completely different broke the test without user impact. As we got more and more processes around our tests, these got less and less frequent but still happened. New errors always find a way.

Test users are often handled differently than real users by backend applications.

In our example we had a test user (in the live production database) that was creating 1000+ transfers — every single day. Some of our KYC (Know Your Customer) and AML (anti-money laundering) systems regularly look for suspicious transactions and behavioural patterns to help prevent malicious actors misusing or taking advantage of our systems. The test user had to be exempted, as it would’ve tripped all the alarms. So in this sense our measurement (in order to keep the company running) was no longer doing the exact same thing users do… It did most of it, but had to pass fewer checks. What if those checks we skipped had bugs? We’d miss reporting their downtime. So for these reasons we turned away from synthetic availability/uptime measurements.

White Box testing: track business operations of real users

These shortcomings above made us consider turning away from synthetic load and blackbox testing to measure organic load of real users. This approach is more representative of user-experienced availability and uptime, and scales much better for our vast product offering. However it comes with its own trade-offs.

Traffic is no longer perfectly even, as people tend to move money in their business hours, so we have quite the seasonality depending on which part of the world is in office hours at any given time. So uptime and availability is no longer the same nominal number.

Measuring uptime with real world traffic is not as straightforward as availability.

As discussed above, given that people often turn ratios into “downtime seconds”, we believe it is better to report uptime to support this subconscious translation. In the next sections we will focus on getting uptime/downtime from real user actions.

Most real users interact with our API through our clients, which have aggressive retries and fallbacks to make their experience smooth even if the backend is having issues. Transient API errors are not reflected in our website/mobile app experience. However this is very hard to factor in. From our backend’s perspective, retries are independent requests; one fails, the other succeeds. We also have a large customer base who are directly consuming our production API (Wise Platform is used by many partners across the globe), so we cannot rely on their retries.

All in all, counting even the transient errors is accurate for direct API customers, and even though it is a less nominal uptime than the ones our web and mobile users actually experience it is a sufficient “at least this good, probably better” number

Downtime vs. approximate downtime of operations

Calculating real downtime from real (non-generated) traffic needs us to enumerate all requests and count the times elapsed between first failed and the next successful request. This is only easy if you have a single queue of requests. Distributed systems designed to be redundant and reliable are just not that simple — they are never “completely down”: some parts might work while others don’t at the same time.

A simple way of approximating downtime is to aggregate away the traffic-pattern of availability. This will not produce the exact same result, but is much easier to calculate and reasonably similar.

Given a short time window (in Wise, 1 minute) we can consider the traffic “constant”. With that assumption we can represent the uptime of an API endpoint for a short period by the availability of it. To get its downtime, we simply subtract the uptime from the window length itself.

The smaller this time window is the better the approximation. 5 minute window is commonly used. Wise used 2 minutes for a few months, and then we have improved to use 1 minute windows.

From there, to get the downtime of a meaningful period we simply sum up the downtime of the small windows the period consists of.

Aggregate downtime of operations to flows

As described above we have a good way of approximating the downtime of each and all important business operations (that each have a single API endpoint we can measure the availability of). The general public (customers, auditors, stakeholders) want a “Single number” representing the full business flow uptime — not 8–10 uptimes for the operations it consists of. To get that reportable number we need to make a trade-off:

First of all let’s declare what do we mean by “down” for a business flow: The business flow is down if at least one of the operations it consists of is down.

Option one: “the sum of downtimes of all operations in a business flow is the overall downtime of the flow”.

This assumes that each operation was down at strictly separate times. I.e. at all times there were exactly 1 operation down.

This is a good “worst case” scenario. It is the equivalent of “at least this good, but probably better”.

Option two: “the highest of downtimes of all operations in a business flow is the overall downtime (i.e. minimum uptime represents all)”.

This assumes that all operations were down at exactly the same time. No exceptions. Either everything worked or nothing.

This is a kind of “best case” scenario, and for simpler business flows it might be the closest to the truth. Engineers don’t really like it as it is very rare that everything is down. Mostly this is too kind to the business. It is the equivalent of “we were down for at least this amount of time, and maybe some more, but this is the biggest contributor”

Option three: “the average of downtimes of all operations in a business flow is the overall uptime”.

This aggregation tries to make up outliers by averaging them out. It is used by some key players, e.g. Revolut does “Daily uptime is defined as the percentage of minutes in the day where the average technical error rate was below 5%.”

They do an extremely detailed breakdown though and they are transparent about this averaging.

At Wise we believe it is always too kind to the business to do averages of business operation uptime/availability for operations that are part of the same business flow, as it will always be more “up” than option one, which we know is a good minimum.

Option four: “a middle point of the known good minimum and known good maximum”

I.e. We say we are sure the downtime of a business flow is always between the sum of its operations (option 1) and max of them (option 2). So we get the middle point (average) of this number, and highlight our confidence interval.

Example:

Uptime of business operations for 2024 Q2:

• Minimum uptime of any single operation: 99.958%
• Sum of all downtimes: 7514.5 seconds (There were 7862399 seconds in 2024/Q2) making uptime 99.904%

So we know that our API customers were able to complete the full flow between 99.904% and 99.958% in 2024/Q2.

The middle point of it is 99.931% (with +- 0.027% confidence band)

How Wise will report business flow uptime from now on:

• We will shift where we can to report uptime, not availability
• We will report our API uptime regardless of our client-side web/mobile retries as we have a large direct API customer base and deciding a threshold of transient issues would just cause more confusion for a slightly greater nominal uptime value that only applies to some of our customers.
• We track our uptime per business operation
• We aggregate that number for business flows:
• We add all downtimes of all operations of a flow to get the worst possible value (assuming exactly one of them were down the same time)
• We choose the maximum downtime of a single operation (assuming all of them were down the same time)
• Then since neither exactly 1, nor all of our operations are down at the same time, we average these numbers to get a middle point and report the difference between min/max and average for transparency.

Onwards.

Measuring meaningful availability / uptime of Wise was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the Wise Cloud Platform squad we take resiliency, capacity planning and costs seriously. We are always looking for ways to improve how services running on our self-managed Kubernetes clusters in AWS can be made more resistant to system failures without significantly increasing cost.

One of the mechanisms we use are Pod Topology Spread Constraints. This is a built-in Kubernetes feature used to distribute workloads across a topology domain, such as Availability Zones or Nodes.

In this blog post, we consider Availability Zones to be our topology domain. We discuss how to configure Pod Topology Spread Constraints correctly and how a naive configuration creates unexpected Pod skew.

If Pod Topology Spread Constraints are misconfigured and an Availability Zone were to go down, you could lose 2/3rds of your Pods instead of the expected 1/3rd. This will likely negatively impact your customers' ability to use your product.

What is wrong with Kubernetes Pod Topology Spread Constraints?

Skew is the difference in the number of pods between the most populated and least populated Availability Zone. If we have 3 Availability Zones and 3 Pods, ideally we want a skew of 0, as seen in the diagram below.

If we were to add another pod, it could be placed in any of the Availability Zones and our pods would still be distributed as evenly as possible. Our skew would then be 1.

If the pods are not evenly distributed, the skew will always be greater than 1. In the case of the diagram below, Availability Zone 1 contains 2 pods and Availability Zone 3 contains zero pods, so the skew is 2.

You can configure the Pod Topology Spread Constraints through the topologySpreadConstraints section of a Pods’ spec. The Pods to which the configuration is applied is determined through LabelSelectors and what you deem an acceptable distribution is determined by the maxSkew field.

Below is an example of a naive topologySpreadConstraints configuration:

apiVersion: v1
kind: Pod
metadata:
 labels:
   app: example-service
spec:
 topologySpreadConstraints:
   - labelSelector:
       matchLabels:
         app: example-service
     maxSkew: 1
     topologyKey: topology.kubernetes.io/zone
     whenUnsatisfiable: DoNotSchedule

With this configuration, if we have at least 3 pods, we expect to have a minimum of 1 Pod running in each Availability Zone and our skew to never exceed 1.

To understand why we saw a deviance from what was expected, we need to explain how scheduling occurs in Kubernetes. A component called kube-scheduler decides which Nodes a Pod will be scheduled on, and a configurable set of rules informs where a pod is allowed to be scheduled. However, the problem is that these rules consider the state of the cluster at the point in time when scheduling occurs. Kube-scheduler will not rebalance pods after they are scheduled or consider the rules when removing pods.

A rolling update of a Deployment in Kubernetes will create a new ReplicaSet version and schedule new Pods and terminate the old Pods. The exact way this rollout happens is configurable, but let us assume we want to schedule 1 Pod at a time. Kube-scheduler will schedule a new Pod and the Deployment Controller waits for this Pod to report itself ready. Once ready, one pod from the old ReplicaSet is terminated and the Deployment Controller requests that another Pod from the new ReplicaSet is scheduled. This process repeats until all Pods are rotated. Note that the termination of Pods is not intelligent in any way; one is chosen at random.

As LabelSelectors are used to define which pods to apply Pod Topology Spread Constraints to, if you use only the app name or something similar the kube-scheduler will consider Pods from both old and new ReplicaSets. This makes it possible for a Pod to be scheduled in a way which violates the given Pod Topology Spread Constraints once a Pod from the old ReplicaSet is terminated.

Let’s consider an example where we are in the middle of a rollout. A new Pod has just been scheduled and reported ready, and an old Pod terminated. Kube-scheduler is now deciding which Availability Zone it can place the new Pod:

According to kube-scheduler the current skew is 0, and it is allowed to place the Pod in any Availability Zone without violating the Pod Topology Spread Constraints. If it places the Pod in one of the Availability Zones that already contain a new Pod, we will end up in a state that violates the constraints once the remaining old Pod is terminated.

The Solution

To prevent this issue, we need kube-scheduler to distinguish between old ReplicaSet Pods and new ReplicaSet Pods. As Pod Topology Spread Constraints use LabelSelectors we can ensure the old and new Pods have a unique label in the LabelSelector of our topologySpreadConstraints configuration.

To do this, we created a mutating admission webhook which adds a label, topology.wise.com/id, to the Pod metadata. The value of this label is a uuid. The mutating admission webhook updates the Pod Topology Spread Constraints LabelSelector accordingly whenever a Deployment object passes through the Kubernetes API.

If our example configuration were passed through this webhook, the result would be:

apiVersion: v1
kind: Pod
metadata:
 labels:
   app: example-service
   topology.wise.com/id: <uuid-value>
spec:
 topologySpreadConstraints:
   - labelSelector:
       matchLabels:
         app: example-service
         topology.wise.com/id: <uuid-value>
     maxSkew: 1
     topologyKey: topology.kubernetes.io/zone
     whenUnsatisfiable: DoNotSchedule

Now, kube-scheduler will not consider the Pods from the old ReplicaSet when scheduling the new Pods.

Alternative Approaches

Instead of using a mutating webhook, it would have been possible to update tooling to edit the manifests, adding the unique label and associated LabelSelector, before they reach the Kubernetes API. This could have saved us having to deploy and monitor a new component. But to do this we would have to change how deployment manifests are created in every place they are generated, which is not scalable.

It is also the case that Kubernetes v1.25 adds a matchLabelKeys field in alpha to the Pod Topology Spread Constraints that can be enabled using a feature gate. Kube-scheduler will assume the values automatically so you can take advantage of the existing pod-template-hash label to achieve the same behaviour we do with the mutating admission webhook.

Both solutions described will not protect you against imbalances after scaling down a deployment. If you regularly scale down workloads, you may want to consider using a tool like descheduler.

You could also consider rearchitecting the way you deploy workloads, such as doing a deployment per Availability Zone or even running a cluster per Availability Zone.

Summary

Out of the box, Pod Topology Spread Constraints are not enough to ensure even Pod distribution. LabelSelectors that capture Pods from both old and new ReplicaSet versions will cause your availability expectations to be violated. If using Pod Topology Spread Constraints to distribute Pods through Availability Zones, this could mean that in the event of an Availability Zone failure, 2/3rds of a service’s pods are lost.

We solved this by implementing a mutating admission webhook to add a new label to Pods controlled by a Deployment. This label has a unique ID as a value per ReplicaSet and is referred to in the LabelSelectors of the Pod Topology Spread Constraints configuration. This allows kube-scheduler to distinguish between old and new Pods, preventing us from ending up in an undesirable state.

If you enjoyed reading this post and like the presented challenges, our Platform Engineering team is hiring! Check out our open Engineering roles here.

Avoiding Kubernetes Pod Topology Spread Constraint Pitfalls was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Deploying your deployment tool

In our 2 part series, we discussed the state of our CI/CD systems and wrapped up with our vision for CD at Wise, and why we chose Spinnaker. Here’s how we set up Spinnaker from scratch, overcoming the technical challenges of bootstrapping a CD platform.

We wanted a hands-off and automated way to bootstrap Spinnaker. When researching across industry peers, one approach was to use a staging instance of Spinnaker to manage your production instance. This method of dual Spinnakers, wasn’t viable for Wise, due to security concerns with a staging tool having access to change a production environment. It also meant accepting a potentially manual error prone process when managing your staging setup. Instead, we wanted to achieve a fully automatic installation for both of our Spinnaker environments and settled on using ArgoCD.

Spinnaker & ArgoCD

Spinnaker consists of around 11 Java microservices, which operates on a centralised deployment model — push based deployments. It utilises pipelines heavily as its deployment strategy, which can be triggered via various input methods e.g. manual, artefact upload etc. Implementing deployments as pipelines allows engineers to describe a release as a set of steps, with powerful concepts such as canary analysis and quality gates built in. This is attractive for Wise, as use of standardised and reusable templates can enforce end consumers to follow a particular paved road e.g. canary deployments with automatic rollbacks.

ArgoCD describes itself as “declarative GitOps for Kubernetes”. It comprises fewer golang microservices and can be configured to be either centralised or decentralised. By default, ArgoCD differs by not utilising pipeline based deployments, instead operating via a GitOps reconciliation loop as its source of truth. Whatever is defined in Git, ArgoCD will try to reconcile within the cluster. ArgoCD is Kubernetes native and therefore can’t be used for deploying infrastructure to other cloud platforms, e.g. VMs, serverless etc.

Making Spinnaker installation repeatable, toil free and quick

Each of our new Kubernetes clusters come with an ArgoCD installation. ArgoCD is relatively standalone, installation is done via Helm charts and can be bootstrapped easily.

To manage our Spinnaker environment we’ve adopted Argo’s App of Apps pattern — think of it as one master application that instructs Argo to deploy other child apps.

The below diagram illustrates our setup.

Contained within our app-of-apps are several other ArgoCD Application definitions — these can either point to plain manifests, Helm charts and Kustomize templates etc. We use Kustomize to apply application/environment specific configuration changes. Here’s an example of our Kustomize definition for Spinnaker.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: kustomize-spinnaker
 namespace: argocd
 finalizers:
 - resources-finalizer.argocd.argoproj.io
spec:
 destination:
   namespace: spinnaker
   server: https://kubernetes.default.svc
 project: default
 source:
   path: manifests/kustomize/spinnaker
   repoURL: <repo-url-here>
   targetRevision: main
 syncPolicy:
   automated:
     prune: true
     selfHeal: true
   syncOptions:
   - CreateNamespace=true

ArgoCD’s responsibility ends once the manifest definitions are applied to the cluster. If any deployments fail Argo will continuously try to fix the state of the cluster (you can see in our example we have 2 apps with yellow arrows — this indicates the state of the cluster doesn’t match what Argo expects).

Our deployment contains both the Spinnaker Operator and Kustomize Spinnaker manifest, the operator includes a CRD (named SpinnakerService) which translates Spinnaker manifests into valid Spinnaker config & Kubernetes objects. You can read more about the operator on Armory’s website, but effectively it bundles Halyard into the deployment.

When changes to the SpinnakerService manifest definition are deployed (by ArgoCD), the Spinnaker Operator picks this up and begins rolling out changes: new replicasets, services and ingress. Below, we’ve included a screenshot of our Operator deployment within Argo (it only shows a subset of objects).

Now whenever we need to deploy to a fresh cluster, it’s as simple as connecting our ArgoCD instance to our Git repository and hitting deploy.

Setting up mTLS between Spinnaker microservices

One challenging part of our Spinnaker configuration to automate consistently was our mTLS setup.

In the case of mTLS, we were bound by some security requirements:

• We didn’t want engineers to see the contents of either private or public keys.
• We didn’t want engineers to have to interact with the key i.e. create and copy them into the cluster manually.
• We didn’t want to setup and manage an internal CA e.g. HashiCorp vault
• We wanted to still use internal cluster DNS e.g. <service>.<namespace>.svc.cluster saving us having to manage new DNS entries.
• Therefore we would need to use a self signed certificate as we don’t own the domain.

Solving the certificate provisioning issue is relatively easy as we can simply use cert-manager to create our certificates based on manifest definitions. We then need to get our CA & mTLS certificate into the Spinnaker service pods and import our CA into the trust store. Using trust-manager (another cert manager project) we can define a bundle which takes in source files and distributes them to destinations within the cluster, below is an example definition.

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
 name: spinnaker-cert-trust-bundle
spec:
 sources:
 - secret:
     name: selfsigned-certificate-authority
     key: "ca.crt"
 target:
   configMap: # what our destination file will look like
     key: mtls-cacerts
   namespaceSelector: # copy to namespace if label match is true
     matchLabels:
       kubernetes.io/metadata.name: spinnaker

This takes care of getting the CA certificate file into our Spinnaker namespace and mountable by service pods. To then import it into our trust store we used init containers, pulled in the default Java trust store, our CA and an empty directory. An empty directory is necessary as secrets & configmaps are read only so we need a place to store our altered file. After we’ve imported our CA, the empty directory mount is then consumed by the main service container and accepts our self signed CA — here’s all that wrapped up in a manifest.

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 kustomize:
   gate: &keystore-init
     deployment:
       patches:
         - |
           spec:
             template:
               spec:
                 initContainers:
                 - name: keystore-init
                   image: java-base-image
                   env:
                     - name: TRUSTSTORE_PASSWORD
                       valueFrom:
                         secretKeyRef:
                           name: certificate-secrets
                           key: TLS_INTERNAL_KEYSTORE_PASSWORD
                   volumeMounts:
                     - name: java-default-truststore
                       mountPath: /mnt/default-truststore
                     - name: spinnaker-ca-trust-bundle
                       mountPath: /mnt/internal-ca
                     - name: empty-dir
                       mountPath: /mnt/resulting-truststore
                   command: ['/bin/sh', '-c']
                   args:
                     - >-
                       keytool -importkeystore -srckeystore /mnt/default-truststore -srcstorepass "$TRUSTSTORE_PASSWORD" -destkeystore /mnt/resulting-truststore -deststorepass "$TRUSTSTORE_PASSWORD" --noprompt -v;
                       keytool -import -file /mnt/internal-ca -keystore /mnt/resulting-truststore -storepass "$TRUSTSTORE_PASSWORD" --noprompt -v;
   …

Now everytime a new pod is created, it will mount and configure all the necessary files it needs to communicate over mTLS.

Here’s what our simplified version of our finished implementation looks like:

1. Cert manager issues a self signed Certificate which we use as our authority.
2. Trust bundler copies the CA to our required namespaces.
3. Our CA issuer generates a certificate which we’ll use for mTLS.
4. Spinnaker pods automatically mount both certificates, adding the CA to their trust store and begin communicating over HTTPS.

Our vision for managing Spinnaker

Our production Spinnaker environment will be the norm for how most deployments are done across Wise. We’ll operate it centrally and manage connections for it to reach out and into our other Kubernetes environments. Our staging instance will be a testing ground for us within the Continuous Delivery team to develop and build features into Spinnaker. So, we’ll avoid rolling out broken artefacts and taking down deployments for our entire engineering team.

And if you’re wondering why we haven’t considered moving deployments entirely to ArgoCD? Well: 1) the need for non-K8S deployments 2) pipeline + release engineering philosophy 3) powerful integrations require additional CNCF projects e.g. keptn

What does our deployment workflow for Spinnaker look like with ArgoCD

We’ve ultimately forked all the microservices (and the Operator + Halyard) to meet our needs. When we are ready to release new changes, our CI system builds and packages images to a container registry. Previous users of Spinnaker will know that it uses a Bill of Materials to determine what versions of each microservice are bundled to make a Spinnaker platform version. Updating this is currently a manual process, along with generating our CHANGELOG, but once this has been merged to master, our CI process takes over once again and stores it in versioned object storage which Spinnaker can later pull from. So, we can easily rollback to older versions if we need to.

Rolling out these new versions is as simple as updating the version string for Spinnaker in our repo and merging to main. ArgoCD will then sync the latest commit, notice the change of state and reconcile the difference.

We’re currently exploring evolving our development workflow to incorporate Ambassador labs Telepresence, to speed up our feedback loop when developing new features into Spinnaker services. Keep an eye out for our findings later in the year.

Wrapping up

Our CD team has been hard at work getting Spinnaker into a place where they are confident in moving over identified early adopters and aim to get our migration underway before Q2 of 2023. Beyond trying to automate away our initial App of Apps setup, any future work that we carry out on Spinnaker will now be shaped by requests from our internal engineers as they begin to onboard and use the tool.

How Wise bootstraps Spinnaker using ArgoCD was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the third in a series of articles on the state of Application Security at Wise, describing our integration of security in the Software Development Lifecycle.

In part three of our blog post series we will be focusing on how we can improve the developers’ experience around fixing vulnerabilities. We explore how we identified the main pain points developers were facing, what we did to improve this experience and future improvement ideas that we are planning to work on.

Recap: What happened so far?

Over the course of the last year we have created our new setup for identifying vulnerabilities at Wise. It’s centred around DefectDojo as our vulnerability management tool and we use scanners, such as Trivy for Software Composition Analysis, as inputs. You can learn more about our setup in part one of this blog series.

Since then we have also created various dashboards to report vulnerabilities to our stakeholders, and also improved our Service-level agreement (SLA) with our developers to resolve new vulnerabilities. To learn more about that, please read part two of this series.

What issues are our developers facing?

As a next step, it’s important to check in with our developers to see how they’re interacting with our program. It’s one thing to set up a vulnerability management program, but we also need to make sure that it actually works for our users and that the recommendations provided are actioned in a timely manner.

While the initial feedback we received from developers was promising, we decided to share a survey with them to collect more actionable feedback. Our survey had a mix of multiple-choice and free-form questions and was mostly focused on how developers are using our tools and what issues they might be facing with them. We also interviewed them for analogous use cases, including what works well and what doesn’t work well in other tools they’re using. For example, we asked:

1. When did you last check the status of your team’s vulnerabilities?
2. If you already receive alerts, how do you receive the majority of them?
3. How often would you want to be notified about vulnerabilities that were found?

Our survey concluded that there were two main pain points that developers were experiencing: Our dashboards were slow to load, and second, that teams had to be proactive and needed to remember to check the dashboards we provided them with, sometimes clashing with their already packed schedules and deadlines.

These two issues raised could potentially result in developers getting easily frustrated by our tooling, but also that they might forget or miss vulnerabilities, because they didn’t check a dashboard in time. The developer experience suffered from that, but it also made it harder for teams to stay within the SLAs that we established.

What can we do about it?

One simple way to tackle both of these issues at the same time is to introduce alerts. It solves both the speed issue, as developers don’t even need to visit the dashboards, unless we tell them there is something worth seeing there, but also the proactivity problem: Instead of developers constantly needing to check whether there are new vulnerabilities the alerts can just tell them.

More than half of the survey participants requested to get a weekly report of all vulnerabilities that are approaching their SLA due date, so that they would have enough time left to fix them. In general, team’s ways of working differ widely across the company though, as our teams have a lot of autonomy. To address that, we added a version-controlled configuration file that teams can append a cron expression to in order to receive notifications on their own schedule.

[
 {
   "teamName": "application-security",
   "slackChannel": "appsec-alerts",
   "cronSchedule": "0 9 * * MON"
 },
 // ...
]

Code Snippet: Teams define their team name, preferred slack channel as well as a cron syntax schedule to receive alerts. In the example above, the Application Security team would receive their notifications at 09:00 am on Monday.

Almost all of our alerting in the company is delivered via Slack, which is why we decided to create a Slack notification as well. Our new bot now sends teams an automatic vulnerability report whenever they need it, which for most teams means once a week. The report shows all vulnerabilities that will breach their SLA in the next two weeks or less. For each vulnerability, the report also shows some metadata, including how many days are remaining to fix it, what the mitigation is and what the CVE of the vulnerability is.

Additionally, teams can report vulnerabilities as a false positive, directly from the slack message, by raising a pre-filled Jira ticket with us. One month after releasing this new feature more than half of all teams in Wise had already signed up to receive these notifications which is a great indicator.

What’s next?

Currently, the bot still has a few limitations that we are working on improving. Since releasing the bot we have also massively improved the loading speed of our vulnerability dashboards. Over the next months we will continue to improve our service to developers based on the feedback we receive.

We are also really excited about the developer portal that was recently launched in Wise and is discussed in more detail in our recent wise tech stack blog post. One aspect that’s particularly interesting about it is to display vulnerability data in the developer portal, similar to what Spotify recently described in their post about vulnerability management.

If you enjoyed reading this post and like the presented challenges, keep an eye out for open Application Security Engineering roles here.

Disclaimer: Product names mentioned in this article are property of their respective owners. This article is not aimed at publicising or endorsing any of the brands used or mentioned in the article.

Improving the Developer Experience — Our Application Security Journey (Part 3) was originally published in Wise Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.