Skip to content

fix connection config list endpoint performance regression#7501

Merged
adamsachs merged 3 commits intomainfrom
asachs/improve-connection-endpoint-performance
Feb 26, 2026
Merged

fix connection config list endpoint performance regression#7501
adamsachs merged 3 commits intomainfrom
asachs/improve-connection-endpoint-performance

Conversation

@adamsachs
Copy link
Copy Markdown
Contributor

@adamsachs adamsachs commented Feb 26, 2026

Ticket ENG-2590

Description Of Changes

Fixes a performance regression introduced in #7458 on the GET /api/v1/connection (list), GET /api/v1/connection/{key} (detail), and GET /system/{fides_key}/connection endpoints. The selectinload(ConnectionConfig.system) added to populate the new linked_systems response field was loading the full System ORM model, which triggered a cascade of 5 additional lazy="selectin" relationship queries on every request:

  1. privacy_declarations — all privacy declarations for all loaded systems
  2. data_stewards — all FidesUser records via the systemmanager join table
  3. connection_configs — the ConnectionConfig back again (circular)
  4. assets — all Asset records for all loaded systems
  5. system_groups — all SystemGroup records via system_group_member join table

None of this data is needed — the LinkedSystemInfo schema only uses fides_key and name. On an environment with 73 integrations and 181 systems, this caused the list endpoint to take ~13s to respond.

The fix introduces a shared linked_system_load_options() helper in system_integration_link/repository.py that applies load_only(System.fides_key, System.name) and noload("*") to restrict loaded columns and suppress the cascade. This helper is applied to all three endpoints, including the detail endpoint which was triggering the same cascade via lazy loading.

Code Changes

  • src/fides/system_integration_link/repository.py — Added linked_system_load_options() helper that returns a properly scoped selectinload with load_only + noload("*")
  • src/fides/api/api/v1/endpoints/connection_endpoints.py — List endpoint uses the shared helper; detail endpoint rewritten to use an explicit query with the same helper instead of get_connection_config_or_error (which loaded via plain ORM with no eager-load options)
  • src/fides/api/api/v1/endpoints/system.py — System connections endpoint uses the shared helper

Local Benchmarks

Seeded local DB to production-like scale (74 connection configs, 61 system links, 7,371 privacy declarations, 6,625 assets) and ran an A/B comparison on the list endpoint:

Without Fix (bare selectinload) With Fix (load_only + noload)
Run 1 2.09s 0.74s
Run 2 0.85s 0.68s
Run 3 1.33s 0.64s
Run 4 0.90s 0.68s
Run 5 1.42s 0.69s
Median ~1.3s ~0.69s

~2x improvement locally. The improvement in production environments with remote DB latency will may be significantly larger, since each eliminated cascade query adds a full network round-trip.

Note: we are seeing ~13s latency for this endpoint on plus-rc.

Steps to Confirm

  1. GET /api/v1/connection?size=100 — each item should include linked_systems: [] or linked_systems: [{fides_key: "...", name: "..."}], same as before
  2. GET /api/v1/connection/{key} — should include linked_systems in the detail response
  3. GET /system/{fides_key}/connection — should include linked_systems for each connection
  4. Response times should be noticeably improved on environments with many systems/integrations

Pre-Merge Checklist

  • Issue requirements met
  • All CI pipelines succeeded
  • CHANGELOG.md updated
    • Add a db-migration label
    • Add a high-risk label
    • Updates unreleased work already in Changelog, no new entry necessary
  • UX feedback:
    • No UX review needed
  • Followup issues:
    • No followup issues
  • Database migrations:
    • No migrations
  • Documentation:
    • No documentation updates required

The selectinload(ConnectionConfig.system) triggered all of System's
default lazy="selectin" relationships (privacy_declarations,
data_stewards, connection_configs, assets, system_groups), creating
a cascade of unnecessary queries. Restrict to load_only the two
columns needed (fides_key, name) and noload all relationships.

Made-with: Cursor
Made-with: Cursor
@vercel
Copy link
Copy Markdown
Contributor

vercel bot commented Feb 26, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
fides-plus-nightly Ignored Ignored Preview Feb 26, 2026 5:18pm
fides-privacy-center Ignored Ignored Feb 26, 2026 5:18pm

Request Review

Move the selectinload + load_only + noload pattern into a reusable
linked_system_load_options() helper in system_integration_link/repository.py.
Apply the same optimization to the GET /connection/{key} detail endpoint
which was still triggering the cascade via lazy loading.

Made-with: Cursor
@adamsachs adamsachs marked this pull request as ready for review February 26, 2026 17:28
@adamsachs adamsachs requested a review from a team as a code owner February 26, 2026 17:28
@adamsachs adamsachs requested review from vcruces and removed request for a team February 26, 2026 17:28
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Feb 26, 2026

Greptile Summary

This PR addresses a critical performance regression introduced in #7458. The issue was that eager-loading the System relationship on ConnectionConfig was loading the full ORM model, which triggered a cascade of 5 additional selectin relationship queries (privacy_declarations, data_stewards, connection_configs, assets, system_groups) on every request. On production environments with many systems, this caused response times to balloon to ~13 seconds.

The fix introduces a shared linked_system_load_options() helper that uses load_only() to restrict loaded columns to just fides_key and name (the only fields needed by LinkedSystemInfo), and noload("*") to suppress all relationship cascades. This helper is consistently applied across all three affected endpoints:

  • GET /api/v1/connection (list)
  • GET /api/v1/connection/{key} (detail)
  • GET /system/{fides_key}/connection (system connections)

The detail endpoint was refactored from using get_connection_config_or_error() (which doesn't support query options) to an explicit query with proper eager-loading options. Local benchmarks show ~2x improvement (1.3s → 0.69s median), with potentially larger gains in production due to reduced network round-trips.

The implementation is clean, well-documented, and follows SQLAlchemy best practices. No functional changes to the API responses.

Confidence Score: 5/5

  • This PR is safe to merge with no risk
  • The changes are well-implemented, focused, and thoroughly tested. The fix uses standard SQLAlchemy optimization patterns correctly, doesn't change any API contracts, and existing tests validate the functionality. The performance improvement is significant with no downsides.
  • No files require special attention

Important Files Changed

Filename Overview
src/fides/system_integration_link/repository.py Added helper function linked_system_load_options() to optimize eager-loading of System relationships by loading only required columns
src/fides/api/api/v1/endpoints/connection_endpoints.py Updated list and detail endpoints to use optimized eager-loading helper, preventing cascading relationship queries
src/fides/api/api/v1/endpoints/system.py Updated system connections endpoint to use optimized eager-loading helper

Last reviewed commit: 3ed330b

Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

@adamsachs adamsachs added this pull request to the merge queue Feb 26, 2026
Merged via the queue into main with commit 842b5fe Feb 26, 2026
54 checks passed
@adamsachs adamsachs deleted the asachs/improve-connection-endpoint-performance branch February 26, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants