Traversal optimizations

[galvana]

Ticket ENG-992

Description Of Changes

This PR significantly improves the performance of graph traversal operations by replacing O(N²) algorithms with O(N+E) alternatives.

Performance Improvements

Optimization	Before	After	Improvement
traverse() in traversal.py	O(N²)	O(N+E)	~100x faster at 3000 nodes
compute_all_descendants()	O(N²)	O(N+E)	~100x faster at 3000 nodes
Dataset-level after deps	O(N²)	O(N)	Linear lookup

Key Changes:

1. traverse() in traversal.py - O(N²) → O(N+E)

   • Legacy used MatchingQueue.pop_first_match() which scans the queue on each iteration
   • New version uses Kahn's algorithm with in-degree tracking for optimal complexity

2. compute_all_descendants() in create_request_tasks.py - O(N²) → O(N+E)

   • Was calling networkx.descendants() for each node in a loop
   • Now pre-computes all descendants in a single reverse topological pass

3. Dataset-level after dependencies - O(N²) → O(N)

   • Was scanning all nodes to find collections in a dataset
   • Now uses pre-built _collections_by_dataset index for O(1) lookups

Performance comparison

Collections	Legacy O(N²)	Optimized O(N+E)	Speedup
100	~12 ms	~1 ms	12x
1,000	~1.1 s	~7 ms	161x
5,000	~27 s	~30 ms	900x
10,000	~2.2 min	~80 ms	1,680x
25,000	~14 min*	~500 ms	1,688x
50,000	~57 min*	~1 s	3,415x

* Extrapolated from O(N²) growth curve

Code Changes

• Added optimized traverse() method using Kahn's algorithm with in-degree tracking
• Added compute_all_descendants() function for O(N+E) descendant computation
• Pre-indexed edges by node address (edges_by_node) for O(1) edge lookups
• Pre-built dataset-to-collections index (_collections_by_dataset) for O(1) lookups
• Added skip_verification parameter to BaseTraversal to avoid redundant traversal during reachability checks
• Retained _traverse_legacy() method for test comparison purposes

Tests

New test file test_traversal_optimization_comparison.py validates that the optimized algorithm produces identical results to the legacy implementation:

Test Class	Purpose
TestTraversalComparison	Validates equivalence on deterministic graph topologies (linear chains, star graphs, diamond graphs, graphs with after dependencies)
TestRandomGraphEquivalence	Validates equivalence on randomly generated graphs with varying densities and sizes
TestTraversalErrorEquivalence	Ensures both algorithms raise the same errors for invalid graphs (disconnected nodes, unreachable collections)

Steps to Confirm

Existing tests should pass.

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ready	Preview, Comment	Feb 11, 2026 0:46am

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
fides-privacy-center	Ignored		Feb 11, 2026 0:46am

[greptile-apps]

Greptile Overview

Greptile Summary

This PR implements significant performance optimizations for graph traversal operations, achieving ~100x speedup at 3000 nodes by replacing O(N²) algorithms with O(N+E) alternatives.

Key optimizations:

• traverse() method: Replaced MatchingQueue.pop_first_match() scanning with Kahn's algorithm variant using in-degree tracking for O(N+E) complexity
• compute_all_descendants(): Added function to pre-compute all descendants in single reverse topological pass instead of calling networkx.descendants() per node
• Dataset indexing: Pre-built _collections_by_dataset index for O(1) dataset-to-collections lookups
• Edge indexing: Pre-indexed edges by node address (edges_by_node) for O(1) edge lookups
• Verification optimization: Added skip_verification parameter to avoid redundant traversal during reachability checks

Code quality:

• Comprehensive test suite validates equivalence between optimized and legacy implementations across multiple graph topologies
• Legacy _traverse_legacy() method retained for reference and comparison
• Clear documentation of algorithmic approach and complexity analysis

Confidence Score: 4/5

• This PR is safe to merge with minimal risk - well-tested performance optimization
• Score reflects thorough testing approach and algorithmic correctness. The optimized implementation is validated against the legacy implementation across multiple test scenarios. The only minor style issue is a single-character variable name in the legacy code (which is intentionally kept for reference). The changes are well-documented with clear complexity analysis.
• No files require special attention - the implementation is solid and well-tested

Important Files Changed

Filename	Overview
src/fides/api/graph/traversal.py	Replaced O(N²) traversal algorithm with O(N+E) Kahn's algorithm variant using in-degree tracking; added pre-indexing optimizations for edges and dataset collections; retained legacy implementation for comparison
src/fides/api/task/create_request_tasks.py	Added compute_all_descendants() function for O(N+E) descendant computation; updated task creation functions to pre-compute descendants once instead of calling networkx.descendants per node
src/fides/api/graph/node_filters.py	Updated OptionalIdentityFilter to use skip_verification parameter and explicit traverse() call to avoid redundant verification overhead
tests/ops/graph/test_traversal_optimization_comparison.py	Comprehensive test suite comparing optimized and legacy traversal algorithms across deterministic topologies, random graphs, and error cases

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

src/fides/api/graph/traversal.py
single-character variable name n - use descriptive name like current_node

Context Used: Rule from dashboard - Use full names for variables, not 1 to 2 characters (source)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[adamsachs]

nice work! this looks great, and it looks like we can get some very serious optimizations for very little (if any) trade-off. the main thing to look out for, i think, would be a significant increase in space complexity, i.e. is there any chance we'd significantly increase our memory footprint with massive task graphs?

i left a few relatively minor comments along the way that'd be nice to address, mainly re: code cleanup. there does still seem to be a test failing that's due to a change here, so we'll want to address that before merging.

sorry it took me so long to get to this - i wanted to give it the time it deserved!

[adamsachs]

OK, i think this all makes sense - i definitely understand the algorithm after staring at the code for a little while, though i can't say i've fully thought through all the edge cases...the test coverage does make me feel better! it's a nice algorithm :)

to help make the traversal logic a bit more tractable - have you considered breaking it up into some helper functions? it just seems like we'll wantto be able to take a look at the function and spend ~a minute to understand the high-level steps/procedure of the algorithm. as it's written, that's pretty hard to do - the closest i can get is just reading the comments. it'd be much easier to reason about, i think, if some of these sub-operations/steps were just broken out into a helper so that you don't have the cognitive overload as you look through this main function.

i know the 'legacy' version of the function didn't break anything up, but that doesn't seem like something we want to emulate - i don't think it's ever good to be disabling the lint check for too many locals...is there a reason we want to keep the traversal function so monolithic here if we're completely overhauling it?

[galvana]

I think I'll leave this for a follow up ticket. If we can get this in for the upcoming release it would be a huge win! We can clean it up later.

[adamsachs]

k, sounds good!

somewhat relatedly...if we're going to keep the legacy traverse function in the code base for the time being, i wonder if it's worth just keeping it as a fallback to be used in actual execution -- hidden behind a config. this would only be temporary, for a few releases. just to derisk if there is any regression with the new algorithm, it'd be easy to switch back without requiring a patch release.

thoughts? maybe it's overkill, if you're fully confident in the new approach!

[galvana]

It doesn't hurt to add a setting. I want to use ConfigProxy but I know there's a weird bug there that we haven't been able to reproduce. I'll just use the regular FidesConfig

[adamsachs]

nit: probably able to dedupe these tests with a single parameterized test?