Add SQLAlchemy deferred loading

[JadeCara]

Ticket ENG-2126

Description Of Changes

🎯 Out of memory errors on workers when processing DSRs tasks

One area of opportunity identified was loading large data while querying tasks. This follows the pattern of defering loading of large columns for Privacy Request objects. This PR updates the following:

• query_with_deferred_data() method for lazy loading
• Applied to cached_erasure_results_by_collection_key()
• Applied to cached_consent_results_by_collection_key()
• Applied to scheduler and service layer queries

Code Changes

• src/fides/api/models/privacy_request/privacy_request.py - Updated to use query with deferred data
• src/fides/api/models/privacy_request/request_task.py - Added query_with_deferred_data
• src/fides/api/service/privacy_request/request_service.py - Updated to use query with deferred data
• src/fides/api/task/scheduler_utils.py - Updated to use query with deferred data

Steps to Confirm

1. Start FidesPlus pointed at this endpoint
2. Run 1 or more access requests, run 1 or more erasure requests.
3. All tests should pass.

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Review	Updated (UTC)
fides-plus-nightly	Ignored	Preview	Jan 26, 2026 10:23pm
fides-privacy-center	Ignored		Jan 26, 2026 10:23pm

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds SQLAlchemy deferred loading to prevent out-of-memory (OOM) errors when processing DSR tasks. The changes implement a new query_with_deferred_data() method on RequestTask that defers loading of large JSON columns (_access_data, _data_for_erasures, collection, traversal_details) when only metadata is needed.

Key changes:

• Added RequestTask.query_with_deferred_data() with configurable deferred loading for access and erasure data columns
• Applied deferred loading to methods that only need task metadata: get_pending_downstream_tasks(), upstream_tasks_objects(), cached_erasure_results_by_collection_key(), cached_consent_results_by_collection_key(), and requeue_polling_tasks()
• Refactored use_dsr_3_0_scheduler() to check for RequestTasks first (cheap SQL count) before checking cache, avoiding the expensive get_raw_access_results() call that was causing 622MB+ memory usage per call

The implementation follows established patterns from PrivacyRequest.query_with_deferred_data() and correctly applies deferred loading only where metadata queries are performed, not where actual data payloads are needed.

Confidence Score: 4/5

• This PR is safe to merge with minimal risk, addressing critical OOM issues through deferred loading
• Score reflects solid implementation following established patterns and addressing a critical performance issue. Minor concern about lack of test coverage for the new deferred loading functionality, but the changes are straightforward and well-documented. The refactoring of use_dsr_3_0_scheduler() is a significant optimization that prevents OOM errors on the hot path.
• Pay close attention to src/fides/api/task/scheduler_utils.py to ensure the cache key prefix logic is correct

Important Files Changed

Filename	Overview
src/fides/api/models/privacy_request/request_task.py	Added query_with_deferred_data() method with comprehensive docstring; applied to get_pending_downstream_tasks() and upstream_tasks_objects()
src/fides/api/models/privacy_request/privacy_request.py	Applied deferred loading to get_raw_masking_counts() and get_consent_results() to avoid loading large JSON columns
src/fides/api/task/scheduler_utils.py	Refactored to check RequestTasks first (fast path), then check cache for DSR 2.0 data instead of loading access results (prevents OOM)

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}