Eng 2185 remove unused cols by JadeCara · Pull Request #7172 · ethyca/fides

JadeCara · 2025-12-30T14:41:05Z

Description Of Changes

🎯 Create manual tasks on the Consent Graph.

There are going to be a few PRs cleaning up some un-used functionality or adjusting functionality for improved usage patterns. This PR removes the columns which are now un-used and have been replaced by the JSONB storage tree. It is a clean up based on the first few rounds of customer use and learning actual usage patterns.

Past PRs:

The logging being used by manual tasks and front end data is entirely the general audit logs so the manual task log table and utilities are now redundant. ENG-2185 #7124
Update the Conditional Dependency table with a JSONB dictionary column. This will allow us to thoroughly test the migration without being destructive. [ENG-2185] #7133

This PR:

Remove the extra columns and rows for conditional dependencies once the tree has been implemented and tested.

Future PRs:

Allow conditional dependencies to be assigned to a specific policy type or id
Allow users to be assigned to specific submissions rather than the whole task.

Code Changes

.fides/db_dataset.yml - removed un used columns
src/fides/api/alembic/migrations/versions/ remove the unused columns.
removed un-used columns and multiple row based functionality.
- src/fides/api/models/conditional_dependency/conditional_dependency_base.py
- src/fides/api/models/digest/conditional_dependencies.py
- src/fides/api/models/manual_task/conditional_dependency.py
Updated all tests removing the manual task log.

Steps to Confirm

There should be NO change in functionality.
Manual tasks, conditional dependencies etc should continue to function as before.
Please test with FidesPlus 2951
Run FidesPlus branch above pointed at this branch.
Create a ManualTask with several submissions required and several conditions, both dataset and privacy request based.
Create privacy requests that meet and do not meet the requirements. Verify that they create/do not create as expected.
Verify that you can complete the manual tasks with no errors.
Verify that you can see the skipped logs and the manual task and manual task actions appear on the DSR activity log.
Verify that you receive the access package with all expected manual task inclusions (attachments and text)
Update the ManualTask submission types, delete one or more.
Delete manual task integration

Pre-Merge Checklist

…des into ENG-2185-add-new-jsob-tree-col

vercel · 2025-12-30T14:41:09Z

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Review	Updated (UTC)
fides-plus-nightly	Ignored	Preview	Jan 12, 2026 9:49pm
fides-privacy-center	Ignored		Jan 12, 2026 9:49pm

…used-cols

codecov · 2025-12-30T16:21:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.04%. Comparing base (ba6556f) to head (2b2c07f).

Additional details and impacted files

@@                        Coverage Diff                         @@
##           ENG-2185-add-new-jsob-tree-col    #7172      +/-   ##
==================================================================
- Coverage                           87.24%   85.04%   -2.20%     
==================================================================
  Files                                 535      535              
  Lines                               35324    35264      -60     
  Branches                             4117     4107      -10     
==================================================================
- Hits                                30817    29992     -825     
- Misses                               3615     4348     +733     
- Partials                              892      924      +32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

greptile-apps

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

...rations/versions/xx_2025_12_30_1421_9cf7bb472a7c_remove_un_used_columns_from_conditional_.py

JadeCara · 2026-01-12T17:52:05Z

@greptile please review

greptile-apps

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

…used-cols

src/fides/api/models/conditional_dependency/conditional_dependency_base.py

src/fides/api/models/manual_task/conditional_dependency.py

tests/api/task/manual/test_manual_task_conditional_evaluation.py

erosselli · 2026-01-12T20:53:12Z

src/fides/api/models/manual_task/conditional_dependency.py

        condition_row = (
-            db.query(cls)
-            .filter(cls.manual_task_id == manual_task_id, cls.parent_id.is_(None))
-            .first()
+            db.query(cls).filter(cls.manual_task_id == manual_task_id).first()
        )


nit: using .one() here would make it so we throw an error if there's somehow more than one matching row

erosselli · 2026-01-12T20:54:19Z

tests/api/models/digest/conftest.py

        "conditions": [
            {
-                "logical_operator": "and",
+                "logical_operator": GroupOperator.and_,


is this change intentional?

yeah - moving away from strings to the actual enums etc.

erosselli · 2026-01-12T20:57:03Z

tests/api/task/manual/test_manual_task_conditional_evaluation.py

                privacy_request=privacy_request,
            )

-            # Verify the extracted location convenience fields match expected


are the comments removed on purpose? I like comments in long tests, it helps read through what the test is doing

Some of them felt redundant, I am probably just too close to it! Thanks for the feedback, I will put them back!

erosselli · 2026-01-12T20:57:16Z

tests/api/task/manual/test_manual_task_conditional_evaluation.py

-            # No conditional dependencies created
-
-            # Extract the data


same question for this test

Putting them back :)

erosselli · 2026-01-12T20:57:45Z

tests/api/task/manual/test_manual_task_integration.py

    """Test that manual task traversal executes correctly with conditional dependencies and upstream data"""

-    @pytest.mark.usefixtures("condition_gt_18", "condition_eq_active")
+    @pytest.mark.usefixtures("group_condition")


before we were using some other conditions here, are those not relevant for this test anymore?

Since all the conditions will now just be in the condition tree instead of spread out across rows. We only need the group condition.

erosselli · 2026-01-12T20:58:49Z

tests/api/task/manual/conftest.py

+@pytest.fixture()
+def condition_gt_18(db: Session, manual_task: ManualTask):
+    """Create a conditional dependency with field_address 'user.age' and operator 'gte' and value 18"""


there were tests that used this fixture but no longer do so, is that an intentional decision?

Its being used, but it looks like it is only used in 1 place now. I can probably just clean that up. It is using the create_condition_gt_18_tree() which adds the condition in dict form and is used a lot more widely now.

Co-authored-by: Jade Wibbels <jade@ethyca.com>

Jade Wibbels and others added 23 commits December 15, 2025 18:43

initial

45ad401

added jsonb col and updated tests

45396d4

Merge branch 'main' into ENG-2185-add-new-jsob-tree-col

c83780f

update db yml

a89bbce

test updates

f5cf36a

clean up alembic file

fdef91d

remove extra definition

2c3bc15

remove extra definition

9a788df

fix return

fccc79b

some clean ups

ae56487

Merge branch 'main' into ENG-2185-add-new-jsob-tree-col

57599c9

Merge branch 'main' into ENG-2185-add-new-jsob-tree-col

10c24be

missed one

744aa5e

Merge branch 'main' into ENG-2185-add-new-jsob-tree-col

0400766

fix tests

e025468

Merge branch 'ENG-2185-add-new-jsob-tree-col' of github.com:ethyca/fi…

b9f38c5

…des into ENG-2185-add-new-jsob-tree-col

clean up

3891837

clean up

f889e04

Merge branch 'main' into ENG-2185-add-new-jsob-tree-col

bf574bd

fix

68d1c4e

.

a774f76

migration

8927dd0

update to remove orphan cols and update db yml

98acd32

Jade Wibbels and others added 4 commits December 30, 2025 15:03

add commit

62e3fe5

Apply suggestion from @JadeCara

eb5d0aa

Merge branch 'ENG-2185-add-new-jsob-tree-col' into ENG-2185-remove-un…

f78c3dd

…used-cols

more test updates

075ff2f

more test updates

41fd4a6

greptile-apps bot reviewed Jan 12, 2026

View reviewed changes

...rations/versions/xx_2025_12_30_1421_9cf7bb472a7c_remove_un_used_columns_from_conditional_.py Outdated Show resolved Hide resolved

...rations/versions/xx_2025_12_30_1421_9cf7bb472a7c_remove_un_used_columns_from_conditional_.py Outdated Show resolved Hide resolved

fix downgrade

90da3a8

greptile-apps bot reviewed Jan 12, 2026

View reviewed changes

JadeCara and others added 6 commits January 12, 2026 11:00

Apply suggestion from @JadeCara

8d17149

Apply suggestion from @JadeCara

40265f0

Merge branch 'ENG-2185-add-new-jsob-tree-col' into ENG-2185-remove-un…

58cd0de

…used-cols

remove redundant test

b2273a4

Merge branch 'ENG-2185-add-new-jsob-tree-col' into ENG-2185-remove-un…

3ea27fc

…used-cols

changelog

be462d8

Base automatically changed from ENG-2185-add-new-jsob-tree-col to main January 12, 2026 20:18

Merge branch 'main' into ENG-2185-remove-unused-cols

c90c738

JadeCara commented Jan 12, 2026

View reviewed changes

src/fides/api/models/conditional_dependency/conditional_dependency_base.py Show resolved Hide resolved

Apply suggestion from @JadeCara

214af8d

JadeCara commented Jan 12, 2026

View reviewed changes

src/fides/api/models/manual_task/conditional_dependency.py Outdated Show resolved Hide resolved

Apply suggestion from @JadeCara

efc027d

JadeCara commented Jan 12, 2026

View reviewed changes

tests/api/task/manual/test_manual_task_conditional_evaluation.py Show resolved Hide resolved

Apply suggestion from @JadeCara

fe919b1

JadeCara commented Jan 12, 2026

View reviewed changes

tests/api/task/manual/test_manual_task_conditional_evaluation.py Show resolved Hide resolved

Apply suggestion from @JadeCara

67e6022

erosselli reviewed Jan 12, 2026

View reviewed changes

updated query and put back some comments

be79f8e

erosselli approved these changes Jan 12, 2026

View reviewed changes

updated to one_or_none for none cases

941a2d9

JadeCara added this pull request to the merge queue Jan 12, 2026

Merged via the queue into main with commit bcfc6ad Jan 12, 2026
51 of 52 checks passed

JadeCara deleted the ENG-2185-remove-unused-cols branch January 12, 2026 22:27

JadeCara added a commit that referenced this pull request Jan 13, 2026

Eng 2185 remove unused cols (#7172)

89de7af

Co-authored-by: Jade Wibbels <jade@ethyca.com>

mfbrown pushed a commit that referenced this pull request Jan 27, 2026

Eng 2185 remove unused cols (#7172)

5409e27

Co-authored-by: Jade Wibbels <jade@ethyca.com>

Conversation

JadeCara commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

Uh oh!

vercel bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JadeCara commented Jan 12, 2026

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JadeCara commented Dec 30, 2025 •

edited

Loading

vercel bot commented Dec 30, 2025 •

edited

Loading

codecov bot commented Dec 30, 2025 •

edited

Loading