ENG-2806: Migrate IDP staged resources to unified IDP_APP type

[dsill-ethyca]

Ticket ENG-2806

Description Of Changes

Add a data migration to unify IDP staged resource types. Existing Okta App and Entra App rows are consolidated to a single IDP App resource type, with the provider stored in meta["provider"]. Also renames the legacy okta_app_id key to the generic app_id in the meta JSON.

Dependency: Fidesplus PR ethyca/fidesplus#3267 consumes the new IDP App resource type. This migration should be merged first.

No customer data is affected — IDP monitor resources are not yet in production. This migration keeps continuity in dev/staging environments.

Code Changes

• src/fides/api/alembic/migrations/versions/xx_2026_03_20_1745_ad1bb600715b_migrate_idp_staged_resources.py - Data migration: rename okta_app_id → app_id in meta, add provider key, update resource_type from Okta App/Entra App to IDP App

Steps to Confirm

1. Run alembic upgrade head — verify migration applies without errors
2. Query SELECT resource_type, meta->>'provider', meta->>'app_id' FROM stagedresource WHERE resource_type = 'IDP App' — verify all IDP resources have the new type and provider
3. Run alembic downgrade 38071fffda39 — verify downgrade restores original Okta App/Entra App types and okta_app_id key

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ignored	Preview	Mar 23, 2026 4:30pm
fides-privacy-center	Ignored		Mar 23, 2026 4:30pm

[greptile-apps]

Greptile Summary

This PR adds an Alembic data migration that consolidates two provider-specific staged resource types (Okta App and Entra App) into a single IDP App type, storing the provider identity in meta["provider"] and normalising the Okta-specific okta_app_id meta key to the generic app_id. The migration is dev/staging-only and has no production impact.

• The upgrade step ordering is correct: key rename → add provider → update resource type.
• The downgrade restores resource types first, then renames keys back — also correct.
• The COALESCE(meta->>'provider', 'okta') default in the downgrade silently classifies provider-less IDP App rows as Okta App, which could misclassify rows inserted by application code after the upgrade if they lack a provider key.
• The okta_app_id → app_id rename in the upgrade is not scoped to Okta App rows (unlike the corresponding downgrade step), creating a minor asymmetry that could leave non-Okta rows permanently modified if they happened to have that key.

Confidence Score: 4/5

• Safe to merge after addressing the COALESCE default and rename-scoping asymmetry; no production data is at risk.
• The migration logic is sound and well-ordered. The two flagged issues (COALESCE default and unscoped rename) are edge cases that are very unlikely to surface in practice given that this data is confirmed non-production, but they represent real asymmetries between upgrade and downgrade that could cause silent data misclassification on rollback.
• Pay close attention to the downgrade logic in xx_2026_03_20_1745_ad1bb600715b_migrate_idp_staged_resources.py, specifically the COALESCE provider default and the scope of the okta_app_id rename.

Important Files Changed

Filename	Overview
src/fides/api/alembic/migrations/versions/xx_2026_03_20_1745_ad1bb600715b_migrate_idp_staged_resources.py	Data migration consolidating Okta App/Entra App staged resource types into a unified IDP App type with provider stored in meta; upgrade ordering is correct but the COALESCE default in the downgrade silently maps provider-less rows to Okta, and the okta_app_id rename is not scoped to Okta rows creating a minor asymmetry.
changelog/7712-idp-staged-resource-migration.yaml	Changelog entry correctly typed as Changed with the db-migration label; no issues.

_{Reviews (1): Last reviewed commit: "Add changelog fragment for IDP staged re..." | Re-trigger Greptile}

[greptile-apps]

Silent Okta App default for provider-less rows

COALESCE(meta->>'provider', 'okta') = 'okta' means any IDP App row where meta is NULL or the provider key is absent will be silently restored as Okta App during downgrade. The upgrade always stamps a provider value for the two known types, but if any IDP App rows are inserted after the upgrade runs (by application code) without a provider in meta, they would be misclassified as Okta App on rollback rather than being skipped or surfaced as an error.

Consider being explicit about the NULL case, e.g.:

Suggested change

	AND COALESCE(meta->>'provider', 'okta') = 'okta'
	AND meta->>'provider' = 'okta'

…and adding a separate safety check for the NULL provider case (e.g., logging or leaving those rows as-is) to make rollback behaviour fully predictable.

[greptile-apps]

Rename not scoped to Okta App rows, asymmetric with downgrade

The okta_app_id → app_id rename runs on all rows in stagedresource that have okta_app_id in their meta (no resource_type filter). The corresponding downgrade step (line 89–98) only renames back for resource_type = 'Okta App' rows. If any non-Okta resource happened to have an okta_app_id key in its meta, the upgrade would rename it to app_id but the downgrade would not restore it, silently losing the original key.

Scoping the upgrade rename to match is safer:

UPDATE stagedresource
SET meta = (meta - 'okta_app_id') || jsonb_build_object('app_id', meta->'okta_app_id')
WHERE resource_type = 'Okta App'
AND jsonb_exists(meta, 'okta_app_id')

[claude]

Code Review

This is a clean, focused migration. The SQL logic is sound and the overall approach — rename the key, add provider, update the type, with a symmetrical downgrade — is correct. The down_revision is properly set to 38071fffda39 (the current chain head). A few minor points below.

Suggestions

1. Rename scope in upgrade (line 22): The okta_app_id → app_id rename is not restricted to resource_type = 'Okta App'. The downgrade's reverse step is restricted to Okta rows, which creates a subtle asymmetry if any other row type ever has that key. Low risk given the current data model, but tightening the WHERE clause would make the upgrade/downgrade pair perfectly symmetric.

2. COALESCE default in downgrade (line 69): The COALESCE(meta->>'provider', 'okta') default treats provider-less IDP App rows as Okta on downgrade. This is safe for a clean upgrade→downgrade cycle but could silently misclassify rows inserted by application code without a provider key. An explicit meta->>'provider' = 'okta' (or a comment explaining the intentional default) would be clearer.

Nice to Have

3. Unknown providers in downgrade (line 80): IDP App rows with a provider other than 'okta' or 'entra' would survive the downgrade unchanged (still IDP App, still with provider key). Not a concern today, but worth a comment for future maintainers.

No critical issues. The migration is well-structured for its stated scope.

[claude]

Suggestion: Scope the okta_app_id rename to Okta App rows only

The upgrade renames okta_app_id → app_id for all rows that have the key in meta, regardless of resource_type. The downgrade (step 3) only reverses this for rows whose resource_type was restored to Okta App. If any Entra App row happened to have okta_app_id in its meta, the downgrade would leave it with app_id instead of restoring the original key.

Scoping the upgrade to match the downgrade would make the two more symmetric:

UPDATE stagedresource
SET meta = (meta - 'okta_app_id') || jsonb_build_object('app_id', meta->'okta_app_id')
WHERE resource_type = 'Okta App'
AND jsonb_exists(meta, 'okta_app_id')

This is low-risk given okta_app_id is semantically Okta-specific, but it makes the intention clearer and the downgrade fully invertible.

[claude]

Suggestion: Potential over-matching in downgrade for IDP App rows

AND COALESCE(meta->>'provider', 'okta') = 'okta'

This treats any IDP App row that lacks a provider key as an Okta row. The upgrade only adds provider for rows that don't already have it, so any pre-existing IDP App rows without a provider key in meta would be (incorrectly) converted to Okta App during downgrade.

The PR description notes IDP monitor resources aren't in production yet, so the practical risk is low. But it's worth documenting this assumption with a comment, or restricting the downgrade to only rows that were explicitly migrated — for example by checking that meta->>'provider' is not null rather than using COALESCE:

WHERE resource_type = 'IDP App'
AND meta->>'provider' = 'okta'

This would leave any IDP App rows with no provider key untouched (their resource_type would remain IDP App after downgrade), which is safer than incorrectly labelling them as Okta.

[claude]

Nice to have: Minor round-trip data change for null-meta Okta App rows

An Okta App row with meta = NULL will end up with meta = '{}'::jsonb (empty JSON object) after a full upgrade → downgrade cycle. This happens because:

1. Upgrade step 2 sets meta = COALESCE(NULL, '{}') || '{"provider":"okta"}' → '{"provider":"okta"}'
2. Downgrade step 4 removes the provider key → '{}'::jsonb

The result is semantically equivalent and practically harmless, but worth being aware of. A comment noting this side-effect would help future maintainers.

[claude]

Suggestion: No migration tests

Other data migrations in this repo have corresponding tests (see tests/ops/migration_tests/test_custom_reports_migration.py and tests/ops/migration_tests/test_system_dictionary_data_migration.py for examples). Those tests exercise the upgrade and downgrade functions directly against a real DB session.

Given this migration makes four separate SQL updates (some of which are order-dependent), a test that:

1. Seeds stagedresource rows for Okta App (with okta_app_id), Entra App, and a null-meta Okta App
2. Runs the upgrade statements
3. Asserts resource_type = 'IDP App', provider is set correctly, app_id present, okta_app_id absent
4. Runs the downgrade statements
5. Asserts original state is restored

...would give confidence that the order of operations is correct and would catch any future regressions from other migrations touching this table.

[dsill-ethyca]

Migration verified end-to-end against a live DB with 40 real Okta resources — upgrade, API verification, downgrade, and API verification again all passed. Skipping formal test for this data-only migration given no customers are using IDP resources yet.

[claude]

Suggestion: The okta_app_id → app_id rename applies to all rows where meta contains okta_app_id, regardless of resource_type. If any non-Okta row ever has that key, it will be renamed here. Consider tightening the filter:

WHERE jsonb_exists(meta, 'okta_app_id')
  AND resource_type = 'Okta App'

This also creates a downgrade asymmetry: the reverse step (app_id → okta_app_id) is scoped to WHERE resource_type = 'Okta App', so any non-Okta row affected by the upgrade rename would not be restored on downgrade.

[dsill-ethyca]

Scoped the rename to resource_type = 'Okta App' for symmetry with the downgrade.

[claude]

Suggestion: The COALESCE(meta->>'provider', 'okta') = 'okta' default silently classifies any IDP App row that has no provider key as an Okta resource. For a clean upgrade→downgrade cycle this is safe (the upgrade always writes a provider key), but if application code ever inserts an IDP App row without a provider, a subsequent downgrade would silently promote it to Okta App without warning. Consider being explicit and using meta->>'provider' = 'okta' — or at minimum add a comment explaining the intentional default.

[dsill-ethyca]

Updated to use explicit meta->>'provider' = 'okta' — rows without a provider key will be left as-is during downgrade.

[claude]

Nice to have: Any IDP App row whose provider is neither 'okta' nor 'entra' (e.g. a future provider added later) will not match either of the first two UPDATE statements and will remain as resource_type = 'IDP App' after the downgrade. The provider removal step below would also skip it (since it only targets 'Okta App' and 'Entra App'). This is not a concern today, but worth a short comment acknowledging the gap so it's not surprising if/when new providers are added.

[dsill-ethyca]

Acknowledged — only okta and entra providers exist today. Future providers would need this downgrade updated, but that's expected for any migration that predates them.