Disable a reference (without deleting it)

I'm wondering if someone with a bit more knowledge about entity reference internals could point me in the right direction here:

In my usecase, it would be particularly useful if an entity reference could be disabled, without removing it all together.

The result should be the same as if the reference was deleted, except an administrator should see a checkbox at the reference, which would be unchecked if disabled (default checked).

See also screenshot for illustration.

I'm particularly interested if there are hooks available to accomplish this, without forking the module too much.

Related:
#1914736: Programmatically disable an entity reference because there is no entity it can reference anymore