Skip to content

BindTemporaryResource hangs on new IDMLDevice after prior device removal during operator initializer dispatch #735

Open
Open
BindTemporaryResource hangs on new IDMLDevice after prior device removal during operator initializer dispatch#735

Description

@ALWAYSxREADY
ALWAYSxREADY
opened on May 1, 2026

Summary

IDMLBindingTable::BindTemporaryResource with a non-null buffer deadlocks CPU-side (never returns, no exception, no error HRESULT) on a freshly-created IDMLDevice when a prior IDMLDevice in the same process experienced device removal during operator initializer dispatch.

The deadlock is conditional on the cause of the prior device removal: device removal during inference does NOT cause this hang on the revival device. Only device removal during the initial operator initializer dispatch poisons subsequent device creation in the same process.

Environment

  • DirectML.dll version: 1.15.4
  • OS: Windows 11 Pro Version 25H2 (OS Build 26200.8117)
  • GPU: NVIDIA RTX 4060 Ti 16GB
  • GPU driver version: 595.97
  • D3D12 feature level: 12 (12_2)

Reproduction Conditions

The deadlock requires all three of the following:

  1. A prior IDMLDevice in the same process experienced device removal (GetDeviceRemovedReason returns DXGI_ERROR_DEVICE_HUNG / 0x887A0005) during an IDMLOperatorInitializer dispatch
  2. A new IDMLDevice is created via DMLCreateDevice after the prior device's removal (the underlying ID3D12Device is still alive — only the DML device died)
  3. A binding table is created on the new device for an IDMLOperatorInitializer whose GetBindingProperties().TemporaryResourceSize > 0, and BindTemporaryResource is called with a non-null DML_BUFFER_BINDING

When all three hold, BindTemporaryResource does not return. The calling thread blocks indefinitely.

Reproduction Sequence

// 1. First IDMLDevice runs a 2225-op operator initializer.
//    The dispatch's ExecuteCommandLists returns success and the wait fence advances,
//    but GetDeviceRemovedReason reports 0x887A0005 immediately afterward.
//    (Specific model: RAFT optical flow, 2225 ops, tempSz=50,323,456 bytes,
//    8 persistent-buffer Conv ops at persistSz=86,528 each.)

// 2. Create new IDMLDevice on the same (still-alive) ID3D12Device
ComPtr<IDMLDevice> revivalDml;
DMLCreateDevice(d3d12Device.Get(), DML_CREATE_DEVICE_FLAG_NONE,
                IID_PPV_ARGS(&revivalDml));  // succeeds

// 3. Build operator initializer for the same 2225 ops
ComPtr<IDMLOperatorInitializer> init;
revivalDml->CreateOperatorInitializer(2225, compiledOps, IID_PPV_ARGS(&init));

DML_BINDING_PROPERTIES bp = init->GetBindingProperties();
// bp.TemporaryResourceSize    = 50,323,456
// bp.PersistentResourceSize   = 692,224  (8 * 86,528)
// bp.RequiredDescriptorCount  = 84

// 4. Allocate fresh resources on revival device
//    (NOT reused from prior device — all freshly created)
ComPtr<ID3D12DescriptorHeap> heap = CreateDescriptorHeap(84);
ComPtr<IDMLBindingTable>     bt   = CreateBindingTable(init.Get(), heap.Get(), 84);
ComPtr<ID3D12Resource>       tempBuf = CreateUAVBuffer(50'323'456);

// 5. Bind temporary resource — DEADLOCKS HERE, never returns
DML_BUFFER_BINDING bb = { tempBuf.Get(), 0, 50'323'456 };
DML_BINDING_DESC   desc = { DML_BINDING_TYPE_BUFFER, &bb };
bt->BindTemporaryResource(&desc);   // CPU hang

Activity

  1. ALWAYSxREADY commented on May 1, 2026

    @ALWAYSxREADY
    ALWAYSxREADY
    on May 1, 2026
    Author

    Conditions That Do NOT Trigger The Hang

    I have verified the following cases all complete normally:

    Scenario | Result -- | -- BindTemporaryResource(nullptr) (null binding) on revival device | Passes Prior DML device removal caused by inference (~600 frames after a successful smaller batch init) rather than operator initializer dispatch | Passes — revival device is not poisoned Operator initializer with TemporaryResourceSize == 0 | Passes (binding skipped/null) First-ever DML device in process (no prior device removal) | Passes

    Every batch with TemporaryResourceSize > 0 triggered initial device removal. The 8-op subset has TemporaryResourceSize = 0 and survives trivially via the null-bind path. Determinism was confirmed with multiple runs at op counts 1112 and 2225.

    The same TemporaryResourceSize > 0 condition controls whether the revival device's BindTemporaryResource deadlocks.

    Eliminated Hypotheses

    • Stale temporary buffer state — fresh buffer on revival device, same hang
    • Stale descriptor heap state — fresh heap on revival device, same hang
    • D3D12 queue/fence issues — same D3D12 device passes other workloads, fences advance normally
    • Specific op type (initially suspected InstanceNormalization) — falsified by the threshold investigation; op type is irrelevant
    • TDR — ruled out by timing and absence of TDR signals

    What I'd Find Useful

    • Confirmation whether this is a known issue in DirectML 1.15.4
    • Whether a newer DirectML release addresses it
    • Whether there is a recommended pattern for recovering from operator-initializer-caused device removal within a single process, or whether process restart is the only viable recovery path

    Additional Context

    The model is a RAFT-style optical flow network compiled to DirectML with 2225 fused ops. The initial operator initializer's ExecuteCommandLists returns success and the wait fence advances, but GetDeviceRemovedReason reports DXGI_ERROR_DEVICE_HUNG (0x887A0005) immediately afterward.

    Happy to provide a standalone C++ reproducer, PIX or DRED captures, or the compiled DML graph if useful — let me know what would help most.

  2. ALWAYSxREADY
    closed this as completedon May 1, 2026
  3. ALWAYSxREADY
    reopened this on May 1, 2026

  4. ALWAYSxREADY commented on May 1, 2026

    @ALWAYSxREADY
    ALWAYSxREADY
    on May 1, 2026
    Author

    Update: I tested whether ORT's DirectML EP avoids this deadlock by using a different DML device lifecycle. It does not — ORT's CreateSession hangs identically when a prior IDMLDevice in the same process experienced operator-initializer-caused device removal. ORT calls AppendExecutionProvider_DML1, which creates its own IDMLDevice via DMLCreateDevice. That device is poisoned per the conditions in the original issue, and BindTemporaryResource deadlocks during op initialization. This rules out the ONNX Runtime DML EP as a workaround for callers hitting this bug. Both the standalone DirectML API and ORT's DML EP share the same DirectML.dll codepath and exhibit identical behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions