Skip to content

fix(cuda.core): serialize pinned-memory kernel-access test on Windows - #3436

Merged
rwgk merged 14 commits into
NVIDIA:mainfrom
jpascucci-nv:issue3426
Nov 13, 2026
Merged

rwgk merged 14 commits into
NVIDIA:mainfrom
jpascucci-nv:issue3426

Conversation

@jpascucci-nv

@jpascucci-nv jpascucci-nv commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

related to #3426

Add @thread_unsafe_on_windows to the pinned-memory kernel-access test. The test accesses host memory while parallel workers may execute GPU work; synchronizing its own stream does not coordinate those workers.

This follows the guard used by similar tests on Windows and WSL. Windows CI is needed to confirm it resolves the reported crashes. The separate OOM failures remain under investigation.

Validation: Ruff and git diff --check pass.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

closes NVIDIA#3426

Add @thread_unsafe_on_windows to the pinned-memory kernel-access test.
The test accesses host memory while parallel workers may execute GPU
work; synchronizing its own stream does not coordinate those workers.

This follows the guard used by similar tests on Windows and WSL.
Windows CI is needed to confirm it resolves the reported crashes.
The separate OOM failures remain under investigation.

Validation: Ruff and git diff --check pass.
@jpascucci-nv jpascucci-nv added this to the cuda.core 1.3.0 milestone Nov 10, 2026
@jpascucci-nv jpascucci-nv self-assigned this Nov 10, 2026
@jpascucci-nv jpascucci-nv added bug Something isn't working P1 Medium priority - Should do test Improvements or additions to tests labels Nov 10, 2026
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Nov 10, 2026
@jpascucci-nv jpascucci-nv changed the title fix(cuda.core): serialize pinned-memory kernel-access test on Windows fix(cuda.core): serialize pinned-memory kernel-access test on Windows Nov 10, 2026
@jpascucci-nv jpascucci-nv changed the title fix(cuda.core): serialize pinned-memory kernel-access test on Windows fix(cuda.core): serialize pinned-memory kernel-access test on Windows Nov 10, 2026
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Updated cleanup behavior for host and NUMA memory pools to preserve peer access; other pool types and IPC-imported pools retain their existing cleanup behavior.
  • Tests
    • Updated Windows handling for a pinned-memory test and marked a memory-pool reset test as thread-unsafe.
    • Improved CUDA test cleanup so the current context is cleared after synchronization, including when synchronization raises an error. Added coverage for cleanup when the test wrapper has already removed the context.

The changes adjust peer-access cleanup for HOST and HOST_NUMA memory pools, update CUDA fixture teardown to synchronize through the bound device and attempt context cleanup, and add thread-safety markers to two memory tests.

Changes: Memory pool peer-access cleanup

Layer / File(s) Summary
Apply location-specific peer-access cleanup
cuda_core/cuda/core/_cpp/rt/memory.cpp
The pool wrapper now accepts a flag that controls peer-access revocation. HOST and HOST_NUMA pools disable revocation during destruction. Other pool locations and IPC-imported pools retain revocation.

CUDA fixture teardown

Layer / File(s) Summary
Synchronize and unset the current context
cuda_core/tests/conftest.py, cuda_core/tests/memory/test_init_cuda_collect_gate.py
The teardown calls device.sync() and attempts to unset the current context in a finally block. Tests cover intact and already-popped fixture contexts, and verify that a synchronization CUDAError propagates while cleanup runs.

Memory test thread-safety guards

Layer / File(s) Summary
Mark thread-sensitive memory tests
cuda_core/tests/test_memory.py
The pinned-memory kernel-access test receives a Windows thread-safety decorator. The watermark-reset test receives a thread-unsafe marker for a legacy-default-stream mempool teardown and synchronization race.

Possibly related issues

  • NVIDIA/cuda-python#3426: The change adds the suggested Windows guard to the pinned-memory kernel-access test. The separate pinned-memory OOM failures remain outside this change.
  • NVIDIA/cuda-python#3414: The change affects the Windows execution of the pinned-memory kernel-access test, but does not add the broader kernel-test parametrization described by the issue.
  • NVIDIA/cuda-python#3350: The watermark-reset test receives a thread-unsafe marker for a related mempool teardown and synchronization race.

Suggested reviewers: mdboom

Change: Bug fix

Merge Risk: Low · previous review e2512

The change is mergeable with awareness that simultaneous fixture teardown failures can obscure the original test failure. The reported Windows crashes still need CI confirmation.

Security architecture risk: Low · up to e2512

Device-pool and imported-pool cleanup retain their existing controls. Host-pool destruction now skips device-access revocation on every platform, but the effect on access permissions after handle reuse remains unverified. No cross-process compromise is established.

Retained concerns

  • Low · security · inferred: Owned HOST and HOST_NUMA pools no longer revoke device access before destruction. Because creation grants the current device READWRITE access, a subsequent pool could inherit an unintended device grant if host handles exhibit the recycling behavior documented for device pools. The change applies without a platform guard. Host-specific inheritance remains unverified, so this is an isolation-contract concern rather than a demonstrated vulnerability.
Security review details

Security Blast Radius

  • inferred — The changed policy reaches newly created owned HOST and HOST_NUMA pools on all platforms using this creator, not just the marked Windows test. The supported concern is device-access state across local pool lifetimes. The evidence does not establish an attacker-controlled remote entry point, cross-tenant exposure, or automatic cross-process access.

Security Findings and Attack Paths

  • inferred — The conditional failure path is: a host pool grants device access, closes without revocation, and its handle is reused for a pool with different intended device access. An old device grant could then outlive its intended pool lifetime. Actual host-pool inheritance, GPU reachability of later allocations, and attacker control are not established; the existing reuse regression test exercises device pools only.

Trust Boundaries and Controls

  • observed — Device-created pools and IPC-imported pools retain revocation before destruction. IPC access also requires an exported pool capability and allocation descriptor. Existing device-memory IPC tests show that an importing process must establish its own peer grants; those tests do not resolve host-specific IPC behavior.

Resilience and Maintainability Implications

  • observed — Ownership remains shared-pointer managed, with destruction attempted on final owning-reference release. Creation and import failures still return before wrapping an owned handle. Explicit close and outstanding asynchronous-allocation handling are unchanged, so their unresolved driver semantics are not identified as new PR risks.

Hardening Proposals

  • proposed — Validate HOST and HOST_NUMA destruction and handle reuse with differing device grants, including driver support for revocation. Use the result to establish whether destruction alone preserves the access-reset invariant or an alternative cleanup control is required.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between bcfd4c4 and e251232.

Files selected for processing: 1

  • cuda_core/cuda/core/_cpp/rt/memory.cpp

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@jpascucci-nv
jpascucci-nv enabled auto-merge (squash) November 10, 2026 19:50
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Doc Preview CI
Preview removed because the pull request was closed or merged.

Check the cuCtxSynchronize result in init_cuda so asynchronous errors
surface during the triggering test's teardown instead of going unnoticed
until a later test. Always attempt context cleanup via finally.

Related to NVIDIA#3426.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between aa83d4b and 730fcbe.

Files selected for processing: 1

  • cuda_core/tests/conftest.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Comment thread cuda_core/tests/conftest.py Resolved
Use device.sync() because the test wrapper may pop the current context
before init_cuda teardown. This avoids spurious INVALID_CONTEXT errors
while preserving synchronization error reporting and context cleanup.

Add regression coverage for a missing current context and cleanup after
synchronization failures.

Related to NVIDIA#3426.
Mark test_mempool_watermark_reset thread_unsafe, matching test_mempool_trim.
Both exercise the pool teardown and synchronization sequence associated
with the suspected driver race in NVIDIA#3350.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick comments: 1

cuda_core/tests/memory/test_init_cuda_collect_gate.py: 1

254-276: Trivial · Maintainability & Code Quality · Heavy lift

suggestion: Assert that fixture-context work completes

When the context is already popped, this test checks only context cleanup. A regression can leave work undrained and still pass. Queue controlled work in the fixture context and assert its completion after teardown.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 587c111 and bcfd4c4.

Files selected for processing: 1

  • cuda_core/tests/test_memory.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Skip peer-access revocation for locally created HOST and HOST_NUMA pools
to investigate pinned-pool teardown OOMs and crashes. Continue destroying
the pools and preserve cleanup for device, managed, and imported pools.

This is a diagnostic workaround pending Windows CI validation.

Related to NVIDIA#3426.
Keep setup or synchronization failures primary and attach subsequent
cleanup failures as exception notes. Always attempt to pop the context.

Add regression coverage for combined failures, including interrupts.
@rwgk

rwgk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@jpascucci-nv @Andy-Jost — I'd like to get this merged before I rebase and run final CI for the #3322 stack tonight, starting around 5pm PST. Today's Pathfinder release CI only went green on the seventh attempt, and I'd really like to avoid repeating that while landing the stack.

An earlier codex's CI log review already found 88 direct pinned-memory kernel-test crashes across 39 runs, concentrated in three free-threaded Windows configurations. The latest candidate here passed all 21 Windows cuda.core jobs on their first attempt, including those three configurations. That's encouraging evidence for a substantial reduction in this crash family.

Codex also flagged a remaining scope issue here: the host-pool cleanup bypass also affects host-preferred managed-memory pools, beyond the stated pinned-memory scope. Since I already have the investigation and review context in a Codex session, I'll add follow-up fix commits here, in hopes that we can get the updated diff reviewed and merged this afternoon.

@jpascucci-nv

Copy link
Copy Markdown
Contributor Author

@rwgk That's fine, go ahead and make your updates here.

HOST and HOST_NUMA also describe managed-pool preferences. Check the allocation type so the pinned-memory workaround preserves managed-pool peer-access revocation.

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@juenglin

juenglin commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

test_pinned_memory_resource_kernel_access allocates through _allocate_pinned_buffer_or_xfail (cuda_core/tests/test_memory.py:1278), which turns CUDA_ERROR_OUT_OF_MEMORY into pytest.xfail. The out-of-memory reason checker added for #2381 only looks at failed reports, so those errors produce no diagnostics.

On Windows L4 (MCDM) with Python 3.14 and CUDA 13.5.0, kernel_access[current] ends in that xfail, and so do the two older tests that construct PinnedMemoryResource() without options: test_non_managed_resources_report_not_managed[pinned] and test_pinned_memory_resource_initialization. This happens in passing runs too. The push run at #3405's commit passed with all three recorded as xfail for CUDA_ERROR_OUT_OF_MEMORY, and in the last nightly before #3405, which also passed, the two older tests show the same xfail in all five repetitions. So allocating from the default host pool already ran out of memory on this configuration before #3405, and the TODO(#9999) xfail has been hiding it.

This matters for the PR in two ways. First, [current] stops at the allocation on this configuration, before the kernel launch, so only [new] and [new_numa] validate kernel access there. Second, the xfails at 49% on the PR's L4 job aren't what starts its cascade at 52%. They also occur in runs that pass, and in the first failing nightly after #3405 the first out-of-memory error came at 15%, before any of these tests ran. The checker's "first CUDA_ERROR_OUT_OF_MEMORY" label is wrong, but the error it reports is still the first one that isn't this known error.

Calling mr.allocate directly here would make the diagnostics worse. [current] would then fail on this configuration, and because the checker captures only once per session, it would spend that capture on this known error whenever this test runs before the cascade begins. The default-host-pool error needs its own issue, which the TODO(#9999) placeholder can then reference.

Edit: rewritten after checking more runs. The xfail is a pre-existing out-of-memory error, so I've withdrawn my suggestion to call mr.allocate directly.

juenglin added a commit to juenglin/cuda-python that referenced this pull request Nov 12, 2026
…op redundant gh install from build-wheel.yml

test_pinned_memory_resource_kernel_access crashes under pytest-run-parallel
on free-threaded Windows builds (py3.14t, py3.15t) with a Windows fatal
access violation. The fix is in progress in NVIDIA#3436; skip on Windows until
it lands.

gh is pre-installed on all runners that build-wheel.yml uses (GitHub-hosted
Windows and NVIDIA self-hosted Linux/aarch64), so the install-gh-cli step
there was always a no-op. CI on the latest commit confirms this. Drop it.
test-wheel-linux.yml keeps its step because the ubuntu:24.04 container
does not ship gh.
rwgk added 2 commits November 12, 2026 14:32
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Nov 12, 2026
@rwgk

rwgk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Quick update: my codex session jumped on addressing:

  • @juenglin's comment above
  • and root-causing a second flake we hit here ("watermark/OOM cascade")

at the same time, before I got a chance to ask it to separate the concerns.
I'm holding off stopping it for the minute, to see how far it gets with what it started already.

rwgk added 2 commits November 12, 2026 15:13
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@rwgk

rwgk commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This PR should be back on track now. The root-causing of the watermark/OOM cascade flakiness continues separately. Sorry for conflating those things accidentally.

@juenglin

Copy link
Copy Markdown
Contributor

@rwgk our updates crossed: I rewrote my comment above before I saw 8812d2b, and the rewrite withdraws the suggestion that commit implements. The L4 job at c4f114a13a5 shows why. With mr.allocate called directly, kernel_access[current] fails at 49% with the known default-host-pool error, and the checker spends its one capture there, with the verdict "inconclusive: all probes succeeded". The cascade still starts at 52%, in the teardown of test_mempool_watermark_reset[PinnedMR-used], and runs to an access violation at 93% without diagnostics.

9bdfbe3 moved the rest of 8812d2b out of this PR, but its test_memory.py line is still here. Could you revert that line too?

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@rwgk

rwgk commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@jpascucci-nv @Andy-Jost are you still around, I asked codex if the PR title is still suitable, because the changes in cuda_core/cuda/core/_cpp/rt/memory.cpp look like a production code bug fix. Here is the response:

codex GPT-6.1-Sol ultra:

The title describes the original test change, but now understates the PR’s scope. My earlier framing was too focused on test mitigation.

The runtime change changes teardown for real applications: locally created pinned host/NUMA pools skip GPU-access revocation before attempting destruction. That applies on all platforms.

I’d suggest:

fix(cuda.core): avoid revoking access during pinned host pool teardown

The description should lead with that production workaround, then explain the test serialization and fixture improvements. “Workaround” reflects the evidence accurately while the underlying crash mechanism remains incompletely understood.

@juenglin

Copy link
Copy Markdown
Contributor

On the [current] round trip: my original suggestion was meant to expose what the xfail hid, in case it was the cascade's real first out-of-memory error. [current] fails the same way on six Windows jobs (WDDM and MCDM; RTX 4090, RTX 2080, A100, L4), those jobs finish with that one failure and no cascade, and where the checker could probe, every probe succeeded with about 93% of device memory free. So it's a separate default-host-pool problem, and restoring the xfail keeps the checker's one capture for the cascade.

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@rwgk

rwgk commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Heads up: after consulting with @Andy-Jost, we decided to backtrack even more here, by backing out the production code changes. Most likely we'll still get a substantial reduction in flakiness. The other changes need more thought and/or root causing.

@rwgk

rwgk commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

I just checked: @jpascucci-nv's test and fixture changes are unchanged after today's back and forth. Apart from merging main, the only net change is removal of the production workaround in memory.cpp, leaving this PR test-only.

@rwgk rwgk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To alleviate the very substantial CI flakiness, I'm approving this based on intense reviewing with codex, and consultation with @Andy-Jost; offline Andy wrote:

The test changes seem safe enough.

All other changes were backed out, pending follow-on investigations.

@rwgk
rwgk enabled auto-merge (squash) November 13, 2026 01:17
@rwgk
rwgk merged commit b715078 into NVIDIA:main Nov 13, 2026
217 of 219 checks passed
github-actions Bot pushed a commit that referenced this pull request Nov 13, 2026
Removed preview folders for the following PRs:
- PR #3323
- PR #3357
- PR #3433
- PR #3436
- PR #3439
- PR #3449
Andy-Jost added a commit that referenced this pull request Nov 18, 2026
* [doc-only] docs(cuda.core): fix the floor-move wording; advise against pinning cuda-bindings (#3441)

* docs(cuda.core): drop the Breaking Changes section name from the floor policy

The release notes no longer have a Breaking Changes section for a floor
move; 1.3.0 lists it under Highlights. Say only that the release notes
list every move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cuda.core): advise against pinning cuda-bindings next to cuda-core

Users who pin cuda-bindings themselves can conflict with the floor of a
later cuda-core release. Tell them to depend on cuda-core with the cu12
or cu13 extra and let cuda-core choose the cuda-bindings version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 83a3422)

* Mark cuda-gdb source-line test thread_unsafe (#3433)

Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
(cherry picked from commit 1a22eec)

* fix(cuda.core): serialize pinned-memory kernel-access test on Windows (#3436)

* fix(cuda.core): serialize pinned-memory kernel-access test on Windows

closes #3426

Add @thread_unsafe_on_windows to the pinned-memory kernel-access test.
The test accesses host memory while parallel workers may execute GPU
work; synchronizing its own stream does not coordinate those workers.

This follows the guard used by similar tests on Windows and WSL.
Windows CI is needed to confirm it resolves the reported crashes.
The separate OOM failures remain under investigation.

Validation: Ruff and git diff --check pass.

* fix(cuda.core): report CUDA synchronization errors during test teardown

Check the cuCtxSynchronize result in init_cuda so asynchronous errors
surface during the triggering test's teardown instead of going unnoticed
until a later test. Always attempt context cleanup via finally.

Related to #3426.

* fix(cuda.core): synchronize the bound context during test teardown

Use device.sync() because the test wrapper may pop the current context
before init_cuda teardown. This avoids spurious INVALID_CONTEXT errors
while preserving synchronization error reporting and context cleanup.

Add regression coverage for a missing current context and cleanup after
synchronization failures.

Related to #3426.

* fix(cuda.core): serialize memory-pool watermark reset tests

Mark test_mempool_watermark_reset thread_unsafe, matching test_mempool_trim.
Both exercise the pool teardown and synchronization sequence associated
with the suspected driver race in #3350.

* fix(cuda.core): bypass access revocation when destroying host pools

Skip peer-access revocation for locally created HOST and HOST_NUMA pools
to investigate pinned-pool teardown OOMs and crashes. Continue destroying
the pools and preserve cleanup for device, managed, and imported pools.

This is a diagnostic workaround pending Windows CI validation.

Related to #3426.

* fix(cuda.core): preserve the first exception during fixture cleanup

Keep setup or synchronization failures primary and attach subsequent
cleanup failures as exception notes. Always attempt to pop the context.

Add regression coverage for combined failures, including interrupts.

* fix(cuda.core): limit host-pool cleanup workaround to pinned allocations

HOST and HOST_NUMA also describe managed-pool preferences. Check the allocation type so the pinned-memory workaround preserves managed-pool peer-access revocation.

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* test(cuda.core): expose pinned allocation OOM and retain its traceback

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: diagnose PR3436 pinned pool teardown on Windows

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: move watermark investigation out of PR3436

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* test(cuda.core): restore expected pinned allocation OOM handling

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* revert(cuda.core): defer pinned host pool cleanup workaround

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

---------

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
(cherry picked from commit b715078)

* [no-ci] ci: exclude unavailable historical GitHub references (#3471)

* ci: exclude unavailable historical GitHub references

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: require manual copy-pr-bot approval

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

---------

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
(cherry picked from commit 3991a69)

* cuda.core: map ml_dtypes bfloat16 to kDLBfloat in DLPack export (#3359)

_smv_dtype_numpy_to_dlpack dispatched on numpy dtype kind, but
ml_dtypes.bfloat16 has kind "V", so bf16 views fell through to the
unsupported-dtype BufferError and could not be re-exported, even though
the DLPack import path already accepts kDLBfloat and _tensor_map.pyx
maps bfloat16. Detect bfloat16 up front and export it as kDLBfloat
(16 bits).

The new round-trip test goes through StridedMemoryView.from_dlpack so
it cannot skip on stacks where NumPy's own DLPack support for extended
dtypes is unavailable, unlike the existing parametrized round-trip
test's NumPy-consumer probe.

Signed-off-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Co-authored-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Co-authored-by: Sebastian Berg <sebastianb@nvidia.com>
(cherry picked from commit d1d1457)

* fix(cuda.core): make IPC imports wait for a pending release of the same descriptor (#3450)

* fix(cuda.core): make IPC imports wait for a pending release of the same descriptor

deviceptr_import_ipc() shares one handle per descriptor because the
driver's first free of an imported pointer unmaps every import of it
(nvbug 5570902). The handle's deleter took ipc_import_mutex only after
the last reference was gone. In that gap an import found the cache entry
expired, re-imported the still-mapped pointer and registered a new entry.
The deleter then erased that new entry and freed the pointer, which
unmapped it under the new Buffer.

The cache entry now outlives the last reference and only its own deleter
erases it, under the mutex, together with the free. An import that finds
an expired entry waits on a condition variable until that release is
done, then imports fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Use CTAD for the ipc_import_mutex locks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2f1185e)

* test(cuda.core): skip HOST graph allocation before CUDA 13 (#3446)

(cherry picked from commit 11cc666)

* test(cuda.core): query green stream contexts with v2 (#3448)

Signed-off-by: Rui Luo <ruluo@nvidia.com>
(cherry picked from commit 7076a67)

* test(cuda.core): skip the VMM RDMA policy test on WSL (#3478)

GPUDirect RDMA is not supported on WSL, but some drivers report the
capability there, and the RDMA-enabled VMM allocation in
test_vmm_allocator_policy_configuration then fails. Skip the test on
WSL; the capability checks stay as they are.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 36bc530)

* docs(cuda.core): add 1.3.0 release note for the IPC import/release race fix (#3479)

* docs(cuda.core): add 1.3.0 release note for the IPC import/release race fix

Follow-up to #3450, which merged without a release note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cuda.core): scope the IPC race release note to memory pool buffers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit de4c60d)

* Fix conditional graph lifetime tests to use supported nodes (#3469)

* Fix conditional graph lifetime tests to use supported nodes

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

* ci: require manual copy-pr-bot approval

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>

---------

Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Co-authored-by: Andy Jost <ajost@nvidia.com>
(cherry picked from commit b80866b)

* fix(cuda.core): run cuMemPoolSetAccess in relaxed mode in PinnedMemoryResource (#3475)

Since #3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool.  The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread.  Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.

Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since #2834.
Call it from _PMR_init instead of the bare with-nogil block.

The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.

Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.

Closes #3451

Co-authored-by: Andy Jost <ajost@nvidia.com>
(cherry picked from commit d94f19c)

* cuda.core: carry the cuda-bindings floors through minimal-sysctk12/13 extras (#3460)

* cuda.core: carry the cuda-bindings floors through minimal-sysctk extras

The floors lived only in the cu12/cu13 extras, so a package that depends
on plain cuda-core never passed them to pip or uv, and an older
cuda-bindings failed only at import (#3459).

Declare the floors in new minimal-sysctk12/13 extras, include them from
cu12/cu13, and depend on cuda-bindings without a version. The floor
parser, the CI wheel-floor reader, the pre-commit hook and the docs read
the new extras.

* cuda.core: read cu<N> floors through the minimal-sysctk self-reference

Drop the unversioned cuda-bindings from the base dependencies, so a
plain cuda-core install keeps asking for an extra.

Keep the floor readers on the cu<N> extras, as before, and teach them
to follow a cuda-core[...] self-reference and to skip an unversioned
cuda-bindings[all]. The cu<N> layout then yields the floor written in
minimal-sysctk<N>, and the existing tests of the cu<N> form stay as
they are. New tests cover the self-reference.

---------

Co-authored-by: Andy Jost <ajost@nvidia.com>
(cherry picked from commit ad38141)

---------

Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Signed-off-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Signed-off-by: Rui Luo <ruluo@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ralf Juengling <rjuengling@nvidia.com>
Co-authored-by: jpascucci-nv <jpascucci@nvidia.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgkio@gmail.com>
Co-authored-by: B Sainath Reddy <bsainathreddy4@gmail.com>
Co-authored-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com>
Co-authored-by: Sebastian Berg <sebastianb@nvidia.com>
Co-authored-by: Rob Parolin <rparolin@users.noreply.github.com>
Co-authored-by: Rui Luo <ruluo@nvidia.com>
Co-authored-by: Leo Fang <leof@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module P1 Medium priority - Should do test Improvements or additions to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants