Skip to content

gh-105766: Add Locking Around Custom Allocators #105619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericsnowcurrently wants to merge 8 commits into from
Closed

gh-105766: Add Locking Around Custom Allocators #105619

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fcbeb6d
Add wrapper allocators with locking when needed.
ericsnowcurrently Jun 7, 2023
fbe26eb
Fix set_allocator_unlocked.
ericsnowcurrently Jun 10, 2023
7191629
Implement should_lock().
ericsnowcurrently Jun 7, 2023
dd60989
Implement acquire_custom_allocator_lock() and release_custom_allocato…
ericsnowcurrently Jun 7, 2023
0f2a242
Merge branch 'main' into lock-around-custom-allocators
ericsnowcurrently Jul 27, 2023
321d1c2
Check for overflow/underflow.
ericsnowcurrently Jul 27, 2023
52c01ea
Use an unsigned int.
ericsnowcurrently Jul 27, 2023
a213597
Only modify num_gils if the interpreter has its own GIL.
ericsnowcurrently Jul 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Implement should_lock().
  • Loading branch information
@ericsnowcurrently
ericsnowcurrently committed Jun 10, 2023
commit 71916298ec746f53c8790f64d0f9e8c86ceb3679
1 change: 1 addition & 0 deletions Include/internal/pycore_pymem.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ struct _pymem_allocators {
debug_alloc_api_t obj;
} debug;
PyObjectArenaAllocator obj_arena;
int num_gils;
Copy link
Member

@gpshead gpshead Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest unsigned int

struct {
PyMemAllocatorEx mem;
PyMemAllocatorEx obj;
Expand Down
9 changes: 8 additions & 1 deletion Objects/obmalloc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -667,7 +667,14 @@ PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)
static int
should_lock(void)
{
// XXX check for main interpreter, etc.
if (_PyRuntime.allocators.num_gils <= 1) {
return 0;
}
PyInterpreterState *interp = _PyInterpreterState_GET();
PyInterpreterState *main_interp = _PyInterpreterState_Main();
if (interp->ceval.gil == main_interp->ceval.gil) {
return 0;
}
return 1;
}

Expand Down
9 changes: 9 additions & 0 deletions Python/pylifecycle.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,7 @@ static PyStatus
init_interp_create_gil(PyThreadState *tstate, int own_gil)
{
PyStatus status;
_PyRuntimeState *runtime = tstate->interp->runtime;

/* finalize_interp_delete() comment explains why _PyEval_FiniGIL() is
only called here. */
Expand All @@ -603,6 +604,9 @@ init_interp_create_gil(PyThreadState *tstate, int own_gil)
if (_PyStatus_EXCEPTION(status)) {
return status;
}
HEAD_LOCK(runtime);
runtime->allocators.num_gils++;
Copy link
Member

@gpshead gpshead Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for overflow. if the value is already INT_MAX pre-increment, we need to bail, even if that means SystemError.

HEAD_UNLOCK(runtime);

return _PyStatus_OK();
}
Expand Down Expand Up @@ -1730,6 +1734,11 @@ finalize_interp_delete(PyInterpreterState *interp)
/* Cleanup auto-thread-state */
_PyGILState_Fini(interp);

_PyRuntimeState *runtime = interp->runtime;
HEAD_LOCK(runtime);
runtime->allocators.num_gils--;
Copy link
Member

@gpshead gpshead Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add an assert(runtime->allocators.num_gils > 0); before this.

HEAD_UNLOCK(runtime);

/* We can't call _PyEval_FiniGIL() here because destroying the GIL lock can
fail when it is being awaited by another running daemon thread (see
bpo-9901). Instead pycore_create_interpreter() destroys the previously
Expand Down