v2.0: Process-per-context architecture with performance improvements

CHANGELOG.md

@@ -4,6 +4,18 @@
 
 ### Added
 
+- **`erlang.reactor` module** - FD-based protocol handling for building custom servers
+  - `reactor.Protocol` - Base class for implementing protocols
+  - `reactor.serve(sock, protocol_factory)` - Serve connections using a protocol
+  - `reactor.run_fd(fd, protocol_factory)` - Handle a single FD with a protocol
+  - Integrates with Erlang's `enif_select` for efficient I/O multiplexing
+  - Zero-copy buffer management for high-throughput scenarios
+
+- **ETF encoding for PIDs and References** - Full Erlang term format support
+  - Erlang PIDs encode/decode properly in ETF binary format
+  - Erlang References encode/decode properly in ETF binary format
+  - Enables proper serialization for distributed Erlang communication
+
 - **PID serialization** - Erlang PIDs now convert to `erlang.Pid` objects in Python
   and back to real PIDs when returned to Erlang. Previously, PIDs fell through to
   `None` (Erlang→Python) or string representation (Python→Erlang).
@@ -16,12 +28,106 @@
   Subclass of `Exception`, so it's catchable with `except Exception` or
   `except erlang.ProcessError`.
 
+- **Audit hook sandbox** - Block dangerous operations when running inside Erlang VM
+  - Uses Python's `sys.addaudithook()` (PEP 578) for low-level blocking
+  - Blocks: `os.fork`, `os.system`, `os.popen`, `os.exec*`, `os.spawn*`, `subprocess.Popen`
+  - Raises `RuntimeError` with clear message about using Erlang ports instead
+  - Automatically installed when `py_event_loop` NIF is available
+
+- **Process-per-context architecture** - Each Python context runs in dedicated process
+  - `py_context_process` - Gen_server managing a single Python context
+  - `py_context_sup` - Supervisor for context processes
+  - `py_context_router` - Routes calls to appropriate context process
+  - Improved isolation between contexts
+  - Better crash recovery and resource management
+
+- **Worker thread pool** - High-throughput Python operations
+  - Configurable pool size for parallel execution
+  - Efficient work distribution across threads
+
+- **`py:contexts_started/0`** - Helper to check if contexts are ready
+
 ### Changed
 
+- **`py:call_async` renamed to `py:cast`** - Follows gen_server convention where
+  `call` is synchronous and `cast` is asynchronous. The semantics are identical,
+  only the name changed.
+
+- **Unified `erlang` Python module** - Consolidated callback and event loop APIs
+  - `erlang.run(coro)` - Run coroutine with ErlangEventLoop (like uvloop.run)
+  - `erlang.new_event_loop()` - Create new ErlangEventLoop instance
+  - `erlang.install()` - Install ErlangEventLoopPolicy (deprecated in 3.12+)
+  - `erlang.EventLoopPolicy` - Alias for ErlangEventLoopPolicy
+  - Removed separate `erlang_asyncio` module - all functionality now in `erlang`
+
+- **Async worker backend replaced with event loop model** - The pthread+usleep
+  polling async workers have been replaced with an event-driven model using
+  `py_event_loop` and `enif_select`:
+  - Removed `py_async_worker.erl` and `py_async_worker_sup.erl`
+  - Removed `py_async_worker_t` and `async_pending_t` structs from C code
+  - Deprecated `async_worker_new`, `async_call`, `async_gather`, `async_stream` NIFs
+  - Added `py_event_loop_pool.erl` for managing event loop-based async execution
+  - Added `py_event_loop:run_async/2` for submitting coroutines to event loops
+  - Added `nif_event_loop_run_async` NIF for direct coroutine submission
+  - Added `_run_and_send` wrapper in Python for result delivery via `erlang.send()`
+  - **Internal change**: `py:async_call/3,4` and `py:await/1,2` API unchanged
+
 - **`SuspensionRequired` base class** - Now inherits from `BaseException` instead
   of `Exception`. This prevents ASGI/WSGI middleware `except Exception` handlers
   from intercepting the suspension control flow used by `erlang.call()`.
 
+- **Per-interpreter isolation in py_event_loop.c** - Removed global state for
+  proper subinterpreter support. Each interpreter now has isolated event loop state.
+
+- **ErlangEventLoopPolicy always returns ErlangEventLoop** - Previously only
+  returned ErlangEventLoop for main thread; now consistent across all threads.
+
+### Removed
+
+- **Context affinity functions** - Removed `py:bind`, `py:unbind`, `py:is_bound`,
+  `py:with_context`, and `py:ctx_*` functions. The new `py_context_router` provides
+  automatic scheduler-affinity routing. For explicit context control, use
+  `py_context_router:bind_context/1` and `py_context:call/5`.
+
+- **Signal handling support** - Removed `add_signal_handler`/`remove_signal_handler`
+  from ErlangEventLoop. Signal handling should be done at the Erlang VM level.
+  Methods now raise `NotImplementedError` with guidance.
+
+- **Subprocess support** - ErlangEventLoop raises `NotImplementedError` for
+  `subprocess_shell` and `subprocess_exec`. Use Erlang ports (`open_port/2`)
+  for subprocess management instead.
+
+### Fixed
+
+- **FD stealing and UDP connected socket issues** - Fixed file descriptor handling
+  for UDP sockets in connected mode
+
+- **Context test expectations** - Updated tests for Python contextvars behavior
+
+- **Unawaited coroutine warnings** - Fixed warnings in test suite
+
+- **Timer scheduling for standalone ErlangEventLoop** - Fixed timer callbacks not
+  firing for loops created outside the main event loop infrastructure
+
+- **Subinterpreter cleanup and thread worker re-registration** - Fixed cleanup
+  issues when subinterpreters are destroyed and recreated
+
+- **Thread worker handlers not re-registering after app restart** - Workers now
+  properly re-register when application restarts
+
+- **Timeout handling** - Improved timeout handling across the codebase
+
+- **Eval locals_term initialization** - Fixed uninitialized variable in eval
+
+- **Two race conditions in worker pool** - Fixed concurrent access issues
+
+### Performance
+
+- **Async coroutine latency reduced from ~10-20ms to <1ms** - The event loop model
+  eliminates pthread polling overhead
+- **Zero CPU usage when idle** - Event-driven instead of usleep-based polling
+- **No extra threads** - Coroutines run on the existing event loop infrastructure
+
 ## 1.8.1 (2026-02-25)
 
 ### Fixed
@@ -102,16 +208,15 @@
 ### Added
 
 - **Shared Router Architecture for Event Loops**
-  - Single `py_event_router` process handles all event loops (both shared and isolated)
+  - Single `py_event_router` process handles all event loops
   - Timer and FD messages include loop identity for correct dispatch
   - Eliminates need for per-loop router processes
   - Handle-based Python C API using PyCapsule for loop references
 
-- **Isolated Event Loops** - Create isolated event loops with `ErlangEventLoop(isolated=True)`
-  - Default (`isolated=False`): uses the shared global loop managed by Erlang
-  - Isolated (`isolated=True`): creates a dedicated loop with its own pending queue
-  - Full asyncio support (timers, FD operations) for both modes
-  - Useful for multi-threaded Python applications where each thread needs its own loop
+- **Per-Loop Capsule Architecture** - Each `ErlangEventLoop` instance has its own isolated capsule
+  - Dedicated pending queue per loop for proper event routing
+  - Full asyncio support (timers, FD operations) with correct loop isolation
+  - Safe for multi-threaded Python applications where each thread needs its own loop
   - See `docs/asyncio.md` for usage and architecture details
 
 ## 1.6.1 (2026-02-22)

README.md

@@ -32,6 +32,7 @@ Key features:
 - **AI/ML ready** - Examples for embeddings, semantic search, RAG, and LLMs
 - **Logging integration** - Python logging forwarded to Erlang logger
 - **Distributed tracing** - Span-based tracing from Python code
+- **Security sandbox** - Blocks fork/exec operations that would corrupt the VM
 
 ## Requirements
 
@@ -66,7 +67,7 @@ application:ensure_all_started(erlang_python).
 {ok, 25} = py:eval(<<"x * y">>, #{x => 5, y => 5}).
 
 %% Async calls
-Ref = py:call_async(math, factorial, [100]),
+Ref = py:cast(math, factorial, [100]),
 {ok, Result} = py:await(Ref).
 
 %% Streaming from generators
@@ -443,7 +444,7 @@ escript examples/logging_example.erl
 {ok, Result} = py:call(Module, Function, Args, KwArgs, Timeout).
 
 %% Async
-Ref = py:call_async(Module, Function, Args).
+Ref = py:cast(Module, Function, Args).
 {ok, Result} = py:await(Ref).
 {ok, Result} = py:await(Ref, Timeout).
 ```
@@ -557,8 +558,10 @@ py:execution_mode().  %% => free_threaded | subinterp | multi_executor
 ## Error Handling
 
 ```erlang
-{error, {'NameError', "name 'x' is not defined"}} = py:eval(<<"x">>).
-{error, {'ZeroDivisionError', "division by zero"}} = py:eval(<<"1/0">>).
+%% Python exceptions return {error, {TypeString, Message}}
+%% Note: Exception types are strings (not atoms) since v2.0 to prevent atom table exhaustion
+{error, {"NameError", "name 'x' is not defined"}} = py:eval(<<"x">>).
+{error, {"ZeroDivisionError", "division by zero"}} = py:eval(<<"1/0">>).
 {error, timeout} = py:eval(<<"sum(range(10**9))">>, #{}, 100).
 ```
 
@@ -573,6 +576,8 @@ py:execution_mode().  %% => free_threaded | subinterp | multi_executor
 - [Threading](docs/threading.md)
 - [Logging and Tracing](docs/logging.md)
 - [Asyncio Event Loop](docs/asyncio.md) - Erlang-native asyncio with TCP/UDP support
+- [Reactor](docs/reactor.md) - FD-based protocol handling
+- [Security](docs/security.md) - Sandbox and blocked operations
 - [Web Frameworks](docs/web-frameworks.md) - ASGI/WSGI integration
 - [Changelog](https://github.com/benoitc/erlang-python/releases)
 

benchmark_results/baseline_20260224_133948.txt.log

@@ -0,0 +1,10 @@
+Error! Failed to eval: 
+            application:ensure_all_started(erlang_python),
+            Results = py_scalable_io_bench:run_all(),
+            py_scalable_io_bench:save_results(Results, "/Users/benoitc/Projects/erlang-python/benchmark_results/baseline_20260224_133948.txt"),
+            init:stop()
+
+
+Runtime terminating during boot ({undef,[{py_scalable_io_bench,run_all,[],[]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,920}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,668}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,276}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
+
+Crash dump is being written to: erl_crash.dump...done

benchmark_results/current_20260224_133950.txt.log

@@ -0,0 +1,10 @@
+Error! Failed to eval: 
+            application:ensure_all_started(erlang_python),
+            Results = py_scalable_io_bench:run_all(),
+            py_scalable_io_bench:save_results(Results, "/Users/benoitc/Projects/erlang-python/benchmark_results/current_20260224_133950.txt"),
+            init:stop()
+
+
+Runtime terminating during boot ({undef,[{py_scalable_io_bench,run_all,[],[]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,920}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,668}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,276}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
+
+Crash dump is being written to: erl_crash.dump...done

c_src/CMakeLists.txt

@@ -50,6 +50,13 @@ endif()
 # Performance build option (for maximum optimization)
 option(PERF_BUILD "Enable aggressive performance optimizations (-O3, LTO, native arch)" OFF)
 
+# ASGI profiling option (for internal timing analysis)
+option(ASGI_PROFILING "Enable ASGI internal profiling" OFF)
+if(ASGI_PROFILING)
+    message(STATUS "ASGI profiling enabled - timing instrumentation active")
+    add_definitions(-DASGI_PROFILING)
+endif()
+
 if(PERF_BUILD)
     message(STATUS "Performance build enabled - using aggressive optimizations")
     # Override compiler flags for maximum performance
@@ -63,7 +70,12 @@ include(FindErlang)
 include_directories(${ERLANG_ERTS_INCLUDE_PATH})
 
 # Find Python using CMake's built-in FindPython3
-# Use PYTHON_CONFIG env variable to hint which Python to use
+#
+# To specify a particular Python installation, set PYTHON_CONFIG env variable:
+#   PYTHON_CONFIG=/opt/local/bin/python3.14-config cmake ...
+#
+# CMake will use its default search order otherwise.
+
 if(DEFINED ENV{PYTHON_CONFIG})
     # Extract prefix from python-config for hinting
     execute_process(
@@ -82,6 +94,67 @@ message(STATUS "Python3 include dirs: ${Python3_INCLUDE_DIRS}")
 message(STATUS "Python3 libraries: ${Python3_LIBRARIES}")
 message(STATUS "Python3 library: ${Python3_LIBRARY}")
 
+# Detect Python features for worker pool optimization
+# We use both version checks and compile tests to verify actual API availability
+
+include(CheckCSourceCompiles)
+
+# First check Python version - subinterpreters with OWN_GIL require Python 3.12+
+if(Python3_VERSION VERSION_GREATER_EQUAL "3.12")
+    message(STATUS "Python ${Python3_VERSION} >= 3.12, checking subinterpreter API...")
+
+    # Save and set required variables for compile test
+    set(CMAKE_REQUIRED_INCLUDES ${Python3_INCLUDE_DIRS})
+    set(CMAKE_REQUIRED_LIBRARIES Python3::Python)
+
+    # Clear any cached result to ensure fresh detection
+    unset(HAVE_SUBINTERPRETERS CACHE)
+
+    # Check for subinterpreter API with per-interpreter GIL (PEP 684, Python 3.12+)
+    # This verifies PyInterpreterConfig and PyInterpreterConfig_OWN_GIL are available
+    check_c_source_compiles("
+#define PY_SSIZE_T_CLEAN
+#include <Python.h>
+int main(void) {
+    PyInterpreterConfig config = {
+        .use_main_obmalloc = 0,
+        .allow_fork = 0,
+        .allow_exec = 0,
+        .allow_threads = 1,
+        .allow_daemon_threads = 0,
+        .check_multi_interp_extensions = 1,
+        .gil = PyInterpreterConfig_OWN_GIL,
+    };
+    (void)config;
+    return 0;
+}
+" HAVE_SUBINTERPRETERS)
+
+    if(HAVE_SUBINTERPRETERS)
+        message(STATUS "Subinterpreter API detected (PyInterpreterConfig_OWN_GIL available)")
+    else()
+        message(STATUS "Subinterpreter API compile test failed, using shared GIL fallback")
+    endif()
+else()
+    message(STATUS "Python ${Python3_VERSION} < 3.12, subinterpreter API not available")
+    set(HAVE_SUBINTERPRETERS FALSE)
+endif()
+
+# Check for free-threaded Python (Python 3.13+ with --disable-gil / nogil build)
+# Free-threaded builds have Py_GIL_DISABLED defined in sysconfig
+execute_process(
+    COMMAND ${Python3_EXECUTABLE} -c "import sysconfig; print('yes' if sysconfig.get_config_var('Py_GIL_DISABLED') else 'no')"
+    OUTPUT_VARIABLE Python3_FREE_THREADED
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+    ERROR_QUIET
+)
+if(Python3_FREE_THREADED STREQUAL "yes")
+    set(HAVE_FREE_THREADED TRUE)
+    message(STATUS "Free-threaded Python detected: GIL disabled at runtime")
+else()
+    set(HAVE_FREE_THREADED FALSE)
+endif()
+
 # Create the NIF shared library
 add_library(py_nif MODULE py_nif.c)
 
@@ -97,6 +170,14 @@ elseif(Python3_LIBRARIES)
     message(STATUS "Using Python library path for dlopen: ${Python3_FIRST_LIB}")
 endif()
 
+# Add Python feature compile definitions for worker pool optimization
+if(HAVE_SUBINTERPRETERS)
+    target_compile_definitions(py_nif PRIVATE HAVE_SUBINTERPRETERS=1)
+endif()
+if(HAVE_FREE_THREADED)
+    target_compile_definitions(py_nif PRIVATE HAVE_FREE_THREADED=1)
+endif()
+
 # Set output name
 set_target_properties(py_nif PROPERTIES
     PREFIX ""