Repository navigation
Preserve small logaddexp gradients - #4865
Open
DimitriChrysafis wants to merge 1 commit into
Open
DimitriChrysafis wants to merge 1 commit into
DimitriChrysafis wants to merge 1 commit into
Conversation
Compute single-input derivative weights directly instead of subtracting from one. Share one exponential in a two-output primitive when both inputs need derivatives, with vectorized CPU evaluation and GPU backend support. Cover small gradients, broadcasting and layouts, higher derivatives, batching, shapeless compilation, export/import, and binary cross-entropy. Include a benchmark for eager and compiled VJP/JVP performance.
DimitriChrysafis
marked this pull request as ready for review
November 10, 2026 21:22
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
logaddexpcomputes the second input's derivative as1 - sigmoid(a - b).When the sigmoid rounds to one, this drops a derivative that is still
representable. Swapping the inputs changes gradient accuracy even though the
forward operation is symmetric.
The error also affects JVPs, second derivatives, and binary cross-entropy with
negative logits and zero targets. It appears at smaller input gaps in float16
and bfloat16.
For a single differentiated input, compute its sigmoid weight directly. When
both inputs need derivatives, an internal two-output primitive computes both
weights using one exponential. With
e = exp(-abs(a - b)), the smaller weight ise / (1 + e), rather than a subtraction from one. CPU, Metal, and CUDA backendsshare this rule. The CPU implementation vectorizes contiguous inputs; the
primitive also supports higher derivatives, vmap, and export/import.
Tests cover both argument positions, all supported floating types on CPU,
broadcasting and strided layouts, empty arrays, non-unit cotangents, nonfinite
inputs, higher derivatives, batching, compiled gradients, export/import, and
the loss-function regression. Small-gradient checks use zero absolute tolerance.
Validation on an Apple M3 Max, macOS 27.2.1, Python 3.12.9:
The benchmark below uses 262,144 elements, 10 warmups and 100 timed evaluations,
with the median result across three fresh processes. The comparison restores
main's VJP/JVP rules in the same release build, then restores the revised rules;
the added primitive is unused in the baseline.
Some cases remain slower: eager float16 CPU VJP/JVP by about 14%, and compiled
Metal JVP by about 8–11% in this benchmark. The script also measures 4,096 elements,
single-input gradients, and float16/bfloat16/float32 on both devices.
Reproduce with
python benchmarks/python/logaddexp_grad_bench.py --device cpuand repeat with
--device gpuor--dtype float16/--dtype bfloat16.