Negative users mocked the suggestion to block attention gradients to prevent J-space formation as it would amount to a transformer with no transformer.
Based on 1 visible X reactions from 5 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@willdepue > preventing attention gradients into past tokens The transformer with no transformer
Maybe, but my guess is the optimization pressure just finds another communication channel. If a shared workspace is genuinely useful for multi-step reasoning, I’d expect it to re-emerge through residual streams or other circuits even if you block one path. Let’s do a paper on it?
can't you prevent J-space from forming just by preventing attention gradients into past tokens? i assume that is the major reason why models do lookahead computation, without it would be really hard to form from just circuit sharing
seems like it works and obviously destroys performance: https://arxiv.org/html/2404.00859v2?utm_source=chatgpt.com
@willdepue > preventing attention gradients into past tokens The transformer with no transformer
Maybe, but my guess is the optimization pressure just finds another communication channel. If a shared workspace is genuinely useful for multi-step reasoning, I’d expect it to re-emerge through residual streams or other circuits even if you block one path. Let’s do a paper on it?
can't you prevent J-space from forming just by preventing attention gradients into past tokens? i assume that is the major reason why models do lookahead computation, without it would be really hard to form from just circuit sharing
seems like it works and obviously destroys performance: https://arxiv.org/html/2404.00859v2?utm_source=chatgpt.com
Negative users mocked the suggestion to block attention gradients to prevent J-space formation as it would amount to a transformer with no transformer.
Based on 1 visible X reactions from 5 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@torchcompiled P-zombie transformer