You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
MemoryError: Cannot allocate write+execute memory for ffi.callback() #873
Found originally here: scrapy/scrapy#4117
Seems like pyOpenSSL issue, that's why creating it here.
Any advice or workaround is appreciated.
Traceback:
2019-10-31 20:24:51 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET https://xxx.yyy/robots.txt>: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Traceback (most recent call last):
.... skipped as non-relevant ...
File "/home/scrapy/env/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1709, in _makeContext
ctx.set_verify(verifyFlags, _verifyCallback)
File "/home/scrapy/env/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1103, in set_verify
self._verify_helper = _VerifyHelper(callback)
Reacted by Lucas Costa, Will Burgess, Matthew Wynn, Alex Martens, Fisher Sun, Lukas Mertens, Changsheng Wu, Zunway, philipkd, Tobias Bergkvist and 2 more
FWIW, I get the same message, on Ubuntu 18.04.3. It actually results in the same stack trace even when I go to execute other things (like install unrelated packages via pip). Uninstalling pyOpenSSL (after the scrapy install) allowed other installations to proceed, showing that it was pyOpenSSL related, but of course scrapy and other dependants can't fully function without it.
Andrii's first link above suggests some workarounds, but not for Ubuntu.
Root of the issue might be a grsecurity override, in (say) a shared webhosting environment. Use of ffi.callback() is apparently frowned-upon now as insecure, and that link above has suggested workarounds.
The end of my stack trace, just to note slight variations in line numbers from the above:
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 286, in create_urllib3_context
context.verify_mode = cert_reqs
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1119, in set_verify
self._verify_helper = _VerifyHelper(callback)
File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 338, in __init__
"int (*)(int, X509_STORE_CTX *)", wrapper)
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL's callback system.
Yeah, seems like there's a ready way to fix the code by changing to new-style "Extern Python" callbacks in CFFI. Here's another project that ran into the same issue and fixed it that way, with a real fast turnaround:
I'm not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI.
I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure's CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux).
Reading the answers above, does it mean every library using openssl has to made that kind of changes ?
When @tiran mentions "use another TLS library", is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ?
Reacted by Calum Smith, Skyler, Romain Besombes, nickd-ss, Nathan Henrie, Alexander Seifert, Martin, Florian, Will Burgess, Anton Sokolchenko and 12 more
tl;dr it's a problem in pyOpenSSL. OpenSSL is fine.
I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python's standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL.
It's an implementation artifact of pyOpenSSL's glue code that wraps OpenSSL's C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks.
There are ways to work around the problem, but it's complicated and lots of work. Nobody has contributed a solution yet.
Reacted by Jérôme Pin, nickd-ss, Jonathan de Jong, william, Doyeon Baek, YB and Songhua Liu
Reacted by Geraud d'Aspremont, John Lynch, Florian, Krishna Kalyan, YB, martinidea, Erik Kastman, Zunway and why-not
This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs.
Is there anyone who could take a look at this?
Reacted by Songhua Liu, Ezequiel Raigorodsky, Daniel Quer, Andy, Patrick Ha, Cyrus Xi, Del Scott, Jason Tam, Suprateem Banerjee, Samuel Ainsworth and 15 more
It would take a serious effort and redesign to address the issue. Somebody would have to replace all existing callbacks in PyCA cryptography and pyOpenSSL with a different design. It's a non-trivial effort, because the implementation must be thread-safe and multi subinterpreter-safe. The effort could easily take two weeks or more even for an expert.
At work we had similar issues with SELinux enforcement of read-only executable mmap pages. We followed a different approach and replaced pyOpenSSL with Python's ssl module.
Reacted by Bernhard Valenti, Jiahao, Peter Hanneman, Srđan Džombić, Rohit Verma, keyserwood, Jerick Órdenes, Songhua Liu, Pedro L. C. Rodrigues, Leo and 30 more
Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞
Reacted by 余半盏, Vlad Lialin, Ashim Mahara and Siddharth Das
That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful.
Reacted by DangerBoos, Steven Sherry, Ashim Mahara, Tomas Rojo, anothercookiecrumbles, Andrew Spott, Loïc Giraudel, Zach Jones and Jean
Reacted by Suprateem Banerjee, 刘思源, Sergey Aleynikov and Ajeet Kokatay
Reacted by Ajeet Kokatay
Reacted by Steve Estes, Nathan Henrie, Pedro L. C. Rodrigues, PretiumBrad, Thomas Aarholt, Ezequiel Raigorodsky, Brian Candler, Josh Friedlander, Mohamed Oun, tjstarak and 12 more
And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1.
FWIW, I've been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I've come back to it every few months to see if it was still crashing (and it has been).
After seeing @alex's comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.
M1 Macbook Pro
MacOS 12.0.1
Python 3.10.0
Scrapy==2.5.1
My test case has always been:
$ scrapy shell 'http://httpforever.com/'# <- succeeds
$ scrapy shell 'https://n8henrie.com'# <- crashes with `Cannot allocate write+execute memory for ffi.callback()`
For me, both seem to now work without error on http or https sites, so perhaps some of the others following this thread might recheck to see if it is still an issue for them or not.
Activity
steve-estes commentedon Jan 11, 2020
FWIW, I get the same message, on Ubuntu 18.04.3. It actually results in the same stack trace even when I go to execute other things (like install unrelated packages via pip). Uninstalling pyOpenSSL (after the scrapy install) allowed other installations to proceed, showing that it was pyOpenSSL related, but of course scrapy and other dependants can't fully function without it.
Andrii's first link above suggests some workarounds, but not for Ubuntu.
Root of the issue might be a grsecurity override, in (say) a shared webhosting environment. Use of ffi.callback() is apparently frowned-upon now as insecure, and that link above has suggested workarounds.
The end of my stack trace, just to note slight variations in line numbers from the above:
tiran commentedon Jan 11, 2020
pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL's callback system.
steve-estes commentedon Jan 12, 2020
Yeah, seems like there's a ready way to fix the code by changing to new-style "Extern Python" callbacks in CFFI. Here's another project that ran into the same issue and fixed it that way, with a real fast turnaround:
Issue: https://github.com/FSX/misaka/issues/72
Commit: FSX/misaka@321357b#diff-cd1259298a2efe394f2d1bbea8b0d39e
I'm not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI.
jeromepin commentedon Feb 13, 2021
I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure's CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux).
Reading the answers above, does it mean every library using openssl has to made that kind of changes ?
When @tiran mentions "use another TLS library", is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ?
tiran commentedon Feb 13, 2021
tl;dr it's a problem in pyOpenSSL. OpenSSL is fine.
I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python's standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL.
It's an implementation artifact of pyOpenSSL's glue code that wraps OpenSSL's C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks.
There are ways to work around the problem, but it's complicated and lots of work. Nobody has contributed a solution yet.
thomasaarholt commentedon Sep 8, 2021
This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs.
Is there anyone who could take a look at this?
tiran commentedon Sep 8, 2021
It would take a serious effort and redesign to address the issue. Somebody would have to replace all existing callbacks in PyCA cryptography and pyOpenSSL with a different design. It's a non-trivial effort, because the implementation must be thread-safe and multi subinterpreter-safe. The effort could easily take two weeks or more even for an expert.
At work we had similar issues with SELinux enforcement of read-only executable mmap pages. We followed a different approach and replaced pyOpenSSL with Python's ssl module.
thomasaarholt commentedon Sep 8, 2021
@tiran Thanks for the comment! I'll suggest dbt/snowflake-connector do the same thing you folks did.
ImAlien commentedon Oct 23, 2021
encounter the same problems on m1 mac
keyserwood commentedon Nov 14, 2021
Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞
fgiroud commentedon Nov 18, 2021
Same issue with the new macbook pros M1 (no rosetta)
xros commentedon Nov 27, 2021
Same problem on M1 MacBook Pro. It's going to be year 2022 and Apple Silicon is still not supported.
reaperhulk commentedon Nov 27, 2021
That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful.
alex commentedon Nov 27, 2021
And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1.
n8henrie commentedon Nov 29, 2021
FWIW, I've been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I've come back to it every few months to see if it was still crashing (and it has been).
After seeing @alex's comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.
My test case has always been:
For me, both seem to now work without error on
httporhttpssites, so perhaps some of the others following this thread might recheck to see if it is still an issue for them or not.47 remaining items
ftmoztl commentedon May 7, 2023
As an M1 user, it worked for me also without using conda virtual env. Thank you so much!
python310Packages.cffi: patch closures to work on M1 machines
python310Packages.cffi: patch closures to work on M1 machines
python310Packages.cffi: patch closures to work on M1 machines
python310Packages.cffi: patch closures to work on M1 machines
python310Packages.cffi: patch closures to work on M1 machines