Skip to content

MemoryError: Cannot allocate write+execute memory for ffi.callback() #873

Open

Description

@askaliuk

For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

Found originally here: scrapy/scrapy#4117
Seems like pyOpenSSL issue, that's why creating it here.

Any advice or workaround is appreciated.

Traceback:

2019-10-31 20:24:51 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET https://xxx.yyy/robots.txt>: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
Traceback (most recent call last):
.... skipped as non-relevant ...
 File "/home/scrapy/env/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1709, in _makeContext
    ctx.set_verify(verifyFlags, _verifyCallback)
  File "/home/scrapy/env/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1103, in set_verify
    self._verify_helper = _VerifyHelper(callback)

OS:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"

Activity

  1. steve-estes commented on Jan 11, 2020

    @steve-estes

    FWIW, I get the same message, on Ubuntu 18.04.3. It actually results in the same stack trace even when I go to execute other things (like install unrelated packages via pip). Uninstalling pyOpenSSL (after the scrapy install) allowed other installations to proceed, showing that it was pyOpenSSL related, but of course scrapy and other dependants can't fully function without it.

    Andrii's first link above suggests some workarounds, but not for Ubuntu.

    Root of the issue might be a grsecurity override, in (say) a shared webhosting environment. Use of ffi.callback() is apparently frowned-upon now as insecure, and that link above has suggested workarounds.

    The end of my stack trace, just to note slight variations in line numbers from the above:

      File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 286, in create_urllib3_context
        context.verify_mode = cert_reqs
      File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
        self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
      File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1119, in set_verify
        self._verify_helper = _VerifyHelper(callback)
      File "/home/username/opt/python-2.7.15/lib/python2.7/site-packages/OpenSSL/SSL.py", line 338, in __init__
        "int (*)(int, X509_STORE_CTX *)", wrapper)
    MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
    
  2. tiran commented on Jan 11, 2020

    @tiran

    pyOpenSSL is not compatible with systems that prevent writeable and executable memory pages. You either have to disable the security feature, use another TLS library like the builtin ssl module, or redesign+rewrite pyOpenSSL's callback system.

  3. steve-estes commented on Jan 12, 2020

    @steve-estes

    Yeah, seems like there's a ready way to fix the code by changing to new-style "Extern Python" callbacks in CFFI. Here's another project that ran into the same issue and fixed it that way, with a real fast turnaround:

    Issue: https://github.com/FSX/misaka/issues/72
    Commit: FSX/misaka@321357b#diff-cd1259298a2efe394f2d1bbea8b0d39e

    I'm not familiar enough with the pyOpenSSL codebase to risk giving it a try, but otherwise it seems fairly straightforward - and would add some performance too, according to CFFI.

  4. jeromepin commented on Feb 13, 2021

    @jeromepin

    I know this is almost a year later, but I too have the issue using my M1 Silicon mac. I first encountered it using Azure's CLI and then with home-made softwares, while it was working great on Intel platforms (both darwin and linux).

    Reading the answers above, does it mean every library using openssl has to made that kind of changes ?

    When @tiran mentions "use another TLS library", is this at the OS level (like replacing OpenSSL by LibreSSL or BoringSSL) or inside the Python software ?

  5. tiran commented on Feb 13, 2021

    @tiran

    tl;dr it's a problem in pyOpenSSL. OpenSSL is fine.

    I see how my comment can be understand the wrong way. I was referring to another Python TLS library such as ssl module from Python's standard library. OpenSSL does not require executable+writable memory. The problem only effects pyOpenSSL.

    It's an implementation artifact of pyOpenSSL's glue code that wraps OpenSSL's C-API and makes it available for Python. pyOpenSSL uses CFFI, which is a Python interface to libffi. libffi (library for foreign function interface) uses dynamic code creation for dynamic callbacks. Every time the code passes a Python method to OpenSSL, it has to wrap the Python method into machine code, so it looks like a C function to the C-API of OpenSSL. It writes dynamic code to an executable memory page. CFFI calls this old style callbacks.

    There are ways to work around the problem, but it's complicated and lots of work. Nobody has contributed a solution yet.

  6. thomasaarholt commented on Sep 8, 2021

    @thomasaarholt

    This issue seems to be blocking snowflake-python-connector and hence dbt, as well as some other packages (see mentions above), from running on M1 (arm64) Macs.

    Is there anyone who could take a look at this?

  7. tiran commented on Sep 8, 2021

    @tiran

    It would take a serious effort and redesign to address the issue. Somebody would have to replace all existing callbacks in PyCA cryptography and pyOpenSSL with a different design. It's a non-trivial effort, because the implementation must be thread-safe and multi subinterpreter-safe. The effort could easily take two weeks or more even for an expert.

    At work we had similar issues with SELinux enforcement of read-only executable mmap pages. We followed a different approach and replaced pyOpenSSL with Python's ssl module.

  8. thomasaarholt commented on Sep 8, 2021

    @thomasaarholt

    @tiran Thanks for the comment! I'll suggest dbt/snowflake-connector do the same thing you folks did.

  9. ImAlien commented on Oct 23, 2021

    @ImAlien

    encounter the same problems on m1 mac

  10. keyserwood commented on Nov 14, 2021

    @keyserwood

    Still having this issue on apple m1 2020, when launching API calls via asyncio [python 3.8.12 & natively installed - no rosetta here]. Hope that folks buying new MacBook Pro with M1 chip will push to fix this kind of issues 🤞

  11. fgiroud commented on Nov 18, 2021

    @fgiroud

    Same issue with the new macbook pros M1 (no rosetta)

  12. xros commented on Nov 27, 2021

    @xros

    Same problem on M1 MacBook Pro. It's going to be year 2022 and Apple Silicon is still not supported.

  13. reaperhulk commented on Nov 27, 2021

    @reaperhulk
    Member

    That attitude is extremely unproductive. As always, please try to remember that these are open source projects and someone has to actually do the work to fully understand the problem and implement a fix. Posts like yours are at best an irritant and quite frequently actively harmful.

  14. alex commented on Nov 27, 2021

    @alex
    Member

    And not for nothing, but our test suite runs without segfaulting on an M1, so either our coverage is lousy (quite possible!) or there's slightly more involved in reproducing this than just using an M1.

  15. n8henrie commented on Nov 29, 2021

    @n8henrie

    FWIW, I've been following this issue for about a year after (like the OP) encountering it on Scrapy on my M1 MBP (0, 1). I've come back to it every few months to see if it was still crashing (and it has been).

    After seeing @alex's comment I wanted to see if I could contribute to the test suite, and lo and behold the crash is gone.

    • M1 Macbook Pro
    • MacOS 12.0.1
    • Python 3.10.0
    • Scrapy==2.5.1

    My test case has always been:

    $ scrapy shell 'http://httpforever.com/' # <- succeeds
    $ scrapy shell 'https://n8henrie.com' # <- crashes with `Cannot allocate write+execute memory for ffi.callback()`

    For me, both seem to now work without error on http or https sites, so perhaps some of the others following this thread might recheck to see if it is still an issue for them or not.

  16. 47 remaining items

  17. ftmoztl commented on May 7, 2023

    @ftmoztl

    @WittmannF Yep. Until build 3 of cffi is available, I have been activating the conda/mamba env and:

    (my_env) $ pip install --force-reinstall 'cffi>=1.15.1'

    It worked here. thanks a lot!

    As an M1 user, it worked for me also without using conda virtual env. Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions