Install fails on MacOS - Failed to build google-crc32c

[JmeCS]

Attempting to pip install google-cloud-storage fails on MacOS

Environment details

• OS type and version: MacOS 11.3.1
• Python version: 3.9.6
• pip version: 21.2.4
• google-cloud-storage version: N/A (attempting to install 1.42.0)

Steps to reproduce

pip install virtualenv
virtualenv
source /bin/activate
/bin/pip install google-cloud-storage

Stack trace

Collecting google-cloud-storage
  Using cached google_cloud_storage-1.42.0-py2.py3-none-any.whl (105 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting google-api-core<3.0dev,>=1.29.0
  Using cached google_api_core-2.0.1-py2.py3-none-any.whl (92 kB)
Collecting google-auth<3.0dev,>=1.25.0
  Using cached google_auth-2.0.2-py2.py3-none-any.whl (152 kB)
Collecting google-resumable-media<3.0dev,>=1.3.0
  Using cached google_resumable_media-2.0.1-py2.py3-none-any.whl (75 kB)
Collecting google-cloud-core<3.0dev,>=1.6.0
  Using cached google_cloud_core-2.0.0-py2.py3-none-any.whl (27 kB)
Collecting protobuf>=3.12.0
  Using cached protobuf-3.17.3-py2.py3-none-any.whl (173 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Requirement already satisfied: setuptools>=40.3.0 in ./venv/lib/python3.9/site-packages (from google-api-core<3.0dev,>=1.29.0->google-cloud-storage) (57.4.0)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting google-crc32c<2.0dev,>=1.0
  Using cached google-crc32c-1.1.3.tar.gz (12 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six>=1.9
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Building wheels for collected packages: google-crc32c
  Building wheel for google-crc32c (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/jameysmith/Documents/code/gcsio/venv/bin/python /Users/jameysmith/Documents/code/gcsio/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/gm/1js6fzpn0nx1l783nsyc4rs40000gn/T/tmpv_41ue8d
       cwd: /private/var/folders/gm/1js6fzpn0nx1l783nsyc4rs40000gn/T/pip-install-r1bkxzhc/google-crc32c_a25c1a35d7f84c8d997ce57d79f83174
  Complete output (22 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.3-arm64-3.9
  creating build/lib.macosx-11.3-arm64-3.9/google_crc32c
  copying src/google_crc32c/_checksum.py -> build/lib.macosx-11.3-arm64-3.9/google_crc32c
  copying src/google_crc32c/cext.py -> build/lib.macosx-11.3-arm64-3.9/google_crc32c
  copying src/google_crc32c/__init__.py -> build/lib.macosx-11.3-arm64-3.9/google_crc32c
  copying src/google_crc32c/__config__.py -> build/lib.macosx-11.3-arm64-3.9/google_crc32c
  copying src/google_crc32c/python.py -> build/lib.macosx-11.3-arm64-3.9/google_crc32c
  running build_ext
  building 'google_crc32c._crc32c' extension
  creating build/temp.macosx-11.3-arm64-3.9
  creating build/temp.macosx-11.3-arm64-3.9/src
  creating build/temp.macosx-11.3-arm64-3.9/src/google_crc32c
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/jameysmith/Documents/code/gcsio/venv/include -I/Users/jameysmith/.pyenv/versions/3.9.6/include/python3.9 -c src/google_crc32c/_crc32c.c -o build/temp.macosx-11.3-arm64-3.9/src/google_crc32c/_crc32c.o
  src/google_crc32c/_crc32c.c:3:10: fatal error: 'crc32c/crc32c.h' file not found
  #include <crc32c/crc32c.h>
           ^~~~~~~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for google-crc32c
Failed to build google-crc32c
ERROR: Could not build wheels for google-crc32c which use PEP 517 and cannot be installed directly

[ddelgrosso1]

Hi @JmeCS,

This is potentially an issue with the virtual environment. One thing you might try is making sure setuptools and wheel are updated: pip install --upgrade setuptools wheel

[JmeCS]

Thanks for the response @ddelgrosso1 !

Sorry I should have mentioned - I tried installing inside and outside of a virtual environment. I also tried upgrading pip, setuptools, and wheel. Same result.

[ddelgrosso1]

@JmeCS would it be possible to try python 3.8 and see if the problem still exists? Thanks.

[JmeCS]

@ddelgrosso1 Just tried with a few different 3.8 versions - same problem. Thanks

[andrewsg]

We've published google-crc32c v.1.1.4 just now in the last few minutes, which should resolve this issue. Please try again, or if that doesn't work, pip install --upgrade google-crc32c.

If you are on Apple Silicon (M1 architecture) you may have a different problem, in which case, please comment with your results with 1.1.4. Thanks!

[JmeCS]

Thanks @andrewsg - I tried again, with the same result. Also tried pip install --upgrade google-crc32c with the same outcome.

I am, in fact, on an M1 chip - that was sort of my hunch from the beginning. It's been causing me all sorts of headaches. Should have mentioned that up front. Thanks!

[andrewsg]

Okay, thanks. Could you please paste the full output of the --upgrade attempt?

[JmeCS]

Collecting google-crc32c
  Using cached google-crc32c-1.1.4.tar.gz (12 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: google-crc32c
  Building wheel for google-crc32c (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/jameysmith/Documents/code/gcsio/venv/bin/python /Users/jameysmith/Documents/code/gcsio/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/gm/1js6fzpn0nx1l783nsyc4rs40000gn/T/tmph6x3tqu_
       cwd: /private/var/folders/gm/1js6fzpn0nx1l783nsyc4rs40000gn/T/pip-install-mffzketi/google-crc32c_459a20a761dc49058f551cff5f78e6ff
  Complete output (23 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.3-arm64-3.8
  creating build/lib.macosx-11.3-arm64-3.8/google_crc32c
  copying src/google_crc32c/_checksum.py -> build/lib.macosx-11.3-arm64-3.8/google_crc32c
  copying src/google_crc32c/cext.py -> build/lib.macosx-11.3-arm64-3.8/google_crc32c
  copying src/google_crc32c/__init__.py -> build/lib.macosx-11.3-arm64-3.8/google_crc32c
  copying src/google_crc32c/__config__.py -> build/lib.macosx-11.3-arm64-3.8/google_crc32c
  copying src/google_crc32c/python.py -> build/lib.macosx-11.3-arm64-3.8/google_crc32c
  running build_ext
  building 'google_crc32c._crc32c' extension
  creating build/temp.macosx-11.3-arm64-3.8
  creating build/temp.macosx-11.3-arm64-3.8/src
  creating build/temp.macosx-11.3-arm64-3.8/src/google_crc32c
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/jameysmith/Documents/code/gcsio/venv/include -I/Users/jameysmith/.pyenv/versions/3.8.10/include/python3.8 -c src/google_crc32c/_crc32c.c -o build/temp.macosx-11.3-arm64-3.8/src/google_crc32c/_crc32c.o
  src/google_crc32c/_crc32c.c:3:10: fatal error: 'crc32c/crc32c.h' file not found
  #include <crc32c/crc32c.h>
           ^~~~~~~~~~~~~~~~~
  1 error generated.
  ERROR:root:Compiling the C Extension for the crc32c library failed. To enable building / installing a pure-Python-only version, set 'CRC32C_PURE_PYTHON=1' in the environment.
  error: command 'clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for google-crc32c
Failed to build google-crc32c
ERROR: Could not build wheels for google-crc32c which use PEP 517 and cannot be installed directly

[andrewsg]

Thank you. We are working on a fix in 1.1.5. In the meantime, to mitigate, you can either pin to 1.1.2 or run CRC32C_PURE_PYTHON=1 pip install --upgrade google-crc32c.

Closing this bug as it's duplicated on the google-crc32c issue tracker, but if you continue to experience issues and are unable to install python-storage even after this mitigation, please reopen it (as it does affect storage). Thanks for your patience as we work our M1 build strategy out.

[JmeCS]

Thank you all - much appreciated! The google-cloud-storage install worked after CRC32C_PURE_PYTHON=1 pip install --upgrade google-crc32c. Will reopen if any issues come up.