sentry internal pypi

this repository contains the tools to import and/or build packages from public pypi for the platforms and achitectures required for sentry development.

packages are configured in the packages.ini file.

the easiest way to add a package and its dependencies is to use:

python3 -m add_pkg PKGNAME

each section is an individual package and has some additional instructions which helps for building.

don't worry too much about the formatting, an auto-formatter will ensure the format is correct.

most packages won't need special build instructions and the section contents can be left blank:

[botocore==1.25.12]

[simplejson==3.17.2]
[simplejson==3.17.6]

Anyone in the Engineering team can approve Pull Requests, but it's preferred to get somebody from your team with context to approve your PR.

some packages require special system-level build dependencies, these can be configured using apt_requires (linux) and brew_requires (macos)

[xmlsec==1.3.12]
apt_requires =
    libxmlsec1-dev
    pkg-config
brew_requires =
    libxmlsec1
    pkg-config

sometimes the dependencies aren't packaged for apt / brew and you need a custom script to set them up. this script will be passed a single "prefix" directory (which contains the standard bin / lib / include / etc. structure).

the script should set up whatever tools are necessary inside only that directory

[google-crc32c==1.3.0]
custom_prebuild = prebuild/crc32c 1.1.2

when the sdist has source files of a compiled language (see BINARY_EXTS), we would expect compiled things (e.g. binary executables, shared objects) in the wheel. sometimes this isn't the case, and you can ignore those source files:

[salt==3006.6]
likely_binary_ignore =
    salt-3006.6/pkg/old/smartos/esky/sodium_grabber.c

some packages are only intended for particular python versions (or don't otherwise build cleanly). the builds can be filtered using python_versions (though usually you should try and upgrade the relevant package).

[backports-zoneinfo==0.2.1]
python_versions = <3.9

after building the packages will be checked that they can install and import

sometimes you may need to hint the validation tooling of additional requirements

an example is black which has a blackd top-level but requires an optional dependency to use (the black[d] extra). you can hint at this via validate_extras

[black==22.3.0]
validate_extras = d

sometimes packages incorrectly specify their dependencies. you can use this option to add import-time dependencies (though you should try and send a PR to fix those packages!)

one example is dictpath which depends on six but doesn't list it:

[dictpath==0.1.3]
validate_incorrect_missing_deps = six

this should usually not be used but sometimes you need to skip importing some top-level modules due to side-effects or weird runtime requirements

[pyuwsgi==2.0.20]
validate_skip_imports = uwsgidecorators