CLI tool to upload Python packages listed in a requirements.txt file into a Sonatype Nexus (from Pypi), and return the list of the artifact URLs
Python tools to help with the development & deployment of company-private Python packages.
It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.
It is composed of 2 distincts :
- nexus_uploader : a simple module to be used as a setup_requires entry in setup.py, in order to easily upload Python packages onto a Nexus.
- pyRequirements2nexus : a CLI tool to convert standard Python requirements.txt into nexus-requirements.txt files, made of URLs pointing to installable packages that pyRequirements2nexus mirrored on your Nexus.
- easy integration in setup.py
- HTTP BasicAuth to connect to Sonatype Nexus
- full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus
- to install packages, the end machine only require a connexion to the Nexus host, not to the Internet
- support -e editable packages in requirements.txt
- support package URL fallbacks as comments in requirements.txt
- a list of packages already included in the base environment can be specified (e.g. if you are using Anaconda), so that they will be excluded from the final nexus-requirements.txt
- only support == version locking in requirements.txt (not >=). This is intentional, to ensure package versions do not change unexpectedly.
How to upload home-made Python packages to my Nexus ?
Here is a handy recipe you can put in your setup.py :
setup( ... setup_requires=['nexus_uploader'] )
$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD
Contrary to requirements.txt files, there is no way to include URLs to dependencies in setup.py install_requires (nor setup_requires). To work around this limitation, pyrequirements2nexus takes advantage of the deprecated “dependency_links” mechanism. Using them, it is able to discover the full chain of Python packages dependencies, both from Pypi AND from your local Nexus.
To easily generate the “dependency_links.txt” in your Nexus-hosted private package out of your requirements.txt, use the following recipe in your its setup.py :
with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file: requirements = requirements_file.readlines() dependency_links = [req for req in requirements if req.startswith('http')] install_requires = [req for req in requirements if not req.startswith('http')] setup( ... install_requires=install_requires, dependency_links=dependency_links, )
Because “dependency_links” are not supported since pip 1.6, they will NOT be installed by pip install normal recursive dependency-retrieval algorithm. You should always use the flat nexus-requirements.txt with pip install.
pip install nexus_uploader pyRequirements2nexus --help
Also take a look at jenkins-install-python-requirements.sh for an example of how we use it on our Jenkins.
Installation of nexus-requirements.txt on an end machine
pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt
Supported requirements.txt format
http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz nose==1.3.7 # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/... -e ../my/relative/path # http://nexus/content/repositories/...fallback_url...
pre-commit hooks installation:
pip install -r dev-requirements pre-commit install
Smoke tests using Pypi:
ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200
pip install - Download error on https://pypi.python.org / Couldn’t find index page for
The stack trace :
Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12)) Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB) Complete output from command python setup.py egg_info: Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found! Couldn't find index page for 'vcversioner' (maybe misspelled?) Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found! No local packages or download links found for vcversioner
Explanation : https://github.com/Julian/jsonschema/issues/276
$ cat <<EOF > ~/.pydistutils.cfg [easy_install] allow_hosts = nexus find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/22.214.171.124/ EOF
How to generate a “–default-packages” file out of an Anaconda .sh installer
grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \ | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \ | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt
Tip for easily removing packages from your nexus
pip install --user repositorytools export REPOSITORY_USER=... export REPOSITORY_PASSWORD= artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz
- Python 2 support
- detect if package releases on Pypi require gcc compilation (are they using setuptools/distutils Extension in setup.py ?)
- classifier-based selection of Python packages
- add support for md5 & sha1 upload/checks
All notable changes to this project will be documented in this file.
[1.0.13] - 2017-07-03
- The resolver now handles properly uppercase versions (like X.Y-SNAPSHOT)
[1.0.12] - 2017-07-03
- Changelog.md wasn’t included in tarball (due to MANIFEST.in)
[1.0.10] - 2017-07-03
- a bug introduced in v1.0.9: piptools.exceptions.UnsupportedConstraint: pip-compile does not support URLs as packages, unless they are editable
- changelog rendering in Pypi rst description
[1.0.9] - 2017-06-30
- Upgrading pip-tools dependency to 1.9.0 to include commit b8043be (support for pip 8.1.2) -> https://github.com/voyages-sncf-technologies/nexus_uploader/commit/38f031b
- Fixing support for classifiers to be able to only retrieve Python2-compatible pkgs -> https://github.com/voyages-sncf-technologies/nexus_uploader/commit/35e78fc
- Fixing repository files permissions -> https://github.com/voyages-sncf-technologies/nexus_uploader/commit/38f031b
- Made the lib compatible for Python 2.7 -> https://github.com/voyages-sncf-technologies/nexus_uploader/commit/5833c5f
- More tests
[1.0.8] - 2016-12-13
- –pypi-json-api-url parameter : source of Python packages to feed the Nexus with
- –allowed-pkg-classifiers parameter : when no source release is available for a given package==version, select a package matching on of those classifiers as a 2nd choice (introduced to handle docutils==0.13.1 that only exist as a wheel).
- The –allowed-pkg-classifiers comes with a default value of py3-none-any changing the existing behaviour. To keep the previous behaviour you’ll need to pass an empty value to this parameter. An PypiQueryError will then be raised if no source package is available.