Skip to main content
Warning: You are using the test version of PyPI. This is a pre-production deployment of Warehouse. Changes made here affect the production instance of TestPyPI (testpypi.python.org).
Help us improve Python packaging - Donate today!

CLI tool to upload Python packages listed in a requirements.txt file into a Sonatype Nexus (from Pypi), and return the list of the artifact URLs

Project Description

Python tools to help with the development & deployment of company-private Python packages.

It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.

It is composed of 2 distincts :

  • nexus_uploader : a simple module to be used as a setup_requires entry in setup.py, in order to easily upload Python packages onto a Nexus.
  • pyRequirements2nexus : a CLI tool to convert standard Python requirements.txt into nexus-requirements.txt files, made of URLs pointing to installable packages that pyRequirements2nexus mirrored on your Nexus.

Features

nexus_uploader features

  • easy integration in setup.py
  • HTTP BasicAuth to connect to Sonatype Nexus

pyRequirements2nexus features

  • full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus
  • to install packages, the end machine only require a connexion to the Nexus host, not to the Internet
  • support -e editable packages in requirements.txt
  • support package URL fallbacks as comments in requirements.txt
  • a list of packages already included in the base environment can be specified (e.g. if you are using Anaconda), so that they will be excluded from the final nexus-requirements.txt

Limitations

  • only support == version locking in requirements.txt (not >=). This is intentional, to ensure package versions do not change unexpectedly.

How to upload home-made Python packages to my Nexus ?

Here is a handy recipe you can put in your setup.py :

setup(
    ...
    setup_requires=['nexus_uploader']
)

Usage:

$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD

Important note

Contrary to requirements.txt files, there is no way to include URLs to dependencies in setup.py install_requires (nor setup_requires). To work around this limitation, pyrequirements2nexus takes advantage of the deprecated “dependency_links” mechanism. Using them, it is able to discover the full chain of Python packages dependencies, both from Pypi AND from your local Nexus.

To easily generate the “dependency_links.txt” in your Nexus-hosted private package out of your requirements.txt, use the following recipe in your its setup.py :

with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file:
    requirements = requirements_file.readlines()
    dependency_links = [req for req in requirements if req.startswith('http')]
    install_requires = [req for req in requirements if not req.startswith('http')]

setup(
    ...
    install_requires=install_requires,
    dependency_links=dependency_links,
)

Because “dependency_links” are not supported since pip 1.6, they will NOT be installed by pip install normal recursive dependency-retrieval algorithm. You should always use the flat nexus-requirements.txt with pip install.

pyRequirements2nexus usage

pip install nexus_uploader
pyRequirements2nexus --help

Also take a look at jenkins-install-python-requirements.sh for an example of how we use it on our Jenkins.

Installation of nexus-requirements.txt on an end machine

pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt

Supported requirements.txt format

http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz
nose==1.3.7   # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/...
-e ../my/relative/path  # http://nexus/content/repositories/...fallback_url...

Contributing

pre-commit hooks installation:

pip install -r dev-requirements
pre-commit install

Unit tests:

py.test tests/

Smoke tests using Pypi:

ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200

FAQ

pip install - Download error on https://pypi.python.org / Couldn’t find index page for

The stack trace :

Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12))
  Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found!
    Couldn't find index page for 'vcversioner' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
    No local packages or download links found for vcversioner

Explanation : https://github.com/Julian/jsonschema/issues/276

Solution :

$ cat <<EOF > ~/.pydistutils.cfg
[easy_install]
allow_hosts = nexus
find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/2.14.0.0/
EOF

How to generate a “–default-packages” file out of an Anaconda .sh installer

grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \
    | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \
    | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt

Tip for easily removing packages from your nexus

pip install --user repositorytools
export REPOSITORY_USER=...
export REPOSITORY_PASSWORD=
artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz

ToDo

  • Python 2 support
  • detect if package releases on Pypi require gcc compilation (are they using setuptools/distutils Extension in setup.py ?)
  • classifier-based selection of Python packages
  • add support for md5 & sha1 upload/checks

Change Log

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/).

[1.0.13] - 2017-07-03

Fixed

  • The resolver now handles properly uppercase versions (like X.Y-SNAPSHOT)

[1.0.12] - 2017-07-03

Fixed

  • Changelog.md wasn’t included in tarball (due to MANIFEST.in)

[1.0.10] - 2017-07-03

Fixed

  • a bug introduced in v1.0.9: piptools.exceptions.UnsupportedConstraint: pip-compile does not support URLs as packages, unless they are editable
  • changelog rendering in Pypi rst description

[1.0.9] - 2017-06-30

Fixed

Added

[1.0.8] - 2016-12-13

Added

  • –pypi-json-api-url parameter : source of Python packages to feed the Nexus with
  • –allowed-pkg-classifiers parameter : when no source release is available for a given package==version, select a package matching on of those classifiers as a 2nd choice (introduced to handle docutils==0.13.1 that only exist as a wheel).

Changed

  • The –allowed-pkg-classifiers comes with a default value of py3-none-any changing the existing behaviour. To keep the previous behaviour you’ll need to pass an empty value to this parameter. An PypiQueryError will then be raised if no source package is available.

Release History

This version
History Node

1.0.13

History Node

1.0.12

History Node

1.0.11

History Node

1.0.10

History Node

1.0.9

History Node

1.0.8

History Node

1.0.6

History Node

1.0.5

History Node

1.0.4

History Node

1.0.4rc0

History Node

1.0.4b0

History Node

1.0.4a0

History Node

1.0.3

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Hash SHA256 Hash Help Version File Type Upload Date
nexus_uploader-1.0.13.tar.gz
(18.9 kB) Copy SHA256 Hash SHA256
Source Jul 3, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting