DevOps Hacking

Maintaining a Private PIP Repository

September 30, 2018

This is highly usable in setup’s where you have computers that do not have direct access to the internet and. You have to options, depending on what access and rights you have to solve this.

Using a Proxy server

option 1. is the most obvious, simply setting up a cache server (Nexus or Nginx) somewhere in you’re organizations, the restricted computers can access.

I did this with Neuxs, creating a Proxy repository, called pypi-proxy, setting the “Proxy” -> “Remote storage” to https://pypi.org see pypi Proxy repository

And then using ansible I pushed the following file to all the computers,

[global]
index=http://172.20.3.24/repository/pypi-proxy/pypi
index-url=http://172.20.3.24/repository/pypi-proxy/simple
trusted-host=172.20.3.24

Make sure to change the pypi-proxy in the URL’s, to whatever you called your repository! And of course the IP :)

Using PyInstaller

See pyinstaller they have that restriction, that you can’t build cross-platform dist. This means that it will not work if your build environment, is pure Linux, and you’re restricted computers are windows. If you can get around that restriction, this is a very good tool, that solves it quite nicely. Currently, I can’t, if I ever get to a place where I can, I will update with my, hands-on experience.

Using Distutil

To Do.

Using a shared folder

If you don’t have access to implement these options, you can use the second option, which is a privately maintained index. This requires, that you have access to a single computer with internet access, that also has access to a file share, that is shared with the restricted computers

Firstly create a requirements.txt file, this file will have to have references to all the packages you will have in you’re index aka. all the packages that are required, on the restricted computers. when you have completed the adding stuff to the file, you can run

pip download -r {you\'re-requirements-file.txt} -d {path-to-you\'re-shared-network-folder}

and then on you restricted computers you can run

pip install --quiet  -r {you\'re-requirements-file.txt} --no-index --find-links {path-to-you\'re-shared-network-folder}

which will install all you’re required packages, from you’re synced share.

On a side note, the update of you’re index should, of course, be automated, so that each time you’re, requirement.txt file is updated the index is as well :)

Tagged: #Python #Pip #Proxy #Enterprise-Hack