1 Demand Analysis

  • Private development sources: Development teams need a convenient python private package distribution mechanism

  • Private image source: Create an official image to improve access speed, avoid occasional network problems, and facilitate private deployment in offline environments

2 Use Docker to deploy the PypiServer server

2.1 Downloading the PypiServer Image

docker pull pypiserver/pypiserver
Copy the code

2.2 Generating Auth Information

# install dependencies
apt-get install -y apache2-utilssudo pip3 install passlib 
Generate htpass file
mkdir -p /opt/pypiserver/auth /opt/pypiserver/packages
# indicates that all users can read and write but cannot execute files/folders
chmod -R 666 /opt/pypiserver/packages
# prompt password input, repeat twice
cd /opt/pypiserver/auth && htpasswd -sc .htaccess ${username}
Copy the code

2.3 Container Deployment

docker run -d \
-p ${port}:8080 \ 
--restart=always \ --name=pypiserver \ 
-v /opt/pypiserver/packages/:/data/packages \
-v /opt/pypiserver/auth:/data/auth/ \ 
pypiserver/pypiserver -P /data/auth/.htaccess -a update /data/packages
Copy the code

2.4 Nginx Reverse proxy

  • Use Docker to deploy Nginx services with HTTPS support

    echo 'server {
listen 80;
server_name ${sever_name];
rewrite ^(.*)$ https://${server_name}The $1 permanent;
}
server {
listen 443 ssl;
server_name ${server_name}; Ssl_certificate /etc/nginx/ SSL /ps-cert.pem; Ssl_certificate_key /etc/nginx/ SSL /ps-cert.key; ssl_session_timeout 10m; Ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:! NULL:! aNULL:! MD5:! ADH:! RC4; ssl_prefer_server_ciphers on; location / { proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $host; proxy_set_header X-Real-IP $remote_addr; The service on the Intranet can be mapped to proxy_pass http:// on the public network using FRP${public_ip}:${port};
}
}' >> /opt/pypi/pypi.conf
Copy the code

  • Deploy the Nginx container

    docker run -d \
--restart always \
-v /opt/pypi/pypi.conf:/etc/nginx/conf.d/pypi.conf \
-v /opt/pypi/ssl/ps-cert.pem:/etc/nginx/ssl/ps-cert.pem \
-v /opt/pypi/ssl/ps-cert.key:/etc/nginx/ssl/ps-cert.key \
-p ${port}:80 \
--name=pypi_nginx
nginx
Copy the code

3 Install the BanderSnatch local Source synchronization tool

3.1 Local Configuration

The configuration file

mkdir -p /opt/bandersnatch/log && touch /opt/bandersnatch/bandersnatch.conf /opt/bandersnatch/bandersnatch-log.conf

echo '[mirror]
directory = /opt/bandersnatchjson = false
release-files = true
cleanup = false
master = https://pypi.org
timeout = 10
global-timeout = 1800
workers = 3hash-index = false
stop-on-error = false
storage-backend = filesystem
;log-config = /opt/bandersnatch/bandersnatch-log.conf
; root_uri = https://example.comverifiers = 3
;keep_index_versions = 0
;vim: set ft=cfg:
;diff-file = /srv/pypi/mirrored-files
;diff-append-epoch = true
[plugins]
enabled = all
[blacklist]
; https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html
; https://pypi.org/stats/
[whitelist]
packages = 
cntk 
tensorflow-gpu 
tensorflow 
tensorflow-cpu 
torch' > /opt/bandersnatch/bandersnatch.conf \
&& echo '
[loggers]
keys=root,file
[handlers]
keys=root,file
[formatters]
keys=common
[logger_root]
level=NOTSEThandlers=root
[logger_file]
level=INFO
handlers=file
propagate=1qual
name=bandersnatch
[formatter_common]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s
[handler_root]
class=StreamHandlerlevel=DEBUGformatter=commonargs=(sys.stdout,) 
[handler_file]
class=handlers.Rotating
FileHandlerlevel=INFO
formatter=commonargs=('/opt/bandersnatch/log/bandersnatch.log', 'D', 1, 'UTF-8')
# will manage one file a day' > /opt/bandersnatch/bandersnatch-log.conf
Copy the code

The deployment of the container

docker run -d \ --restart=always \ --name=bandersnatch \ -v /opt/bandersnatch/bandersnatch.conf:/etc/bandersnatch.conf \  -v /opt/bandersnatch:/opt/bandersnatch \ pypa/bandersnatch bandersnatch mirrorCopy the code

3.2 Nginx Reverse Proxy Configuration

Using Docker to deploy the Nginx service, the Nginx configuration file is as follows

server {
listen 80;
server_name ${server_name};
rewrite^ (. *) $ https://${server_name}The $1 permanent;
}
server {
listen 443 ssl;
server_name ${server_name};
SSL certificate file location (common certificate file format: CRT /pem)
ssl_certificate /etc/nginx/ssl/bs-cert.pem;
# SSL certificate key location
ssl_certificate_key /etc/nginx/ssl/bs-cert.key;
ssl_session_timeout 10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphersECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:! NULL:! aNULL:! MD5:! ADH:! RC4;ssl_prefer_server_ciphers on;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $host;
proxy_set_header X-Real-IP $remote_addr;
FRP can be used to map services on the internal network to the public network
proxy_pass http://${public_ip}:${port}; }}Copy the code

5 reference

  • PypiServer

    • Pypi.org/project/pyp…
    • Github.com/pypiserver/…
    • PypiServer Docker Hub

  • bandersnatch

    • Hub.docker.com/r/pypa/band…
    • Github.com/pypa/bander…
    • Mirror configuration