===========================
Mirroring a PyPI repository
===========================

.. contents::
   :local:
   :depth: 1


Before you start
================

You need to have already installed and configured
your Repository instance. Due
to the size of Repository, it is important that you have
configured a file storage location with sufficient disk space. If
necessary, see the :ref:`requirements for the file storage
location <repo-hardware-reqs>`.

The full PyPI mirror requires approximately 120 GB.

Running the PyPI mirror command
===============================

To create a PyPI mirror::

    anaconda-server-sync-pypi

This command loads all of the packages on pypi.python.org into
the ~pypi binstar user account.

Verify that the command ran successfully by opening your browser
to ``http://your-anaconda-repo/pypi/~pypi``.

NOTE: Replace ``your-anaconda-repo`` with the URL to your
installation of Repository.

Customizing the mirror
======================

It is possible to customize the mirror behavior by creating a
configuration file such as
``$PREFIX/etc/anaconda-server/mirror/pypi.yaml`` and using the
``--mirror-config`` option::

  anaconda-server-sync-pypi --mirror-config /etc/binstar/mirrors/pypi.yaml



The following configuration options are available:

+----------------------+-------------------------------------------------------+
| Name                 | Description                                           |
+======================+=======================================================+
| ``user``             | The local user under which the PyPI packages are      |
|                      | imported. Default: ``pypi``.                          |
+----------------------+-------------------------------------------------------+
| ``pkg_list``         | A list of packages to mirror. Only packages listed    |
|                      | are mirrored. If this is set, ``blacklist`` and       |
|                      | ``whitelist`` settings are ignored. **Default:**      |
|                      | ``[]``.                                               |
+----------------------+-------------------------------------------------------+
| ``whitelist``        | A list of packages to mirror. Only packages listed    |
|                      | are mirrored. If the list is empty, all packages are  |
|                      | checked. Default: ``[]``.                             |
+----------------------+-------------------------------------------------------+
| ``blacklist``        | A list of packages to skip. The packages listed are   |
|                      | ignored. Default: ``[]``.                             |
+----------------------+-------------------------------------------------------+
| ``latest_only``      | Only download the latest versions of the packages.    |
|                      | Default: ``false``.                                   |
+----------------------+-------------------------------------------------------+
| ``remote_url``       | The URL of the PyPI mirror. ``/pypi`` is appended to  |
|                      | build the XML RPC API URL, ``/simple`` for the simple |
|                      | index and ``/pypi/{package}/{version}/json`` for the  |
|                      | JSON API. Default: ``https://pypi.python.org/``.      |
+----------------------+-------------------------------------------------------+
| ``xml_rpc_api_url``  | A custom value for XML RPC URL. If this value is      |
|                      | present, it takes precedence over the URL built using |
|                      | ``remote_url``. Default: ``null``.                    |
+----------------------+-------------------------------------------------------+
| ``simple_index_url`` | A custom value for the simple index URL. If this      |
|                      | value is present, it takes precedence over the URL    |
|                      | built using ``remote_url``. Default: ``null``.        |
+----------------------+-------------------------------------------------------+
| ``use_xml_rpc``      | Whether to use the XML RPC API as specified by        |
|                      | `PEP381                                               |
|                      | <https://www.python.org/dev/peps/pep-0381/>`_. If     |
|                      | this is set to ``true``, the simple index is used to  |
|                      | determine which packages to check. The simple index   |
|                      | is also used if the XML RPC API fails for any reason. |
|                      | Default: ``true``.                                    |
+----------------------+-------------------------------------------------------+
| ``use_serial``       | Whether to use the serial number provided by the XML  |
|                      | RPC API. Only packages updated since the last serial  |
|                      | saved are checked. If this is set to false, all PyPI  |
|                      | packages are checked for updates. Default: ``true``.  |
+----------------------+-------------------------------------------------------+
| ``create_org``       | Create the mirror user as an organization instead of  |
|                      | a regular user account. All superusers are added to   |
|                      | the "Owners" group of the organization. Default:      |
|                      | ``false``.                                            |
+----------------------+-------------------------------------------------------+
| ``private``          | Save the mirrored packages as private. **Default:**   |
|                      | ``false``.                                            |
+----------------------+-------------------------------------------------------+
 
EXAMPLE:

.. code-block:: yaml

    whitelist:
      - requests
      - six
      - numpy
      - simplejson
    latest_only: true
    remote_url: http://pypimirror.local/
    use_xml_rpc: false

Configuring pip
===============

To configure pip to use this new mirror, you must edit your
``~/.pip/pip.conf`` file::

  [global]
  index-url = http://your-anaconda-repo:<port>/pypi/~pypi/simple

NOTE: Replace ``your-anaconda-repo`` with the URL to your
installation of Repository.
