===================
Customizing mirrors
===================

.. contents::
   :local:
   :depth: 1


This section explains how you can customize your PyPI or
Anaconda repository mirror.

NOTE: This documentation applies to versions of Repository 2.26
and newer. For versions between 2.23 and 2.26, see
:ref:`2.23 documentation <mirrors-anaconda-repository-2.23>`. For
older versions, see
:ref:`2.20 documentation <mirrors-sync-configuration-2.2>`.

Before you start
================

Your Anaconda Repository should already be installed and configured, as well
as the mirroring tool ``cas-mirror``. See :doc:`mirroring an Anaconda
repository <mirror-anaconda-repository>`.

Customizing your mirror
=======================

The following commands can be used with the ``cas-mirror`` tool:

* ``cas-sync``: Creates or updates an existing local Anaconda
  package repository. The packages are saved into a local directory.
  This directory is defined by the ``mirror_dir`` configuration setting.

* ``cas-sync-api-v4``: Creates or updates an existing remote Anaconda
  package repository. The packages are uploaded into an account of an
  existing Anaconda repository instance. The account and instance are
  controlled by the ``dest_channel`` and ``dest_site`` configuration
  settings.

* ``cas-merge``: Combines delta repositories into an existing
  Anaconda package repository. A delta repository contains the changes
  between the local and remote Anaconda package repositories. Delta
  repositories are generated by ``cas-sync`` using the ``delta`` configuration
  setting.

All the commands can read a YAML configuration file specified by
the ``--file`` (or ``-f``) command line option. The YAML file can
contain any of the following valid keys:

remote_url
~~~~~~~~~~

Conda packages, Anaconda installers and Miniconda installers are fetched from
this remote URL.

DEFAULT: ``https://repo.continuum.io/``

channels
~~~~~~~~

Conda packages are fetched from these remote channels.

DEFAULT: A list of these channels:

* ``<remote_url>/pkgs/main/``

* ``<remote_url>/pkgs/free/``

* ``<remote_url>/pkgs/pro/``

mirror_dir
~~~~~~~~~~

The mirror is saved in this local directory.

NOTE: A full mirror of the Anaconda repository uses about 100 GB of disk space.
Make sure the mirror directory has enough space.

DEFAULT: ``/opt/cas-mirror``

platforms
~~~~~~~~~

Conda packages and installers for this list of platforms are mirrored.

DEFAULT: A list of all platforms. This is:

``['osx-32', 'osx-64', 'win-32', 'win-64', 'linux-32', 'linux-64', 'linux-armv6l', 'linux-armv7l', 'linux-ppc64le']``

fetch_installers
~~~~~~~~~~~~~~~~

Whether to fetch all Anaconda and Miniconda installers from ``remote_url``.

If ``fetch_installers`` is set to ``false`` then installers are not fetched.

DEFAULT: ``true``

python_versions
~~~~~~~~~~~~~~~~

Python versions to mirror.

DEFAULT: All versions.

EXAMPLE: ``['2.7', '3.6']``

pkg_list
~~~~~~~~

An explicit list of package names to be mirrored.

When this list is provided, the ``license_blacklist``, ``blacklist`` and
``whitelist`` keys are not allowed to be set.

license_blacklist
~~~~~~~~~~~~~~~~~

A list of licenses to be excluded from the mirror.

The license families that can be blacklisted are:

* AGPL
* APACHE
* BSD
* GPL2
* GPL3
* LGPL
* MIT
* PROPRIETARY
* PUBLICDOMAIN
* PSF
* OTHER
* NONE

blacklist
~~~~~~~~~

A list of package names to be excluded from the mirror.

whitelist
~~~~~~~~~

A list of package names to be included in the mirror.

The whitelist overrides the blacklists. If a package is both blacklisted and
whitelisted, then it is included and mirrored.

EXAMPLE: The package ``numpy`` has a license in the license family ``BSD``.

If ``license_blacklist`` contains ``BSD`` and ``whitelist`` is empty,
then ``numpy`` and all other BSD licensed packages are excluded and not
mirrored.

If ``license_blacklist`` contains ``BSD`` and ``whitelist`` contains
``numpy``, then the ``numpy`` package is included and mirrored and other BSD
licensed packages are excluded and not mirrored.

dest_channel
~~~~~~~~~~~~

Optional channel to use when synchronizing with a local Repository instance.

DEFAULT: "anaconda"

dest_site
~~~~~~~~~

Optional site to use when synchronizing with a local Repository instance.

DEFAULT: None

delta
~~~~~

If ``delta`` is true, then a delta is generated from ``mirror_dir``.

If ``delta`` is false, then the changes are applied directly.

delta_dir
~~~~~~~~~

The delta is generated (or merged) onto this directory.

If this is not specified, the generated delta directory is named
``delta-<timestamp>-pkgs``. ``<timestamp>`` is replaced with a timestamp.

max_retries
~~~~~~~~~~~

The number of retries to allow before failing.

When it is set to 0, ``cas-mirror`` fails at the first error. Default is 0.

This is supported only for the ``cas-sync`` and ``cas-sync-api-v4`` commands.

DEFAULT: 0

safe
~~~~

If ``safe`` is ``true``, synchronizing repositories or merging delta
directories never delete anything.

DEFAULT: ``false``

repodata_source
~~~~~~~~~~~~~~~

Uses a ``repodata.json`` (or ``repodata.json.bz2``) as the source of
existing packages.

This file must be in the mirror directory for a specific platform.

EXAMPLE: ``/mirror/linux-64/repodata.json``

Mirroring a platform-specific list
==================================

By default, ``cas-sync`` and ``cas-sync-api-v4`` mirror all platforms.
If you do not need all platforms, you can save time and disk
space by editing the ``yaml`` file to specify which platform(s) you want
to mirror.

EXAMPLE::

  platforms:
    - linux-64
    - win-32


Mirroring a package-specific list
=================================

You may want to mirror only a small subset of Repository. Rather than
blacklisting a long list of packages you do not want to be mirrored, you
can instead simply enumerate the list of packages you do want to mirror.

EXAMPLE: This example mirrors only the three packages Accelerate,
PyQt and Zope. All other packages are ignored::

  package_list:
  - accelerate
  - pyqt
  - zope


Mirroring Python version-specific packages
==========================================

You may want to mirror only a subset of versions.

EXAMPLE: This example mirrors only Anaconda packages built for
Python 3.3::

  python_versions:
    - 3.3


Mirroring with a license blacklist
==================================

As of Repository 2.26.0, the Anaconda mirroring script supports license
blacklisting for the following license families:

* AGPL
* APACHE
* BSD
* GPL2
* GPL3
* LGPL
* MIT
* PROPRIETARY
* PUBLICDOMAIN
* PSF
* OTHER
* NONE

EXAMPLE: This example mirrors all the packages in the repository
except those that are GPL2-, GPL3- or BSD-licensed::

  license_blacklist:
    - GPL2
    - GPL3
    - BSD

NOTE: Older versions of Anaconda mirror support only license
blacklisting for GPL. If you are using an older version of
Repository, see the documentation for :ref:`customizing your
PyPI or Anaconda Repository mirror
<mirrors-sync-configuration-2.2>`.


Mirroring with a blacklist
==========================

The ``blacklist`` allows access to all packages except those
explicitly listed.

EXAMPLE: This example mirrors the entire Repository except the
bzip2, tk and openssl packages::

  blacklist:
    - bzip2
    - tk
    - openssl


Mirroring with a whitelist
==========================

The whitelist functions in combination with either the
license_blacklist or blacklist arguments, and re-adds packages
that were excluded by a previous argument.

EXAMPLE: This example mirrors the entire Repository except any
GPL2- or GPL3-licensed packages, but including readline, despite
the fact that it is GPL3-licensed::

  license_blacklist:
    - GPL2
    - GPL3
  whitelist:
    - readline


Combining multiple mirror configurations
========================================

You may find that combining two or more of the arguments above is the
simplest way to get the exact combination of packages that you want.

The platforms argument is evaluated before any other argument.

EXAMPLE: This example mirrors only linux-64 distributions of the
dnspython, shapely and gdal packages::

  platforms:
    - linux-64
  package_list:
    - dnspython
    - shapely
    - gdal

If the license_blacklist and blacklist arguments are combined,
the license_blacklist is evaluated first, and the blacklist is
a supplemental modifier.

EXAMPLE: In this example, the mirror configuration does not
mirror GPL2-licensed packages. It does not mirror the
GPL3-licensed package PyQt because it has been blacklisted. It
does mirror all other packages in Repository::

  license_blacklist:
    - GPL2
  blacklist:
    - pyqt

If the blacklist and whitelist arguments are both employed, the
blacklist is evaluated first, with the whitelist functioning
as a modifier.

EXAMPLE: This example mirrors all packages in the repository
except astropy and pygments. Despite being listed on the
blacklist, Accelerate is mirrored because it is listed on the
whitelist::

  blacklist:
   - accelerate
   - astropy
   - pygments
  whitelist:
   - accelerate
