=================
General functions
=================

Installing conda packages
=========================

Install conda packages on the cluster nodes using
``adam scale conda install``.

EXAMPLE: To install NumPy and pandas and their dependencies into the
root conda environment on a cluster named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda install numpy pandas
    All nodes (x4) response:
    {
      "actions": {
        "LINK": [
          "mkl-11.3.3-0 /opt/continuum/anaconda/pkgs 1",
          "openssl-1.0.2h-1 /opt/continuum/anaconda/pkgs 1",
          "sqlite-3.13.0-0 /opt/continuum/anaconda/pkgs 1",
          "zlib-1.2.8-3 /opt/continuum/anaconda/pkgs 1",
          "python-2.7.11-5 /opt/continuum/anaconda/pkgs 1",
          "conda-env-2.5.1-py27_0 /opt/continuum/anaconda/pkgs 1",
          "numpy-1.11.0-py27_2 /opt/continuum/anaconda/pkgs 1",
          "pycosat-0.6.1-py27_1 /opt/continuum/anaconda/pkgs 1",
          "pytz-2016.4-py27_0 /opt/continuum/anaconda/pkgs 1",
          "pyyaml-3.11-py27_4 /opt/continuum/anaconda/pkgs 1",
          "requests-2.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
          "ruamel_yaml-0.11.7-py27_0 /opt/continuum/anaconda/pkgs 1",
          "setuptools-23.0.0-py27_0 /opt/continuum/anaconda/pkgs 1",
          "six-1.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
          "wheel-0.29.0-py27_0 /opt/continuum/anaconda/pkgs 1",
          "conda-4.1.4-py27_0 /opt/continuum/anaconda/pkgs 1",
          "pip-8.1.2-py27_0 /opt/continuum/anaconda/pkgs 1",
          "python-dateutil-2.5.3-py27_0 /opt/continuum/anaconda/pkgs 1",
          "pandas-0.18.1-np111py27_0 /opt/continuum/anaconda/pkgs 1"
          ],
          "PREFIX": "/opt/continuum/anaconda",
        },
        "success": true
      }

Creating conda environments
===========================

Create multiple conda environments on the cluster nodes using
``adam scale conda create``.

EXAMPLE: To create a conda environment called ``py3`` with Python 3 on a
cluster named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda create -n py3 python=3
    All nodes (x4) response:
    {
      "actions": {
        "LINK": [
          "openssl-1.0.2h-1 1 None",
          "readline-6.2-2 1 None",
          "sqlite-3.13.0-0 1 None",
          "tk-8.5.18-0 1 None",
          "xz-5.2.2-0 1 None",
          "zlib-1.2.8-3 1 None",
          "python-3.5.1-5 1 None",
          "setuptools-23.0.0-py35_0 1 None",
          "wheel-0.29.0-py35_0 1 None",
          "pip-8.1.2-py35_0 1 None"
        ],
        "PREFIX": "/opt/continuum/anaconda/envs/py3",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
      },
      "success": true
    }

Updating conda packages
=======================

Update conda packages on the cluster nodes using
``adam scale conda update``.

EXAMPLE: To update pandas and its dependencies in the root conda
environment on a cluster named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda update pandas
    All nodes (x4) response:
    {
      "actions": {
        "LINK": [
          "pandas-0.18.1-np111py27_0 1 None"
        ],
        "PREFIX": "/opt/continuum/anaconda",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
        "UNLINK": [
          "pandas-0.18.0-np111py27_0"
        ],
      },
      "success": true
    }

Removing conda packages or environments
=======================================

Remove conda packages on the cluster nodes using
``adam scale conda remove``.

EXAMPLE: To remove pandas from the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda remove pandas
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda",
        "UNLINK": [
          "pandas-0.18.1-np111py27_0"
        ],
      },
      "success": true
    }

Remove conda environments on the cluster nodes using
``adam scale conda remove``.

EXAMPLE: To remove an environment called ``py3`` on a cluster named
``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda remove -n py3 --all
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda/envs/py3",
        "UNLINK": [
          "openssl-1.0.2j-0",
          "pip-8.1.2-py35_0",
          "python-3.5.2-0",
          "readline-6.2-2",
          "setuptools-27.2.0-py35_0",
          "sqlite-3.13.0-0",
          "tk-8.5.18-0",
          "wheel-0.29.0-py35_0",
          "xz-5.2.2-0",
          "zlib-1.2.8-3"
        ]
      },
      "success": true
    }

Listing conda packages
======================

List conda packages on the cluster nodes using
``adam scale conda list``.

EXAMPLE: To list conda packages in the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda list
    All nodes (x4) response:
    +-----------------+---------+--------+
    | Package         | Version | Build  |
    +-----------------+---------+--------+
    | numpy           | 1.11.0  | py27_2 |
    | conda-env       | 2.5.1   | py27_0 |
    | tk              | 8.5.18  | 0      |
    | pip             | 8.1.2   | py27_0 |
    | readline        | 6.2     | 2      |
    | setuptools      | 23.0.0  | py27_0 |
    | openssl         | 1.0.2h  | 1      |
    | pycosat         | 0.6.1   | py27_1 |
    | python-dateutil | 2.5.3   | py27_0 |
    | sqlite          | 3.13.0  | 0      |
    | pytz            | 2016.4  | py27_0 |
    | mkl             | 11.3.3  | 0      |
    | yaml            | 0.1.6   | 0      |
    | ruamel_yaml     | 0.11.7  | py27_0 |
    | zlib            | 1.2.8   | 3      |
    | pyyaml          | 3.11    | py27_4 |
    | requests        | 2.10.0  | py27_0 |
    | wheel           | 0.29.0  | py27_0 |
    | python          | 2.7.11  | 5      |
    | conda           | 4.1.4   | py27_0 |
    | pycrypto        | 2.6.1   | py27_0 |
    | six             | 1.10.0  | py27_0 |
    +-----------------+---------+--------+

Getting conda information
=========================

View information about the conda installation on the cluster nodes using
``adam scale conda info``.

EXAMPLE: To view information about the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda info
    All nodes (x4) response:
    Current conda install:

                 platform : linux-64
            conda version : 4.1.4
        conda-env version : 2.5.1
      conda-build version : not installed
           python version : 2.7.11.final.0
         requests version : 2.10.0
         root environment : /opt/continuum/anaconda  (writable)
      default environment : /opt/continuum/anaconda
         envs directories : /opt/continuum/anaconda/envs
            package cache : /opt/continuum/anaconda/pkgs
             channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
                            https://repo.continuum.io/pkgs/free/noarch/
                            https://repo.continuum.io/pkgs/pro/linux-64/
                            https://repo.continuum.io/pkgs/pro/noarch/
              config file : None
             offline mode : False
        is foreign system : False

Or to view a list of conda environments on the cluster nodes:

.. code-block:: none

  $ adam scale -n cluster conda info -e
  All nodes (x4) response:
  # conda environments:
  #
  py27                     /opt/continuum/anaconda/envs/py27
  py35                     /opt/continuum/anaconda/envs/py35
  r                        /opt/continuum/anaconda/envs/r
  root                  *  /opt/continuum/anaconda


Managing conda with environment definition files
================================================

Manage conda environments on the cluster nodes using
``adam scale env``:

.. code-block:: none

    $ adam scale -n cluster env
    Usage: adam scale env [OPTIONS] COMMAND [ARGS]...

    Options:
      -h, --help  Show this message and exit.

    Commands:
      create  Create conda environment
      list    List conda environments
      remove  Remove conda environment
      update  Update conda environment

This functionality uses
`conda-env <http://conda.pydata.org/docs/commands.html#conda-environment-commands>`_
and conda environment environment definition--YAML--files.

EXAMPLE: The following ``environment.yml`` file describes a conda
environment named ``my-env`` with Python 2.7 and additional packages:

.. code-block:: yaml

    name: my-env
    channels:
    - defaults
    dependencies:
    - python=2.7.*
    - bokeh
    - dask
    - distributed
    - pandas

Use ``adam scale env create`` to create a conda environment based on the
above conda environment definition file ``environment.yml``:

.. code-block:: none

    $ adam scale -n cluster env create -f environment.yml
    All nodes (x4) response:
    {
      "actions": {
        "EXTRACT": [
          "msgpack-python-0.4.8-py27_0",
          "numpy-1.11.2-py27_0",
          "psutil-5.0.0-py27_0",
          "pytz-2016.7-py27_0",
          "tblib-1.3.0-py27_0",
          "python-dateutil-2.6.0-py27_0",
          "zict-0.1.0-py27_0",
          "pandas-0.19.1-np111py27_0",
          "tornado-4.4.2-py27_0",
          "bokeh-0.12.3-py27_0",
          "dask-0.12.0-py27_0",
          "distributed-1.14.1-py27_0"
        ],
        "FETCH": [
          "msgpack-python-0.4.8-py27_0",
          "numpy-1.11.2-py27_0",
          "psutil-5.0.0-py27_0",
          "pytz-2016.7-py27_0",
          "tblib-1.3.0-py27_0",
          "python-dateutil-2.6.0-py27_0",
          "zict-0.1.0-py27_0",
          "pandas-0.19.1-np111py27_0",
          "tornado-4.4.2-py27_0",
          "bokeh-0.12.3-py27_0",
          "dask-0.12.0-py27_0",
          "distributed-1.14.1-py27_0"
        ],
        "LINK": [
          "mkl-11.3.3-0 1",
          "openssl-1.0.2j-0 1",
          "readline-6.2-2 1",
          "sqlite-3.13.0-0 1",
          "tk-8.5.18-0 1",
          "yaml-0.1.6-0 1",
          "zlib-1.2.8-3 1",
          "python-2.7.12-1 1",
          "backports-1.0-py27_0 1",
          "backports_abc-0.4-py27_0 1",
          "click-6.6-py27_0 1",
          "cloudpickle-0.2.1-py27_0 1",
          "futures-3.0.5-py27_0 1",
          "heapdict-1.0.0-py27_1 1",
          "locket-0.2.0-py27_1 1",
          "markupsafe-0.23-py27_2 1",
          "msgpack-python-0.4.8-py27_0 1",
          "numpy-1.11.2-py27_0 1",
          "psutil-5.0.0-py27_0 1",
          "pytz-2016.7-py27_0 1",
          "pyyaml-3.12-py27_0 1",
          "requests-2.11.1-py27_0 1",
          "setuptools-27.2.0-py27_0 1",
          "six-1.10.0-py27_0 1",
          "tblib-1.3.0-py27_0 1",
          "toolz-0.8.0-py27_0 1",
          "wheel-0.29.0-py27_0 1",
          "chest-0.2.3-py27_0 1",
          "jinja2-2.8-py27_1 1",
          "partd-0.3.6-py27_0 1",
          "pip-9.0.1-py27_0 1",
          "python-dateutil-2.6.0-py27_0 1",
          "singledispatch-3.4.0.3-py27_0 1",
          "ssl_match_hostname-3.4.0.2-py27_1 1",
          "zict-0.1.0-py27_0 1",
          "pandas-0.19.1-np111py27_0 1",
          "tornado-4.4.2-py27_0 1",
          "bokeh-0.12.3-py27_0 1",
          "dask-0.12.0-py27_0 1",
          "distributed-1.14.1-py27_0 1"
        ],
        "PREFIX": "/opt/continuum/anaconda/envs/my-env",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
        "op_order": [
          "RM_FETCHED",
          "FETCH",
          "RM_EXTRACTED",
          "EXTRACT",
          "UNLINK",
          "LINK",
          "SYMLINK_CONDA"
        ]
      },
      "success": true
    }

NOTE: The command ``adam scale env create`` is compatible only with conda dependencies
and ignores any pip or other non-conda dependencies that are specified
in the conda environment definition file.

Use ``adam scale env list`` to list conda environments on the cluster:

.. code-block:: none

    $ adam scale -n cluster env list
    All nodes (x4) response:
    # conda environments:
    #
    my-env                   /opt/continuum/anaconda/envs/my-env
    root                  *  /opt/continuum/anaconda

Use ``adam scale env update`` to update packages in an existing conda
environment based on a conda environment definition file:

.. code-block:: none

    $ adam scale -n cluster env update -f environment.yml
        All nodes (x4) response:
        {
          "message": "All requested packages already installed.",
          "success": true
        }

Use ``adam scale env remove`` to remove a conda environment from the
cluster:

.. code-block:: none

    $ adam scale -n cluster env remove -n my-env
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda/envs/my-env",
        "UNLINK": [
          "mkl-11.3.3-0",
          "numpy-1.11.2-py27_0",
          "openssl-1.0.2j-0",
          "pandas-0.18.0-np111py27_0",
          "pip-9.0.1-py27_0",
          "python-2.7.12-1",
          "python-dateutil-2.6.0-py27_0",
          "pytz-2016.7-py27_0",
          "readline-6.2-2",
          "setuptools-27.2.0-py27_0",
          "six-1.10.0-py27_0",
          "sqlite-3.13.0-0",
          "tk-8.5.18-0",
          "wheel-0.29.0-py27_0",
          "zlib-1.2.8-3"
        ]
      },
      "success": true
    }
