============================
Backing up and restoring AEN
============================

.. contents::
   :local:
   :depth: 3

Document purpose
================

This document lays out the steps to backup and restore Anaconda Enterprise
Notebooks (AEN) for Disaster Recovery. It is not intended to provide High
Availability. Each of the components (Server, Gateway and Compute) has its own
instructions and each may be done individually as needed. The steps primarily
involve creating tar files of important configuration files and data.

This document is written for a system administrator who is comfortable with
basic Linux command line navigation and usage.

To migrate to a new cluster, use these backup and restore instructions to
back up the system from the old cluster and restore it to the new cluster.

Important notes
===============

Review the :doc:`../concepts` page to become familiar with the different
components and how they work together.

Root or sudo access is required for some commands.

**CAUTION:** All commands MUST be run by ``$AEN_SRVC_ACCT`` (the account used to
run AEN) except for those commands explicitly indicated to run as root or sudo.
If the commands are not run by the correct user, the installation will not
work, and a full uninstallation and reinstallation will be required!

These instructions assume that the fully qualified domain name (FQDN) has not
changed for any of the component nodes. If any of the FQDNs are not the same,
additional steps will be needed.

Server component steps
======================

Backup
------

Mongo database
^^^^^^^^^^^^^^

This will create a single tar file called ``aen_mongo_backup.tar`` that
includes only the database named "wakari" that is used by AEN. It also
generates a log of the database backup.

NOTE: These commands must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	mongodump -db wakari -o aen_main >> mongo_backup.log
	tar -cvf aen_mongo_backup.tar aen_main


AEN Server config files (including License file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create a tar file of all of the configuration files, including any license
files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -cvf aen_server_config.tar -C /opt/wakari/ wakari-server/etc/wakari/


Nginx config (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^

Make a copy of the nginx configuration file if it has been customized.
The default configuration for the AEN server is a symlink.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	/etc/nginx/conf.d/www.enterprise.conf -> /opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf


SSL certificates (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Make a copy of the SSL certificates files (certfiles) for the server, including
the key file, and a copy of the certfile for the gateway, which is needed for
verification if using self-signed or private CA signed certs.


Restore
-------

Reinstall AEN-Server
^^^^^^^^^^^^^^^^^^^^

See :doc:`the instructions for installing the current version of AEN-Server
<../install/install-server-node>`.

It is not necessary to upload the license, because it will be restored with the
config files.

NOTE: The new installation will generate a new password for the local ``$AEN_SRVC_ACCT``
account.

Restore Mongo database
^^^^^^^^^^^^^^^^^^^^^^

This assumes that mongo was reinstalled as part of the reinstallation of the
server component. Untar the mongo database and restore it.

NOTE: These commands must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_mongo_backup.tar
	mongorestore --drop aen_main


NOTE: The ``--drop`` option resets the ``$AEN_SRVC_ACCT`` user password and restores
the database to the exact state it was in at the time of backup. Please see the
`MongoDB documentation <https://docs.mongodb.com/v2.6/reference/program/mongorestore/>`_
for more information about mongorestore options for Mongo 2.6.

NOTE: AEN uses Mongo 2.6 by default. If you are using a different version,
consult the documentation for your version.

AEN Server config files (including License file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Untar the tar file of all of the configuration files, including any license files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_server_config.tar -C /opt/wakari/


Make sure the files are in ``/opt/wakari/wakari-server/etc/wakari/``
and are owned by the ``$AEN_SRVC_ACCT``.


Nginx config (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^

Make sure any modifications to the nginx configuration are either in
``/etc/nginx/conf.d`` or in ``/opt/wakari/wakari-server/etc/nginx/conf.d/``
with a proper symlink.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	/etc/nginx/conf.d/www.enterprise.conf -> /opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf


SSL certificates (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Move any SSL certificate files to the locations indicated in the config files.


Restart server
^^^^^^^^^^^^^^

Restart the server application.

NOTE: This command must be run as root or with sudo.

.. code-block:: bash

	service wakari-server restart


Gateway component steps
=======================

Backup
------

Config files
^^^^^^^^^^^^

Create a tar file of all of the configuration files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -cvf aen_gateway_config.tar -C /opt/wakari/ wakari-gateway/etc/wakari/


Custom .condarc file (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Make a copy of any ``/opt/wakari/miniconda/.condarc`` if it has been modified.

SSL certificates (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Make a copy of SSL certificate files for the gateway (including the key file)
and the certfile for the server (needed for verification if using self-signed
or private CA signed certs).


Restore
-------

Reinstall AEN-Gateway
^^^^^^^^^^^^^^^^^^^^^

Setting variables and changing permissions
""""""""""""""""""""""""""""""""""""""""""

NOTE: These commands must be run by ``$AEN_SRVC_ACCT``.

.. include:: ../install/install-gateway-node.rst
   :start-after: AEN_GW_INST_VARS_PERMS_START
   :end-before: AEN_GW_INST_VARS_PERMS_END

Running the AEN gateway installer
"""""""""""""""""""""""""""""""""

.. include:: ../install/install-gateway-node.rst
   :start-after: AEN_GW_INST_RUN_START
   :end-before: AEN_GW_INST_RUN_END

Config files
^^^^^^^^^^^^

Untar the configuration files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_gateway_config.tar -C /opt/wakari


Verify that the files are in ``/opt/wakari/wakari-gateway/etc/wakari/``
and are owned by the ``$AEN_SRVC_ACCT``.

Custom .condarc file (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Move the custom .condarc file to ``/opt/wakari/miniconda/.condarc``.

SSL certificates (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Move any SSL certificate files to the locations indicated in the config files.

Restart gateway
^^^^^^^^^^^^^^^

Restart the gateway application.

NOTE: This command must be run as root or with sudo.

.. code-block:: bash

	service wakari-gateway restart


Compute component steps
=======================

Backup
------

Config files
^^^^^^^^^^^^

Create a tar file of all of the configuration files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -cvf aen_compute_config.tar -C /opt/wakari/ wakari-compute/etc/wakari

Custom Changes (rare)
^^^^^^^^^^^^^^^^^^^^^

Manually backup any custom changes that were applied to the code.
One change might be additional files in the skeleton folder:

``/opt/wakari/wakari-compute/lib/node_modules/wakari-compute-launcher/skeleton``


Create user list
^^^^^^^^^^^^^^^^

AEN uses POSIX access control lists (ACLs) for project sharing, so the backup
must preserve the ACL information. This is done with a script that creates a
file named ``users.lst`` containing a list of all users that have access to
projects on a given compute node. Download and run the script.

NOTE: These commands must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	wget https://s3.amazonaws.com/continuum-airgap/misc/wk-compute-get-acl-users.py
	chmod 755 wk-compute-get-acl-users.py
	./wk-compute-get-acl-users.py


Project files
^^^^^^^^^^^^^

Create a tar of the projects directory with ACLs enabled. The default projects
base location is ``/projects``.

NOTE: This command must be run as root or with sudo.

.. code-block:: bash

	tar --acls -cpvf projects.tar -C <projects base location>/*


Full Anaconda (option 1)
^^^^^^^^^^^^^^^^^^^^^^^^

If any changes have been made to the default Anaconda installation (additional
packages installed or packages removed), it is necessary to backup the entire
Anaconda installation.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -cvf aen_anaconda.tar -C /opt/wakari/anaconda/*


If no changes have been made to the default installation of Anaconda, you may
just backup the ``.condarc`` file and any custom environments.


Partial Anaconda (option 2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Custom ``.condarc`` file
""""""""""""""""""""""""

Make a copy of ``/opt/wakari/anaconda/.condarc``.

Custom environments (if needed)
"""""""""""""""""""""""""""""""

Create a tar file of any custom shared environments.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -cvf aen_compute_envs.tar -C /opt/wakari/ anaconda/envs


NOTE: If no custom shared environments have been created, the ``envs`` folder
will not be present.


Restore
-------

Reinstall AEN-Compute
^^^^^^^^^^^^^^^^^^^^^

Setting variables and changing permissions
""""""""""""""""""""""""""""""""""""""""""

NOTE: These commands must be run by ``$AEN_SRVC_ACCT``.

.. include:: ../install/install-compute-node.rst
   :start-after: AEN_C_INST_VARS_PERMS_START
   :end-before: AEN_C_INST_VARS_PERMS_END

Running the AEN compute installer
"""""""""""""""""""""""""""""""""

.. include:: ../install/install-compute-node.rst
   :start-after: AEN_C_INST_RUN_START
   :end-before: AEN_C_INST_RUN_END

Config files
^^^^^^^^^^^^

Untar the config files.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_compute_config.tar -C /opt/wakari


NOTE: Verify that they are located in ``/opt/wakari/wakari-compute/etc/wakari``
and are owned by the ``$AEN_SRVC_ACCT``.

Custom changes (rare)
^^^^^^^^^^^^^^^^^^^^^

Manually restore any custom changes you saved in the backup section.
If there are changes in the skeleton directory, these files must be world readable or
projects will refuse to start.

Create users
^^^^^^^^^^^^

NOTE: Only create users with these instructions if your Linux machine is not
bound to LDAP.

In order for the ACLs to be set properly on restore, all users that have
permissions to the files must be available on the machine. Ask your system
administrator for the proper way to do this for your system, such as using the
"useradd" tool. A list of users that are needed was created in the backup
process as a file named ``users.lst``.

A process similar to the following ``useradd`` example will be suitable for most
Linux systems.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	xargs -0 -n 1 useradd --user-group < users.lst


Project files
^^^^^^^^^^^^^

Create the projects directory in the location specified in projectRoot in
``wk-compute-launcher-config.json``.

NOTE: By default this directory is ``/projects``.

Then untar the projects directory with ACLs.

NOTE: This command must be run as root or with sudo:

.. code-block:: bash

	tar --acls -xpvf projects.tar -C <projects base location>

Full Anaconda (option 1)
^^^^^^^^^^^^^^^^^^^^^^^^

If you did a full backup of the full Anaconda installation, untar this file to
``/opt/wakari/anaconda``.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_anaconda.tar -C /opt/wakari


Partial Anaconda (option 2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Restore the custom ``.condarc`` file.

If you did a partial backup of the Anaconda installation, move the copy of the
``.condarc`` file to ``/opt/wakari/anaconda/.condarc``.

Custom environments (if needed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Untar any custom environments that were created to ``/opt/wakari/anaconda/envs``.

NOTE: This command must be run by ``$AEN_SRVC_ACCT``.

.. code-block:: bash

	tar -xvf aen_compute_envs.tar -C /opt/wakari

Restart compute node
^^^^^^^^^^^^^^^^^^^^

Restart the compute-launcher application.

NOTE: This command must be run as root or with sudo.

.. code-block:: bash

	service wakari-compute restart
