Skip to content

Commit

Permalink
Merge pull request #47 from EarthCubeInGeo/develop
Browse files Browse the repository at this point in the history 
[MNT]: Release v2019.1.0
  • Loading branch information
@asreimer
asreimer authored Nov 16, 2019
2 parents 102a365 + aed89ea commit 20b179f
Show file tree
Hide file tree
Showing 10 changed files with 1,324 additions and 938 deletions.
15 changes: 15 additions & 0 deletions docs/Continuous_Integration.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Continuous Integration

This is the practice of frequently pushing new changes to the codebase to the repository and having each change verified as working by an automated build and test tool. Github provides hooks for automated build services to watch a repo and trigger a new build/test.

There are several options for testing utilties out there. This provides a nice summary of three of them and of the CI process itself: https://hackernoon.com/continuous-integration-circleci-vs-travis-ci-vs-jenkins-41a1c2bd95f5



# TravisCI

For linux and macos

# Appveyor

For windows
2 changes: 1 addition & 1 deletion docs/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
author = 'Resen Team'

# The full version, including alpha/beta/rc tags
release = 'v2019.1.0rc2'
release = 'v2019.1.0'

master_doc = 'index'

Expand Down
11 changes: 2 additions & 9 deletions docs/installation/installation.general.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,7 @@ Docker CE is the recomended version of Docker to use with Resen. `Installation
Resen
=====

Install Resen by first cloning the resen GitHub repo (https://github.com/EarthCubeInGeo/resen)::
Install Resen from a python 3 environment using ``pip``::

git clone https://github.com/EarthCubeInGeo/resen.git
pip install git+https://github.com/EarthCubeInGeo/resen.git@v2019.1.0

Change into the ``resen`` directory::

cd resen

In a python 3 environment, use pip to install Resen::

pip install .
12 changes: 2 additions & 10 deletions docs/installation/installation.windows.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,9 @@ Install Anaconda and Resen
We recommend downloading and installing the Python 3 Anaconda Distribution (https://www.anaconda.com/distribution/). This simplifies the installation and usage of several common software tools needed to install and run Resen.

**Resen**:
Using the start menu search, open the "Anaconda Powershell Prompt" and navigate to a directory where you wish to host the Resen source code. Next, install Resen by first cloning the resen GitHub repo (https://github.com/EarthCubeInGeo/resen)::
Using the start menu search, open the "Anaconda Powershell Prompt" and install Resen using ``pip``::

git clone https://github.com/EarthCubeInGeo/resen.git

Change into the ``resen`` directory::

cd resen

Finally, install Resen::

pip install .
pip install git+https://github.com/EarthCubeInGeo/resen.git@v2019.1.0

Once complete, this will provide the command line command ``resen``. Next, we need to install Docker.

Expand Down
168 changes: 126 additions & 42 deletions docs/usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ To use resen, simply enter ``resen`` at the command line::

This will open the resen tool::

___ ___ ___ ___ _ _
___ ___ ___ ___ _ _
| _ \ __/ __| __| \| |
| / _|\__ \ _|| .` |
|_|_\___|___/___|_|\_|
Resen 2019.1.0rc2 -- Reproducible Software Environment
[resen] >>>

Resen 2019.1.0 -- Reproducible Software Environment

[resen] >>>

Type ``help`` to see available commands::

Expand All @@ -24,24 +24,28 @@ This will produce a list of resen commands you will use to manage your resen buc

Documented commands (type help <topic>):
========================================
EOF exit quit start_jupyter stop_jupyter
create_bucket help remove_bucket status
EOF exit help list remove status
create export import quit start stop

To get more information about a specific command, enter ``help <command>``.

Resen Workflow
==============

Use Resen to create and remove buckets. Buckets are portable, system independent environments where code can be developed and run. Buckets can be shared between Windows, Linux, and macos systems and all analysis within the bucket will be run exactly the same. Resen buckets come preinstalled with a variety of common geospace software that can be used immediately in analysis.
To create, import, export, and remove buckets, we use Resen. Buckets are portable, system independent environments where code can be developed and run. Buckets can be shared between Windows, Linux, and macos systems and all analysis within the bucket will be run exactly the same. Resen buckets come preinstalled with a variety of common geospace software that can be used immediately in analysis.

The interface to a resen bucket is a jupyter lab server and access to the bucket is provided through a web browser. The user home directory is ``/home/jovyan``. Any mounted storage directories are available in ``mount``.

Below are instructions on how to use Resen to work with buckets. A typical workflow will involve: creation of a bucket, performing scientific data analysis inside the bucket, exporting the bucket and sharing it with colleagues. Collaborators can then import the bucket, perform additional analysis, and then export the bucket for publication in an open access citable repository, such as `Zenodo <https://zenodo.org/>`_.

Setup a New Bucket
------------------

1. Creating a new bucket is performed with the command::

[resen] >>> create_bucket
[resen] >>> create

The ``create_bucket`` command queries the user for several pieces of information required to create a bucket. First it asks for the bucket name. Creating a bucket named ``amber``::
The ``create`` command queries the user for several pieces of information required to create a bucket. First it asks for the bucket name. Creating a bucket named ``amber``::

Please enter a name for your bucket.
Valid names may not contain spaces and must start with a letter and be less than 20 characters long.``
Expand All @@ -50,27 +54,19 @@ Setup a New Bucket
Next, the user is asked to specify the version of resen-core to use::

Please choose a version of resen-core.
Available versions: 2019.1.0rc2
>>> Select a version: 2019.1.0.rc2
Available versions: 2019.1.0
>>> Select a version: 2019.1.0

Optionally, one may then specify a local directory to mount into the bucket at ``/home/jovyan/work``::
Optionally, one may then specify a local directory to mount into the bucket at ``/home/jovyan/mount``::

Local directories can be mounted to either /home/jovyan/work or /home/jovyan/mount/ in
a bucket. The /home/jovyan/work location is a workspace and /home/jovyan/mount/ is intended
for mounting in data. You will have rw privileges to everything mounted in work, but can
specify permissions as either r or rw for directories in mount. Code and data created in a
bucket can ONLY be accessed outside the bucket or after the bucket has been deleted if it is
saved in a mounted local directory.
>>> Mount storage to /home/jovyan/work? (y/n): y
>>> Enter local path: /some/local/path

Followed by additional local directories that can be mounted under ``/home/jovyan/mount``::

>>> Mount storage to /home/jovyan/mount? (y/n): y
>>> Enter local path: /some/other/local/path
>>> Enter bucket path: /home/jovyan/mount/data001
>>> Enter permissions (r/rw): r
>>> Mount additional storage to /home/jovyan/mount? (y/n): n
>>> Enter local path: /some/local/path

Finally, the user is asked if they want jupyterlab to be started::

Expand All @@ -79,60 +75,148 @@ Setup a New Bucket
after which resen will begin creating the bucket. Example output for a new bucket named ``amber`` with jupyterlab started is::

...adding core...
...adding ports...
...adding mounts...
Bucket created successfully!
...starting jupyterlab...
Jupyter lab can be accessed in a browser at: http://localhost:9000/?token=61469c2ccef5dd27dbf9a8ba7c296f40e04278a89e6cf76a
Jupyter lab can be accessed in a browser at: http://localhost:9002/?token=e7a11fc1ea42a445807b4e24146b9908e1abff82bacbf6f2

2. Check the status of the bucket::

[resen] >>> status amber
{'bucket': {'name': 'amber'}, 'docker': {'image': '2019.1.0rc2', 'container': 'a6501d441a9f025dc7dd913bf6d531b6b452d0a3bd6d5bad0eedca791e1d92ca', 'port': [[9000, 9000, True]], 'storage': [['/some/local/path', '/home/jovyan/work', 'rw'], ['/some/other/local/path', '/home/jovyan/mount/data001', 'ro']], 'status': 'running', 'jupyter': {'token': '61469c2ccef5dd27dbf9a8ba7c296f40e04278a89e6cf76a', 'port': 9000}, 'image_id': 'sha256:3ba43e401c1b1a8eca8969aec8426a22d99bca349fd837270fa06dbcaefaeb47', 'pull_image': 'earthcubeingeo/resen-core@sha256:c3783e3b7f05ec17f9381a01009b794666107780d964e8087c62f7baaa00049d'}}

At this point, the bucket should have a name, an image, at least one port, and optionally one or more storage location. Status should be ``running`` if the user decided to have jupyterlab started, otherwise the status will be ``None``.
[resen] >>> status amber

amber
=====
Resen-core Version: 2019.1.0
Status: running
Jupyter Token: e7a11fc1ea42a445807b4e24146b9908e1abff82bacbf6f2
Jupyter Port: 9002
Jupyter lab URL: http://localhost:9002/?token=e7a11fc1ea42a445807b4e24146b9908e1abff82bacbf6f2
Storage:
Local Bucket Permissions
/some/local/path /home/jovyan/mount/path rw
Ports:
Local Bucket
9002 9002

At this point, the bucket should have a name, an image, at least one port, and optionally one or more storage locations. Status should be ``running`` if the user decided to have jupyterlab started, otherwise the status will be ``None``.

Work with a Bucket
------------------
1. Check what buckets are available with ``status``::
1. Check what buckets are available with ``list``::

[resen] >>> status
[resen] >>> list
Bucket Name Docker Image Status
amber 2019.1.0rc2 running
amber 2019.1.0 running

If a bucket is running, it will consume system resources accordingly.

2. Stop jupyter lab from a bucket::
2. Stop the bucket ``amber``::

[resen] >>> stop_jupyter amber
[resen] >>> stop amber

The status of ``amber`` should now be ``exited``::

[resen] >>> status
[resen] >>> list
Bucket Name Docker Image Status
amber 2019.1.0rc2 exited
amber 2019.1.0 exited

The bucket will still exist and can be restarted at any time, even after quitting and restarting resen.

3. Start a jupyter lab in bucket ``amber`` that has been stopped::
3. Start the bucket ``amber`` that was just stopped::

[resen] >>> start_jupyter amber
[resen] >>> start amber

The status of ``amber`` should now be ``running``::

[resen] >>> status
Bucket Name Docker Image Status
amber 2019.1.0rc2 running
amber 2019.1.0 running

4. Export bucket ``amber``::

[resen] >>> export amber

The ``export`` command will ask a series of question. First, provide a name for the output *.tar file::

>>> Enter name for output tar file: /path/for/output/amber.tar

If desired, change the default name and tag for the exported image::

By default, the output image will be named "amber" and tagged "latest".
>>> Would you like to change the name and tag? (y/n): y
>>> Image name: custom_name
>>> Image tag: custom_tag

Specify if you want all mounted directories to be included in the exported bucket. Answering `n` to this query will allow you to see how large each mount is and specify which you would like to include. Consider excluding any mounts that are not nessesary for the analysis to reduce the size of the output file::

The following local directories are mounted to the bucket (total 2212 MB):
/home/usr/mount1
/home/usr/mount2
/home/usr/mount3
>>> Would you like to include all of these in the exported bucket? (y/n): n
>>> Include /home/usr/mount1 [154.68095 MB]? (y/n): y
>>> Include /home/usr/mount2 [2005.28493 MB]? (y/n): y
>>> Include /home/usr/mount3 [53.59823 MB]? (y/n): y

The jupyter lab server starts in the ``/home/jovyan`` directory, which should include the persistent storage directories ``work`` and ``mount``.
The user can alternate between the jupyter lab and the classic notebook view by changing the url in the browser from ``http://localhost:8000/lab`` to ``http://localhost:8000/tree``. Alternatively one can switch from the lab to the notebook through Menu -> Help -> Launch Classic Notebook.
Confirm that you want to continue with the export. The values shown should be considered a "high-side" approximation and may not be the actual final size::

This export could require up to 13337 MB of disk space to complete and will produce an output file up to 4600 MB.
>>> Are you sure you would like to continue? (y/n): y
Exporting bucket amber. This will take several minutes.

5. Import a new bucket, ``amber2``, from a tar file ``amber.tar``::

[resen] >>> import

This command will also ask a series of questions. First provide a name for the imported bucket::

Please enter a name for your bucket.
Valid names may not contain spaces and must start with a letter and be less than 20 characters long.
>>> Enter bucket name: amber2

Specify the *.tar file to import the bucket from::

>>> Enter name for input tar file: /path/to/file/amber.tar

If desired, enter a custom image name and tag. If not provided, the name an image saved on export will be used::

>>> Would you like to keep the default name and tag for the imported image? (y/n): n
>>> Image name: amber2
>>> Image tag: new_tag

When a bucket that had mounts is imported, the mounted directories must be extracted and saved on the local machine. Resen will do this automatically, but you have the option to specify where these files should be saved instead of the default location::

The default directory to extract the bucket metadata and mounts to is /default/save/path/resen_amber2.
>>> Would you like to specify and alternate directory? (y/n): y
>>> Enter path to directory: /new_save_path

Aside from the existing mounts, you can add new mounts to a imported bucket. This is useful if you would like to repeat the analysis with a different dataset::

>>> Mount additional storage to the imported bucket? (y/n): y
>>> Enter local path: /new/local/path/new_mount
>>> Enter bucket path: /home/jovyan/mount/new_mount
>>> Enter permissions (r/rw): r
>>> Mount additional storage to /home/jovyan/mount? (y/n): n

Similar to ``create``, you have the option to start jupyter lab immediately after the bucket is imported::

>>> Start bucket and jupyterlab? (y/n): y
...starting jupyterlab...
Jupyter lab can be accessed in a browser at: http://localhost:9003/?token=70532767bab0ddc4febe2790efaaf974961e961e78e6025a


Remove a Bucket
---------------
**WARNING**: This will permanently delete the bucket. Any work that was not saved in a mounted storage directory or downloaded from the bucket will be **permanently lost**.

The user can delete a bucket with the following command::

[resen] >>> remove_bucket amber
[resen] >>> remove amber

A bucket that is running needs to be stopped before being removed.


A bucket that is running needs to be stopped before removed.
WARNING: This will permanently delete the bucket. Any work that was not saved in a mounted storage directory will be lost.
Loading

0 comments on commit 20b179f

Please sign in to comment.