Merge pull request #70 from EarthCubeInGeo/develop

Release candidate for v2020.2.0

HISTORY.rst

@@ -1,20 +1,29 @@
 
 .. :changelog:
 
+2020.2.0 (2020-11-06)
++++++++++++++++++++++
+
+- Changed valid core list to be read from JSON file downloaded from resen-core repo rather than hardcoded in source code
+- Fixed #63, bug which caused odd behavior with progress bar when downloading cores in narrow terminal window
+- Read version number from init file
+- Improve documentation on mounting, bucket exporting, and Linux post-installation steps
+- Fixed #14, an issue with the resen lockfile
+
 2020.1.0 (2020-06-24)
 +++++++++++++++++++++
 
 - Add new available resen-core: 2020.1.0
-- bug fixed when importing a bucket: now tar files of mounted paths are removed after a successful import.
-- fixed bug causing enabling sudo access for user jovyan failed
+- Bug fixed when importing a bucket: now tar files of mounted paths are removed after a successful import.
+- Fixed bug causing enabling sudo access for user jovyan failed
 
 2019.1.1 (2019-12-10)
 +++++++++++++++++++++
 
 - Raise exceptions added when local storage not found
 - Add detection of resen running on a windows system
-- if docker toolbox, change path to be the docker VM path instead of the host machine path
-- find a port that is available
+- If docker toolbox, change path to be the docker VM path instead of the host machine path
+- Find a port that is available
 
 2019.1.0 (2019-11-24)
 +++++++++++++++++++++

README.rst

@@ -18,7 +18,7 @@ Resen requires both python 3 and docker to be installed.
 
 1. Install `Python 3 <https://www.python.org/>`_
 2. Install `docker <https://docs.docker.com/install/>`_
-3. Install Resen with pip ``pip install git+https://github.com/EarthCubeInGeo/resen.git@v2020.1.0``
+3. Install Resen with pip ``pip install git+https://github.com/EarthCubeInGeo/resen.git@v2020.2.0``
 
 Please refer to the :ref:`installation documentation <installation>` for more detailed instructions.
 

docs/index.rst

@@ -8,6 +8,7 @@ Contents
    readme
    installation/index
    usage
+   tutorials
    mounting
 
 Changelog

docs/installation/installation.general.rst

@@ -35,4 +35,4 @@ Resen
 
 Install Resen from a python 3 environment using ``pip``::
 
-    pip install git+https://github.com/EarthCubeInGeo/resen.git@v2020.1.0
+    pip install git+https://github.com/EarthCubeInGeo/resen.git@v2020.2.0

docs/tutorials.rst

@@ -0,0 +1,38 @@
+Tutorials
+*********
+
+The InGeO team has collected a variety of `tutorials <https://github.com/EarthCubeInGeo/resen-core/tree/master/tutorials>`_ to help new users get started with everything from python basics to using specific geospace packages included in Resen.  These tutorials are available as jupyter notebooks and are automatically mounted in `Resen Online <https://ingeo.datatransport.org/home/resen/resen-online>`_, however, in deference to users who may want a cleaner starting environment, they have not been automatically included in the resen-core image.  The simplest way to access a tutorial is to download it directly into your resen bucket.
+
+1. Start a bucket.
+
+2. Launch a terminal window.
+
+3. Navigate to where you would like the tutorial to be saved with `cd`.  For instance, to save tutorials to the work directory::
+
+    $ cd /home/jovyan/work
+
+4. Copy/paste the appropriate fetching command from the table below to download the tutorial, for example::
+
+    $ wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/GetStartedWithPython.ipynb
+
+After downloading completes, the tutorial notebook should be visible in the navigation toolbar.  By double-clicking, you can launch the jupyter notebook, read through descriptions, try example code, and even modify the notebook to try new things.
+
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| Tutorials                                                                                                                              | Fetch Command                                                                                                          |
++========================================================================================================================================+========================================================================================================================+
+| `GetStartedWithPython.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/GetStartedWithPython.ipynb>`_       | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/GetStartedWithPython.ipynb``    |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `GetStartedWithJupyter.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/GetStartedWithJupyter.ipynb>`_     | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/GetStartedWithJupyter.ipynb``   |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `GetStartedWithResen.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/GetStartedWithReSEn.ipynb>`_         | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/GetStartedWithReSEn.ipynb``     |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `GeospacePackagesInResen.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/GeospacePackagesInResen.ipynb>`_ | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/GeospacePackagesInResen.ipynb`` |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `CaseStudy.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/CaseStudy.ipynb>`_                             | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/CaseStudy.ipynb``               |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `Intro-Swarm-viresclient.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/Intro-Swarm-viresclient.ipynb>`_ | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/Intro-Swarm-viresclient.ipynb`` |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `ICON-Data-Demo.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/ICON-Data-Demo.ipynb>`_                   | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/ICON-Data-Demo.ipynb``          |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
+| `CitationHelper.ipynb <https://github.com/EarthCubeInGeo/resen-core/blob/v2020.2.0/tutorials/CitationHelper.ipynb>`_                   | ``wget https://raw.githubusercontent.com/EarthCubeInGeo/resen-core/v2020.2.0/tutorials/CitationHelper.ipynb``          |
++----------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+

docs/usage.rst

@@ -12,7 +12,7 @@ This will open the resen tool::
        |   / _|\__ \ _|| .` |
        |_|_\___|___/___|_|\_|
 
-    Resen 2020.1.0 -- Reproducible Software Environment
+    Resen 2020.2.0 -- Reproducible Software Environment
 
     [resen] >>>
 
@@ -56,8 +56,8 @@ Setup a New Bucket
    Next, the user is asked to specify the version of resen-core to use::
 
      Please choose a version of resen-core.
-     Available versions: 2019.1.0, 2020.1.0
-     >>> Select a version: 2020.1.0
+     Available versions: 2019.1.0, 2020.1.0, 2020.2.0
+     >>> Select a version: 2020.2.0
 
    Optionally, one may then specify a local directory to :ref:`mount <mounting>` into the bucket at ``/home/jovyan/mount``::
 
@@ -90,7 +90,7 @@ Setup a New Bucket
      amber
      =====
 
-     Resen-core Version:  2020.1.0
+     Resen-core Version:  2020.2.0
      Status:  running
      Jupyter Token:  e7a11fc1ea42a445807b4e24146b9908e1abff82bacbf6f2
      Jupyter Port:  9002
@@ -112,7 +112,7 @@ Work with a Bucket
 
     [resen] >>> list
     Bucket Name         Docker Image             Status
-    amber               2020.1.0                 running
+    amber               2020.2.0                 running
 
    If a bucket is running, it will consume system resources accordingly.
 
@@ -124,7 +124,7 @@ Work with a Bucket
 
     [resen] >>> list
     Bucket Name         Docker Image             Status
-    amber               2020.1.0                 exited
+    amber               2020.2.0                 exited
 
    The bucket will still exist and can be restarted at any time, even after quitting and restarting resen.
 
@@ -136,7 +136,7 @@ Work with a Bucket
 
     [resen] >>> status
     Bucket Name         Docker Image             Status
-    amber               2020.1.0                 running
+    amber               2020.2.0                 running
 
 4. Export bucket ``amber``::
 

docs/version_update.py

@@ -0,0 +1,17 @@
+# Script to replace the version numbers in all *.rst files EXCEPT HISTORY.rst
+
+import re
+import os
+import fileinput
+
+old_version = '2020.2.0'
+new_version = '2020.1.0'
+
+for root, dirs, files in os.walk('..'):
+    for fn in files:
+        if fn.endswith('.rst') and fn!='HISTORY.rst':
+
+            filename = os.path.join(root,fn)
+            with fileinput.FileInput(filename, inplace=True) as f:
+                for line in f:
+                    print(line.replace(old_version, new_version), end='')

resen/DockerHelper.py

@@ -119,18 +119,43 @@ def stream_pull_image(self,image):
         Pull image from dockerhub.
         '''
         import datetime
+        import os
+
         # time formatting
         def truncate_secs(delta_time, fmt=":%.2d"):
             delta_str = str(delta_time).split(':')
             return ":".join(delta_str[:-1]) + fmt%(float(delta_str[-1]))
+
+        # get terminal dimensions
+        def get_terminal_dims():
+            """Returns integers rows and columns of the terminal."""
+            return [int(x) for x in os.popen('stty size', 'r').read().split()]
+
         # progress bar
-        def update_bar(sum_total,accumulated,t0,current_time, scale=0.5):
+        def update_bar(sum_total,accumulated,t0,current_time, init_bar_chars=34):
+            """Updates the progress bar.
+
+            Args:
+                sum_total: total number of bytes to download
+                accumulated: bytes already downloaded
+                t0: time when pulling image started
+                current_time: current time
+                init_bar_chars: default number of characters of the bar
+                              including [,>, spaces,  and ],
+                              e.g. [===>   ] would be 9 characters.
+            """
             percentage = accumulated/sum_total*100
-            nchars = int(percentage*scale)
-            bar = "\r["+nchars*"="+">"+(int(100*scale)-nchars)*" "+"]"
-            time_info = "Elapsed time: %s"%truncate_secs(current_time - t0)
-            print(bar+" %6.2f %%, %5.3f/%4.2fGB %s"%(percentage,
-                accumulated/1024**3,sum_total/1024**3,time_info),end="")
+            time_info = "Elapsed t: %s"%truncate_secs(current_time - t0)
+            bartext = "%5.1f %%, %5.3f/%4.2fGB %s"%(percentage,
+                accumulated/1024**3,sum_total/1024**3,time_info)
+            rows, columns = get_terminal_dims()
+            max_bar_length = max(5,columns - len(bartext)-1)
+            bar_chars = min(init_bar_chars,max_bar_length)
+            nchars = int(percentage*(bar_chars-3)/100)
+            bar = "\r["+nchars*"="+">"+(bar_chars-3-nchars)*" "+"]"
+            total_out = bar+bartext
+            max_out = min(columns,len(total_out)) # don't print beyond columns
+            print(total_out[:max_out],end="")
 
         print('Pulling image: {}:{}'.format(image['repo'],image['version']))
         print('   This may take some time...')

resen/Resen.py

@@ -521,10 +521,13 @@ def start_jupyter(self,bucket_name,local_port=None,container_port=None):
             local_port = bucket['port'][0][0]
             container_port = bucket['port'][0][1]
 
+        # Get the python environment path, if none found, default to py36
+        envpath = bucket['image'].get('envpath','/home/jovyan/envs/py36')
+
         # set a random token and form
         token = '%048x' % random.randrange(16**48)
-        command = "bash -cl 'source /home/jovyan/envs/py36/bin/activate py36 && jupyter lab --no-browser --ip 0.0.0.0 --port %s --NotebookApp.token=%s --KernelSpecManager.ensure_native_kernel=False'"
-        command = command % (container_port, token)
+        command = "bash -cl 'source %s/bin/activate && jupyter lab --no-browser --ip 0.0.0.0 --port %s --NotebookApp.token=%s --KernelSpecManager.ensure_native_kernel=False'"
+        command = command % (envpath,container_port, token)
 
         # exectute command to start jupyter server
         self.execute_command(bucket_name,command,detach=True)
@@ -563,12 +566,15 @@ def stop_jupyter(self,bucket_name):
         if pid is None:
             return True
 
+        # Get the python environment path, if none found, default to py36
+        envpath = bucket['image'].get('envpath','/home/jovyan/envs/py36')
+
         # form python command to stop jupyter and execute it
         port = bucket['jupyter']['port']
         python_cmd = 'from notebook.notebookapp import shutdown_server, list_running_servers; '
         python_cmd += 'svrs = [x for x in list_running_servers() if x[\\\"port\\\"] == %s]; ' % (port)
         python_cmd += 'sts = True if len(svrs) == 0 else shutdown_server(svrs[0]); print(sts)'
-        command = "bash -cl '/home/jovyan/envs/py36/bin/python -c \"%s \"'" % (python_cmd)
+        command = "bash -cl '%s/bin/python -c \"%s \"'" % (envpath,python_cmd)
         status = self.execute_command(bucket_name,command,detach=False)
 
         # now verify it is dead
@@ -889,13 +895,46 @@ def _get_home_dir(self):
         return os.path.join(homedir,appname)
 
 
+    def __process_exists(self,pid):
+        # Need to do different things for *nix vs. Windows
+        # see https://stackoverflow.com/questions/568271/how-to-check-if-there-exists-a-process-with-a-given-pid-in-python
+
+        if os.name == 'nt':
+            # only works on windows
+            from win32com.client import GetObject
+            WMI = GetObject('winmgmts:')
+            processes = WMI.InstancesOf('Win32_Process')
+            pids = [process.Properties_('ProcessID').Value for process in processes]
+            return pid in pids
+        else:
+            # only works on *nix systems
+            try:
+                os.kill(pid,0)
+            except ProcessLookupError:
+                return False
+            except PermissionError: # errno.EPERM
+                return True # Operation not permitted (i.e., process exists)
+            else:
+                return True # no error, we can send a signal to the process
+
+
     def __lock(self):
+        # dev note: if we want to be more advanced, need psutil as dependency
+        # get some telemetry to fingerprint with
+        cur_pid = os.getpid()      # process id
+
+        # check if lockfile exists
         self.__lockfile = os.path.join(self.resen_root_dir,'lock')
         if os.path.exists(self.__lockfile):
-            raise RuntimeError('Another instance of Resen is already running!')
+            #parse existing file
+            with open(self.__lockfile,'r') as f:
+                pid = int(f.read())
+                if self.__process_exists(pid):
+                    raise RuntimeError('Another instance of Resen is already running!')
 
+        telem = '%d' % (cur_pid)
         with open(self.__lockfile,'w') as f:
-            f.write('locked')
+            f.write(telem)
         self.__locked = True
 
 

resen/__init__.py

@@ -1,3 +1,3 @@
-__version__ = '2020.1.0'
+__version__ = '2020.2.0'
 
 from .Resen import Resen

resen/resencmd.py

@@ -233,7 +233,7 @@ def do_export(self,args):
             return
         else:
             try:
-                print('Exporting bucket %s.  This will take several mintues.' % bucket_name)
+                print('Exporting bucket %s.  This will take several minutes.' % bucket_name)
                 self.program.export_bucket(bucket_name, file_name, exclude_mounts=exclude_list, img_repo=img_name, img_tag=img_tag)
             except (ValueError, RuntimeError) as e:
                 print(e)