Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test installing R and RStudio with the scripts #25

Closed
Tracked by #14
eeholmes opened this issue Oct 8, 2024 · 6 comments
Closed
Tracked by #14

test installing R and RStudio with the scripts #25

eeholmes opened this issue Oct 8, 2024 · 6 comments

Comments

@eeholmes
Copy link
Member

eeholmes commented Oct 8, 2024 •
edited
Loading

The install_r script fails in the make part (which is part of the R source).

Created a dev branch where I am root in the docker image so I can test the script and figure out what is going on.

Update: Setting path temporarily fixes the problem. Here the PATH var is only for the RUN cmds.

  RUN PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin && \
    mkdir /rocker_scripts && \
    cp ${REPO_DIR}/scripts/install_R_source.sh /rocker_scripts/install_R_source.sh && \
    chmod +x /rocker_scripts/install_R_source.sh && \
    cd / && \
    /rocker_scripts/install_R_source.sh
@eeholmes eeholmes mentioned this issue Oct 8, 2024
Closed
3 tasks
@eeholmes
Copy link
Member Author

eeholmes commented Oct 9, 2024

Debug notes. Trying to get this to run

Start Codespace on py-rocket-2

docker run -d -p 8888:8888 --name test ghcr.io/nmfs-opensci/py-rocket-2:8e17b0058367
# run as root bash
docker exec -u root -it test /bin/bash

Then install stuff

cd /
source ${REPO_DIR}/rocker.sh # get env vars
# Read the Dockerfile and process each line
while IFS= read -r line; do
    # Check if the line starts with ENV or RUN
    if [[ "$line" == ENV* ]]; then
        # Export the environment variable from the ENV line
        eval $(echo "$line" | sed 's/^ENV //g')
    elif [[ "$line" == RUN* ]]; then
        # Run the command from the RUN line
        cmd=$(echo "$line" | sed 's/^RUN //g') || echo ${cmd}" encountered error. Continuing"
        echo "Executing: $cmd"
        eval "$cmd"
    fi
done < /rocker_scripts/r-ver_4.4.1.Dockerfile

@eeholmes
Copy link
Member Author

eeholmes commented Oct 9, 2024 •
edited
Loading

Where I am at.

  • I forked rocker
  • Started codespace
  • ran docker build . -f dockerfiles/r-ver_4.4.1.Dockerfile
  • build fine. takes 5 minutes or so
  • testing with builddeps:jammy. Seems ok

So what is different with the repo2docker image?

  • testing just the install_R_source.sh from appendix in dev2 branch -- does not work
  • test install_R_source.sh from Codespace docker image -- does not work
  • test with a minimal repo2docker image

@eeholmes
Copy link
Member Author

eeholmes commented Oct 9, 2024

Here is the Dockerfile that repo2docker is making

FROM docker.io/library/buildpack-deps:jammy
  # Avoid prompts from apt
  ENV DEBIAN_FRONTEND=noninteractive
  # Set up locales properly
  RUN apt-get -qq update && \
      apt-get -qq install --yes --no-install-recommends locales > /dev/null && \
      apt-get -qq purge && \
      apt-get -qq clean && \
      rm -rf /var/lib/apt/lists/*
  RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
      locale-gen
  ENV LC_ALL=en_US.UTF-8 \
      LANG=en_US.UTF-8 \
      LANGUAGE=en_US.UTF-8
  # Use bash as default shell, rather than sh
  ENV SHELL=/bin/bash
  # Set up user
  ARG NB_USER
  ARG NB_UID
  ENV USER=${NB_USER} \
      HOME=/home/${NB_USER}
  RUN groupadd \
          --gid ${NB_UID} \
          ${NB_USER} && \
      useradd \
          --comment "Default user" \
          --create-home \
          --gid ${NB_UID} \
          --no-log-init \
          --shell /bin/bash \
          --uid ${NB_UID} \
          ${NB_USER}
  # Base package installs are not super interesting to users, so hide their outputs
  # If install fails for some reason, errors will still be printed
  RUN apt-get -qq update && \
      apt-get -qq install --yes --no-install-recommends \
         gettext-base \
         less \
         unzip \
         > /dev/null && \
      apt-get -qq purge && \
      apt-get -qq clean && \
      rm -rf /var/lib/apt/lists/*
  EXPOSE 8888
  # Environment variables required for build
  ENV APP_BASE=/srv
  ENV CONDA_DIR=${APP_BASE}/conda
  ENV NB_PYTHON_PREFIX=${CONDA_DIR}/envs/notebook
  ENV NPM_DIR=${APP_BASE}/npm
  ENV NPM_CONFIG_GLOBALCONFIG=${NPM_DIR}/npmrc
  ENV NB_ENVIRONMENT_FILE=/tmp/env/environment.lock
  ENV MAMBA_ROOT_PREFIX=${CONDA_DIR}
  ENV MAMBA_EXE=${CONDA_DIR}/bin/mamba
  ENV CONDA_PLATFORM=linux-64
  ENV KERNEL_PYTHON_PREFIX=${NB_PYTHON_PREFIX}
  # Special case PATH
  ENV PATH=${NB_PYTHON_PREFIX}/bin:${CONDA_DIR}/bin:${NPM_DIR}/bin:${PATH}
  # If scripts required during build are present, copy them
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e70a7b /etc/profile.d/activate-conda.sh
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e10-2dlinux-2d64-2elock-8fa955 /tmp/env/environment.lock
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-6a6072 /tmp/install-base-env.bash
  RUN TIMEFORMAT='time: %3R' \
  bash -c 'time /tmp/install-base-env.bash' && \
  rm -rf /tmp/install-base-env.bash /tmp/env
  RUN mkdir -p ${NPM_DIR} && \
  chown -R ${NB_USER}:${NB_USER} ${NPM_DIR}
  # ensure root user after build scripts
  USER root
  # Allow target path repo is cloned to be configurable
  ARG REPO_DIR=${HOME}
  ENV REPO_DIR=${REPO_DIR}
  # Create a folder and grant the user permissions if it doesn't exist
  RUN if [ ! -d "${REPO_DIR}" ]; then \
          /usr/bin/install -o ${NB_USER} -g ${NB_USER} -d "${REPO_DIR}"; \
      fi
  WORKDIR ${REPO_DIR}
  RUN chown ${NB_USER}:${NB_USER} ${REPO_DIR}
  # We want to allow two things:
  #   1. If there's a .local/bin directory in the repo, things there
  #      should automatically be in path
  #   2. postBuild and users should be able to install things into ~/.local/bin
  #      and have them be automatically in path
  #
  # The XDG standard suggests ~/.local/bin as the path for local user-specific
  # installs. See https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
  ENV PATH=${HOME}/.local/bin:${REPO_DIR}/.local/bin:${PATH}
  # The rest of the environment
  ENV CONDA_DEFAULT_ENV=${KERNEL_PYTHON_PREFIX}
  # Run pre-assemble scripts! These are instructions that depend on the content
  # of the repository but don't access any files in the repository. By executing
  # them before copying the repository itself we can cache these steps. For
  # example installing APT packages.
  # ensure root user after preassemble scripts
  USER root
  # Copy stuff.
  COPY --chown=1000:1000 src/ ${REPO_DIR}/
  # Run assemble scripts! These will actually turn the specification
  # in the repository into an image.
  # Container image Labels!
  # Put these at the end, since we don't want to rebuild everything
  # when these change! Did I mention I hate Dockerfile cache semantics?
  LABEL repo2docker.ref="refs/heads/dev2"
  LABEL repo2docker.repo="https://github.com/nmfs-opensci/py-rocket-2"
  LABEL repo2docker.version="[202](https://github.com/nmfs-opensci/py-rocket-2/actions/runs/11257663103/job/31302445744#step:5:205)4.07.0+28.g239c4f5"
  # We always want containers to run as non-root
  USER ${NB_USER}
  # Make sure that postBuild scripts are marked executable before executing them
  RUN chmod +x postBuild
  RUN ./postBuild
  # Add start script
  RUN chmod +x "${REPO_DIR}/start"
  ENV R2D_ENTRYPOINT="${REPO_DIR}/start"
  # Add entrypoint
  ENV PYTHONUNBUFFERED=1
  COPY /python3-login /usr/local/bin/python3-login
  COPY /repo2docker-entrypoint /usr/local/bin/repo2docker-entrypoint
  ENTRYPOINT ["/usr/local/bin/repo2docker-entrypoint"]
  # Specify the default command to run
  CMD ["jupyter", "notebook", "--ip", "0.0.0.0"]
  # Appendix:
  # Re-enable man pages disabled in Ubuntu 18 minimal image
  # https://wiki.ubuntu.com/Minimal
  USER root
  ENV R_VERSION="4.4.1"
  ENV R_HOME="/usr/local/lib/R"
  ENV TZ="Etc/UTC"
  RUN mkdir /rocker_scripts && \
    cp ${REPO_DIR}/install_R_source.sh /rocker_scripts/install_R_source.sh && \
    chmod +x /rocker_scripts/install_R_source.sh && \
    cd / && \
    /rocker_scripts/install_R_source.sh
  # Revert to default user
  USER ${NB_USER}

@eeholmes
Copy link
Member Author

eeholmes commented Oct 9, 2024

This runs. This only biggish thing removed

  # If scripts required during build are present, copy them
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-e70a7b /etc/profile.d/activate-conda.sh
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e10-2dlinux-2d64-2elock-8fa955 /tmp/env/environment.lock
  COPY --chown=1000:1000 build_script_files/-2fopt-2fvenv-2flib-2fpython3-2e11-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dbase-2denv-2ebash-6a6072 /tmp/install-base-env.bash
  RUN TIMEFORMAT='time: %3R' \
  bash -c 'time /tmp/install-base-env.bash' && \
  rm -rf /tmp/install-base-env.bash /tmp/env

FROM docker.io/library/buildpack-deps:jammy
  # Avoid prompts from apt
  ENV DEBIAN_FRONTEND=noninteractive
  # Set up locales properly
  RUN apt-get -qq update && \
      apt-get -qq install --yes --no-install-recommends locales > /dev/null && \
      apt-get -qq purge && \
      apt-get -qq clean && \
      rm -rf /var/lib/apt/lists/*
  RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
      locale-gen
  ENV LC_ALL=en_US.UTF-8 \
      LANG=en_US.UTF-8 \
      LANGUAGE=en_US.UTF-8
  # Use bash as default shell, rather than sh
  ENV SHELL=/bin/bash
  # Set up user
  ENV NB_USER="jovyan"
  ENV NB_UID=1000
  ENV USER=${NB_USER} \
      HOME=/home/${NB_USER}
  RUN groupadd \
          --gid ${NB_UID} \
          ${NB_USER} && \
      useradd \
          --comment "Default user" \
          --create-home \
          --gid ${NB_UID} \
          --no-log-init \
          --shell /bin/bash \
          --uid ${NB_UID} \
          ${NB_USER}
  # Base package installs are not super interesting to users, so hide their outputs
  # If install fails for some reason, errors will still be printed
  RUN apt-get -qq update && \
      apt-get -qq install --yes --no-install-recommends \
         gettext-base \
         less \
         unzip \
         > /dev/null && \
      apt-get -qq purge && \
      apt-get -qq clean && \
      rm -rf /var/lib/apt/lists/*
  EXPOSE 8888
  # Environment variables required for build
  ENV APP_BASE=/srv
  ENV CONDA_DIR=${APP_BASE}/conda
  ENV NB_PYTHON_PREFIX=${CONDA_DIR}/envs/notebook
  ENV NPM_DIR=${APP_BASE}/npm
  ENV NPM_CONFIG_GLOBALCONFIG=${NPM_DIR}/npmrc
  ENV NB_ENVIRONMENT_FILE=/tmp/env/environment.lock
  ENV MAMBA_ROOT_PREFIX=${CONDA_DIR}
  ENV MAMBA_EXE=${CONDA_DIR}/bin/mamba
  ENV CONDA_PLATFORM=linux-64
  ENV KERNEL_PYTHON_PREFIX=${NB_PYTHON_PREFIX}
  # Special case PATH
  ENV PATH=${NB_PYTHON_PREFIX}/bin:${CONDA_DIR}/bin:${NPM_DIR}/bin:${PATH}
  RUN mkdir -p ${NPM_DIR} && \
  chown -R ${NB_USER}:${NB_USER} ${NPM_DIR}
  # ensure root user after build scripts
  USER root
  ENV REPO_DIR="/srv/repo"
  # Create a folder and grant the user permissions if it doesn't exist
  RUN if [ ! -d "${REPO_DIR}" ]; then \
          /usr/bin/install -o ${NB_USER} -g ${NB_USER} -d "${REPO_DIR}"; \
      fi
  WORKDIR ${REPO_DIR}
  RUN chown ${NB_USER}:${NB_USER} ${REPO_DIR}
  # We want to allow two things:
  #   1. If there's a .local/bin directory in the repo, things there
  #      should automatically be in path
  #   2. postBuild and users should be able to install things into ~/.local/bin
  #      and have them be automatically in path
  #
  # The XDG standard suggests ~/.local/bin as the path for local user-specific
  # installs. See https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
  ENV PATH=${HOME}/.local/bin:${REPO_DIR}/.local/bin:${PATH}
  # The rest of the environment
  ENV CONDA_DEFAULT_ENV=${KERNEL_PYTHON_PREFIX}
  # Run pre-assemble scripts! These are instructions that depend on the content
  # of the repository but don't access any files in the repository. By executing
  # them before copying the repository itself we can cache these steps. For
  # example installing APT packages.
  # ensure root user after preassemble scripts
  USER root
  # Copy stuff.
  COPY --chown=1000:1000 . ${REPO_DIR}/
  # Run assemble scripts! These will actually turn the specification
  # in the repository into an image.
  # Container image Labels!
  # Put these at the end, since we don't want to rebuild everything
  # when these change! Did I mention I hate Dockerfile cache semantics?
  LABEL repo2docker.ref="refs/heads/dev2"
  LABEL repo2docker.repo="https://github.com/nmfs-opensci/py-rocket-2"
  LABEL repo2docker.version="[202](https://github.com/nmfs-opensci/py-rocket-2/actions/runs/11257663103/job/31302445744#step:5:205)4.07.0+28.g239c4f5"
  # We always want containers to run as non-root
  USER ${NB_USER}
  # Add entrypoint
  ENV PYTHONUNBUFFERED=1
  # Specify the default command to run
  CMD ["jupyter", "notebook", "--ip", "0.0.0.0"]
  # Appendix:
  # Re-enable man pages disabled in Ubuntu 18 minimal image
  # https://wiki.ubuntu.com/Minimal
  USER root
  ENV R_VERSION="4.4.1"
  ENV R_HOME="/usr/local/lib/R"
  ENV TZ="Etc/UTC"
  RUN mkdir /rocker_scripts && \
    cp ${REPO_DIR}/scripts/install_R_source.sh /rocker_scripts/install_R_source.sh && \
    chmod +x /rocker_scripts/install_R_source.sh && \
    cd / && \
    /rocker_scripts/install_R_source.sh
  # Revert to default user
  USER ${NB_USER}

@eeholmes
Copy link
Member Author

ok I was able to get myself out of libcurl header hell with

 ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
 ENV PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig:$PKG_CONFIG_PATH
 ENV CURL_CONFIG=/usr/bin/curl-config

but this basic problem is the conda bins stuck in the path. Those ENV vars are telling it to ignore all the conda stuff.

/srv/conda/condabin:/home/codespace/.local/bin:/home/codespace/.local/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:

Setting the PATH temporarily back to what it was before micromamba was installed works and seems easiest.

  RUN PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin && \
    mkdir /rocker_scripts && \
    cp ${REPO_DIR}/scripts/install_R_source.sh /rocker_scripts/install_R_source.sh && \
    chmod +x /rocker_scripts/install_R_source.sh && \
    cd / && \
    /rocker_scripts/install_R_source.sh

Question is, it the conda stuff in the path in a repo2docker image generally going to cause a problem for R/RStudio.

@eeholmes
Copy link
Member Author

Long journey but fixed #46 #45 #44 #44 #43 #42 #41 #40 #39 #38 and on and on

hard part was getting the ENV variables passed to the scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@eeholmes