Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in installation of Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild 4.9.2 #21078

Open
gkaf89 opened this issue Jul 28, 2024 · 8 comments
Open

Comments

@gkaf89
Copy link

gkaf89 commented Jul 28, 2024

When trying to install Python-3.11.5-GCCcore-13.2.0.eb with EasyBuild version 4.9.2, the installation fails due to an error in the installation of a file. The installation script seems to be using the log file path to store intermediate files which causes an error.

  • EasyBuild command:

    eb --configfiles=iris_broadwell.cfg Python-3.11.5-GCCcore-13.2.0.eb
  • Contents of configuration file iris_broadwell.cfg:

    [MAIN]
    # Enable debug log mode (default: False)
    #debug=
    # Enable info log mode (default: False)
    #info=
    # Enable quiet/warning log mode (default: False)
    #quiet=
    
    [basic]
    # Submit the build as a job (default: False)
    job=False
    # Enable dependency resolution, optionally consider additional paths to search for easyconfigs (type pathsep-separated list)
    robot=/work/projects/software_stack_alpha/backup/easybuild/easyconfigs
    # Additional paths to consider by robot for easyconfigs (--robot paths get priority) (type pathsep-separated list; default: /mnt/aiongpfs/projects/software_stack_alpha/backup/easybuild/easyconfigs:/work/projects/software_stack_alpha/easybuild/iris/2023b/epyc/software/EasyBuild/4.9.1/easybuild/easyconfigs)
    robot-paths=%(DEFAULT_ROBOT_PATHS)s
    # Additional locations to consider in --search (next to --robot and --robot-paths paths) (type pathsep-separated list)
    #search-paths=
    
    [config]
    # Temporary build path (default: /home/users/gkafanas/.local/easybuild/build)
    buildpath=/tmp/easybuild/iris/2023b/epyc/build
    # Change prefix for buildpath, installpath, sourcepath and repositorypath (used prefix for defaults /home/users/gkafanas/.local/easybuild)
    # NOTE: it seems that installpath is not determined by prefix
    prefix=/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell
    # Install path for software and modules (default: /home/users/gkafanas/.local/easybuild)
    installpath=%(prefix)s
    # Location where container recipe & image will be stored (default: /home/users/gkafanas/.local/easybuild/containers)
    containerpath=%(prefix)s/containers
    # The destination path for the packages built by package-tool (default: /home/users/gkafanas/.local/easybuild/packages)
    packagepath=%(prefix)s/packages
    # Module naming scheme to use (default: EasyBuildMNS)
    module-naming-scheme=CategorizedModuleNamingScheme
    # Repository path, used by repository (is passed as list of arguments to create the repository instance). For more info, use --avail-repositories. (type comma-separated list; default: /home/users/gkafanas/.local/easybuild/ebfiles_repo)
    repositorypath=%(prefix)s/ebfiles_repo
    # Path(s) to where sources should be downloaded (string, colon-separated) (default: /home/users/gkafanas/.local/easybuild/sources)
    sourcepath=%(prefix)s/sources
    # Location of Python module with hook implementations (type str)
    hooks=configuration/hooks/ulhpc.py
    
    # Directory name and format of the log file (type comma-separated tuple; default: easybuild,easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log)
    logfile-format=%(prefix)s/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log
    
    # Backend to use for submitting jobs (type choice; default: GC3Pie) (choices: GC3Pie, PbsPython, Slurm)
    job-backend=Slurm
    
    [container]
    [easyconfig]
    [github]
    [informative]
    [job]
    
    [override]
    # Wait interval (in seconds) to use when waiting for existing lock to be removed (type <class 'int'>; default: 60)
    #wait-on-lock-interval=
    # Maximum amount of time (in seconds) to wait until lock is released (0 means no waiting at all, exit with error; -1 means no waiting limit, keep waiting) (type <class 'int'>; default: 0)
    #wait-on-lock-limit=
    [package]
    [regtest]
    [software]
    [unittest]
  • Error message produced during the installation step (sub-step: == taking care of extensions...):

    ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/20240728_023527/python passed to update_paths which only expects relative paths.')
    

Further details

During the failed installation sub-step the file

/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/%(name)s-%(version)s/%(date)s_%(time)s/python/sitecustomize.py

is created in the log directory (with the references in the log file path not resolved). This seems like a file that should be installed in the installation path according to the file contents:

# sitecustomize.py script installed by EasyBuild,
# to pick up Python packages installed with `--prefix` into folders listed in $EBPYTHONPREFIXES

import os
import site
import sys

# print debug messages when $EBPYTHONPREFIXES_DEBUG is defined
debug = os.getenv('EBPYTHONPREFIXES_DEBUG')

# use prefixes from $EBPYTHONPREFIXES, so they have lower priority than
# virtualenv-installed packages, unlike $PYTHONPATH

ebpythonprefixes = os.getenv('EBPYTHONPREFIXES')

if ebpythonprefixes:
    postfix = os.path.join('lib', 'python' + '.'.join(map(str, sys.version_info[:2])), 'site-packages')
    if debug:
        print("[EBPYTHONPREFIXES] postfix subdirectory to consider in installation directories: %s" % postfix)

    potential_sys_prefixes = (getattr(sys, attr, None) for attr in ("real_prefix", "base_prefix", "prefix"))
    sys_prefix = next(p for p in potential_sys_prefixes if p)
    base_paths = [p for p in sys.path if p.startswith(sys_prefix)]

    for prefix in ebpythonprefixes.split(os.pathsep):
        if debug:
            print("[EBPYTHONPREFIXES] prefix: %s" % prefix)
        sitedir = os.path.join(prefix, postfix)
        if os.path.isdir(sitedir):
            if debug:
                print("[EBPYTHONPREFIXES] adding site dir: %s" % sitedir)
            site.addsitedir(sitedir)

    # Move base python paths to the end of sys.path so modules can override packages from the core Python module
    sys.path = [p for p in sys.path if p not in base_paths] + base_paths

The installation fails when coping sitecustomize.py to its final path.

Following the installation error, the log files for Python are not created. Log files were created without issues for all packages on which Python depends.

@gkaf89
Copy link
Author

gkaf89 commented Jul 29, 2024

Call stack produced with traceback.print_stack() at exceptions in lines 986 and 1460 of file easybuild-framework/easybuild/tools/module_generator.py:

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 789, in <module>
    main(prepared_cfg_data=(init_session_state, eb_go, cfg_settings))
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 742, in main
    do_cleanup = process_easystack(options.easystack, args, logfile, testing, init_session_state, do_build)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 289, in process_easystack
    do_cleanup &= process_eb_args([path], eb_go, cfg_settings, modtool, testing, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 567, in process_eb_args
    ecs_with_res = build_and_install_software(ordered_ecs, init_session_state,
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/main.py", line 137, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4264, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 4143, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3978, in run_step
    step_method(self)()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 2880, in extensions_step
    fake_mod_data = self.load_fake_module(purge=True, extra_modules=build_dep_mods)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 1708, in load_fake_module
    fake_mod_path = self.make_module_step(fake=True)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/framework/easyblock.py", line 3751, in make_module_step
    txt += self.make_module_extra()
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599, in make_module_extra
    txt += self.module_generator.prepend_paths('PYTHONPATH', self.pythonpath)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 265, in prepend_paths
    return self.update_paths(key, paths, prepend=True, allow_abs=allow_abs, expand_relpaths=expand_relpaths)
  File "/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/tools/module_generator.py", line 1461, in update_paths
    traceback.print_stack()
ERROR: Build of /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/easybuild/easyconfigs/p/Python/Python-3.11.5-GCCcore-13.2.0.eb failed (err: 'build failed (first 300 chars): Absolute path /work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/logs/   %(name)s-%(version)s/20240729_095326/python passed to update_paths which only expects relative paths.') 

@gkaf89
Copy link
Author

gkaf89 commented Jul 29, 2024

Working back from

"/work/projects/software_stack_alpha/test/easybuild/iris/2023b/broadwell/software/EasyBuild/4.9.2/lib/python3.11/site-packages/easybuild/easyblocks/p/python.py", line 599

the problem seems to be caused by the definition of the pythonpath class member variable of the EB_Python block:

# Used for EBPYTHONPREFIXES handler script
self.pythonpath = os.path.join(log_path(), 'python')

Should pythonpath be set to the log directory? Would using build_path instead of log_path make more sense?

gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Jul 30, 2024
There are tow problems with the way the `pythonpath` instance field is
defined:

- it uses the log directory to store a build artifact, and
- the functions that use it do not accept a full path.

We cannot use the `build_path()` function instead of `log_path()` as the
function provides a full path as well. Thus we opted to hard-code a
relative path with respect to the build directory.

Issue: easybuilders/easybuild-easyconfigs#21078
@gkaf89
Copy link
Author

gkaf89 commented Jul 30, 2024

Using build_path suffers from the same issue with the log_path, the function that uses the pythonpath field variable expects a relative path.

I created a pull request in the easyblocks repository: easybuilders/easybuild-easyblocks#3400

gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Jul 31, 2024
There are 2 problems with the way the `pythonpath` instance field is defined:

- it uses the log directory to store a build artifact, and
- the functions that use `pythonpath` do not accept a full path.

We cannot use the `build_path()` function instead of `log_path()` as the
function provides a full path as well. Thus we opted to hard-code a
relative path with respect to the build directory.

Issue: easybuilders/easybuild-easyconfigs#21078
gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Jul 31, 2024
The `pythonpath` variable in the `python.py` EasyBlock is used to create
temporary directories to store build artifacts. However, there are 2
problems with the current definition of the `pythonpath` instance field:

- it uses the log directory to store build artifacts, and
- the functions that use `pythonpath` do not accept a full path.

The `build_path()` function cannot be used instead of `log_path()` as the
function provides a full path as well. Thus we opted to hard-code a
relative path with respect to the build directory.

Issue: easybuilders/easybuild-easyconfigs#21078
@boegel boegel added this to the release after 4.9.2 milestone Jul 31, 2024
@boegel
Copy link
Member

boegel commented Aug 6, 2024

I took a quick look, and it seems like the self.pythonpath in the Python easyblock is set up assuming that the log_path() function will return a relative path, while in your case it's returning an absolute path.

With a default EasyBuild configuration, I'm seeing:

$ python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
easybuild

When I mimic the relevant part of your configuration though, that changes:

export EASYBUILD_LOGFILE_FORMAT='/prefix/logs/%%(name)s-%%(version)s/%%(date)s_%%(time)s,easybuild-%%(name)s-%%(version)s.log'
python3 -c "from easybuild.tools.options import set_up_configuration; from easybuild.tools.config import log_path; set_up_configuration(silent=True); print(log_path())"
/prefix/logs/%(name)s-%(version)s/%(date)s_%(time)s

The reason the Python easyblock is using log_path() is to figure out the subdirectory of the installation directory in which the EasyBuild installation log file is stored (which by default is easybuild).

So in some sense this problem is self-inflicted, it only occurs because you've customized the logfile-format configuration setting.

That said, the Python easyblock should be robust against this, of course...

@boegel
Copy link
Member

boegel commented Aug 6, 2024

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

@gkaf89
Copy link
Author

gkaf89 commented Aug 6, 2024

It seems a bit restrictive the force the installation of the from log files in a path relative to the installation path.

This does not affects our case however, I can change the path to a relative path. It is already pointing inside the installation directory anyway.

@gkaf89
Copy link
Author

gkaf89 commented Aug 12, 2024

After giving this some more thought, I think the directory part of logfile-format is supposed to be a path that's relative to the software installation directory. EasyBuild isn't enforcing that though (it probably should).

I think we are dealing with 2 issues here.

  • logfile-format is expected to set a relative path, but this requirement is not documented or enforced. In my opinion it is a bit restrictive to force the log files into the directory specified by the easybuild prefix setting, but I would say this is a bug.
  • The pythonpath field is set to the directory containing the log files. Since the pythonpath is used to store build artifacts, I believe it is an error to store build artifacts inside a directory used to hold log files.

Am I understanding the use of pythonpath correctly? Is there a more appropriate directory to store the python artifacts?

@boegel
Copy link
Member

boegel commented Aug 14, 2024

@gkaf89 Your understanding of the pythonpath variable is correct (we should probably rename it too, because it's a bit confusing).

I think you're right that we should reconsider where the sitecustomize.py script is being stored.
That should definitely be somewhere in the installation directory.

That said, calling the easybuild subdirectory of the installation directory the "directory containing the log files" is somewhat incorrect, since other artifacts are also already stored in there, like a copy of the easyconfig file being used, patch files, the reprod directory that contains a copy of the easyblock, etc.

However, since sitecustomize.py is really part of the installation itself, it probably should be stored outside of the easybuild directory.

I guess storing it directory in the installation directory is not a good option, and then the question is in which (custom) subdirectory it should be stored (definitely not <installdir>/python/)...

Any suggestions here?

@boegel boegel modified the milestones: 4.9.3, release after 4.9.3 Sep 11, 2024
gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Sep 25, 2024
The `pythonpath` variable in the `python.py` EasyBlock is used to create
temporary directories to store build artifacts. However, there are 2
problems with the current definition of the `pythonpath` instance field:

- it uses the log directory to store build artifacts, and
- the functions that use `pythonpath` do not accept a full path.

The `build_path()` function cannot be used instead of `log_path()` as the
function provides a full path as well. Thus we opted to hard-code a
relative path with respect to the build directory.

Issue: easybuilders/easybuild-easyconfigs#21078
gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Nov 20, 2024
The `pythonpath` variable in the `python.py` EasyBlock is used to create
temporary directories to store build artifacts. However, there are 2
problems with the current definition of the `pythonpath` instance field:

- it uses the log directory to store build artifacts, and
- the functions that use `pythonpath` do not accept a full path.

The `build_path()` function cannot be used instead of `log_path()` as the
function provides a full path as well. Thus we opted to hard-code a
relative path with respect to the build directory.

Issue: easybuilders/easybuild-easyconfigs#21078
gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Nov 24, 2024
The `pythonpath` variable was used to store python scripts in a
temporary location and later copy them in the installation path. The
scripts were stored in the path were log files are stored, `log_path()`,
in both the build and installation directories.

- Rename `pythonpath` to `python_script_dir_relative_path`.
- Use the more appropriate location `etc/python` to store the python
  scripts in the build and then installation directories.

Note that all functions handling the python script path accept relative
paths only. Currently, only the `sitecustomize.py` python environment
initialization script is stored in script directory.

Issue: easybuilders/easybuild-easyconfigs#21078
gkaf89 added a commit to gkaf89/easybuild-easyblocks that referenced this issue Nov 24, 2024
The `pythonpath` variable was used to store python scripts in a
temporary location and later copy them in the installation path. The
scripts were stored in the path were log files are stored, `log_path()`,
in both the build and installation directories.

- Rename `pythonpath` to `python_script_dir_relative_path`.
- Use the more appropriate location `etc/python` to store the python
  scripts in the build and then installation directories.

Note that all functions handling the python script path accept relative
paths only. Currently, only the `sitecustomize.py` python environment
initialization script is stored in script directory.

Issue: easybuilders/easybuild-easyconfigs#21078
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants