Web and server: Add web-based job submission for BUDA #5918
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
variants are stored in <project>/buda_apps/<app>/<variant>.
This includes app files (copied from sandbox) and templates
(generated by handler)
- Add web interface for submitting BUDA jobs (not finished)
- Change implementation of user file sandbox
old: sandbox dir had 'link files' containing md5 and size;
actual file is in download hierarchy with sb_md5 name
new: sandbox dir has actual files.
parallel .md5/ dir has 'info files' (md5 size)
Files are not stored in download hierarchy.
New philosophy: names in the download hierarchy include
not only an MD5 (for uniqueness)
but also text describing the use of the file
(input file for a batch, part of a BUDA app, etc.).
This may allow duplicate files,
but it makes it possible to always clean up unused files.
- use readdir() instead of opendir()/scandir()
changes:
- creating a variant creates a JSON file, variant.json,
describing the dockerfile, app files, and in/out files.
Template files are now generated during job submission.
- no aliasing of files. If your main prog is foo,
your Dockerfile must end with CMD ./foo
- batch zip file must have shared input files
in a directory shared_input_files/.
All other directories are jobs.
It works now, to the extent of staging files properly and creating plausible-looking workunits. Next step: see if they work.
buda: output files are <copy_file/>
When you run a shell script on Unix, and it has Windows line endings (CRLF), it fails with a misleading 'file not found' error message. This can cause problems with BUDA apps, which can involve shell scripts, and all files go through the user sandbox. For example: if you put the script (with Unix endings) into Github and check it out on a Win machine, all of a sudden it has Win endings! If you upload it to your sandbox, it won't work. So I added a sandbox feature where you can add a file by pasting text into a web form. Surprisingly, even this changed the LF to a CRLF! I changed the form handler to convert CRLF to LF, and now it works. How many man-years have been wasted on this line-ending BS? I'm guessing the blame goes to Microsoft.
For BUDA batches, description is '<app> (<variant>)' Move job submission admin functions to their own page Lay the groundwork for unifying output file handling for remote job submission.
old model:
no assimilator
output files live in upload hierarchy w/ physical names
WUs are marked as assimulated when batch is retired;
file_deleter deletes output files after that.
new model:
assimilator (e.g. sample_assimilate.py and sample_assimilator.cpp)
moves output files of canonical results to project/results/<batch_id>,
with names that include the batch name and the logical name.
WU is marked as assimilated; file_deleter deletes
output files of non-canonical results.
advantages of new model:
can see all output files of a batch on cmdline with ls
can zip output files of a batch without copying them
unified naming scheme for output files that encodes
the batch, the job (e.g. the BUDA job dir name)
and the logical name of the file.
------------
script assimilator: pass logical names to the script
Support both models. Choice of model is per app.
The project.inc file says which app uses which model.
Principle: when showing jobs or files, use names that the user provided, rather than DB IDs or names that BOINC generated.
There was a problem hiding this comment.
Copilot reviewed 6 out of 26 changed files in this pull request and generated no suggestions.
Files not reviewed (20)
- html/inc/db_conn.inc: Language not supported
- html/inc/dir_hier.inc: Language not supported
- html/inc/result.inc: Language not supported
- html/inc/sandbox.inc: Language not supported
- html/inc/submit_util.inc: Language not supported
- html/inc/util.inc: Language not supported
- html/user/buda.php: Language not supported
- html/user/buda_submit.php: Language not supported
- html/user/get_output3.php: Language not supported
- html/user/sandbox.php: Language not supported
- html/user/submit.php: Language not supported
- samples/docker_wrapper/docker_wrapper.cpp: Language not supported
- samples/docker_wrapper/test_buda/Dockerfile_worker_1: Language not supported
- samples/docker_wrapper/test_buda/main_2.sh: Language not supported
- sched/assimilator.cpp: Language not supported
- sched/file_deleter.cpp: Language not supported
- sched/sample_assimilator.cpp: Language not supported
- sched/sample_dummy_assimilator.cpp: Language not supported
- sched/script_assimilator.cpp: Language not supported
- sched/validate_util.cpp: Language not supported
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5918 +/- ##
============================================
- Coverage 10.73% 10.73% -0.01%
Complexity 1068 1068
============================================
Files 280 280
Lines 36604 36618 +14
Branches 8484 8488 +4
============================================
Hits 3930 3930
- Misses 32285 32299 +14
Partials 389 389
|
AenBleidd
requested changes
Nov 25, 2024
- validate URL args that are used as filenames; prevent ../ stuff.
Do this by checking for '/'; is that sufficient?
- add 'delete app' function
- remove binary test file
There was a problem hiding this comment.
Copilot reviewed 6 out of 26 changed files in this pull request and generated no suggestions.
Files not reviewed (20)
- html/inc/db_conn.inc: Language not supported
- html/inc/dir_hier.inc: Language not supported
- html/inc/result.inc: Language not supported
- html/inc/sandbox.inc: Language not supported
- html/inc/submit_util.inc: Language not supported
- html/inc/util.inc: Language not supported
- html/inc/util_basic.inc: Language not supported
- html/user/buda.php: Language not supported
- html/user/buda_submit.php: Language not supported
- html/user/get_output3.php: Language not supported
- html/user/sandbox.php: Language not supported
- html/user/submit.php: Language not supported
- samples/docker_wrapper/docker_wrapper.cpp: Language not supported
- samples/docker_wrapper/test_buda/Dockerfile_worker_1: Language not supported
- samples/docker_wrapper/test_buda/main_2.sh: Language not supported
- sched/assimilator.cpp: Language not supported
- sched/file_deleter.cpp: Language not supported
- sched/sample_assimilator.cpp: Language not supported
- sched/sample_dummy_assimilator.cpp: Language not supported
- sched/script_assimilator.cpp: Language not supported
AenBleidd
approved these changes
Nov 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Design goals:
rather than DB IDs or names generated by BOINC.
When users submit jobs, they use whatever filenames they want.
These files get copied to physical names (which the user doesn't see).
Users can reuse the sames names for other files.
delete all unused files (app, input, output).
To do this, I redesigned a couple of server-side features.
(If any projects are currently using these features,
they'll need to do a reset; I don't think any are.)
The web interface is the same, but I changed the implementation.
used by various job-submission systems.
These are now unified into a single scheme.