Skip to content

2.1.12: Merge pull request #284 from HSF/dev
Compare
Choose a tag to compare
@wguanicedew wguanicedew released this 12 Mar 17:00
· 206 commits to master since this release
2.1.12
9720992
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

new iworkflow (funcation-as-a-task) development
(1) Add new iDDS http rest function to be able to create new iwork, for the workflow to generate new tasks
Add rest function
Add client function
(2) iDDS agents
Update clerk, transformer and carrier to adapt new iworkflow
Also update some ‘core’ functions to adapt it
(3)Add panda plugin in carrier.submitter to support new iworkflow
Add functions to convert python functions to panda tasks
(4) Add several database updates
Sqlalchemy related updates
Sql commands to update oracle db
Alembic scripts to update database automatically (for k8s deployment)
(5) Add a new transformer to panda system
Pilot will at first start a transformer, which starts the payloads.
The new transformer will be started in pilot, to load idds clients, which will load the function from a json parameter and then run the function defined in iworkflow/iwork
Add tools to build the transformer, it needs to add corresponding packages to self-zipped bash scripts. When running the bash transformer, it will automatically unzip itself, setup the python environment and launch the python idds client to run iworkflow
(6) Add iworkflow tests
(7) Update idds common package
Split it to common.client package and common.server package: Some packages used only by server were in the common package, which made some client errors during importing the server packages. The splitting will simplify the common.client. It will be good to run clients on some remote sites.
(8) Test and fix iworkflow during running with container
(9) Prepare HPO test example
Have a working test example locally, fix some old issues because of python2 -> python3
Define and build alma9 container with installed ML softwares, such as xgboost, bayes_opt
Update HPO example to add python decorators to convert it to python-as-a-task workflow
(10)Test and optimize the test example
Develop codes to generate plots (AUC with eta(learning rate))
(11) add is_processed status to include both finished and subfinished

Assets 2
Loading