Skip to content

Meeting Notes May 2020

Jump to bottom
Patricia Grubel edited this page May 13, 2020 · 12 revisions

Wednesday, May 13, 2020

Attendees - Rusty Davis (rstyd), Qiang Guan (guanxyz), Al McPherson (mcpherson), Pat Grubel (pagrubel), Jake Tronge (jtronge)

Announcements

  • Quincy will be joining us starting Monday.
  • Losing Paid Github access
    • Losing wiki and multiple reviewers with the private repo. Perhaps we could switch to a public repo?

PR Review

  1. NONE

Issue Review

  1. Develop an abstract Class for Container Runtime #157(qguanxyz) -
    • Want to support various runtimes
    • Have a system vs user configuration For now we will support the user configuration file with Charliecloud as default. In the future we will look through the system configuration after the user configuration.
      - A user can override the system configuration.
      - Users that want to let the system admins set things up for them can be unaware of setup.

Around the room

  • Rusty
    • Continuing work on uWSGI
    • Looking at Jake's code
    • PR for the integration test
  • Qiang
    • Looking at Kubernetes for ideas on how to implement the orchestrator
    • Read through the documentation for Kubernetes (REST APIs)
    • Setup environment on local server and tried out a tutorial
    • Going to start writing some design documentation
  • Pat
    • WoWoHa - presenting on June 26th, looking for others to contribute
    • Working on container runtime charliecloud class, have a working version
      • Make sure the unit tests still work
      • Need to add using configuration file
  • Jake
    • Working on the BEEStart code
    • Fixed a few issues Tim/Rusty addressed
  • Al
    • Working on database redesign
    • Writing scripts to test new nodes/relationships
    • Figuring out how to possibly work with Steven

Monday, May 11, 2020

Attendees - Rusty Davis (rstyd), Al McPherson (mcpherson), Pat Grubel (pagrubel), Jake Tronge (jtronge)

PR Review

  1. NONE

Issue Review

  1. NONE

Discussion (ToDo?)

Around the room

  • Rusty
    • working on uWSGI will use uWSGI itself
    • will be editing the TM and WFM code to use that instead of debug Flash server
  • Jake
    • working on BEEstart not working with non debug option, trying to figure out why
    • Rusty's change may help; He will check it out.
  • Al
    • working on database redesign and parsing on the side
  • Pat
    • abstract container runtime out, maybe work load manager

Wednesday, May 06, 2020

Attendees - Rusty Davis (rstyd), Qiang Guan (guanxyz), Al McPherson (mcpherson), Pat Grubel (pagrubel), Tim Randles (trandles-lanl), Jake Tronge (jtronge)

PR Review

  1. NONE

Issue Review

  1. 155(rstyd) - Come up with an id scheme for the WFM
    • Al will look how that impacts the database.
  2. 156(guanxyz) - discussed roles of the components need to add resource monitor and arrow back to WFM from TM

Discussion (ToDo?)

  • We need to write up a data model.
  • When we have the database refactor #119, Wiki page: Graph Database Entities.
    • Where's the slurm job id - should be in the task meta data.
    • There is no schema it is freeform. We can write a utility to spit our the metadata at completion of the workflow.
  • Qiang - Do we have a big picture with all the components?

Around the room

  • Rusty
    • for pexpect test, I need to build it into pytest
    • working on WSGI Tim you should be able to have a user run it
    • can we containerize it which should be cleaner
  • Qiang
    • Working on the interface design, looking at Kubernetes design
  • Al
    • Still working on parsing looking at how to get scatter, gather and globbing in the database
    • Also, working on stuff for Steven
  • Pat
    • Separate out the container runtime and maybe the resource manager at the same time
  • Jake
    • Working on BEEstart, debug is working for graph database but not working for TM and WFM
      • Tim - look at stderr and stdout in subprocess attaching
  • Tim
    • Working on FY21 planning
    • Big Picture will be working on it try to get that out by noon Friday

Monday, May 04, 2020

Attendees - Rusty Davis (rstyd), Qiang Guan (guanxyz), Al McPherson (mcpherson), Pat Grubel (pagrubel), Tim Randles (trandles-lanl), Jake Tronge (jtronge)

PR Review

  1. 152(pagrubel) - Image extension fix 146
    • APPROVED
  2. 154(pagrubel) - Parser passes all step hints to the Workflow-153.

Issue Review

  1. 156(guanxyz) - Interface Design and Spec (TM to Resource Manager)
  2. 155(rstyd) - Come up with an id scheme for the WFM

Discussion (ToDo?)

Around the room

  • Rusty
    • pytest
    • had trouble accessing cypher-shell in neo4j
    • doco on failure possibilies WfM-TM
    • WSGI stuff
  • Qiang
    • interface and PySlurm
  • Al
    • parsing difficulties
    • scatter/gather is going to be difficult
      • file globbing in CWL is complicated
      • can't know apriori
    • work with Steven to combine parsing and database redesign
    • maybe more complex communication between WfM/TM
  • Pat
    • looking at CWL requirements
    • abstracting out container runtimes and workload managers
  • Jake
    • added logdir logging ability to BEEStart
    • work with Rusty to talk about how logging works with WSGI
Clone this wiki locally