Homework problems and educational outcomes

[sgbaird]

This is a place to discuss potential extensions of the demo to classroom settings, such as homework problems, teaching outcomes, and general recommendations related to its use.

[sgbaird]

Homework Problems

Other than "Getting Started", these are meant to be problems that have not been done before (at least not exactly), rather than limiting it to a set of problems that already have examples or solutions and trying to "hide" the solutions from the students. Note that some of these may be implemented and made public within this repository eventually, so this is meant to be a living list of ideas. The plan is to add strikethrough text and a link to the relevant resource once an implementation is made public.

Getting Started

• Have each student or team of students set up the demo and run the intro tutorial to completion with their own device.
• Assign one random notebook (other than the intro tutorial, 4.2) to each team of students to work through on their own. Have them present a summary or walkthrough of the assigned notebook to the class.
• Read the reaching critical MASS manuscript and have a class discussion about it.

Basic Optimization

• Create a random forest or gradient-boosted trees model and pair it with an acquisition function that always chooses the max predicted value. Using the same random forest or gradient-boosted trees model, use it with the expected improvement (EI) acquisition function. Compare this with the default Bayesian optimization model implemented in self-driving-lab-demo. Here are some resources that may be of help:
  • https://modal-python.readthedocs.io/en/latest/content/examples/bayesian_optimization.html
  • https://www.kaggle.com/code/neerajmohan/randomforest-model-with-bayesian-optimization
  • https://machinelearningmastery.com/what-is-bayesian-optimization/
• Compare the optimization efficiency for Ax (default implementation) with at least two other Bayesian optimization packages. For example:
  • Dragonfly
  • scikit-optimize
  • bayesian-optimization
• Compare the efficiency of the default BO model with a linear regression model (what to use as the acquisition function since no uncertainty measurements?).

Advanced Optimization

• Compare the efficiency of multi-fidelity optimization vs. evaluating only at the highest fidelity vs. evaluating only at the lowest fidelity via a plot of "best observed so far" vs. "total cost so far".
• Compare the efficiency of high-dimensional optimization via SAASBO vs. standard optimization via Gaussian process expected improvement (GPEI) within the Ax platform as a function of increasing dimensionality (e.g., 1 to 20 dimensions). Use the crabnet-hyperparameter benchmark function within matsci-opt-benchmarks as the objective function.
• Plot the Pareto front results of a two-objective optimization via parallel noisy expected hypervolume improvement (qNEHVI) vs. the Pareto front results using the sum of the two objectives as a scalarized objective (i.e., converted to a single-objective problem).
• Use the parallel negative integrated posterior variance (qNIPV) acquisition function to adaptively sample points in a way that reduces model error rather than minimizing/maximizing an objective function.
• Get every demo in the classroom online and cloud-accessible, and perform asynchronous or batch optimization using all available demos (i.e., a network of cloud-based experiments).

Design Principles

• Divide students into human-in-the-loop, human-centric design, and hardware-centric design groups. Have students discuss the trade-offs in approaches, present their findings, and learn from other groups. How many experiments was each group able to do within a day? After a week?

Data Logging

• Create a free MySQL database and use it as the database backend for storing experiments via Ax. Ax example with SQLAlchemy #159

Extensions and Design

• Use either Bluesky or alab_management as the experiment orchestration package for the demo. Experimental workflow orchestration (exploring Bluesky) #64
• Create a notification system that monitors "resources" and alerts the user when resources are running low (i.e., pretend like there is a limited number of LED flashes before it needs to be "refilled"). Devices that monitor resources (flowmeters, liquid level gauges, mass balances) #172
• Open-ended: allow students to come up with their own ideas and designs for extending the demo to either new teaching examples or real research tasks Roadmap for demo extensions #77
• After finishing the "Getting Started" tasks, ask teams of students to create a proposal for a self-driving lab (essentially the AC's call for white papers)

Kid-friendly Demos

• Match the color of fruit (e.g., banana, orange, grape, strawberry, kiwi, blueberry)

[sgbaird]

Course Outcomes

• Familiarize yourself with the concept of self-driving labs (hardware execution, algorithm planning, experiment orchestration)
• Identify best practices related to self-driving labs (e.g., modular, reconfigurable, reproducible, user-friendly, safe, documented)
• Get hands-on experience with self-driving labs by setting up a toy demo
• Encourage innovative ideas for incorporating autonomy or automation into real-world research tasks
• Learn the principles and best practices of "cloud experimentation" (i.e., allowing users to be geographically distant from the hardware)
• Learn best practices for tracking and storing experimental data (e.g., machine-readable, persistent, flexible, scalable)
• Understand the efficiency benefits of Bayesian optimization over standard DoE methods such as grid search, random search, and quasi-random sampling
• Gain an appreciation for and experience with the complexity/efficiency trade-offs for advanced optimization methods (e.g., multi-objective vs. scalarized objective, multi-fidelity vs. single-fidelity)