Data Science for Dynamical Systems Course Materials

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Course Content

You can find the video lectures on YouTube.

The associated Julia code (in the form of Jupyter Notebooks) can be found in the "notebooks" folder. In case a certain package is missing in your Julia version, you can simply add a cell before the first one containing the two code lines
using Pkg
Pkg.add("name_of_package")

References to related literature can be found at the end of this Readme file.

Section 00: Course introduction

• Data Science for Dynamical Systems Course Introduction 1 (DS4DS 0.01)
• Data Science for Dynamical Systems Course Introduction 2 (DS4DS 0.02)

Section 01: Dynamical Modeling Fundamentals

• Week 1:

  • Solution of scalar linear autonomous ODEs (DS4DS 1.01)
  • Solution of vectorial linear autonomous ODEs (DS4DS 1.02)
  • Matrix exponential solution via diagonalization (DS4DS 1.03)
  • Spring-mass-damper as an autonomous state-space system example (DS4DS 1.04)
  • State stability of autonomous state-space systems (DS4DS 1.05)
  • Nonlinear ODEs (1.06)

• Week 2:

  • Numerical solutions of ODEs 1 (DS4DS 1.07)
  • Numerical solutions of ODEs 2 (DS4DS 1.08)
  • Numerical solutions of ODEs 3 (DS4DS 1.09)
  • State-space models with inputs (DS4DS 1.10)
  • Exact solution of linear state-space models (DS4DS 1.11)
  • Observability 1 (DS4DS 1.12)
  • Observability 2 (DS4DS 1.13)
  • Observability 3 (DS4DS 1.14)
  • Discrete dynamical systems (DS4DS 1.15)

• Week 3:

  • Partial differential equations: Introduction (DS4DS 1.16)
  • Partial differential equations: Derivation of the heat equation (DS4DS 1.17)
  • Partial differential equations: Spatial discretization (DS4DS 1.18)
  • Partial differential equations: Discretization in space and time (DS4DS 1.19)

Section 02: Linear Model Identification

• Week 4:

  • Linear model identification: intro part 1/2 (DS4DS 2.01)
  • Linear model identification: intro part 2/2 (DS4DS 2.02)
  • Linear least squares solution (DS4DS 2.03)
  • Ordinary least squares: static example (DS4DS 2.04)
  • Ordinary least squares: bias (DS4DS 2.05)
  • Ordinary least squares: covariance (DS4DS 2.06)
  • Ordinary least squares: consistency (DS4DS 2.07)

• Week 5:

  • Multicollinearity (DS4DS 2.08)
  • Ridge regression (DS4DS 2.09)
  • Weighted least squares: motivation & solution (DS4DS 2.10)
  • Weighted least squares: example (DS4DS 2.11)
  • Recursive least squares: derivation part 1/2 (DS4DS 2.12)
  • Recursive least squares: derivation part 2/2 (DS4DS 2.13)
  • Recursive least squares: exponential forgetting (DS4DS 2.14)
  • Recursive least squares: example (DS4DS 2.15)

• Week 6:

  • Ordinary Least Squares for Dynamical Systems: Introduction (DS4DS 2.16)
  • Ordinary Least Squares for Dynamical Systems: Example (DS4DS 2.17)
  • Ordinary Least Squares for Dynamical Systems: Limitations (DS4DS 2.18)
  • Singular Value Decomposition: Motivation (DS4DS 2.19)
  • Singular Value Decomposition: Interpretation (DS4DS 2.20)
  • Eigendecomposition of continuous and discrete-time dynamical systems (DS4DS 2.21)
  • Dynamic Mode Decomposition: Concept (DS4DS 2.22)
  • Dynamic Mode Decomposition: Efficient computation (DS4DS 2.23)

Section 03: Optimization for Machine Learning

• Week 7:

  • Training nonlinear models (DS4DS 3.01)
  • The hypothesis space (DS4DS 3.02)
  • Gradient descent 1 (DS4DS 3.03)
  • Gradient descent 2 (DS4DS 3.04)
  • Gradient descent 3 (DS4DS 3.05)
  • Gradient descent 4 (DS4DS 3.06)
  • The Armijo and Wolfe conditions (DS4DS 3.07)
  • Stochastic gradient descent (DS4DS 3.08)

• Week 8:

  • Momentum (DS4DS 3.09)
  • Convergence rates (DS4DS 3.10)
  • Newton's method (DS4DS 3.11)
  • Differentiation Techniques: Overview (DS4DS 3.12)
  • Symbolic Differentiation (DS4DS 3.13)
  • Algorithmic Differentiation: Forward mode (DS4DS 3.14)
  • Algorithmic Differentiation: Backward mode (DS4DS 3.15)

• Week 9:

  • Constrained optimization 1 (DS4DS 3.16)
  • Constrained optimization 2 (DS4DS 3.17)
  • Constrained optimization 3 (DS4DS 3.18)
  • Penalty methods and Sequential Quadratic Programming (DS4DS 3.19)
  • The active set method 1 (DS4DS 3.20)
  • The active set method 2 (DS4DS 3.21)

Section 04: Nonlinear Model Identification

• Week 10:

  • Optimization-based parameter identification (DS4DS 4.01)
  • Impact of the prediction horizon in data-driven modeling (DS4DS 4.02)
  • Nonlinear prediction error method identification (DS4DS 4.03)
  • Global vs. local optima in nonlinear data-driven modeling (DS4DS 4.04)
  • Identification of an unknown initial state (DS4DS 4.05)
  • Utilizing constrained optimization for incorporating a priori knowledge (DS4DS 4.06)

• Week 11:

  • Neural ordinary differential equations - NODEs (DS4DS 4.07)
  • Sampled input data for NODE modeling (DS4DS 4.08)
  • Hammerstein-Wiener models in NODEs (DS4DS 4.09)
  • Extrapolation of data-driven dynamical models (DS4DS 4.10)
  • Discrete-time nonlinear system identification (DS4DS 4.11)

Section 05: Feature Engineering

• Week 12:
  • Feature Engineering - Introduction (DS4DS 5.01)
  • Manual feature selection using dictionaries (DS4DS 5.02)
  • Extracting linear features from data (DS4DS 5.03)
  • Feature engineering using autoencoders (DS4DS 5.04)
  • The role of features in deep learning (DS4DS 5.05)

Section 06: Model Selection

• Week 13:
  • Overfitting and the bias-variance tradeoff (DS4DS 6.01)
  • Cross validation (DS4DS 6.02)
  • Sparsity and the L1 norm (DS4DS 6.03)
  • Subdifferentials of non-differentiable functions (DS4DS 6.04)
  • The LASSO algorithm for model selection (DS4DS 6.05)
  • Sparse identification of nonlinear dynamics (SINDy) (DS4DS 6.06)

Section 07: Control

• Week 14:
  • Optimal control - Introduction (DS4DS 7.01)
  • Optimal control in discrete time (DS4DS 7.02)
  • Optimal control of linear systems 1 - Dynamics (DS4DS 7.03)
  • Optimal control of linear systems 2 - Objective function (DS4DS 7.04)
  • Optimal control of linear systems 3 - Example (DS4DS 7.05)
  • Linear model predictive control (DS4DS 7.06)
  • Data-driven model predictive control using DMD (DS4DS 7.07)
  • Differential predictive control: introduction (DS4DS 7.08)
  • Differential predictive control: global vs. local optima (DS4DS 7.09)

Section 08: Koopman operator

• An introduction to the Koopman Operator (DS4DS 8.01)
• Koopman operator: Spectral decomposition 1 (DS4DS 8.02)
• Koopman operator: Spectral decomposition 2 (DS4DS 8.03)
• Koopman: Observable functions (DS4DS 8.04)
• Extended Dynamic Mode Decomposition 1 - Finite dimensional subspaces (DS4DS 8.05)
• Extended Dynamic Mode Decomposition 2 - The EDMD algorithm (DS4DS 8.06)
• Extended Dynamic Mode Decomposition 3 - Koopman eigenfunctions (DS4DS 8.07)
• Extended Dynamic Mode Decomposition 4 - Koopman modes & Summary (DS4DS 8.08)
• Dynamic Mode Decomposition as a special case of EDMD (DS4DS 8.09)
• kernel EDMD 1 - Motivation & kernel trick (DS4DS 8.10)
• kernel EDMD 2 - Main algorithm (DS4DS 8.11)
• kernel EDMD 3 - Koopman eigenfunctions and modes (DS4DS 8.12)
• The Koopman Generator (DS4DS 8.13)
• Extended DMD for the Koopman Generator (DS4DS 8.14)

Related literature

• Data science and machine learning:

  • Y. S. Abu-Mostafa, M. Magdon-Ismail and H.-T. Lin. "Learning from data: A short course." AMLBook, 2012. (URL)
  • S. L. Brunton and J. N. Kutz. "Data-Driven Science and Engineering." Cambridge University Press, 2019. (DOI)
  • C. M. Bishop. "Pattern Recognition and Machine Learning." Springer, 2007. (URL)

• Dynamical systems and system identification:

  • R. Isermann and M. Münchhof. "Identification of Dynamic Systems." Springer, 2011. (DOI)
  • O. Nelles. "Nonlinear System Identification." Springer, 2001. (DOI)

• Optimization

  • G. Nocedal and S. J. Wright. "Numerical Optimization." (DOI)
  • The book by Nelles also has an extensive introduction to optimization.