💡 [Feature] How to launch dask workers on a second machine from a Jupyter environment

[huard]

Description

At Ouranos we have a second server that could provide additional CPUs for large computations. The idea would be to be able to create a dask client in a notebook, and have an option to create dask workers on the second machine.

Use case:

• User opens Jupyter notebook
• User launches Scheduler and n workers on 2nd machine
• User creates client connected to Scheduler
• User launches job using client

Questions

If jobs are IO bounds, the network between the two machines might become the bottleneck.
This could eventually be improved by the installation of a 10GB network, or a private 1GB connection between the two machines.

References

Dask cluster deployment documentation: https://docs.dask.org/en/stable/deploying.html

I suspect the workers need to live in the same conda environment as the notebook. This might mean running a second docker container on the 2nd machine, and creating a scheduler and a bunch of workers.
Docker deployment: https://docs.dask.org/en/stable/deploying-docker.html

SSH deployment: https://docs.dask.org/en/stable/deploying-ssh.html

Concerned Organizations

@tlvu (Ouranos)