@nmfs-openscapes/2024-science-teams

Reminder! Our HackHour guest speaker today is Rich Signell. He has extensive experience with enabling scalable big data workflows and high performance computing (for oceanography) in the cloud. Come with your questions!

11am PT/2pm ET
Google Meet joining info
Video call link: https://meet.google.com/dur-omus-uvz

Intro to Dask for Parallel Computing (in Python). On Friday, our guest speaker Rich Signell showed us how to start a Dask cluster on Coiled. For those of you who are new to Dask, I have added 3 simple examples that show Dask local, Gateway and Coiled clusters. Hopefully this will help you understand and get started with using Dask clusters. Especially if you are doing heavy computing, Dask will simplify creating distributed computations (meaning lots of computers working for you) if your tasks can be parallelized.

https://nmfs-opensci.github.io/NMFSHackDays-2025/topics-2025/2025-dask/

You learn about local Dask clusters to run tasks on all your CPUs versus Dask Gateway and Coiled clusters. The nmfs-openscapes Jupyter Hub is set-up with Dask Gateway to allow Gateway; it is a common but not a default feature of Jupyter Hubs. Coiled will require a Coiled account and your own AWS, GCP, Azure or other cloud account where your new pods (virtual machines) will be spun up. BTW for those using xarray to work with rasters and data cubes,xarray uses dask under the hood to use all the CPUs in your Jupyter Hub pod; so it basically does a local cluster automatically for you.