thêm chức năng train trên odc predict trên planetary
This commit is contained in:
Victor Phan
@@ -0,0 +1,696 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "416ee5f1-2a79-4952-a901-0d1f027f468d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Introduction to Dask and the Open Data Cube <img align=\"right\" src=\"../../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"**Prerequisites**: This material assumes basic knowledge of the Open Data Cube, Xarray and numerical processing using numpy.\n",
|
||||
"\n",
|
||||
"- [Introduction](#Introduction-to-Dask-and-the-Open-Data-Cube)\n",
|
||||
" - [Planning and writing efficient applications](#Planning-and-writing-efficient-applications)\n",
|
||||
" - [What you will learn](#What-you-will-learn)\n",
|
||||
"- [Performance in Python](#Performance-in-Python)\n",
|
||||
" - [Python list and numpy](#Python-list-and-numpy)\n",
|
||||
" - [Numba - accelerating Python](#Numba---accelerating-Python)\n",
|
||||
"- [Parallelism with Dask](#Parallelism-with-Dask)\n",
|
||||
" - [Review of dask](#Review-of-dask)\n",
|
||||
" - [Dask local cluster](#Dask-local-cluster)\n",
|
||||
"\n",
|
||||
"The Open Data Cube library is written in Python and makes extensive use of scientific and geospatial libraries. For the purposes of this tutorial we will primarily consider five libraries:\n",
|
||||
"\n",
|
||||
" 1. `datacube` - EO datacube\n",
|
||||
" 1. `xarray` - labelled arrays\n",
|
||||
" 1. (optional) `dask` & `distributed` - distributed parallel programming\n",
|
||||
" 1. `numpy` - numerical array processing with vectorisation\n",
|
||||
" 1. (optional) `numba` - a library for high performance python\n",
|
||||
"\n",
|
||||
"Whilst the interrelations are intimate it is useful to conceptualise them according to their primary role and how these roles build from low level numerical array processing (`numpy`) through to high-level EO datacube semantics (`datacube` and `xarray`). If you prefer, viewed from top to bottom we can say:\n",
|
||||
" 1. `datacube.load()` does the necessary file IO and data manipulation to construct a...\n",
|
||||
" 1. `xarray` which will be labelled with the necessary coordinate systems and band names and made up of...\n",
|
||||
" 1. (optionally) `dask.array`s which contain many `chunks` which are...\n",
|
||||
" 1. `numpy` arrays containing the actual data values.\n",
|
||||
"\n",
|
||||
"Each higher level of abstraction thus builds on the lower level components that perform the actual storage and computation.\n",
|
||||
"\n",
|
||||
"Overlaid on this are libraries like `numba`, `dask` and `distributed` that provide computational components that can accelerate and distribute processing across multiple compute cores and computers. The use of `dask`, `distributed` and `numba` are optional - not all applications require the additional complexity of these tools.\n",
|
||||
"\n",
|
||||
"### Planning and writing efficient applications\n",
|
||||
"\n",
|
||||
"Achieving performance and scale requires an understanding of the performance of each library and how it interacts with the others. Moreover, and often counterintuitively, adding more compute cores to a problem may not make it faster; in fact it may slow down (as well as waste resources). Added to that is the _deceptive simplicity_ in that some of the tools can be simply _turned on_ with only a few code changes and significant performance increases can be achieved.\n",
|
||||
"\n",
|
||||
"However, as the application is scaled or an alternative algorithm is used further challenges may arise, in expected I/O or compute efficiency, that require code refactors and changes in algorithmic approach. These challenges can seem to undo some of the earlier work and be frustrating to address. \n",
|
||||
"\n",
|
||||
"The good news is whilst there is complexity (six interrelated libraries mentioned so far), there are common concepts and techniques involved in analysing _how to optimise your algorithm_. If you know from the start your application is going to require scale, then it does help to think in advance where you are heading.\n",
|
||||
"\n",
|
||||
"### What you will learn\n",
|
||||
"\n",
|
||||
"This course will equip readers with concepts and techniques they can utilise in their algorithm and workflow development. The course will be using computer science terms and a variety of libraries but won't be discussing these in detail in order to keep this course concise. The focus will be on demonstration by example and analysis techniques to identify where to focus effort. The reader is encouraged to use their favorite search engine to dig deeper when needed; there are a lot of tutorials online!\n",
|
||||
"\n",
|
||||
"One last thing, in order to maintain a healthy state of mind for \"Dask and ODC\", the reader is encouraged to hold both of these truths in mind at the same time:\n",
|
||||
" 1. The *best* thing about dask is it makes distributed parallel programming in the datacube easy\n",
|
||||
" 1. The *worst* thing about dask it is makes distributed parallel programming in the datacube easy\n",
|
||||
"\n",
|
||||
"Yep, that's contradictory! By the end of this course, and a couple of your own adventures, you will understand why."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f3fb9221-d423-4a3e-baa4-6cdaefd56c1d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# EASI defaults\n",
|
||||
"import git\n",
|
||||
"import sys\n",
|
||||
"import os\n",
|
||||
"os.environ['USE_PYGEOS'] = '0'\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults, notebook_utils"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "78b8349c-6698-45d2-a159-8013222232d1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"easi = EasiDefaults()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5be4a71f-d1f0-4c2a-a0cb-f53757d78cb1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Performance in Python\n",
|
||||
"\n",
|
||||
"In this section we will explore python performance for array processing. Python itself, as you will soon see, is quite slow. It is, however, highly expressive and can orchestrate more complex and faster libraries of numerical code (e.g., `numpy`). Python is also ammendable to being accelerated (e.g. using `numba`) and made to run on multiple CPU cores (e.g. via `dask`). \n",
|
||||
"\n",
|
||||
"### Python `list` and `numpy`\n",
|
||||
"\n",
|
||||
"Let's take a look at the simple addition of two arrays. In Python the nearest data type to an array is a `list` of numbers. This will be our starting point.\n",
|
||||
"\n",
|
||||
"Our focus is on performance so we'll use the Jupyter `%%time` and `%%timeit` magics to run our cells and time their execution. The latter will run the cell multiple times and provide us with more representative statistics of performance and variability.\n",
|
||||
"\n",
|
||||
"First in pure Python using lists :"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f2dfa3fd-0cc7-4804-8790-ed65f05eaa4e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"size_of_vec = 2000*2000\n",
|
||||
"X_list = range(size_of_vec)\n",
|
||||
"Y_list = range(size_of_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d8c79652-2a47-40f4-87a2-8e90ce9cdf9c",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit -n 10 -r 1\n",
|
||||
"Z = [X_list[i] + Y_list[i] for i in range(len(X_list)) ]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9aa9586d-b6ab-49d6-8914-ff016023cae8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now the same processing using `numpy`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5158c9d0-d8a9-48e6-ab1f-df4948dde092",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy\n",
|
||||
"X = numpy.arange(size_of_vec)\n",
|
||||
"Y = numpy.arange(size_of_vec)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a4dc338b-d411-40be-8aa0-4b50bce2201a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit -n 10 -r 1\n",
|
||||
"Z = X + Y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c8e8fb26-0d58-4dc1-b8ea-6c5ad044e4ea",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's check that the two arrays are identical (note that %%timeit does not make the variables above available due to the way that %%timeit works, so we reconstruct the arrays)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "476b9020-ba06-47d4-84f7-f6ede912ce88",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Are the arrays identical?\")\n",
|
||||
"([X_list[i] + Y_list[i] for i in range(len(X_list)) ] == X + Y).all()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7ff1825c-9bf8-4ab8-baa9-13b43f4f4a3f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"At least two orders of magnitude in performance improvement!\n",
|
||||
"\n",
|
||||
"Why?\n",
|
||||
"\n",
|
||||
"`numpy` provides a python interface to an underlying C array library that makes use of CPU `vectorization` - this allows it to process several add operations at the same time.\n",
|
||||
"\n",
|
||||
"`numpy` isn't the only library that does this type of wrapping over a fast optimised library. There are, for example,\n",
|
||||
"- `cuPy` which uses GPUs for array processing\n",
|
||||
"- `tensorflow` uses both CPU and GPU optimisations for machine learning\n",
|
||||
"- `datashader` for large dataset visualisation\n",
|
||||
"\n",
|
||||
"It's a very long list and thanks to a great deal of work by a great many software engineers most of these libraries will work together efficiently. \n",
|
||||
"\n",
|
||||
"> __Tip__: Where possible use high performance libraries that have python wrappers.\n",
|
||||
"\n",
|
||||
"The reader will have noticed the change in abstraction. The pure Python version used list comprehension syntax to add the two arrays, while `numpy` was a much shorter direct addition syntax more in keeping with the mathematics involved. This change in abstraction is seen in most libraries, including the ODC library where `datacube.load()` is shorthand for a complex process of data discovery, reprojection, fusing and array construction. High-level abstractions like this are powerful and greatly simplify development (the good). They can also hide performance bottlenecks and challenges (the bad).\n",
|
||||
"\n",
|
||||
"> __Tip__: Use high level API abstractions but be mindful of their use."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e01d98d-0090-46a7-adee-52534ee49f74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### `Numba` - accelerating Python\n",
|
||||
"\n",
|
||||
"So high performance libraries rock, but what if you don't have one for your purpose and you're back in Python?\n",
|
||||
"`Numba` translates Python functions into optimized machine code at runtime - https://numba.pydata.org.\n",
|
||||
"\n",
|
||||
"Let's see how this works. A more complex example this time with a smoothing function applied over our (random) image, perform an FFT, and save the result.\n",
|
||||
"These examples are (very) slightly modified versions from the [High Performance Python Processing Pipeline video by Matthew Rocklin](https://youtu.be/wANQkgDuTAk). It's such a good introduction its worth repeating.\n",
|
||||
"\n",
|
||||
"We'll also use the `tqdm` library to provide a progress bar."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "30019636-0446-4f0a-85ec-d93139065f74",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"from tqdm.notebook import tqdm\n",
|
||||
"\n",
|
||||
"def load_eo_data():\n",
|
||||
" return np.random.random((1000, 1000))\n",
|
||||
"\n",
|
||||
"def smooth(x):\n",
|
||||
" out = np.empty_like(x)\n",
|
||||
" for i in range(1, x.shape[0] - 1):\n",
|
||||
" for j in range(1, x.shape[1] - 1):\n",
|
||||
" out[i, j] = (x[i + -1, j + -1] + x[i + -1, j + 0] + x[i + -1, j + 1] +\n",
|
||||
" x[i + 0, j + -1] + x[i + 0, j + 0] + x[i + 0, j + 1] +\n",
|
||||
" x[i + 1, j + -1] + x[i + 1, j + 0] + x[i + 1, j + 1]) // 9\n",
|
||||
" return out\n",
|
||||
"\n",
|
||||
"def save(x, filename):\n",
|
||||
" pass "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f83fbe3b-5138-472e-a6f2-7e34f71804bb",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"for i in tqdm(range(5)):\n",
|
||||
" img = load_eo_data()\n",
|
||||
" img = smooth(img)\n",
|
||||
" img = np.fft.fft2(img)\n",
|
||||
" save(img, \"file-\" + str(i) + \"-.dat\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6cc2f3e8-c9e5-4cf2-bb66-bb27c3a7211b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The `smooth(x)` function contains two python loops. Now we could (and would) find a similar high performance library with a `smooth(x)` function but for this example let's use `numba`'s `jit` compiler to translate the python function into optimized machine code at runtime."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "18808bb2-ca62-425c-9e08-f876ceeaaff2",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numba\n",
|
||||
"\n",
|
||||
"fast_smooth = numba.jit(smooth)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "14d8a5e0-30f9-4676-a436-e5643c92dddf",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"\n",
|
||||
"for i in tqdm(range(5)):\n",
|
||||
" img = load_eo_data()\n",
|
||||
" img = fast_smooth(img)\n",
|
||||
" img = np.fft.fft2(img)\n",
|
||||
" save(img, \"file-\" + str(i) + \"-.dat\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "82412220-68b4-4905-b6f0-2b53b71c55f5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Just a bit quicker! Much of the time in the first run was `numba` performing compilation. Run the cell above again and you'll find it runs faster the second time.\n",
|
||||
"\n",
|
||||
"The _recommended_ approach to have `numba` compile a python function is to use python decorator syntax (`@numba.jit`). So the original code now looks like this (single line changed) and we can call `smooth(x)` without having to create `fast_smooth`:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "820dd545-bae8-477b-9e1e-73c806b953bd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"def load_eo_data():\n",
|
||||
" return np.random.random((1000, 1000))\n",
|
||||
"\n",
|
||||
"@numba.jit\n",
|
||||
"def smooth(x):\n",
|
||||
" out = np.empty_like(x)\n",
|
||||
" for i in range(1, x.shape[0] - 1):\n",
|
||||
" for j in range(1, x.shape[1] - 1):\n",
|
||||
" out[i, j] = (x[i + -1, j + -1] + x[i + -1, j + 0] + x[i + -1, j + 1] +\n",
|
||||
" x[i + 0, j + -1] + x[i + 0, j + 0] + x[i + 0, j + 1] +\n",
|
||||
" x[i + 1, j + -1] + x[i + 1, j + 0] + x[i + 1, j + 1]) // 9\n",
|
||||
" return out\n",
|
||||
"\n",
|
||||
"def save(x, filename):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "59f9604e-e0d6-4ac0-bd54-b93f6952131e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"for i in tqdm(range(5)):\n",
|
||||
" img = load_eo_data()\n",
|
||||
" img = smooth(img)\n",
|
||||
" img = np.fft.fft2(img)\n",
|
||||
" save(img, \"file-\" + str(i) + \"-.dat\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4598e9d1-e154-4309-9bc0-0d5252775975",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Why not use `numba` all the time everywhere?\n",
|
||||
"\n",
|
||||
"Like most high level abstractions `numba` makes assumption about code, only accelerates a subset of python libraries (not all `numpy` functions are available via `numba`), and it is entirely possible it can make performance worse or not work at all!\n",
|
||||
"\n",
|
||||
"There's one additional consideration. If you've run all the cells to this point in order, try running the `fast_smooth` cell again, repeated below for convenience:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "01a91155-67a3-4ab2-a2ff-712d0c43c71e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fast_smooth = numba.jit(smooth)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9675cb05-6794-4e9f-8daa-099846dfd3d0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Error!\n",
|
||||
"\n",
|
||||
"The `smooth` function was decorated so is already `jit`-compiled. Attempting to do so again causes this error, and exposes some of the low level changes behind the abstraction.\n",
|
||||
"This can make debugging code difficult if you are not mindful of what is occuring.\n",
|
||||
"\n",
|
||||
"TIP: __Use high level API abstractions but be mindful of their use__"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "25419e83-3fce-49be-b1b4-54f6415d7096",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Parallelism with Dask\n",
|
||||
"\n",
|
||||
"Our fake EO processing pipeline only has 5 images and takes about 1 sec to run. In practice we'll have 1000s of images to process (if not more).\n",
|
||||
"\n",
|
||||
"Let's repeat our example code but now with more iterations. You can understand why we use The `tqdm` library to provide a progress bar for these larger scale examples rather than printing out each iteration number or staring at a blank screen wondering if it works!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "af003cf6-3993-4718-a0d0-48c34f96c0e4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import numba\n",
|
||||
"from tqdm.notebook import tqdm\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def load_eo_data():\n",
|
||||
" return np.random.random((1000, 1000))\n",
|
||||
"\n",
|
||||
"@numba.jit\n",
|
||||
"def smooth(x):\n",
|
||||
" out = np.empty_like(x)\n",
|
||||
" for i in range(1, x.shape[0] - 1):\n",
|
||||
" for j in range(1, x.shape[1] - 1):\n",
|
||||
" out[i, j] = (x[i + -1, j + -1] + x[i + -1, j + 0] + x[i + -1, j + 1] +\n",
|
||||
" x[i + 0, j + -1] + x[i + 0, j + 0] + x[i + 0, j + 1] +\n",
|
||||
" x[i + 1, j + -1] + x[i + 1, j + 0] + x[i + 1, j + 1]) // 9\n",
|
||||
" return out\n",
|
||||
"\n",
|
||||
"def save(x, filename):\n",
|
||||
" pass"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a44bb7e4-7f70-40a1-84cc-9b2f63007378",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Before running the next code, open a terminal window (File>New>Terminal) and run `htop` at the command line to show current CPU usage per core.\n",
|
||||
"\n",
|
||||
"> **TIP:** Drag your terminal window so that it sits below this notebook before you run `htop` to see both windows at the same time."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "88d0fc98-eb4a-445e-b305-84aa493d07b6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"for i in tqdm(range(1000)):\n",
|
||||
" img = load_eo_data()\n",
|
||||
" img = smooth(img)\n",
|
||||
" img = np.fft.fft2(img)\n",
|
||||
" save(img, \"file-\" + str(i) + \"-.dat\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c2ef6e05-6466-4119-b33d-eaa64445ba24",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You'll notice that only one core is showing any load. The above code is not using any of the additional cores.\n",
|
||||
"\n",
|
||||
"`dask` can be useful in this scenario even on a local machine. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d7fcafa1-75cf-4856-987d-bf1154203a01",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Review of dask\n",
|
||||
"\n",
|
||||
"Firstly, a few notes on terminology. A Dask Cluster is comprised of a __client__, a __scheduler__, and __workers__. These terms will be used throughout this tutorial. Figure 1 below shows the relationship between each of these components. The __client__ submits tasks to the __scheduler__, which decides how to submit the tasks to individual workers. During this process, the scheduler creates what is called a __Task Graph__. This is essentially a map of the tasks that need to be carried out. Figure 2 shows an example of a simple task graph (see https://docs.dask.org/en/stable/graphs.html for more information). __Workers__ carry out the actual calculations and either store the results or send them back to the client.\n",
|
||||
"\n",
|
||||
"<div>\n",
|
||||
" <span style=\"border:solid 1px #888;float:left;padding:10px;margin-right:25px;width:550px\">\n",
|
||||
" <img src=\"../../resources/distributed-overview.png\">\n",
|
||||
" <figcaption><em>Figure 1. Overview of a Dask Cluster.</em></figcaption>\n",
|
||||
" </span>\n",
|
||||
" <span style=\"border:solid 1px #888;float:left;padding:10px;margin-left:25px;width:150px\">\n",
|
||||
" <img style=\"float:left\" src=\"../../resources/dask-simple.png\">\n",
|
||||
" <figcaption><em>Figure 2. A simple Task Graph.</em></figcaption>\n",
|
||||
" </span>\n",
|
||||
"</div>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d4cae0a-7557-440e-b6b0-54ffd968cced",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Dask has several core data types, including __Dask DataFrames__ and __Dask Arrays__. Essentially, Dask DataFrames are parallelized Pandas DataFrames (Figure 3) and Dask Arrays are parallelized Numpy arrays (Figure 4).\n",
|
||||
"\n",
|
||||
"<div style=\"width:100%\">\n",
|
||||
" <span style=\"border:solid 1px #888;float:left;padding:10px;margin-right:25px;width:200px\">\n",
|
||||
" <img src=\"../../resources/dask-dataframe.svg\">\n",
|
||||
" <figcaption><em>Figure 3. A Dask DataFrame is comprised of many in-memory pandas DataFrames separated along an index.</em></figcaption>\n",
|
||||
" </span>\n",
|
||||
" <span style=\"border:solid 1px #888;float:left;padding:10px;margin-left:25px;width:300px\">\n",
|
||||
" <img style=\"float:left\" src=\"../../resources/dask-array.svg\">\n",
|
||||
" <figcaption><em>Figure 4. A Dask Array is a subset of the NumPy <code>ndarray</code> interface using blocked algorithms, cutting up the large array into many small arrays.</em></figcaption>\n",
|
||||
" </span>\n",
|
||||
"</div>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d3f64100-b5cc-48a8-9be9-71f31e50dd35",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"EASI and the Open Data Cube primarily make use of __Dask DataArrays__. For more information see https://tutorial.dask.org/02_array.html."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "823a9c12-1c03-4ea8-8a9e-ec498bdbc056",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Dask local cluster\n",
|
||||
"\n",
|
||||
"Let's start by creating a local dask cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e0d95f91-a9e0-44c1-a577-f8e708ccbce6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from dask.distributed import Client, LocalCluster, fire_and_forget\n",
|
||||
"\n",
|
||||
"cluster = LocalCluster()\n",
|
||||
"client = Client(cluster)\n",
|
||||
"client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e96378b8-ec25-4970-9957-8c482112073a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There is also a utility function in `notebook_utils` which you can use to initialize a local dask cluster. This funtion can also be used to initialize a remote cluster using Dask Gateway, but please complete the other Dask tutorials before using a remote cluster.\n",
|
||||
"\n",
|
||||
"The following lines will initialize and display a local dask cluster and can replace the code above.\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"cluster, client = notebook_utils.initialize_dask(use_gateway=False, wait=True)\n",
|
||||
"display(cluster if cluster else client)\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "922c7673-88cf-445c-90c7-ea4c6c2124d9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The Dask Dashboard url will show as \"localhost\" or \"127.0.0.1\" since its running locally in the Jupyter kernel. The dashboard can be accessed via the Jupyter server proxy using the following url:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "31a07d9e-3b83-4081-9db2-cd70ac3853d9",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"notebook_utils.localcluster_dashboard(client=client, server=easi.hub)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bd3c526a-c04f-40ac-b2dc-46701f9ec739",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You will want to have this open when running the next cell.\n",
|
||||
"\n",
|
||||
"We modify our application to `submit` functions to run on the dask cluster:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c8b09db6-aea0-4877-a0e8-4cc2497c8936",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"\n",
|
||||
"for i in tqdm(range(1000)):\n",
|
||||
" img = client.submit(load_eo_data, pure=False)\n",
|
||||
" img = client.submit(smooth, img)\n",
|
||||
" img = client.submit(np.fft.fft2, img)\n",
|
||||
" future = client.submit(save, img, \"file-\" + str(i) + \"-.dat\")\n",
|
||||
" fire_and_forget(future)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "032d7d77-3e37-4fb5-bd9d-fa7f1c6db1ff",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If you watch `htop` in the terminal you'll see all cores become active. The dask dashboard will also provide a view of the tasks being run in parallel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ef6c1656-f68b-4192-a88d-21a8d0b78c7e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You will also see that the cluster remains busy after the cell above finishes. This is because Dask is working in the background processing in parallel the tasks that have been submitted to it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bd40a084-1721-4d10-9400-e2bd563f5823",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"A `dask.distributed.LocalCluster()` will shutdown when this notebook kernel is stopped.\n",
|
||||
"Still it's a good practice to close the client and the cluster so its all cleaned up. This will be more important when using dask distributed clusters as they are independent of the notebook kernel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "70edbc54-df60-4f33-8b56-9a10a8bb3fa7",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.close()\n",
|
||||
"\n",
|
||||
"cluster.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "8c6e908d-1467-4246-bf12-e1a2cfd07be2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,570 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "54240a84-8659-4d34-af38-1249129a221a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Dask Local Cluster - Larger than memory computation <img align=\"right\" src=\"../../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"In the ODC and Dask (LocalCluster) notebook we saw how dask can be used to speed up IO and computation by parallelising operations into _chunks_ and _tasks_, and using _delayed tasks_ and _task graph_ optimization to remove redundant tasks when results are not used.\n",
|
||||
"\n",
|
||||
"Using _chunks_ provides one additional capability beyond parallelisation - _the ability to perform computations that are larger than available memory_.\n",
|
||||
"\n",
|
||||
"Since dask operations are performed on _chunks_ it is possible for dask to perform operations on smaller pieces that each fit into memory. This is particularly useful if you have a large amount of data that is being reduced, say by performing a seasonal mean.\n",
|
||||
"\n",
|
||||
"As with parallelisation, not all algorithms are amenable to being broken into smaller pieces so this won't always be possible. Dask arrays though go a long way to make this easier for a great many operations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8d810c6f-46aa-48b0-a7a9-82151b29bb96",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Firstly, some initial imports..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "10bf42b0-b251-4d68-9af0-cf9ab988c886",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import git\n",
|
||||
"import sys, os\n",
|
||||
"from dateutil.parser import parse\n",
|
||||
"from dateutil.relativedelta import relativedelta\n",
|
||||
"from dask.distributed import Client, LocalCluster\n",
|
||||
"import datacube\n",
|
||||
"from datacube.utils import masking\n",
|
||||
"from datacube.utils.aws import configure_s3_access\n",
|
||||
"\n",
|
||||
"# EASI defaults\n",
|
||||
"os.environ['USE_PYGEOS'] = '0'\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults, notebook_utils\n",
|
||||
"easi = EasiDefaults()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c34ae885-c6bb-4d90-8769-e9a41d8b92cb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We'll continue using the same algorithm as before but this time we're going to modify it's memory usage to exceed the LocalCluster's available memory. This example notebook is setup to run on a compute node with 28 GiB of available memory and 8 cores for the LocalCluster. We'll make that explicit here in case you are blessed with a larger number of resources.\n",
|
||||
"\n",
|
||||
"Let's start the cluster..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2608053d-83db-45fe-9b09-9ea089edaf37",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cluster = LocalCluster(n_workers=2, threads_per_worker=4)\n",
|
||||
"cluster.scale(n=2, memory=\"14GiB\")\n",
|
||||
"client = Client(cluster)\n",
|
||||
"client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a69fa086-3587-4717-9466-8eb3ca6a7fcb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can monitor memory usage on the workers using the dask dashboard URL below and the Status tab. The workers are local so this will be memory on the same compute node that Jupyter is running in. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7303b0e2-abd8-475c-ae4c-14de9e7a3c3d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dashboard_address = notebook_utils.localcluster_dashboard(client=client,server=easi.hub)\n",
|
||||
"print(dashboard_address)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "25bbfd77-faf4-4834-99ef-7688911e02aa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As we will be using __Requester Pays__ buckets in AWS S3, we need to run the `configure_s3_access()` function below with the `client` option to ensure that Jupyter and the cluster have the correct permissions to be able to access the data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "bc032bf6-2105-4237-9121-0f557c7f65a6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dc = datacube.Datacube()\n",
|
||||
"configure_s3_access(aws_unsigned=False, requester_pays=True, client=client);"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "026408c9-f44c-4270-b7fa-325646f22d99",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get the centroid of the coordinates of the default extents\n",
|
||||
"central_lat = sum(easi.latitude)/2\n",
|
||||
"central_lon = sum(easi.longitude)/2\n",
|
||||
"# central_lat = -42.019\n",
|
||||
"# central_lon = 146.615\n",
|
||||
"\n",
|
||||
"# Set the buffer to load around the central coordinates\n",
|
||||
"# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions\n",
|
||||
"buffer = 0.05\n",
|
||||
"\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)\n",
|
||||
"\n",
|
||||
"# Data product\n",
|
||||
"products = easi.product('landsat')\n",
|
||||
"\n",
|
||||
"# Set the date range to load data over\n",
|
||||
"set_time = easi.time\n",
|
||||
"set_time = (set_time[0], parse(set_time[0]) + relativedelta(years=1))\n",
|
||||
"# set_time = (\"2021-01-01\", \"2021-12-31\")\n",
|
||||
"\n",
|
||||
"# Selected measurement names (used in this notebook). None` will load all of them\n",
|
||||
"alias = easi.aliases('landsat')\n",
|
||||
"measurements = None\n",
|
||||
"# measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]\n",
|
||||
"\n",
|
||||
"# Set the QA band name and mask values\n",
|
||||
"qa_band = alias['qa_band']\n",
|
||||
"qa_mask = easi.qa_mask('landsat')\n",
|
||||
"\n",
|
||||
"# Set the resampling method for the bands\n",
|
||||
"resampling = {qa_band: \"nearest\", \"*\": \"average\"}\n",
|
||||
"\n",
|
||||
"# Set the coordinate reference system and output resolution\n",
|
||||
"set_crs = easi.crs('landsat') # If defined, else None\n",
|
||||
"set_resolution = easi.resolution('landsat') # If defined, else None\n",
|
||||
"# set_crs = \"epsg:3577\"\n",
|
||||
"# set_resolution = (-30, 30)\n",
|
||||
"\n",
|
||||
"# Set the scene group_by method\n",
|
||||
"group_by = \"solar_day\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4f52fb17-cd58-439a-921e-775ce94ebcca",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":1},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41cd8e21-cdae-4ee2-a19e-5803617e85bf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can check the total size of the dataset using `nbytes`. We'll divide by 2**30 to have the result display in [gibibytes](https://simple.wikipedia.org/wiki/Gibibyte)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e00f1009-9734-4bb0-b49a-4918af09af8e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "39a30454-290b-4f1f-a8ca-32c697b68b89",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As you can see this Region of Interest (ROI) and spatial range (1 year) is tiny, let's scale up by increasing our ROI by increasing the buffer\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9d53a93b-a050-4d7c-bd50-fc83d35c1126",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"buffer = 1\n",
|
||||
"\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "16b66c21-f5e8-48de-bb6e-1ce34f885bd8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":1},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "08eb169a-7fee-4703-8203-cc871e3b2bcc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Okay, this should now be larger than the available memory that our Jupyter node has available (which you should be able to see at the bottom of your window - probably 24-30 GB). This creates issues for calculation. We need to have a solution that lets us calculate the information that we want without the machine running out of memory. \n",
|
||||
"\n",
|
||||
"Dask can compute many tasks and handle large amounts of data over the course of a series of calculations. Collectively, these calculations might work on more data in total than can fit in RAM, but it is a problem if the final product is too big to fit in RAM. Below we will change the dataset so that the final result can fit in RAM and then use the `.compute()` function to run all the calculations.\n",
|
||||
"\n",
|
||||
"Let's take a look at the memory usage for one of the bands, we'll use `red`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "891babf5-678a-4a40-9502-2418b4820e5e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset[alias['red']]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5ddd4c1c-647a-4a98-8af0-f10ef5f029d0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You can see the year now has more time observations than in the first dataset because we've expanded the area of interest and picked up multiple satellite passes. The spatial dimensions are also much larger.\n",
|
||||
"\n",
|
||||
"Take a note of the _Chunk Bytes_ - probably around 80 MiB. This is the smallest unit of this dataset that dask will do work on. To do an NDVI calculation, dask will need two bands, the mask, the result and a few other temporary variables in memory at once. This means whilst this value is an indicator of memory required on a worker to perform an operation it is not the total, which will depend on the operation.\n",
|
||||
"\n",
|
||||
"We can adjust the amount of memory per chunk further by _chunking_ the spatial dimension. Let's split it into 2048x2048 size pieces."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "497dfc58-4c65-46e9-81b5-80aba1db87b1",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":1, \"x\":2048, \"y\":2048}, ## Adjust the chunking spatially as well\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2da60763-1813-43fa-857c-99153c99a6e5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As you can see the total dataset size stays the same. \n",
|
||||
"\n",
|
||||
"Look at the `red` data variable below. You can see the chunk size has reduced to 8 MiB, and there are now more chunks (around 700-800) - compared with around 60 previously. This will result in a higher number of Tasks for Dask to work on. This makes sense: smaller chunks, more tasks.\n",
|
||||
"\n",
|
||||
"> __TIP__: The _relationship between tasks and chunks_ is a critical tuning parameter.\n",
|
||||
"\n",
|
||||
"Workers have limits in memory and compute capacity. The Dask Scheduler has limits in how many tasks it can manage efficiently (and remember it is tracking all of the data variables, not just this one). The trick with Dask is to give it a good number of chunks of data that aren not too big and don't result in too many tasks. There is always a trade-off and each calculation will be different. Ideally, you want chunks to be aligned with how the data is stored, or how the data is going to be used. If those two things are different, then rechunking can result in large amounts of data needing to be held in the cluster memory, which could result in failures. In the same way, if chunks are too large, they might end up taking up too much memory, causing a crash. This is sometimes down to trial and error.\n",
|
||||
"\n",
|
||||
"Later, when we move to a fully remote and distributed cluster, _chunks_ also become an important element in communicating between workers over networks.\n",
|
||||
"\n",
|
||||
"If you look carefully at the cube-like diagram in the summary below you will see that some internal lines showing the chunk boundaries for the spatial dimensions. 2048 wasn't an even multiplier so dask has made some chunks on the edges smaller. The specification of `chunks` is a guide: the actual data, numpy arrays in this case, are made into `chunk` sized shapes or smaller. These are called `blocks` in dask and represent the actual shape of the numpy array that will be processed.\n",
|
||||
"\n",
|
||||
"Somewhat confusingly the terms `blocks` and `chunks` are also used in dask literature and you'll need to check the context to see if it is referring to the _specification_ or the _actual block of data_. For the moment this differentiation doesn't matter but when performing low level custom operations knowing that your `blocks` might be a different shape does matter."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "32b96e79-6c54-4f03-91a8-6b4a840d3225",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset[alias['red']]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7fa1249-f0af-4e83-a356-31cfc34e1c1c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We won't worry to much about tuning these parameters right now and instead will focus on processing this larger dataset. As before we can exploit dask's ability to use _delayed_ tasks and apply our masking and NDVI directly to the full dataset. We'll also add an unweighted seasonal mean calculation using `groupby(\"time.season\").mean(\"time\")`. Dask will seek to complete the reductions (by chunk) first as they reduce memory usage.\n",
|
||||
"\n",
|
||||
"It's probably worth monitoring the dask cluster memory usage via the dashboard _Workers Memory_ to see just how little ram is actually used during this calculation despite it being performed on a large dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6258f835-1f34-4dc1-8d99-e89d6233528f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(dashboard_address)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb3b1aba-85d4-415d-84e3-745fb38138a2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will now calculate NDVI and group the results by season:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e27e9923-6884-40d1-a53a-2babfae82a66",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Identify pixels that don't have cloud, cloud shadow or water\n",
|
||||
"cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
"\n",
|
||||
"# Apply the mask\n",
|
||||
"cloud_free = dataset.where(cloud_free_mask)\n",
|
||||
"\n",
|
||||
"# Calculate the components that make up the NDVI calculation\n",
|
||||
"band_diff = cloud_free[alias['nir']] - cloud_free[alias['red']]\n",
|
||||
"band_sum = cloud_free[alias['nir']] + cloud_free[alias['red']]\n",
|
||||
"# Calculate NDVI and store it as a measurement in the original dataset ta da\n",
|
||||
"ndvi = None\n",
|
||||
"ndvi = band_diff / band_sum\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = ndvi.groupby(\"time.season\").mean(\"time\") # Calculate the seasonal mean"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "874bfe1d-799c-4449-9b80-19bcc7f99061",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's check the shape of our result - it should have 4 seasons now instead of the individual dates."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "009724fc-3c43-4225-9404-85b01f94d1fa",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ndvi_unweighted"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "db88af8e-beac-4779-b07b-5b001c5505bb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Before we do the `compute()` to get our result we should make sure the final result will fit in memory for the Jupyter kernel"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ced41b22-710e-4603-8f04-49c7f8849bca",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"dataset size (GiB) {ndvi_unweighted.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7dcb1346-242d-412e-b824-1ae6a6b1b2e9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This shows that the resulting data should be around 1 GiB of data, which will fit in local memory.\n",
|
||||
"\n",
|
||||
"If you are monitoring the cluster when you run the cell below, you might notice a delay between running the next cell and actual computation occuring. Dask performs a _task graph optimisation_ step on the _client_ not the cluster. How long this takes depends on the number of tasks and complexity of the graph. The speed of this step has improved recently due to recent Dask updates. We'll talk more about this later.\n",
|
||||
"\n",
|
||||
"In the meantime, run the next cell and watch dask compute the result without running out of memory. You might notice that your cluster spills some data to disk (the grey part of the bars in the _Bytes stored per worker_ graph). This is not normally desirable and slows down the calculation (because reading and writing to/from the disk is slower than to/from RAM), but it is a mechanism used by Dask to help manage large calculations. \n",
|
||||
"\n",
|
||||
">__Tip:__ don't forget to look at your Dask Dashboard (URL a few cells above) to watch what is happening in your cluster"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b4d3915b-1ccc-4ece-9559-cd27295a14f5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"actual_result = ndvi_unweighted.compute()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "181e6946-4a12-4d69-bf0a-3f02d64e2371",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To avoid northern/southern hemisphere differences, the `season` values are represented as acronyms of the months that make them up, so:\n",
|
||||
"- December, January, February = DJF\n",
|
||||
"- March, April, May = MAM\n",
|
||||
"- June, July, August = JJA\n",
|
||||
"- September, October, November = SON\n",
|
||||
"\n",
|
||||
"Let's plot the result for `DJF`. This will take a few seconds, the image is several thousand pixels across."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "289cfdb6-c5c7-4468-a379-9094a0003b82",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"actual_result.sel(season='DJF').plot(robust=True, size=6, aspect='equal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e757725f-c923-403a-8943-d79bb42dafc9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Not the most useful visualisation as a small image, and a little slow. Dask can help with this too but that's a topic for another notebook. There are many other ways to work with Dask and optimize performance. This is just the beginning of how to manage large calculations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "75730dc0-0940-44e2-b656-747f6862915d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Be a good dask user - Clean up the cluster resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "48eeb0ee-6888-4362-8615-19cf217e80c0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.close()\n",
|
||||
"\n",
|
||||
"cluster.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "29d22aa2-f239-4a0a-8b8a-331ca32b09d8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,751 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e374c84f-75e4-4859-bf4f-3a7847aef454",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Go Big or Go Home Part 1 - Dask fully distributed <img align=\"right\" src=\"../../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"In the previous notebooks we've been using `dask.distributed.LocalCluster` to split up our tasks and run them on the same compute node that is running the notebook. We noted that this could then use all cores to run tasks in parallel, greatly speeding up loading, and thanks to _chunks_ we can also process _some_ algorithms, like our NDVI seasonal mean, on datasets larger than available RAM.\n",
|
||||
"\n",
|
||||
"But what happens if your algorithm and dataset are such they cannot fit the compute nodes' RAM, or the result of the calculation is also massive, or its just so big (memory and computation) that it takes hours to compute?\n",
|
||||
"\n",
|
||||
"Well, `dask.distributed.LocalCluster` is just one member of the `dask.distributed` cluster family. There are several others but the one we will be using is Kubernetes Cluster (`KubeCluster`). Kubernetes is an excellent technology that takes advantage of modern Cloud Computing architectures to automatically provision (and remove) compute nodes on demand. It does a lot of other stuff well beyond the scope of this dask tutorial of course. The important point is that using `KubeCluster` we can dramatically expand the number of Compute Nodes to schedule our dask Workers to; potentially very dramatically.\n",
|
||||
"\n",
|
||||
"In this notebook we'll expand our NDVI seasonal mean calculation to a larger area and take it back through two decades of observations. Along the way we'll explore the dask data structures, how computation proceeds, and what we can do to tune the performance. At this spatial size we'll also look at how to interactively visualise a result that is larger than the Jupyter notebook can handle.\n",
|
||||
"\n",
|
||||
"Everything we do next builds on the concepts of _chunks_, _tasks_, _data locality_, and _task graph_ covered previously."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5a59f381-17c5-49c1-9579-00a1a6f22e4b",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Dask Gateway and remote dask schedulers\n",
|
||||
"\n",
|
||||
"Once we have multiple compute nodes to run workers on we have a lot more moving parts in the system. These parts are also fully distributed and will need to communicate between each other to pass results, perform tasks, confirm completion of tasks and ask for more work, etc. It is important to understand what the components are and how they interact because it can impact both _performance_ and _stablity_ of calculation, particularly at very large scales. In addition, _data locality_ really matters when your dataset is large - you don't want to `compute()` a 1 TB result and have it brought back to the Jupyter kernel on a 32 GiB machine! There are also subtleties to be aware of in how data gets from your Jupyter notebook to the dask distributed nodes - it has to be communicated somehow.\n",
|
||||
"\n",
|
||||
"This can all be a bit overwhelming to think about. Thankfully, you don't hit all of these at once as dask does good job of hiding many of the details but then remember our two \"laws\" of dask from the first notebook:\n",
|
||||
"1. The best thing about dask is it makes distributed parallel programming in the datacube easy\n",
|
||||
"1. The worst thing about dask is it makes distributed parallel programming in the datacube easy\n",
|
||||
"\n",
|
||||
"The transition point from gain to pain, and back to gain, is connected to these details. So let's define out various parts and their roles and start building our knowledge."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7b75e69d-237d-4411-bd88-59e2e7346826",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Kubernetes\n",
|
||||
"\n",
|
||||
"In a Kubernetes environment all programs that execute things run in __Pods__. What a _pod_ is and how it works is a subject for another course and you can use an internet search to find out more. For our purposes it is sufficient to understand that the Jupyter notebook, the dask scheduler, the dask workers, and all the components that make this work are running in _Pods_.\n",
|
||||
"\n",
|
||||
"* _Pods_ have resources - memory, cpu, gpu, storage.\n",
|
||||
"* _Pods_ request resources and have resource limits.\n",
|
||||
"* _Pods_ communicate to each other over a network.\n",
|
||||
"* You can think of a _pod_ as being a kind of virtual PC on which you can run your programs.\n",
|
||||
"\n",
|
||||
"_Pods_ run on _Compute Nodes_ - physical hardware with an actual CPU, GPU, memory and storage. _Compute Nodes_ can run more than one _Pod_ so long as the sum of all the requests will fit. For example, if your _Compute Node_ has 64 GiB of RAM and your _worker pods_ request 14 GiB each, then 4 _workers Pods_ will run on 1 _Compute Node_.\n",
|
||||
"\n",
|
||||
"Thankfully, you don't need to figure out what Pods get placed where as Kubernetes will do that automatically. There is more to be said about this relationship and the impacts on performance and operational cost but for now just note that _Pods_ are where your code and data lives and they have requests and limits which you can control.\n",
|
||||
"\n",
|
||||
"This diagram shows a single user (_Joe_) running a Jupyter Notebook and connected to a single dask cluster with 5 Workers (running on 3 Worker Nodes).\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "865c9853-fe04-439c-858f-4f2777eb4c48",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The __Jupyter notebook__ is where your code is typed in. It has a Python kernel of its own and will be the __dask client__ that talks to the __dask cluster__. It is running in a Pod, as are all the components. So the Jupyter notebook is _separate_ from the other components in the system and communicates over a network.\n",
|
||||
"\n",
|
||||
"The __dask cluster__ is the __dask scheduler__ plus the group of __dask workers__ that process the tasks in a distributed manner. The _scheduler_ and the _workers_ are all Pods, which means they are _separate_ from each other and communicate over a network. This is different to `dask.distributed.LocalCluster` in which the Jupyter notebook, dask scheduler and workers all resided on the same machine and all communicated _very_ rapidly on the local machine's communications channel. Now they can be on entirely different _compute nodes_ and are _communicating over a much slower network_. We have the benefit of more compute resources, at the cost of slower communication.\n",
|
||||
"\n",
|
||||
"The __dask gateway__ is a new component and is used to manage __dask clusters__ (note that is plural). The __Jupyter notebook__ acts as a client to the _dask gateway_ and makes requests for a __dask cluster__ (both the _scheduler_ and the _workers_) to the _dask gateway_ to create and destroy them. The _dask gateway_ manages the lifecycle of the cluster on the user's behalf.\n",
|
||||
"\n",
|
||||
"> __Tip__: This means the dask clusters have an independent life cycle compared with the Jupyter notebook. Quitting your Jupyter notebook will not necessarily quit your dask cluster.\n",
|
||||
"\n",
|
||||
"Moreover, you can have more than one Jupyter notebook talking to the _same_ dask cluster simultaneously. There are some good reasons for doing this but we won't be touching on them in this course."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4770317f-ec1b-4365-9a23-d4ef684a1a11",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Running our NDVI seasonal mean on the remote dask cluster\n",
|
||||
"\n",
|
||||
"Let's move our NDVI seasonal mean from the `LocalCluster` to our _dask gateway_ managed cluster, and add some extra compute resources to it.\n",
|
||||
"\n",
|
||||
"The biggest change here is simply how we start and shutdown the dask cluster. The rest of the code, to do the actual computation, is _exactly_ the same.\n",
|
||||
"\n",
|
||||
"The first thing we need to do is create a client so we can connect to the _dask gateway_.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c1fdc377-b388-44bc-ae3b-234848cd620f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize the Gateway client\n",
|
||||
"from dask.distributed import Client\n",
|
||||
"from dask_gateway import Gateway\n",
|
||||
"\n",
|
||||
"gateway = Gateway()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f2529f3c-3b04-4bc0-a036-aab65a3602c9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Easy! We now have a `gateway` client variable. Using this we can start clusters, stop clusters, ask for a list of clusters we have running, set options for our scheduler and workers (like cpu and memory requests).\n",
|
||||
"\n",
|
||||
"Let's see what the `cluster_options` are. We don't need to guess, we can ask the `gateway`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c3fec8f2-91bf-495f-9493-21cfea00f3b7",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gateway.cluster_options()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4d0a08ee-c3aa-4b5a-bda8-20f6fc6b47b0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"That's a lot I know. The majority of these don't need to change and most users will simply tweak _worker_ parameters: _cores_, _threads_ (probably keeping it the same as _cores_), _memory_ and the _worker group_.\n",
|
||||
"\n",
|
||||
"We will be using the defaults for now."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8da34ddb-f856-4234-9ef9-cf45095a83ba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create the Cluster\n",
|
||||
"\n",
|
||||
"Create the cluster with default options if it doesn't already exist. If a cluster exists in your namespace, the code below will connect to the first cluster. List the available clusters with `gateway.list_clusters()`.\n",
|
||||
"\n",
|
||||
"The cluster creation may take a little while (minutes) if a suitable _node_ isn't available for the _scheduler_. The same thing will occur for _workers_ when they start. If a _node_ does exist then this can happens in seconds.\n",
|
||||
"\n",
|
||||
"> __Tip__: Users are often confused by the changing start up time and think something is wrong.\n",
|
||||
"\n",
|
||||
"It can take _minutes_ for a brand new _node_ to be provisioned, please be patient. If it takes 10 minutes then yes, something is wrong and you should probably contact an administrator of the system if that problem persists."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7dc86e6f-6afb-4a8b-83fd-48bed81bd142",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"clusters = gateway.list_clusters()\n",
|
||||
"if not clusters:\n",
|
||||
" print('Creating new cluster. Please wait for this to finish.')\n",
|
||||
" cluster = gateway.new_cluster()\n",
|
||||
"else:\n",
|
||||
" print(f'An existing cluster was found. Connecting to: {clusters[0].name}')\n",
|
||||
" cluster=gateway.connect(clusters[0].name)\n",
|
||||
"cluster"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb7174b3-3e78-4496-9e09-7200aef5780b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Scale the cluster\n",
|
||||
"\n",
|
||||
"Use the GatewayCluster widget (above) to adjust the cluster size. Alternatively use the cluster API methods.\n",
|
||||
"\n",
|
||||
"For many tasks 1 or 2 workers will be sufficient, although for larger areas or more complex tasks 5 to 10 workers may be used. If you are new to Dask, start with one worker and then scale your cluster if needed.\n",
|
||||
"\n",
|
||||
"In this notebook we'll start with 4 workers - that's 4x the resources for workers compared to our previous `LocalCluster`. In addition the _scheduler_ is also on its own node, and so is the Jupyter notebook kernel. Lots more resources for all the components involved.\n",
|
||||
"\n",
|
||||
"The next cell will use the cluster API to add 4 workers programmatically."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "52629b33-88e4-4f2a-8388-9213e4808e2f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cluster.scale(4)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3edb753-4309-4196-b2f4-1d10dc0ea93b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Connect to the cluster\n",
|
||||
"To connect to your cluster and start doing work, use the `get_client()` method. This step will wait until the workers are ready. You don't actually have to wait for the workers. The Jupyter notebook can be doing other things whilst the workers are coming up. We're waiting in this example so you don't end up with an unexpected wait later.\n",
|
||||
"\n",
|
||||
"***This may take a few minutes before your workers will be ready to use. Please wait for the cell to finish and show you the Dask Client.***"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b11f3b60-0981-4090-bb29-a810dbdf15ea",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = cluster.get_client()\n",
|
||||
"# client.wait_for_workers(n_workers=4) # Before release 2023.10.0\n",
|
||||
"client.sync(client._wait_for_workers,n_workers=4) # Since release 2023.10.0\n",
|
||||
"client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fcefb34f-b1fa-4774-92e9-836bbb6f21b6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The client widget provides a clickable __dask dashboard__ link so click that and you'll see your dashboard. It works the same as before despite the fact that everything is now running in a distributed manner. If you click the _Workers_ tab in the _dashboard_ you will see that we now have 32 cores (up from 8) made up of 4x 8-core workers. Lot's of RAM too.\n",
|
||||
"\n",
|
||||
"Go back to the _Status_, so you can watch everything run."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "62379a64-d3ae-43b9-ae09-20024b66e0f9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Perform the computation\n",
|
||||
"\n",
|
||||
"This is the same as in [dask tutorial 03](./03_-_Larger_than_RAM_(LocalCluster).ipynb).\n",
|
||||
"\n",
|
||||
"We don't need to change any of our code to run this now, so let's repeat the full calculation.\n",
|
||||
"\n",
|
||||
"As we will be using __Requester Pays__ buckets in AWS S3, we need to run the `configure_s3_access()` function below with the `client` option to ensure that Jupyter and the cluster have the correct permissions to be able to access the data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5896362b-4d9e-4356-af33-2c2e8a0d07e0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import git\n",
|
||||
"import sys, os\n",
|
||||
"from dateutil.parser import parse\n",
|
||||
"from dateutil.relativedelta import relativedelta\n",
|
||||
"from dask.distributed import Client, LocalCluster\n",
|
||||
"import datacube\n",
|
||||
"from datacube.utils import masking\n",
|
||||
"from datacube.utils.aws import configure_s3_access\n",
|
||||
"\n",
|
||||
"# EASI defaults\n",
|
||||
"os.environ['USE_PYGEOS'] = '0'\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults, notebook_utils\n",
|
||||
"easi = EasiDefaults()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "df12b923-dc2e-4ebe-ad54-59ef7e1402e4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dc = datacube.Datacube()\n",
|
||||
"configure_s3_access(aws_unsigned=False, requester_pays=True, client=client)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c7e594cc-cfcb-41f2-b282-cefc14be13d7",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get the centroid of the coordinates of the default extents\n",
|
||||
"central_lat = sum(easi.latitude)/2\n",
|
||||
"central_lon = sum(easi.longitude)/2\n",
|
||||
"# central_lat = -42.019\n",
|
||||
"# central_lon = 146.615\n",
|
||||
"\n",
|
||||
"# Set the buffer to load around the central coordinates\n",
|
||||
"# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions\n",
|
||||
"buffer = 0.8\n",
|
||||
"\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)\n",
|
||||
"\n",
|
||||
"# Data product\n",
|
||||
"products = easi.product('landsat')\n",
|
||||
"\n",
|
||||
"# Set the date range to load data over\n",
|
||||
"set_time = easi.time\n",
|
||||
"set_time = (set_time[0], parse(set_time[0]) + relativedelta(years=1))\n",
|
||||
"# set_time = (\"2021-01-01\", \"2021-12-31\")\n",
|
||||
"\n",
|
||||
"# Selected measurement names (used in this notebook). None` will load all of them\n",
|
||||
"alias = easi.aliases('landsat')\n",
|
||||
"measurements = None\n",
|
||||
"# measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]\n",
|
||||
"\n",
|
||||
"# Set the QA band name and mask values\n",
|
||||
"qa_band = alias['qa_band']\n",
|
||||
"qa_mask = easi.qa_mask('landsat')\n",
|
||||
"\n",
|
||||
"# Set the resampling method for the bands\n",
|
||||
"resampling = {qa_band: \"nearest\", \"*\": \"average\"}\n",
|
||||
"\n",
|
||||
"# Set the coordinate reference system and output resolution\n",
|
||||
"set_crs = easi.crs('landsat') # If defined, else None\n",
|
||||
"set_resolution = easi.resolution('landsat') # If defined, else None\n",
|
||||
"# set_crs = \"epsg:3577\"\n",
|
||||
"# set_resolution = (-30, 30)\n",
|
||||
"\n",
|
||||
"# Set the scene group_by method\n",
|
||||
"group_by = \"solar_day\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2abeb772-f4c2-4392-968b-5d5a6fd6f0f4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":1, \"x\":2048, \"y\":2048}, ## No change here, chunking spatially just like before\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "04b3c928-43f0-4f53-8f10-6a64e0301c9b",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Identify pixels that are either \"valid\", \"water\" or \"snow\"\n",
|
||||
"cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
"\n",
|
||||
"# Apply the mask\n",
|
||||
"cloud_free = dataset.where(cloud_free_mask)\n",
|
||||
"\n",
|
||||
"# Calculate the components that make up the NDVI calculation\n",
|
||||
"band_diff = cloud_free[alias['nir']] - cloud_free[alias['red']]\n",
|
||||
"band_sum = cloud_free[alias['nir']] + cloud_free[alias['red']]\n",
|
||||
"# Calculate NDVI and store it as a measurement in the original dataset ta da\n",
|
||||
"ndvi = None\n",
|
||||
"ndvi = band_diff / band_sum\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = ndvi.groupby(\"time.season\").mean(\"time\") # Calculate the seasonal mean"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "39611d88-6e76-4ed6-a4da-fad1bc496287",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"actual_result = ndvi_unweighted.compute()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a501db5-d805-4b26-b511-ec2f8d169a52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As before there will be a short delay as the Jupyter kernel (_client_) is used to optimize the task graph before sending it to the _scheduler_ which will then execute _tasks_ on the _workers_.\n",
|
||||
"\n",
|
||||
"> __Tip__: You can open a terminal and use `htop` to monitor the Jupyter notebook CPU usage. You'll see at least one core using nearly 100% cpu usage during the optimisation phase. It will then drop back to idle as the _task graph_ is sent to the _scheduler_ at which point the dask dashboard will show activity on the cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6f9845f5-c3dd-4371-9a50-1fb4e8da2180",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"actual_result.sel(season='DJF').plot(robust=True, size=6, aspect='equal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3f463907-1920-4b3c-a33b-0d29467eefae",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"### Exploiting our new resources - adjusting our chunk size\n",
|
||||
"\n",
|
||||
"Before we \"Go Big\" let's take advantage of our new resources.\n",
|
||||
"\n",
|
||||
"We've gained memory. The more memory we have the more data we can operate on at once _and the less we need to communicate between nodes_. Communication over a network is slow relative to local communcation in a _pod_. So if we change the _chunking_ we may see an improvement in performance.\n",
|
||||
"\n",
|
||||
"_Chunking_ will also impact the number of tasks - the fewer chunks, the fewer tasks. This in turn will impact how much _task graph optimization_ is required, how hard the _scheduler_ has to work, and how much communication of partial results goes between workers (for example, passing the partial means around to get a final mean).\n",
|
||||
"\n",
|
||||
"Let's look at our existing chunk size - `(1,2048,2048)`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f0d65191-67aa-4919-8b32-8b4762d02a41",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset[alias['red']]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c50815c1-fbc8-4853-9539-446e39abb3a7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Each `red` chunk is currently 8 MiB in size. Of course this will vary with data type and stage of computation so when tuning dask you may need to check on the _chunk size_ and _tasks_ as your computation transforms the data. We're doing a simple computation here so we can focus on this initial value. With experience it does get easier to figure out when and where this parameter needs further adjustment. We'll look at some of this later.\n",
|
||||
"\n",
|
||||
"For now there are some things we can observe:\n",
|
||||
"1. 8 MiB is pretty small when our 4 workers have 32 Gigs each so we have room to grow even with all the temporaries to allow for.\n",
|
||||
" * _We should monitor the worker memory usage (shown as `Bytes stored per worker` in the dashboard as we make changes to ensure none are spilling to disk (shown in grey) as that will slow things down and is unnecessary in this case_\n",
|
||||
"1. Geospatial operations - the data load, the reprojection, even the masking - may benefit from having a larger spatial area.\n",
|
||||
" * No point going too large though as the satellite paths have a finite width and we'll just have lots of empty space.\n",
|
||||
"1. The computation involves a seasonal mean, which means some temporal grouping might improve performance.\n",
|
||||
" * That said, _chunks_ are a unit for communication and it may mean that we're passing more information around than is necessary if we group too much together across the seasonal boundaries.\n",
|
||||
"\n",
|
||||
"So we have good reason to increase our chunks both spatially and temporally; just be mindful of the impact on communication of results between nodes. The mean is seasonal and Landsat 8 performs repeat passes nominally every 16 days, so let's do a small grouping in time. We'll also increase the spatial chunking slightly."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "acbc9f88-7b72-48e0-a1e3-700273a85cfe",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":3072}, # This line has changed\n",
|
||||
" group_by=group_by,\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "872da0b0-9a9f-454f-81e7-eac5cd2e3306",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset[alias['red']]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "67a323f5-fc2a-4167-925e-ecd607812bd7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"As you can see the number of tasks has dropped and our chunks are larger at 36 MiB.\n",
|
||||
"\n",
|
||||
"There is a small slither along the bottom because the chunk size isn't a good fit for the actual array size. Let's expand our chunk size in that direction slightly to give us a better fit."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "42b77c6d-4ea3-4198-bc91-ed00aa20f4b3",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from math import ceil\n",
|
||||
"\n",
|
||||
"y_chunk = ceil(dataset.dims['y']/2)\n",
|
||||
"print(f'Y-dim chunk: {y_chunk}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f9d92fab",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":y_chunk},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"dataset[alias['red']]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "809561da-7fe9-4a0e-8ec8-0f7ee28564a0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Notice how that small change to remove the sliver only marginally increased our chunk memory usage but dramatically reduced the number of chunks.\n",
|
||||
"\n",
|
||||
"Let's see what this does to our performance. We need to re-run the code to update all the intermediate variables in our calculation and call `compute()`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a2ec7285-fa1d-479c-8bf8-b58b2785195a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Identify pixels that are either \"valid\", \"water\" or \"snow\"\n",
|
||||
"cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
"\n",
|
||||
"# Apply the mask\n",
|
||||
"cloud_free = dataset.where(cloud_free_mask)\n",
|
||||
"\n",
|
||||
"# Calculate the components that make up the NDVI calculation\n",
|
||||
"band_diff = cloud_free[alias['nir']] - cloud_free[alias['red']]\n",
|
||||
"band_sum = cloud_free[alias['nir']] + cloud_free[alias['red']]\n",
|
||||
"# Calculate NDVI and store it as a measurement in the original dataset ta da\n",
|
||||
"ndvi = None\n",
|
||||
"ndvi = band_diff / band_sum\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = ndvi.groupby(\"time.season\").mean(\"time\") # Calculate the seasonal mean"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1818d321-3019-4ef2-a965-cd99deed996e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"actual_result = ndvi_unweighted.compute()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b08376e4-2473-4f63-a80e-da65347a1d8e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You will notice the time for _task graph optimization_ - the delay between executing the cell above and seeing processing in the cluster dashboard - is down significantly. Fewer tasks means less time in optimization. We've decreased the computation time as well.\n",
|
||||
"\n",
|
||||
"There is one more thing we can do before we \"Go Big\". We've done this before and its simple enough. Save dask the challenge of figuring out which measurements we aren't using by telling it only to load the ones we do use. Let's add our measurements list in."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "37b423b9-a28e-48e2-81c5-b2524802fb05",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]\n",
|
||||
"\n",
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":y_chunk},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"# Identify pixels that are either \"valid\", \"water\" or \"snow\"\n",
|
||||
"cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
"\n",
|
||||
"# Apply the mask\n",
|
||||
"cloud_free = dataset.where(cloud_free_mask)\n",
|
||||
"\n",
|
||||
"# Calculate the components that make up the NDVI calculation\n",
|
||||
"band_diff = cloud_free[alias['nir']] - cloud_free[alias['red']]\n",
|
||||
"band_sum = cloud_free[alias['nir']] + cloud_free[alias['red']]\n",
|
||||
"# Calculate NDVI and store it as a measurement in the original dataset ta da\n",
|
||||
"ndvi = None\n",
|
||||
"ndvi = band_diff / band_sum\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = ndvi.groupby(\"time.season\").mean(\"time\") # Calculate the seasonal mean"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "07d8c1f7-99dd-4907-8cda-b268bb371623",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"actual_result = ndvi_unweighted.compute()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e13f71e8-63bd-4fc6-b6a4-cb95109377aa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This didn't make to much difference to computational time but it has shortened the _task graph optimisation_ phase a little more. That time isn't a problem in this example but as we \"Go Big\" it will be.\n",
|
||||
"\n",
|
||||
"Now you can continue on to [Part 2](./05_-_Go_Big_or_Go_Home_Part_2.ipynb) of this part to test out a much bigger area."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cfcad62-a6e4-4504-9e41-d33f872ade7a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Be a good dask user - Clean up the cluster resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "66acca97-686c-4d09-ba5b-a9aeede4bfba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Disconnecting your client is good practice, but the cluster will still be up so we need to shut it down as well"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0578682e-917e-49b7-9f3e-88584a572936",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.close()\n",
|
||||
"\n",
|
||||
"cluster.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4dfc916f-ad27-44cb-9381-6c7179d6ac34",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,727 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e374c84f-75e4-4859-bf4f-3a7847aef454",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Go Big or Go Home Part 2 - Working and Visualising on cluster <img align=\"right\" src=\"../../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"In this notebook we finally do our larger area. We're going to need some better visualisation tools and it would be great not to bring the results back to the Jupyter notebook but to leverage the dask clusters resources during visualisation. We'll be using some dask-aware visualiation libraries (holoviews and datashader) to do the heavy lifting.\n",
|
||||
"\n",
|
||||
"Let's begin by starting up our cluster and sizing it appropriately to our computational task."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4770317f-ec1b-4365-9a23-d4ef684a1a11",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Time to go big!\n",
|
||||
"\n",
|
||||
"All the code here is the same as the conclusion from the previous notebook, except we'll make the cluster bigger with 10 workers instead of 4. We'll also make the masking and NDVI calculation into a python function since we won't be making any changes to that now.\n",
|
||||
"\n",
|
||||
"We'll use the same ROI and time period for this run and we're using all the techniques so far to reduce the computation time:\n",
|
||||
"1. Dask chunk size selection\n",
|
||||
"2. Only loading the measurements we intend on using in this calculation to save on the task graph optimisation time"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1619a66a-c80f-4345-8022-a81018dc294f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize the Gateway client\n",
|
||||
"from dask.distributed import Client\n",
|
||||
"from dask_gateway import Gateway\n",
|
||||
"\n",
|
||||
"number_of_workers = 10 \n",
|
||||
"\n",
|
||||
"gateway = Gateway()\n",
|
||||
"\n",
|
||||
"clusters = gateway.list_clusters()\n",
|
||||
"if not clusters:\n",
|
||||
" print('Creating new cluster. Please wait for this to finish.')\n",
|
||||
" cluster = gateway.new_cluster()\n",
|
||||
"else:\n",
|
||||
" print(f'An existing cluster was found. Connecting to: {clusters[0].name}')\n",
|
||||
" cluster=gateway.connect(clusters[0].name)\n",
|
||||
"\n",
|
||||
"cluster.scale(number_of_workers)\n",
|
||||
"\n",
|
||||
"client = cluster.get_client()\n",
|
||||
"client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "01c6ebaa-8f17-4767-b120-8917bd75a466",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pyproj\n",
|
||||
"pyproj.set_use_global_context(True)\n",
|
||||
"\n",
|
||||
"import git\n",
|
||||
"import sys, os\n",
|
||||
"from dateutil.parser import parse\n",
|
||||
"from dateutil.relativedelta import relativedelta\n",
|
||||
"from dask.distributed import Client, LocalCluster\n",
|
||||
"import datacube\n",
|
||||
"from datacube.utils import masking\n",
|
||||
"from datacube.utils.aws import configure_s3_access\n",
|
||||
"\n",
|
||||
"# EASI defaults\n",
|
||||
"os.environ['USE_PYGEOS'] = '0'\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults, notebook_utils\n",
|
||||
"easi = EasiDefaults()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "73eaf35c-f851-4121-b3dc-fe34ee025e0a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dc = datacube.Datacube()\n",
|
||||
"configure_s3_access(aws_unsigned=False, requester_pays=True, client=client)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "fca9b521-6b25-4738-be07-b1d575a7f8ea",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get the centroid of the coordinates of the default extents\n",
|
||||
"central_lat = sum(easi.latitude)/2\n",
|
||||
"central_lon = sum(easi.longitude)/2\n",
|
||||
"# central_lat = -42.019\n",
|
||||
"# central_lon = 146.615\n",
|
||||
"\n",
|
||||
"# Set the buffer to load around the central coordinates\n",
|
||||
"# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions\n",
|
||||
"buffer = 0.8\n",
|
||||
"\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)\n",
|
||||
"\n",
|
||||
"# Data product\n",
|
||||
"products = easi.product('landsat')\n",
|
||||
"\n",
|
||||
"# Set the date range to load data over\n",
|
||||
"set_time = easi.time\n",
|
||||
"set_time = (set_time[0], parse(set_time[0]) + relativedelta(years=1))\n",
|
||||
"# set_time = (\"2021-01-01\", \"2021-12-31\")\n",
|
||||
"\n",
|
||||
"# Selected measurement names (used in this notebook)\n",
|
||||
"alias = easi.aliases('landsat')\n",
|
||||
"measurements = [alias[x] for x in ['qa_band', 'red', 'nir']]\n",
|
||||
"\n",
|
||||
"# Set the QA band name and mask values\n",
|
||||
"qa_band = alias['qa_band']\n",
|
||||
"qa_mask = easi.qa_mask('landsat')\n",
|
||||
"\n",
|
||||
"# Set the resampling method for the bands\n",
|
||||
"resampling = {qa_band: \"nearest\", \"*\": \"average\"}\n",
|
||||
"\n",
|
||||
"# Set the coordinate reference system and output resolution\n",
|
||||
"set_crs = easi.crs('landsat') # If defined, else None\n",
|
||||
"set_resolution = easi.resolution('landsat') # If defined, else None\n",
|
||||
"# set_crs = \"epsg:3577\"\n",
|
||||
"# set_resolution = (-30, 30)\n",
|
||||
"\n",
|
||||
"# Set the scene group_by method\n",
|
||||
"group_by = \"solar_day\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e2febae5-307c-4f16-ada5-1236d4a46ffc",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def masked_seasonal_ndvi(dataset):\n",
|
||||
" # Identify pixels that are either \"valid\", \"water\" or \"snow\"\n",
|
||||
" cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
" # Apply the mask\n",
|
||||
" cloud_free = dataset.where(cloud_free_mask)\n",
|
||||
"\n",
|
||||
" # Calculate the components that make up the NDVI calculation\n",
|
||||
" band_diff = cloud_free[alias['nir']] - cloud_free[alias['red']]\n",
|
||||
" band_sum = cloud_free[alias['nir']] + cloud_free[alias['red']]\n",
|
||||
" # Calculate NDVI\n",
|
||||
" ndvi = None\n",
|
||||
" ndvi = band_diff / band_sum\n",
|
||||
"\n",
|
||||
" return ndvi.groupby(\"time.season\").mean(\"time\") # Calculate the seasonal mean\n",
|
||||
"\n",
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":3072},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = masked_seasonal_ndvi(dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "80a6df3e-d808-47e9-a9a0-cf7328253c3b",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")\n",
|
||||
"print(f\"ndvi_unweighted size (GiB) {ndvi_unweighted.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e50c099d-de4d-42c8-9eef-f9f01b34bac8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# client.wait_for_workers(n_workers=10) # Before release 2023.10.0\n",
|
||||
"client.sync(client._wait_for_workers,n_workers=10) # Since release 2023.10.0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "42ee9be6-9c84-4eff-91a0-d876cbfd99e9",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cluster"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ceea43a7-ec03-4413-89b1-91e5fbeabee9",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"actual_result = ndvi_unweighted.compute()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a501db5-d805-4b26-b511-ec2f8d169a52",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"You'll notice the computation time is slightly faster with more workers - we're IO bound so more workers means more available IO bandwidth and threads. It's not 2-3 x faster though - we're wasting a lot of resources because we can't actually use all of that extra power.\n",
|
||||
"\n",
|
||||
"> __Tip__: More isn't always better. Be mindful of your computational resource usage and cost. This size cluster is a tremendous waste for this size computational job. Size things appropriately."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb5ee837-faae-4928-b9a4-dbe6d0d43476",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"And visualise the result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "43c304d8-407d-48e3-95cd-41aa51537382",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"actual_result.sel(season='DJF').plot()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3495ba8f-301c-4751-af73-a5333348e2bc",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"We'll save the coordinates of this section from the array (as slices) so we can use them later for visualising the same ROI from a larger dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "04dc7f0e-b2f0-4093-9d81-7cdc00cd8afa",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"x_slice = slice(ndvi_unweighted.x[0], ndvi_unweighted.x[-1])\n",
|
||||
"y_slice = slice(ndvi_unweighted.y[0], ndvi_unweighted.y[-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2dbfb392-9ffd-418e-a610-e29503980458",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Now for a bigger area\n",
|
||||
"\n",
|
||||
"Let's change the area extent to about 4 degrees square.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e08de486-3ff3-44c3-8d0e-29dfbd929982",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"buffer = 2\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "8ddfb47e-53ec-4e77-8f20-bc687c8f35c0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Check the map below to see if you are including the area that you want. For this example, it would be best to not include too much water.\n",
|
||||
"from dea_tools.plotting import display_map\n",
|
||||
"display_map(study_area_lon, study_area_lat)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "85a90609-1d18-4ab9-ab8c-ce83659db9e4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":3072},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = masked_seasonal_ndvi(dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "449e859a-5cbc-4341-ae78-d5735b11ed35",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Before we compute anything let's take a look at our result's shape and size"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5d66426b-ef44-469a-9675-116f5d3fec1c",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")\n",
|
||||
"print(f\"ndvi_unweighted size (GiB) {ndvi_unweighted.nbytes / 2**30:.2f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "07021f2b-5f82-4a61-9afb-c8e5b304b94e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This is now much bigger!\n",
|
||||
"\n",
|
||||
"The result is getting on the large size for the notebook node __so we will need to pay attention to _data locality_ and the size of results being processed__. The cluster has a LOT more memory than the _notebook node_; bring too much back to the notebook and the notebook will crash.\n",
|
||||
"\n",
|
||||
"> __Tip__: Be mindful of the size of the results and their _data locality_. \n",
|
||||
"\n",
|
||||
"Now let's check the _shape_, _tasks_ and _chunks_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d751b9c3-75b6-4143-a484-68d9fab6751f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cca340a5-1068-4cdc-b4ae-ed7eda75853e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Looking at the `red` data variable we can see about 50 GiB for the array, 36 MiB per chunk and 5526 tasks. Noting the `nir` and `qa_band` will be similarly shaped and size.\n",
|
||||
"\n",
|
||||
"The number of tasks is climbing so we can expect an increase in _task graph optimisation_ time.\n",
|
||||
"\n",
|
||||
"Chunk size and tasks seems okay, but we will monitor the _dask dashboard_ in case there are issues with temporaries causing _workers_ to _spill to disk_ if memory is too full.\n",
|
||||
"\n",
|
||||
"_The chunking is resulting in some slivers, particularly on the y axis._ Let's modify the y chunk size so these slivers don't exist as its blowing out the tasks and is likely unnecessary. We will calculate the x and y chunk sizes below to get a nice fit. _Make sure to check the chunk size afterwards to make sure it doesn't get too large. If it does we can make the chunks smaller to reduce slivers too._"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b2ff1968-87fb-4b02-8957-b78348a30d52",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from math import ceil\n",
|
||||
"\n",
|
||||
"y_chunks = ceil(dataset.dims['y']/5)\n",
|
||||
"y_chunks"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a2785d84-c578-40be-898f-84536ccc45a4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = None # clear results from any previous runs\n",
|
||||
"dataset = dc.load(\n",
|
||||
" product=products,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = {\"time\":2, \"x\":3072, \"y\":y_chunks},\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"ndvi_unweighted = masked_seasonal_ndvi(dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "293f8379-60e4-4fd7-b722-bfeb1b09fc18",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Now recheck our chunk size and tasks for `red`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "fefd593d-741b-423f-95cd-3c501230a6c0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b6ea5908-12e0-4c89-af9e-b09a905e1638",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Very marginal increase in _memory per chunk_ but the _tasks_ have dropped from 5526 to 4661. Note that this occurs for every measurement and operation so the benefit is significant."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "12283353-1f88-4abd-b8de-fc490ea02cb6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ndvi_unweighted"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7a7438cf-1d36-4afc-9230-744765bc11cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"Total task count is sub 100_000 so should be okay but _task graph optimisation_ will take a while. Resulting array is a bit big for the notebook node as stated previously.\n",
|
||||
"\n",
|
||||
"The shape spatially is `y:15686, x:13707`. Standard plots aren't going to work very well for visualising the result in the notebook and the result uses a fair amount of memory so we'll need a different approach.\n",
|
||||
"\n",
|
||||
"For now let's visualize the same ROI as the small area before. We stashed that ROI in `x_slice, y_slice`.\n",
|
||||
"\n",
|
||||
"__If you haven't already, open the dask dashboard so you can watch the cluster make progress__\n",
|
||||
"\n",
|
||||
"The code to do this visualisation is basically the same as before except now we specify a slice"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "21b9258c-010b-4a5c-92a7-1e88c8f09fb6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"ndvi_unweighted.sel(season='DJF', x=x_slice, y=y_slice).compute().plot(robust=True, size=6, aspect='equal')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a81e9086-b7b9-490c-88a8-a36e4c272862",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The computation time is relatively short since we are only materialising the result for a subset of the overall dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd4988-8843-4cc2-8d82-7be1b63f6b92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Visualising all of the data\n",
|
||||
"\n",
|
||||
"To visualise all of the data we will make use of the dask cluster and some dask-aware visualisation capabilties from `holoviews` and `datashader` python libraries. These libraries provide an _interactive_ visualisation capability that leaves the large datasets on the cluster and transmits only the final visualisation to the Jupyter notebook. This is done on the fly so the user can zoom and pan about the dataset in all dimensions and the dask cluster will scale data to fit in the viewport automatically. Details of how this is done and advanced features available is beyond the scope of this dask and ODC course but the manuals are extensive and the basic example here both powerful and useful.\n",
|
||||
"\n",
|
||||
"> __Tip__: The [datashader pipeline](https://datashader.org/getting_started/Pipeline.html) page provides an excellent summary of what's going on."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0fd9c328-5e93-4aed-bb4f-0f6aab70c71d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### `compute()` and `persist()`\n",
|
||||
"\n",
|
||||
"The first thing we will do is `persist()` the results of our calculation to the cluster. This will materialise the results but will keep the result on the cluster (so all lazy tasks are calculated, just like `compute()` but _data locality_ remains on the cluster). This will ensure the result is readily available for the visualisation. The cluster has plenty of (distributed) memory so there is no reason not to materialise the result on the cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ec3ae48-5799-4baf-92b2-e853b8b5bfd8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"`persist()` is non-blocking so will return just as soon as _task graph optimisation_ (which is performed in the notebook kernel) is complete. Run the next cell and you will see it takes a few seconds to do _task graph optimisation_, and once that is complete the Jupyter notebook will be available for use again. At the same time the _dask dashboard_ will show tasks running as the result is computed and left on the cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d9f0ed8f-8a01-4329-9998-d817622d6c4f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"on_cluster_result = ndvi_unweighted.persist()\n",
|
||||
"# wait(on_cluster_result)\n",
|
||||
"on_cluster_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "db6a6d61-4bd8-42b6-8aaa-c9765a8083f6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The `on_cluster_result` will continue to show as a dask array on the cluster - not actual results. Think of it as a handle that links the _Jupyter client_ to the result on the _dask cluster_."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "744fd152-50d4-45f8-ab2e-7311aecb7548",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The cluster will start the computation, but we can continue working in the notebook. Let's import a new visualization library: `hvplot`. We'll be using `datashader.rasterize` via `hvplot` to handle the visualisation of the full dataset which has many more pixels than what what is being displayed in the notebook. `hvplot.xarray` makes visualising `xarray` data a very natural experience, so the code is quite simple, a lot is taken care of for you.\n",
|
||||
"\n",
|
||||
"Notice also there are no bounds set on the dataset, we are viewing the entire result, including the _season_ dimension. `hvplot` will provide an interface for pan, zoom and season selection and you can use the mouse to move around the data.\n",
|
||||
"\n",
|
||||
">__Tip:__ Keep watching your Dask dashboard to see how the calculations are progressing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1e653ad4-155e-4bbe-86d5-fbc4e90149b7",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# quick calculation so the interactive UI is no more than 700 pixels wide and maintains aspect ratio.\n",
|
||||
"aspect = on_cluster_result.sizes['y']/on_cluster_result.sizes['x']\n",
|
||||
"width = 700\n",
|
||||
"height = int(width*aspect)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "728a39dd-ea5a-445b-b3c7-33f39d113dc8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The next cell will display the result - when its ready. The `rasterize` function will calculate a representative pixel for display from the full array on the dask cluster. If you monitor the dashboard you will see small bursts of activity across the workers and quite some waiting whilst data transfers occur to bring all the summary information back and transmit it to the Jupyter notebook. It's a large dataset, only the pixels you can see on the screen are sent to your web browser.\n",
|
||||
"\n",
|
||||
"You can use the controls on the right to pan and zoom around the full image. If you zoom in, `rasterize` will take a moment to generate a new summary for the current zoom level and show more or less detail. Similarly for panning."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "748c7958-4bce-45dc-895f-255347f91c4f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import hvplot.xarray \n",
|
||||
"import xarray as xr\n",
|
||||
"on_cluster_result.hvplot.image(groupby='season', rasterize=True).opts(\n",
|
||||
" title=\"NDVI Seasonal Mean\",\n",
|
||||
" cmap=\"RdYlGn\", # NDVI more green the larger the value. \n",
|
||||
" clim=(-0.3, 0.8), # we'll clamp the range for visualisation to enhance the visualisation\n",
|
||||
" colorbar=True,\n",
|
||||
" frame_width=width,\n",
|
||||
" frame_height=height\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cfcad62-a6e4-4504-9e41-d33f872ade7a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"# Be a good dask user - Clean up the cluster resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "66acca97-686c-4d09-ba5b-a9aeede4bfba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Disconnecting your client is good practice, but the cluster will still be up so we need to shut it down as well"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0578682e-917e-49b7-9f3e-88584a572936",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.close()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d144e53a-fb47-493f-9dbd-adef37566e07",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cluster.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "69b02631-c13f-481b-bc1c-f183538e7f63",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,519 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e4faa01-790c-4c1f-ac68-d46730fb570a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# On Chunks - The Art of Dask Part 1 <img align=\"right\" src=\"../../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"In this notebook we'll be exploring the impact of chunking choices for dask arrays. We'll use an ODC example but this isn't specific to ODC, it applies to all usage of dask `Array`s. Chunking choices have a _significant_ impact on performance for three reasons:\n",
|
||||
"1. Chunks are the unit of work during processing\n",
|
||||
"2. Chunks are the unit of transport in communicating information between workers\n",
|
||||
"3. Chunks are directly related to the number of _tasks_ being executed\n",
|
||||
"\n",
|
||||
"Performance is thus impacted in multiple ways - this is all about tradeoffs:\n",
|
||||
"* if chunks are too small, there will be too many _tasks_ and processing may be inefficient _BUT_\n",
|
||||
"* if chunks are too big, communication may be too long and the combined total of all chunks required for a calculation may exceed worker memory causing spilling to disk or worse, workers are killed\n",
|
||||
"\n",
|
||||
"It's not just size that matters either, the relative contiguity of dimensions matters:\n",
|
||||
"* Temporal processing is enhanced by larger chunks along the time dimension\n",
|
||||
"* Spatial processing is enhanced by larger chunks along the spatial dimensions, _BUT_\n",
|
||||
"* Earth Observation data can be sparse spatially, if chunks are too large spatially there will be a lot of empty chunks\n",
|
||||
"\n",
|
||||
"Thankfully it is possible to _re-chunk_ data for different stages of computation. Whilst _re-chunking is an *expensive* operation_ the efficiency gains for downstream computation can be very significant and sometimes are simply essential to support the numerical processing required. For example, it is often necessary to have a single chunk on the time dimension for temporal calculations.\n",
|
||||
"\n",
|
||||
"To understand the impact of chunking choices on _your code_ (it is very algorithm dependent) it is essential to understand both the:\n",
|
||||
"* _Static_ impact of chunking (e.g. task count, chunk size in memory), and;\n",
|
||||
"* _Dynamic_ impact of chunking (e.g. CPU load, thread utilisation, network communication, task count and scheduler load).\n",
|
||||
"\n",
|
||||
"`Dask` provides tools for viewing all of these when you print out arrays in the notebook (static) and when viewing the various graphs in the dask dashboard (dynamic)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "205583d7-7b6f-413a-819c-f3e2b6a1b0c9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Our example\n",
|
||||
"\n",
|
||||
"The code below will be familiar, it's the same example from previous notebooks (seasonal mean NDVI over a large area). A normalised burn ratio (NBR2) calculation has been added as well to provide some additional load to assist in making the performance differences more noticeable in various graphs. The NBR2 uses two additional bands but is effectively the same type of calculation as the NDVI (a normalised difference ratio).\n",
|
||||
"\n",
|
||||
"The primary difference for this example is the calculation (both NDVI and NBR) is performed 4 times, each with a different chunking regime. See the `chunk_settings` list.\n",
|
||||
"\n",
|
||||
"__When running this notebook, be sure to have the dask dashboard open and preferably visible as calculations proceed.__\n",
|
||||
"\n",
|
||||
"There are several sections to pay attention too:\n",
|
||||
"* Status\n",
|
||||
" * Short term snapshot of the Memory use (total and per worker - also shows Managed, Unmanaged and Spilled to Disk splits), Processing and CPU usage (change the Tab to switch between them)\n",
|
||||
" * Progress of the optimized Task graph\n",
|
||||
" * Near term Task Stream (Red is comms, White space is \"doing nothing\", other colours mostly match the tasks and you can hover over them with the mouse to get more information)\n",
|
||||
"* Tasks\n",
|
||||
" * Longer term Task Stream. This is a more comprehensive and accurate view of the Execution over time\n",
|
||||
"* System\n",
|
||||
" * Scheduler CPU, Memory and Communications load. You can zoom the graphs out using the control to get a longer term view.\n",
|
||||
"* Groups\n",
|
||||
" * High level view of the Task Graph Groups and their execution. The actual task graph is too detailed to display so this provides some insight into how high level aspects of your algorithm are executing.\n",
|
||||
"\n",
|
||||
"_All_ of these graphs are dynamic and should be interpreted over time.\n",
|
||||
"\n",
|
||||
"The dask _scheduler_ itself is also dynamic and as your code executes it stores information about how the tasks are executing and the communication occuring and adjusts scheduling accordingly. It can take a few minutes for the scheduler to settle into a true pattern. That pattern may also change, particularly in latter parts of a computation when work is completing and there are fewer tasks to execute.\n",
|
||||
"\n",
|
||||
"Yes, that is a LOT of information. Thankfully you don't necessarily need to learn it all at once. In time, reading the information available will become easier as will knowing what to do about it.\n",
|
||||
"\n",
|
||||
"Now let's run this notebook, remember to watch the execution in the Dask Dashboard.\n",
|
||||
"\n",
|
||||
"> __Tip__: It's likely you will want to repeat the calculation in this notebook several times. Because the results are `persisted` to the cluster simply calling it again will result in no execution (none is required because it was `persisted`). Rather than doing `cluster.shutdown()` and creating a new cluster each time you can clear the `persisted` result by performing a `client.restart()`. This will clear out all previous calculations so you can `persist` again. You can do this either by creating a new cell or using a Python Console for this Notebook (right click on the notebook and select _New Console for Notebook_)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fc395c36-548d-42bc-bd59-d4f390a6fc0a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a cluster\n",
|
||||
"A modest cluster will do... _and Open the dashboard_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1619a66a-c80f-4345-8022-a81018dc294f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Initialize the Gateway client\n",
|
||||
"from dask.distributed import Client\n",
|
||||
"from dask_gateway import Gateway\n",
|
||||
"\n",
|
||||
"number_of_workers = 5 \n",
|
||||
"\n",
|
||||
"gateway = Gateway()\n",
|
||||
"\n",
|
||||
"clusters = gateway.list_clusters()\n",
|
||||
"if not clusters:\n",
|
||||
" print('Creating new cluster. Please wait for this to finish.')\n",
|
||||
" cluster = gateway.new_cluster()\n",
|
||||
"else:\n",
|
||||
" print(f'An existing cluster was found. Connecting to: {clusters[0].name}')\n",
|
||||
" cluster=gateway.connect(clusters[0].name)\n",
|
||||
"\n",
|
||||
"cluster.scale(number_of_workers)\n",
|
||||
"\n",
|
||||
"client = cluster.get_client()\n",
|
||||
"client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ca446bdc-03b8-49bb-8489-13ad313a9a8f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup all our functions and query parameters\n",
|
||||
"\n",
|
||||
"Nothing special here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "008bc505-e1e7-44bf-9c95-d5bb7d96bda4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pyproj\n",
|
||||
"pyproj.set_use_global_context(True)\n",
|
||||
"\n",
|
||||
"import git\n",
|
||||
"import sys, os\n",
|
||||
"from dateutil.parser import parse\n",
|
||||
"from dateutil.relativedelta import relativedelta\n",
|
||||
"from dask.distributed import Client, LocalCluster, wait\n",
|
||||
"import datacube\n",
|
||||
"from datacube.utils import masking\n",
|
||||
"from datacube.utils.aws import configure_s3_access\n",
|
||||
"\n",
|
||||
"# EASI defaults\n",
|
||||
"os.environ['USE_PYGEOS'] = '0'\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults, notebook_utils\n",
|
||||
"easi = EasiDefaults()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1d8da7e0-5e0b-42a2-833b-ea60962581f4",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dc = datacube.Datacube()\n",
|
||||
"configure_s3_access(aws_unsigned=False, requester_pays=True, client=client)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a518ead6-750d-48ef-88f6-4fd4861eb939",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Get the centroid of the coordinates of the default extents\n",
|
||||
"central_lat = sum(easi.latitude)/2\n",
|
||||
"central_lon = sum(easi.longitude)/2\n",
|
||||
"# central_lat = -42.019\n",
|
||||
"# central_lon = 146.615\n",
|
||||
"\n",
|
||||
"# Set the buffer to load around the central coordinates\n",
|
||||
"# This is a radial distance for the bbox to actual area so bbox 2x buffer in both dimensions\n",
|
||||
"buffer = 1\n",
|
||||
"\n",
|
||||
"# Compute the bounding box for the study area\n",
|
||||
"study_area_lat = (central_lat - buffer, central_lat + buffer)\n",
|
||||
"study_area_lon = (central_lon - buffer, central_lon + buffer)\n",
|
||||
"\n",
|
||||
"# Data product\n",
|
||||
"product = easi.product('landsat')\n",
|
||||
"\n",
|
||||
"# Set the date range to load data over\n",
|
||||
"set_time = easi.time\n",
|
||||
"set_time = (set_time[0], parse(set_time[0]) + relativedelta(months=6))\n",
|
||||
"#set_time = (\"2021-07-01\", \"2021-12-31\")\n",
|
||||
"\n",
|
||||
"# Selected measurement names (used in this notebook)\n",
|
||||
"alias = easi.aliases('landsat')\n",
|
||||
"measurements = [alias[x] for x in ['qa_band', 'red', 'nir', 'swir1', 'swir2']]\n",
|
||||
"\n",
|
||||
"# Set the QA band name and mask values\n",
|
||||
"qa_band = alias['qa_band']\n",
|
||||
"qa_mask = easi.qa_mask('landsat')\n",
|
||||
"\n",
|
||||
"# Set the resampling method for the bands\n",
|
||||
"resampling = {qa_band: \"nearest\", \"*\": \"average\"}\n",
|
||||
"\n",
|
||||
"# Set the coordinate reference system and output resolution\n",
|
||||
"set_crs = easi.crs('landsat') # If defined, else None\n",
|
||||
"set_resolution = easi.resolution('landsat') # If defined, else None\n",
|
||||
"# set_crs = \"epsg:3577\"\n",
|
||||
"# set_resolution = (-30, 30)\n",
|
||||
"\n",
|
||||
"# Set the scene group_by method\n",
|
||||
"group_by = \"solar_day\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "71e6ed74-06b4-4450-a7cd-25e487909878",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def calc_ndvi(dataset):\n",
|
||||
" # Calculate the components that make up the NDVI calculation\n",
|
||||
" band_diff = dataset[alias['nir']] - dataset[alias['red']]\n",
|
||||
" band_sum = dataset[alias['nir']] + dataset[alias['red']]\n",
|
||||
" # Calculate NDVI\n",
|
||||
" ndvi = band_diff / band_sum\n",
|
||||
" return ndvi\n",
|
||||
"\n",
|
||||
"def calc_nbr2(dataset):\n",
|
||||
" # Calculate the components that make up the NDVI calculation\n",
|
||||
" band_diff = dataset[alias['swir1']] - dataset[alias['swir2']]\n",
|
||||
" band_sum = dataset[alias['swir1']] + dataset[alias['swir2']]\n",
|
||||
" # Calculate NBR2\n",
|
||||
" nbr2 = band_diff / band_sum\n",
|
||||
" return nbr2\n",
|
||||
"\n",
|
||||
"def mask(dataset, bands):\n",
|
||||
" # Identify pixels that are either \"valid\", \"water\" or \"snow\"\n",
|
||||
" cloud_free_mask = masking.make_mask(dataset[qa_band], **qa_mask)\n",
|
||||
" # Apply the mask\n",
|
||||
" cloud_free = dataset[bands].astype('float32').where(cloud_free_mask)\n",
|
||||
" return cloud_free\n",
|
||||
"\n",
|
||||
"def seasonal_mean(dataset):\n",
|
||||
" return dataset.resample(time=\"QS-DEC\").mean('time') # perform the seasonal mean for each quarter"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6d30531-5747-4b70-9e47-aabd41f60b2e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We have an array of chunk settings to trial.\n",
|
||||
"\n",
|
||||
"* Notice the `chunk_settings` are nominally the same size\n",
|
||||
"* Notice we're varying the temporal chunking from large to small and adjusting the spatial chunking to keep the overall volume similar (nominally this will be 100 Megs per chunk for the original dataset)\n",
|
||||
"\n",
|
||||
"There are two `time:1` chunks because 50 doesn't have a clean sqrt. The first is the nearest square, the second simply changes the chunks to be rectangles (no one said the spatial dimensions needed to be the same).\n",
|
||||
"\n",
|
||||
"Given the chunk size in memory is roughly the same, the cluster the same, the calculation the same - any differences in execution are a result of the different chunking shape."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1ade0fc8-49a8-43e6-997a-60bae9d5c9c5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"chunk_settings = [\n",
|
||||
" {\"chunks\": {\"time\":100, \"x\":300, \"y\":300}, \"comment\": \"This run has small spatial chunks but each chunk has a lot of time steps. This results in many small file reads, but there are more total tasks for the scheduler to handle.\"},\n",
|
||||
" {\"chunks\": {\"time\":50, \"x\":1*300, \"y\":2*300}, \"comment\": \"This second run has slightly larger spatial chunks but smaller temporal extents in each chunk. This results in fewer total tasks, but each one takes longer to load.\"},\n",
|
||||
" {\"chunks\": {\"time\":1, \"x\":10*300, \"y\":10*300}, \"comment\": \"This run has only a single time step in each chunk, but large, square spatial extents. As a result, workers need to store much more data in memory and some data is spilled to disk.\"},\n",
|
||||
" {\"chunks\": {\"time\":1, \"x\":21*300, \"y\":5*300}, \"comment\": \"Again this run has a single time step per chunk, but the spatial extents are rectangles.\"},\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65b4ff1b-d512-4b6d-9f32-8fb0eeea9398",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we can loop over all our `chunk_settings` and create all the required `delayed task graphs`. This will take a moment as the ODC database will be interogated for all the necessary dataset information.\n",
|
||||
"\n",
|
||||
"_You will notice the calculation is split up so we can see the interim results_ - well the last one at least given its a loop and we're overwriting them.\n",
|
||||
"\n",
|
||||
"Different stages of computation will produce different data types and calculations and thus _chunk_ and _task_ counts. We may find that an interim result has a terrible chunk size (e.g. `int16` data variables become `float64` and thus your chunks are now 4x the size, or a dimension is reduced and chunks are too small). It is thus advisable when tuning to make it possible to view these interim stages to see the _static_ impact.\n",
|
||||
"\n",
|
||||
"__Remember__: there is a single task graph executing to provide the final result. There is no need to `persist()` or `compute()` the interim results to see their _static_ attributes. In fact, it may be unwise to `persist()` as this will chew up resources on the cluster if you don't intend on using the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "89eb9a27-c0d7-44d2-9823-e7f994ddf135",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for chunkset in chunk_settings:\n",
|
||||
" chunks = chunkset[\"chunks\"]\n",
|
||||
" print(chunks)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "e4c2276f-71be-4ac6-b29b-5e9ef96ab69c",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"results = []\n",
|
||||
"for chunkset in chunk_settings:\n",
|
||||
" chunks = chunkset[\"chunks\"]\n",
|
||||
" dataset = dc.load(\n",
|
||||
" product=product,\n",
|
||||
" x=study_area_lon,\n",
|
||||
" y=study_area_lat,\n",
|
||||
" time=set_time,\n",
|
||||
" measurements=measurements,\n",
|
||||
" resampling=resampling,\n",
|
||||
" output_crs=set_crs,\n",
|
||||
" resolution=set_resolution,\n",
|
||||
" dask_chunks = chunks,\n",
|
||||
" group_by=group_by,\n",
|
||||
" )\n",
|
||||
" \n",
|
||||
" num_time = dataset.sizes['time']\n",
|
||||
" time_ind = np.linspace(1, num_time, 100, dtype='int') - 1\n",
|
||||
" dataset = dataset.isel(time=time_ind) # load exactly 100 evenly spaced timesteps so that we can work more easily with different chunks\n",
|
||||
"\n",
|
||||
" masked_dataset = mask(dataset, [alias[x] for x in ['red', 'nir', 'swir1', 'swir2']])\n",
|
||||
" ndvi = calc_ndvi(masked_dataset)\n",
|
||||
" nbr2 = calc_nbr2(masked_dataset)\n",
|
||||
" seasonal_mean_ndvi = seasonal_mean(ndvi)\n",
|
||||
" seasonal_mean_nbr2 = seasonal_mean(nbr2)\n",
|
||||
" seasonal_mean_ndvi.name = 'ndvi'\n",
|
||||
" seasonal_mean_nbr2.name = 'nbr2'\n",
|
||||
" results.append([seasonal_mean_ndvi, seasonal_mean_nbr2])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9ac8a059-82d9-4188-8878-e7cc1dec7efe",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Inspecting _static_ information\n",
|
||||
"\n",
|
||||
"Lets take a look at the vital statistics for the final iteration of the loop. All the calculations are the same, just the `chunk` parameters vary so we can infer easily from these what else is happening for the _static_ parameters.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ffb6bc3b-9872-4b11-8599-ba31f1ade74b",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(f\"dataset size (GiB) {dataset.nbytes / 2**30:.2f}\")\n",
|
||||
"print(f\"seasonal_mean_ndvi size (GiB) {seasonal_mean_ndvi.nbytes / 2**30:.2f}\")\n",
|
||||
"display(dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "31183e67-0f5c-40d8-aee1-13a8f73da9a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"So the source `dataset` is 150 GB in size - mostly `int16` data type. _We need to be mindful that our calculation will convert these to `floats`._ The code above does an explicit type conversion to `float32` which can fully represent an `int16`. Without the explicit type conversion, Python would use `float64` resulting in double the memory usage for no good reason (for this algorithm).\n",
|
||||
"\n",
|
||||
"Open the _cylinder_ to show the `red` dask array details. The chunk is about 100 MiB in size. Generally this is a healthy size though it can be larger and may need to be smaller depending on the calculation involved and communication between workers.\n",
|
||||
"\n",
|
||||
"Now let's look at the results for the NDVI and NBR:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "10ab0c2f-05f8-4ca7-8dd8-789c63d143df",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"display(results[0][0])\n",
|
||||
"display(results[0][1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cde0a747-5629-48d3-b0c7-de9035c96f01",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Notice the result is _much smaller_ in chunk size - 4 MiB. This is due to the seasonal mean. This may have an impact on downstream usage of the result as the _chunks_ may be too small and result in too many tasks reducing later processing performance.\n",
|
||||
"\n",
|
||||
"Notice also the Task count. With both results we're pushing towards 100_000 tasks in the scheduler depending on task graph optimisation. The Scheduler has its own overheads (about 1ms per active task, and memory usage for tracking all tasks, including executed ones as it keeps the history in case it needs to reproduce the results e.g. if a worker is lost). Again, it is possible to have more than 100_000 tasks and be efficient depending on your algorithm but its something to keep an eye on. We will be below it in this case (especially after optimisation)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ee97957f-7959-422d-b9ae-a47750aa4b50",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Persist the results\n",
|
||||
"\n",
|
||||
"Theoretically we could `persist` all of the `results` at once - though we would be well above the 100_000 task limit if we did.\n",
|
||||
"More importantly we actually want to see the difference in the _dynamics_ of the execution.\n",
|
||||
"The loop below will persist each result one at a time and _wait()_ for it to be complete.\n",
|
||||
"\n",
|
||||
"__You should monitor execution in the Dask Dashboard__\n",
|
||||
"\n",
|
||||
"Look at the various tabs as execution proceeds. you will notice differences in memory per worker, Communication between workers (red bars in the Task Stream), white space (idle time), and CPU utilisation (remember to click on the CPU tab to get to this detail).\n",
|
||||
"The `Tasks` section of the dashboard is particularly useful at looking at a comparison of all four runs' dynamics as the length of all calculations means this snapshot still show all four blocks of computation at once.\n",
|
||||
"\n",
|
||||
"Don't forget, if you want to run the code again use `client.restart()` to clear out the previous results from the cluster.\n",
|
||||
"\n",
|
||||
">__Tip:__ If you leave your computer while this step is running, make sure that it doesn't go to sleep by adjusting your power settings."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "43a306aa-86e7-432a-bdfc-b4ecf20e8aa0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# client.wait_for_workers(n_workers=number_of_workers) # Before release 2023.10.0\n",
|
||||
"client.sync(client._wait_for_workers,n_workers=number_of_workers) # Since release 2023.10.0\n",
|
||||
"\n",
|
||||
"for i, result in enumerate(results):\n",
|
||||
" print(f'Run number {i+1}:')\n",
|
||||
" print(f'Chunks: {chunk_settings[i][\"chunks\"]}')\n",
|
||||
" print(chunk_settings[i][\"comment\"])\n",
|
||||
" client.restart()\n",
|
||||
" f = client.persist(result)\n",
|
||||
" %time wait(f)\n",
|
||||
" client.restart() # clearing the cluster out so each run it cleanly separated\n",
|
||||
" print()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "322e2de7-2e11-4edf-bf14-6ca1d0b81a6a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Understanding the dynamics\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cfcad62-a6e4-4504-9e41-d33f872ade7a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Be a good dask user - Clean up the cluster resources"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "66acca97-686c-4d09-ba5b-a9aeede4bfba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Disconnecting your client is good practice, but the cluster will still be up so we need to shut it down as well"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0578682e-917e-49b7-9f3e-88584a572936",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client.close()\n",
|
||||
"\n",
|
||||
"cluster.shutdown()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "3de2e63e-138d-401e-944d-27520d3ad8b9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user