thêm chức năng train trên odc predict trên planetary
This commit is contained in:
Victor Phan
@@ -0,0 +1,458 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1f47c2fa-32af-4ff1-8563-35605ee42edb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Data storage and locality <img align=\"right\" src=\"../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"We have the potential to use and generate huge volumes of data, in different formats, for different purposes and across multiple projects and collaborations.\n",
|
||||
"\n",
|
||||
"How we store this data can have significant impacts on:\n",
|
||||
"\n",
|
||||
"- Our compute efficiency and science productivity\n",
|
||||
"- Our ability to connect with other researchers\n",
|
||||
"- Our budget\n",
|
||||
"- Our cybersecurity and\n",
|
||||
"- Our commitment to Research Data Management policies, procedures, and practices\n",
|
||||
"\n",
|
||||
"So, how do we choose the right storage solution?\n",
|
||||
"\n",
|
||||
"This module will help you explore the different storage options on EASI, their use-cases, their limitations and how you can get started using them. By the end of this module, you will be able to:\n",
|
||||
"\n",
|
||||
"- Identify the different available storage options in EASI and recognize:\n",
|
||||
" - When to use one or more storage options\n",
|
||||
" - Considerations associated with these storage options (cost, security, latency etc)\n",
|
||||
"- Upload data to use with your Jupyter Notebook and where to store data outputs for download or re-use by yourself or others.\n",
|
||||
"- Access resources to help build skills and knowledge to further your learning around data storage options\n",
|
||||
"\n",
|
||||
"**Contents**:\n",
|
||||
"- [How do we normally store data?](#How-do-we-normally-store-data?)\n",
|
||||
"- [Different storage types in EASI](#Different-storage-types-in-EASI)\n",
|
||||
"- [Decision making matrix](#Decision-making-matrix)\n",
|
||||
"- [Scenarios](#Scenarios)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "448eaa6b-f22c-43c5-9bb1-b682180a5a92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How do we normally store data? \n",
|
||||
"\n",
|
||||
"If you have been using a computer, chances are that you are familiar with storing data on your home drive, or on an external hard drive, or maybe even on the cloud. This is typically referred to as **File Storage**.\n",
|
||||
"\n",
|
||||
"With file storage, all data is stored together (usually in one file) and the file extension type helps determine the applications that can read or open the file and access the data (e.g., .jpg, .docx, or .txt).\n",
|
||||
"\n",
|
||||
"File storage systems also make it easier for users to find and manage files using a hierarchical structure that organizes files into folders and subfolders. To access the file, select or enter the path of the file, including subdirectories and file name. Most users manage their file storage using a simple file system such as a file manager.\n",
|
||||
"\n",
|
||||
"There are other alternatives to the everyday local file storage, like network file storage (connect to the network server first then access by path/file) and *Object Storage*, which is most relevant to cloud-computing.\n",
|
||||
"\n",
|
||||
"**Object storage** is like a giant virtual bucket where you can store your belongings in labelled boxes. Each box has a unique ID and can hold different types of objects. You can easily add or remove boxes as needed, and the bucket can expand to accommodate more and more boxes. Plus, you can access any box from anywhere in the world, if you have the ID and access permission. Object storage is ideal for storing vast amounts of unstructured data, such as videos, images, and documents, and is commonly used in cloud storage services. You can have many Buckets where you can store vast amounts of your data and they can all have different access permissions and life cycles. \n",
|
||||
"\n",
|
||||
"EASI supports many types of data storage, local and network file storage and Object Storage, and each method has certain advantages, limitations and requirements. EASI also manages data for all users in the Open Data Cube, and there are third-party datasets accessible directly in the Cloud. \n",
|
||||
"\n",
|
||||
"Some additional terms that may be helpful when talking about Data Storage:\n",
|
||||
"\n",
|
||||
"**Latency**\n",
|
||||
"\n",
|
||||
"Data latency is the time it takes for data packets to be stored or retrieved. \n",
|
||||
"\n",
|
||||
"**Data Retrieval**\n",
|
||||
"\n",
|
||||
"The process of identifying and extracting data from a database or storage system, based on a query provided by the user or application. It enables the fetching of data from storage in order to display it on a monitor and/or use within an application. \n",
|
||||
"\n",
|
||||
"**Cache** \n",
|
||||
"\n",
|
||||
"A high-speed memory or storage device that helps reduce the time required to read and write data to a slower device, such as a hard drive or a remote server (or bucket)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d0b692c6-e989-418b-86de-908818af7891",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Different storage types in EASI\n",
|
||||
"\n",
|
||||
"- [Type 1: Home Directory](#Home-Directory)\n",
|
||||
"- [Type 2: DataCube](#DataCube)\n",
|
||||
"- [Type 3: User Scratch](#User-Scratch)\n",
|
||||
"- [Type 4: Project Data](#Project-Data)\n",
|
||||
"- [Type 5: Cloud Data](#Cloud-Data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "981c0d96-5f23-443e-8c17-88a8f7ab6ca7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Home Directory\n",
|
||||
"\n",
|
||||
"**What is the Home Directory?**\n",
|
||||
"\n",
|
||||
"The Home Directory uses the Amazon Web Service (AWS) Elastic File System (EFS) storage architecture. This is similar to a normal network file system in that it has folders, sub folders and files and it is easy and familiar to navigate and use.\n",
|
||||
"\n",
|
||||
"**When would I use the Home Directory to store data?**\n",
|
||||
"\n",
|
||||
"The Home Directory is best used for small files including source code and notebooks and smaller sets of data outputs.\n",
|
||||
"\n",
|
||||
"You can also use the Home Directory to store files that cannot readily work from other storage options (i.e., files or programs that require a normal file system). Again, if these are large, then you may wish to find an alternative and your EASI Administrator will be happy to advise.\n",
|
||||
"\n",
|
||||
"**What are the advantages of the Home Directory?**\n",
|
||||
"\n",
|
||||
"The data housed in your Home Directory is persistent through log-in/log-out cycles (your home directory will be as you left it). The home directory is also automatically backed-up nightly and retained for 90 days. \n",
|
||||
"\n",
|
||||
"The EASI Admin team can restore a file. However this is not always guaranteed. If you are looking for more stable and long-term storage, there are better solutions out there! \n",
|
||||
"\n",
|
||||
"**What are the limitations of the Home Directory?**\n",
|
||||
"\n",
|
||||
"This is the most expensive form of storage available on EASI. It is ok to store a Gigabyte or so of data, code and outputs. For larger datasets and outputs there are other EASI storage types available. \n",
|
||||
"\n",
|
||||
"Note that your home directory is not visible to *Dask* workers, as they run in a different part of EASI. We will learn about the Python *Dask* library later in the module.\n",
|
||||
"\n",
|
||||
"**How do I upload and use data?**\n",
|
||||
"\n",
|
||||
"There’s a video of this and information in the [Jupyter labs documentation](https://jupyterlab.readthedocs.io/en/stable/user/files.html).\n",
|
||||
"\n",
|
||||
"- For small files, it is simplest to drag-and-drop your files into the JupyterLab file interface directly from your desktop. This will upload the file via your browser into your home directory.\n",
|
||||
"- For larger files, it's best to use the AWS CLI running locally along with your EASI credentials and upload directly to your [EASI S3 User Scratch space](#User-Scratch) space.\n",
|
||||
"\n",
|
||||
"Example usage and EASI documentation: https://docs.csiro.easi-eo.solutions/user-guide/users-guide/07-uploading-data/ \n",
|
||||
"\n",
|
||||
"**Tasks**\n",
|
||||
"\n",
|
||||
"Open an EASI JupyterLab session and a separate browser tab to: \n",
|
||||
"\n",
|
||||
"- [ ] Familiarise yourself with the JupyterLabs documentation.\n",
|
||||
"- [ ] Upload some of your own data or file to your JupyterLab home directory."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5dc7e716-27ab-4b53-ab4c-ae9cd7e4697e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### DataCube\n",
|
||||
"\n",
|
||||
"**What is DataCube?**\n",
|
||||
"\n",
|
||||
"The [Open Data Cube (ODC)](https://github.com/opendatacube) is an Open-Source Geospatial Data Management and Analysis Software project that helps you harness the power of Satellite data. At its core, the ODC is a set of Python libraries and PostgreSQL database that helps you work with geospatial raster data.\n",
|
||||
"\n",
|
||||
"The broad goal of the ODC is to make it easier to access and use large data holdings, without requiring data to be stored in a specific way or in a specific place. What this means is that you can point it at your data repository and index the data where it sits, abstracting the complexity of managing large, distributed data holdings.\n",
|
||||
"\n",
|
||||
"**EASI managed data**\n",
|
||||
"\n",
|
||||
"EASI manages an amount of data in its datacube databases. These can be viewed in each EASI's Explorer website and with the *Datacube()* API.\n",
|
||||
"\n",
|
||||
"- This is the CSIRO Explorer site: https://explorer.csiro.easi-eo.solutions/\n",
|
||||
"- The EASI documentation [lists the URLs for other EASIs](https://docs.csiro.easi-eo.solutions/user-guide/developers/easi-platform-overview/#easi-deployments-services-and-support).\n",
|
||||
"\n",
|
||||
"There are many commonalities in the data processing workflows (into a datacube databases) across the ODC community. We do this so that its easier to move your work between ODC systems (including third-parties). The EASI team are working to make our data processing workflows transparent and open to contributions.\n",
|
||||
"\n",
|
||||
"**When would I use the DataCube**\n",
|
||||
"\n",
|
||||
"If you’re using data that’s available as EASI managed data, then it’s much easier to use via the ODC API and you don’t have to have your own copy! See the [easi-notebooks](https://github.com/csiro-easi/easi-notebooks) for getting started tutorials.\n",
|
||||
"\n",
|
||||
"**What are the advantages of the DataCube?**\n",
|
||||
"\n",
|
||||
"DataCube is managed for you! So, you don’t need to worry about updates, maintenance, storage costs etc. Many EO and large public data collections are available, or becoming available, via the cloud. EASI and ODC connects directly to the current best collection endpoints and API services, and utilises cloud efficiencies and capabilities where possible.\n",
|
||||
"\n",
|
||||
"Many of the data collections and processing routines are open source, and are contributed to by the ODC, research and domain-coordination communities. EASI leverages and contributes to these open resources to enure our users stay up to date. The EASI team welcome suggestions and contributions to the development of current or new data processing workflows.\n",
|
||||
"\n",
|
||||
"**What are the limitations of the DataCube?**\n",
|
||||
"\n",
|
||||
"It may not have the specific dataset you are looking for. However datasets are always being added and updated, and you can build your own and share it!\n",
|
||||
"\n",
|
||||
"The Datacube API may not suit your overall workflow. Just remember that the datacube shows “one way to do it”. Talk to the EASI team about your workflow and we can advise what others options are available or could be combined. For example, you can access the list of files (rather than the data) that the datacube API would return for your query, and you can likely access the same sources that EASI’s workflows do.\n",
|
||||
"\n",
|
||||
"**How do I upload and use data?**\n",
|
||||
"\n",
|
||||
"Getting start with using data from, and uploading to, the Open Data Cube is straightforward. You can explore the tutorial notebooks in the git repository: https://github.com/csiro-easi/easi-notebooks.\n",
|
||||
"\n",
|
||||
"**Tasks**\n",
|
||||
"\n",
|
||||
"Open an EASI JupyterLab session to:\n",
|
||||
"\n",
|
||||
"- [ ] Upload a shapefile (copy from some resource or choose your own)\n",
|
||||
"- [ ] Load EO datacube into an xarray\n",
|
||||
"- [ ] Mask with an uploaded shapefile"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f13ce0ac-7d78-44bf-930d-2beaf66a4add",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### User Scratch\n",
|
||||
"\n",
|
||||
"**What is User Scratch?**\n",
|
||||
"\n",
|
||||
"EASI has a \"scratch\" bucket available for all EASI users to write to. Scratch storage serves as a temporary location for large data sets that are more appropriate to save and access from AWS S3 storage. These could be intermediate datasets in your workflow or interim results for testing and exploration.\n",
|
||||
"\n",
|
||||
"**When would I use the User Scratch?**\n",
|
||||
"\n",
|
||||
"The User Scratch bucket is helpful to save files between processing runs or share files between your projects. You can use Scratch as large, efficient temporary storage for your workflows and projects, e.g. for saving intermediate results from your workflows.\n",
|
||||
"\n",
|
||||
"Scratch can be accessed by *Dask* workers for passing data and saving snapshots of results, which is an advantage over Home Directory storage. Don’t know what *Dask* is? Don’t worry, we’ll cover Dask in detail.\n",
|
||||
"\n",
|
||||
"**What are the advantages of the User Scratch?**\n",
|
||||
"\n",
|
||||
"It’s a dedicated S3 Bucket in EASI that provides:\n",
|
||||
"\n",
|
||||
"- Lots of storage and scale (efficient)\n",
|
||||
"- Works with Dask workers (fast)\n",
|
||||
"- Has a managed lifetime, in case you forget it will (eventually) clean up after you\n",
|
||||
"- You can use the AWS CLI and library functions to read and write files to this bucket\n",
|
||||
"\n",
|
||||
"**What are the limitations of the User Scratch?**\n",
|
||||
"\n",
|
||||
"User Scratch is *Object Storage*, which is a bit different to file storage, so it requires a different method to access and manage the data. Many tools support efficient read/write to cloud object storage directly but some older tools that you use may not. If so, use the AWS CLI to make a temporary copy of the files to your JupyterLab home directory or to dask workers.\n",
|
||||
"\n",
|
||||
"Secondly, objects in the user scratch bucket have a 30-day (since date of creation) lifecycle rule. It is not intended for long-term data storage. Similarly, it is also not a practical solution if you need to share files and resources between projects, particularly beyond 30 days. If sharing data or a different lifecycle is a project requirement, then consider provisioning a \"Project\" bucket tailored to your needs.\n",
|
||||
"\n",
|
||||
"**How do I upload and use data?**\n",
|
||||
"\n",
|
||||
"User guide / additional information:\n",
|
||||
"\n",
|
||||
"- https://docs.csiro.easi-eo.solutions/user-guide/users-guide/07-uploading-data/\n",
|
||||
"- https://github.com/csiro-easi/easi-notebooks\n",
|
||||
"\n",
|
||||
"**Tasks**\n",
|
||||
"\n",
|
||||
"- [ ] Save your xarray masked result (from previous step) to User Scratch"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "129ac062-fc83-46b5-9ac0-a244dfb10da1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Project Data\n",
|
||||
"\n",
|
||||
"**What is Project Data?**\n",
|
||||
"\n",
|
||||
"Like the “User Scratch bucket”, which all EASI users can access, each project can also make additional storage resources available for their users in EASI. This is known as a **Project** bucket, and it can be shared across all project members and have different levels of membership (read/write, read-only).\n",
|
||||
"\n",
|
||||
"Create and manage your own AWS account, and then create a bucket that can be shared with EASI. The EASI team will coordinate with you to enable cross-account authorisation between EASI and your shared project bucket. You retain full control of your AWS resources. Contact EASI for more information.\n",
|
||||
"\n",
|
||||
"**When would I use Project Data?**\n",
|
||||
"\n",
|
||||
"Project Data is best used when you need to share data and results with collaborators in a project, or when you want to manage your own large data in your own project account with your own lifecycle rules.\n",
|
||||
"\n",
|
||||
"**What are the advantages?**\n",
|
||||
"\n",
|
||||
"From EASI, a Project bucket is only accessible by users who have been nominated as a member of the *project*. This allows sharing with colleagues who are part of the same project.\n",
|
||||
"\n",
|
||||
"The project's AWS account administrator can create custom lifecycle rules and can enable other services that might be appropriate for the project.\n",
|
||||
"\n",
|
||||
"**What are the limitations?**\n",
|
||||
"\n",
|
||||
"Only accessible to people that have access to this storage, i.e. colleagues as part of the same project, which is the intent.\n",
|
||||
"\n",
|
||||
"**How do I upload and use data?**\n",
|
||||
"\n",
|
||||
"The methods for uploading or reading data from a project bucket are the same as for the User Scratch bucket. Just change the bucket name!\n",
|
||||
"\n",
|
||||
"**User guide / additional information:**\n",
|
||||
"\n",
|
||||
"In CSIRO, contact the Cloud Platforms Team to set up your project AWS account. Then contact the EASI team to enable cross-account authorisation and to nominate users and their read/write permissions.\n",
|
||||
"\n",
|
||||
"**Tasks**\n",
|
||||
"\n",
|
||||
"- [ ] If you have a project bucket then use the same approach as for User Scratch to upload your xarray result to the project bucket.\n",
|
||||
"- [ ] If not, then the only task is to remember that this option exists!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cee6dfc5-3405-4262-9d3f-1cdbc99e27a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Cloud Data\n",
|
||||
"\n",
|
||||
"**What is cloud-available data and what cloud data services / technologies does EASI support?\n",
|
||||
"\n",
|
||||
"The “cloud” in this context refers to the main global cloud data companies – Amazon Web Services, Google Earth Engine and Microsoft Planetary Computer. However, the ideas mentioned here are also reasonably applicable to many internet-available datasets, although the specific way to access these may differ.\n",
|
||||
"\n",
|
||||
"Many Earth observation and other public data collection providers are making their data available directly in the cloud. In many cases the cloud copy is their ‘authoritative’ public copy of the data. The same data collection may also be available in more than one cloud system.\n",
|
||||
"\n",
|
||||
"The amount and variety of data and providers is vast, and we can’t cover them all here but will highlight that they can in most cases be accessed directly from EASI.\n",
|
||||
"\n",
|
||||
"**When would I use cloud-based data assets?**\n",
|
||||
"\n",
|
||||
"When a provider has data you want and its just as efficient to access it directly as it is to download and store (why bother curating it yourself!)\n",
|
||||
"\n",
|
||||
"**What are the advantages of cloud-based data assets?**\n",
|
||||
"\n",
|
||||
"- You don’t pay for the long-term storage\n",
|
||||
"- Maintained by the data provider\n",
|
||||
"- There’s a lot of it\n",
|
||||
"\n",
|
||||
"Many of the EASI-managed datacube products are indexed directly or indirectly from the cloud.\n",
|
||||
"\n",
|
||||
"**What are the limitations of cloud-based data assets?**\n",
|
||||
"\n",
|
||||
"Cost and efficiency mostly. While these cloud data collections are publicly available they are not necessarily free to access. The cloud companies typically charge network data transfer and related costs associated for moving data within, into and out of their systems.\n",
|
||||
"\n",
|
||||
"Additionally, accessing data over the internet can slow workflow speed and repeated use in workflows will result in more network costs (possibly more than the cost of downloading the data and temporarily storing it). All these need to be managed.\n",
|
||||
"\n",
|
||||
"For small, infrequent access (like one off grabs of data) you can safely just use them. For large data use or frequent re-use then if you are unsure contact your EASI Admin and we’ll help you navigate. It’s not difficult, just awkward at the start to understand what to watch for.\n",
|
||||
"\n",
|
||||
"**How do I access and use cloud-based data?**\n",
|
||||
"\n",
|
||||
"This varies depending on the data source and we can’t cover them all. However, there are common considerations highlighted below to help you think about how to use cloud-based data. Each cloud data company provides a list or summary of their available public data collections. It is important to be able to locate these and to identify and interpret some of the key points.\n",
|
||||
"\n",
|
||||
"- [Registry of Open Data on AWS](https://registry.opendata.aws/)\n",
|
||||
"- [Earth Engine Data Catalog](https://developers.google.com/earth-engine/datasets)\n",
|
||||
"- [Planetary Computer Data Catalog](https://planetarycomputer.microsoft.com/catalog)\n",
|
||||
"\n",
|
||||
"The OpenDataCube provides a tool that can read from the cloud data collections and return an xarray object as if the data had been read from a datacube database. Examples are provided here: https://odc-stac.readthedocs.io/en/latest/examples.html\n",
|
||||
"\n",
|
||||
"**Tasks**\n",
|
||||
"\n",
|
||||
"- [ ] Let's look at a data catalog example from AWS: https://registry.opendata.aws/sentinel-2-l2a-cogs/\n",
|
||||
"\n",
|
||||
"<img align=\"center\" src=\"../resources/aws-opendata-markup.png\">"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ee4e2480-5702-4326-a15a-2276d17c1965",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Decision making matrix\n",
|
||||
"\n",
|
||||
"This table summarises the key attributes of the different storage types.\n",
|
||||
"\n",
|
||||
"<table>\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <th> \n",
|
||||
" <th colspan=2>User access rights\n",
|
||||
" <th colspan=2>Dask workers access rights\n",
|
||||
" <th colspan=2>Collaborator access rights\n",
|
||||
" <th colspan=2>Suitable file size\n",
|
||||
" <th colspan=2>Network costs\n",
|
||||
" <th colspan=2>Storage costs\n",
|
||||
" <th colspan=2>Shelf life\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td> \n",
|
||||
" <td><em>Read<td><em>Write\n",
|
||||
" <td><em>Read<td><em>Write\n",
|
||||
" <td><em>Read<td><em>Write\n",
|
||||
" <td><em>Large<td><em>Small\n",
|
||||
" <td><em>High<td><em>Low\n",
|
||||
" <td><em>High<td><em>Low\n",
|
||||
" <td><em>Short term<td><em>Long term\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td><a href=\"#Home-directory\">Home directory</a><td>Yes<td>Yes<td>No<td>No<td>No<td>No<td> <td>X<td> <td>X<td>X<td> <td>X<td>X\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td><a href=\"#User-scratch\">User scratch</a><td>Yes<td>Yes<td>Yes<td>Yes<td>No<td>No<td>X<td>X<td> <td>X<td> <td>X<td>X<td> \n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td><a href=\"#Project-data\">Project data</a><td>Yes<td>Yes<td>Yes<td>Yes<td>Yes<td>Yes<td>X<td>X<td> <td>X<td> <td>X<td>X<td>X\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td><a href=\"#Data-cube\">Data cube</a><td>Yes<td>No<td>Yes<td>No<td>Yes<td>No<td>X<td>X<td> <td>X<td> <td>X<td> <td>X\n",
|
||||
"<tr align=\"center\">\n",
|
||||
" <td><a href=\"#Cloud-data\">Cloud data</a><td>Yes<td>No<td>Yes<td>No<td>Yes<td>No<td>X<td>X<td>X<td>X<td> <td>X<td>X<td>X\n",
|
||||
"</table>\n",
|
||||
"\n",
|
||||
"Note that the specifics for any cloud data sources can vary depending on the provider, where the data is stored, where it is being accessed and more. For an in-depth conversation, reach out to the EASI team.\n",
|
||||
"\n",
|
||||
"<!--\n",
|
||||
"Markup version - could not find a way to do colspan\n",
|
||||
"| | User access rights | | Dask workers access rights | | Collaborator access rights | | Suitable file size | | Network costs | | Storage cost | | Shelf life | |\n",
|
||||
"|--|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n",
|
||||
"| | *Read* | *Write* | *Read* | *Write* | *Read* | *Write* | *Large* | *Small* | *High* | *Low* | *High* | *Low* | *Short term* | *Long term* |\n",
|
||||
"| [User scratch](#User-scratch) | Yes | Yes | No | No | No | No | | X | | X | X | | X | X |\n",
|
||||
"-->"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fdd2703e-24b1-4d56-8c48-7d7accf8b41d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Scenarios\n",
|
||||
"\n",
|
||||
"### Scenario 1 - Home directory and ODC\n",
|
||||
"\n",
|
||||
"Michael has some small data files that he wants to upload and use in conjunction with datacube data. Where is the best place for him to upload this data?\n",
|
||||
"\n",
|
||||
"*Answer*: Home directory\n",
|
||||
"\n",
|
||||
"The Home Directory is great for smaller data files that you need for your work. It is easy to upload small files from your desktop with drag-and-drop in JupyterLab. There are many python notebook examples (to borrow or adapt from) that query the datacube API (which returns an xarray object) then combine with other data (such as shapefiles, polygons or non-datacube data) with standard python tools.\n",
|
||||
"\n",
|
||||
"For larger files and datasets you may consider uploading to User Scratch or a Project Data bucket."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5628209c-9a27-4eb1-b210-d9512e91664d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Scenario 2 - User scratch and Project data buckets\n",
|
||||
"\n",
|
||||
"Jenny is generating a set of intermediate or final data files from her workflow. The total size is in the Gigabytes. Where should Jenny write the data files to?\n",
|
||||
"\n",
|
||||
"*Answer*: User scratch or Project data (if applicable)\n",
|
||||
"\n",
|
||||
"*Extended response*:\n",
|
||||
"\n",
|
||||
"S3 bucket storage is preferred for data sets larger than a few GBs. Workflows (notebooks and dask workers) can read and write to S3 buckets if permitted. All EASI users have access to the User Scratch bucket. Some users may also have access to a Project data bucket, in which case the data can also be used by project colleagues.\n",
|
||||
"\n",
|
||||
"S3 bucket storage is preferred for larger datasets because it is cheaper, efficient and programmable from notebooks and dask workers. Your home directory is more expensive and not accessible from dask workers, so is less suitable for workflows generating or using large datasets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "42a61595-436a-41e6-91d2-402eca49288d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Scenario 3: Cloud or external online data\n",
|
||||
"\n",
|
||||
"Catherine would like to access a dataset that is available in the cloud or from an external online service. She checks Explorer and notes that the dataset is not available in the datacube database. What choices does Catherine have?\n",
|
||||
"\n",
|
||||
"*Answer*: Consider the details for accessing these Cloud data in EASI and seek advice if required.\n",
|
||||
"\n",
|
||||
"*Extended response*:\n",
|
||||
"\n",
|
||||
"Typical external data access considerations include:\n",
|
||||
"\n",
|
||||
"- Whether there is a programmable request interface, e.g. STAC API, an OGC web service, or a custom API. For STAC APIs, you can use the odc-stac tool to query and read the data into an xarray object. Similarly, there are common python tools for reading from ODC web services. A custom API will likely need specific code to search, download and prepare the data for use.\n",
|
||||
"- Where the data are located. Cloud data services (including EASI’s host) often charge for each data transaction into and out of their data centres, which add up. Moving data across the internet can be slow as well. Ask the EASI team for advice.\n",
|
||||
"- Is this a candidate dataset for including in the datacube database? If yes, then a great start is to document how this dataset can be used by you and others, the dataset source and metadata, and some example code for accessing and using the dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "d3bd1817-31f7-455a-ade6-57822792837c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Reference in New Issue
Block a user