thêm chức năng train trên odc predict trên planetary
This commit is contained in:
Victor Phan
@@ -0,0 +1,373 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Using EASI scratch and project buckets <img align=\"right\" src=\"../resources/csiro_easi_logo.png\">\n",
|
||||
"\n",
|
||||
"EASI has a **Scratch** bucket available for all users.\n",
|
||||
"- **Scratch** means temporary: all files will be deleted after 30 days.\n",
|
||||
"- Use the scratch bucket to save files between processing runs or share files between projects, temporarily.\n",
|
||||
"\n",
|
||||
"**Project** buckets are available to selected users as well. A project bucket can exist in another AWS account and be cross-linked to EASI. An EASI admin will assign users to a \"project\", which will enable their access to the bucket. Files in a project bucket are subject to the bucket owner's life cycle rules, administration and costs.\n",
|
||||
"\n",
|
||||
"> Cross-account **project** buckets may benefit from additional ACL settings. See [User Guide/08-cross-account-storage-usage](https://docs.csiro.easi-eo.solutions/user-guide/users-guide/08-cross-account-storage-usage/) (in your deployment).\n",
|
||||
"\n",
|
||||
"Glossary:\n",
|
||||
"- S3 storage items are called **objects**. Typically these are files but they could be any blob of data.\n",
|
||||
"- An **object**'s name is its **key**. The **key** can be [just about any string](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). Typically we include a `/` in the key to make it look like a directory path, which we're familiar with from regular file systems.\n",
|
||||
"\n",
|
||||
"There are two AWS APIs that can be used to read/write to a **scratch** or **project** bucket. Examples for both are given in this notebook.\n",
|
||||
"- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html) - linux program (use in terminal)\n",
|
||||
"- [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) - python library (use in code)\n",
|
||||
"\n",
|
||||
"We show *writing* first so that you add a test file for the *reading* section.\n",
|
||||
"\n",
|
||||
"- [Writing](#Writing)\n",
|
||||
" - [User ID](#User-ID)\n",
|
||||
" - [Select a test file](#Select-a-test-file)\n",
|
||||
" - [Upload a file](#Upload-a-file)\n",
|
||||
"- [Reading](#Reading)\n",
|
||||
" - [List objects](#List-objects)\n",
|
||||
" - [Read a file directly](#Read-a-file-directly)\n",
|
||||
" - [Copy a file to local](#Copy-a-file-to-local)\n",
|
||||
"\n",
|
||||
"## Imports and setup"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys, os\n",
|
||||
"import boto3\n",
|
||||
"from datetime import datetime as dt\n",
|
||||
"\n",
|
||||
"# EASI tools\n",
|
||||
"import git\n",
|
||||
"repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
|
||||
"if repo not in sys.path: sys.path.append(repo)\n",
|
||||
"from easi_tools import EasiDefaults"
|
||||
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"client = boto3.client('s3')\n",
|
||||
"\n",
|
||||
"easi = EasiDefaults()\n",
|
||||
"bucket = easi.scratch"
|
||||
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Optional, for parallel uploads and downloads of large files\n",
|
||||
"# Add a (..., Config=config) parameter to the relevant upload and download functions\n",
|
||||
"\n",
|
||||
"# from boto3.s3.transfer import TransferConfig\n",
|
||||
"# config = TransferConfig(\n",
|
||||
"# multipart_threshold = 1024 * 25,\n",
|
||||
"# max_concurrency = 10,\n",
|
||||
"# multipart_chunksize = 1024 * 25,\n",
|
||||
"# use_threads = True\n",
|
||||
"# )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"## Writing\n",
|
||||
"\n",
|
||||
"### User ID\n",
|
||||
"\n",
|
||||
"To write to the **scratch** bucket the root of the key must be your AWS **User ID**.\n",
|
||||
"\n",
|
||||
"For a **project** bucket this restriction probably doesn't apply. Any root key conditions are managed by the bucket owner."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash\n",
|
||||
"\n",
|
||||
"userid=`aws sts get-caller-identity --query 'UserId' | sed 's/[\"]//g'`\n",
|
||||
"echo $userid"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"userid = boto3.client('sts').get_caller_identity()['UserId']\n",
|
||||
"print(userid)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Select a test file\n",
|
||||
"\n",
|
||||
"For use in this notebook."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"testfile = '/home/jovyan/test-file.txt'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash -s \"$testfile\"\n",
|
||||
" \n",
|
||||
"testfile=$1\n",
|
||||
"touch $testfile\n",
|
||||
"ls -l $testfile"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload a file"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash -s \"$bucket\" \"$userid\" \"$testfile\"\n",
|
||||
"\n",
|
||||
"bucket=$1\n",
|
||||
"userid=$2\n",
|
||||
"testfile=$3\n",
|
||||
"\n",
|
||||
"aws s3 cp ${testfile} s3://${bucket}/${userid}/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"target = testfile.split('/')[-1]\n",
|
||||
"try:\n",
|
||||
" print(f'upload: {testfile} to s3://{bucket}/{userid}/{target}')\n",
|
||||
" r = client.upload_file(testfile, bucket, f'{userid}/{target}')\n",
|
||||
" print('Success.')\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
" print('Failed.')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Reading\n",
|
||||
"\n",
|
||||
"### List objects\n",
|
||||
"\n",
|
||||
"The `boto3.list_objects_v2` function will return at most 1000 keys. Two options are shown here.\n",
|
||||
"1. Basic use of `list_objects_v2`\n",
|
||||
"2. Paginated list objects, for potentially >1000 keys"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash -s \"$bucket\" \"$userid\"\n",
|
||||
"\n",
|
||||
"bucket=$1\n",
|
||||
"userid=$2\n",
|
||||
"\n",
|
||||
"aws s3 ls s3://${bucket}/${userid}/"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Basic use of list_objects_v2\n",
|
||||
"\n",
|
||||
"response = client.list_objects_v2(Bucket=bucket, Prefix=f'{userid}/')\n",
|
||||
"\n",
|
||||
"# from pprint import pprint\n",
|
||||
"# pprint(response)\n",
|
||||
"\n",
|
||||
"# List each key with its last modified time stamp\n",
|
||||
"if 'Contents' in response:\n",
|
||||
" for c in response['Contents']:\n",
|
||||
" key = c['Key']\n",
|
||||
" lastmodified = c['LastModified'].strftime('%Y-%d-%m %H:%M:%S')\n",
|
||||
" size = c['Size']\n",
|
||||
" print(f'{lastmodified}\\t{size} {key}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Paginated list objects, for potentially >1000 keys\n",
|
||||
"\n",
|
||||
"paginator = client.get_paginator('list_objects_v2')\n",
|
||||
"page_iterator = paginator.paginate(Bucket=bucket, Prefix=f'{userid}/')\n",
|
||||
"\n",
|
||||
"for response in page_iterator:\n",
|
||||
" if 'Contents' in response:\n",
|
||||
" for c in response['Contents']:\n",
|
||||
" key = c['Key']\n",
|
||||
" lastmodified = c['LastModified'].strftime('%Y-%d-%m %H:%M:%S')\n",
|
||||
" psize = c['Size']\n",
|
||||
" print(f'{lastmodified}\\t{size} {key}')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Read a file directly\n",
|
||||
"\n",
|
||||
"Many data reading packages can read a file from an *s3://bucket/key* path into memory. Examples include:\n",
|
||||
"- `rasterio` and `rioxarray`\n",
|
||||
"- `gdal`\n",
|
||||
"\n",
|
||||
"For packages that can not read from an S3 path, first copy the file to your home directory or a temporary directory (e.g., dask workers). Then read the file with a normal file path."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Copy a file to local"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash -s \"$bucket\" \"$userid\" \"$testfile\"\n",
|
||||
"\n",
|
||||
"bucket=$1\n",
|
||||
"userid=$2\n",
|
||||
"testfile=$3\n",
|
||||
"\n",
|
||||
"source=`basename $testfile`\n",
|
||||
"aws s3 cp s3://${bucket}/${userid}/${source} ${testfile}\n",
|
||||
"ls -l $testfile"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"source = testfile.split('/')[-1]\n",
|
||||
"try:\n",
|
||||
" print(f'download: s3://{bucket}/{userid}/{source} to {testfile}')\n",
|
||||
" r = client.download_file(bucket, f'{userid}/{source}', testfile)\n",
|
||||
" print('Success.')\n",
|
||||
"except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
" print('Failed.')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
Reference in New Issue
Block a user