remote-sensing/notebooks/easi-scratch-bucket.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using EASI scratch and project buckets <img align=\"right\" src=\"../resources/csiro_easi_logo.png\">\n",
    "\n",
    "EASI has a **Scratch** bucket available for all users.\n",
    "- **Scratch** means temporary: all files will be deleted after 30 days.\n",
    "- Use the scratch bucket to save files between processing runs or share files between projects, temporarily.\n",
    "\n",
    "**Project** buckets are available to selected users as well. A project bucket can exist in another AWS account and be cross-linked to EASI. An EASI admin will assign users to a \"project\", which will enable their access to the bucket. Files in a project bucket are subject to the bucket owner's life cycle rules, administration and costs.\n",
    "\n",
    "> Cross-account **project** buckets may benefit from additional ACL settings. See [User Guide/08-cross-account-storage-usage](https://docs.csiro.easi-eo.solutions/user-guide/users-guide/08-cross-account-storage-usage/) (in your deployment).\n",
    "\n",
    "Glossary:\n",
    "- S3 storage items are called **objects**. Typically these are files but they could be any blob of data.\n",
    "- An **object**'s name is its **key**. The **key** can be [just about any string](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). Typically we include a `/` in the key to make it look like a directory path, which we're familiar with from regular file systems.\n",
    "\n",
    "There are two AWS APIs that can be used to read/write to a **scratch** or **project** bucket. Examples for both are given in this notebook.\n",
    "- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html) - linux program (use in terminal)\n",
    "- [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) - python library (use in code)\n",
    "\n",
    "We show *writing* first so that you add a test file for the *reading* section.\n",
    "\n",
    "- [Writing](#Writing)\n",
    "   - [User ID](#User-ID)\n",
    "   - [Select a test file](#Select-a-test-file)\n",
    "   - [Upload a file](#Upload-a-file)\n",
    "- [Reading](#Reading)\n",
    "   - [List objects](#List-objects)\n",
    "   - [Read a file directly](#Read-a-file-directly)\n",
    "   - [Copy a file to local](#Copy-a-file-to-local)\n",
    "\n",
    "## Imports and setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys, os\n",
    "import boto3\n",
    "from datetime import datetime as dt\n",
    "\n",
    "# EASI tools\n",
    "import git\n",
    "repo = git.Repo('.', search_parent_directories=True).working_tree_dir\n",
    "if repo not in sys.path: sys.path.append(repo)\n",
    "from easi_tools import EasiDefaults"

   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client = boto3.client('s3')\n",
    "\n",
    "easi = EasiDefaults()\n",
    "bucket = easi.scratch"

   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Optional, for parallel uploads and downloads of large files\n",
    "# Add a (..., Config=config) parameter to the relevant upload and download functions\n",
    "\n",
    "# from boto3.s3.transfer import TransferConfig\n",
    "# config = TransferConfig(\n",
    "#     multipart_threshold = 1024 * 25,\n",
    "#     max_concurrency = 10,\n",
    "#     multipart_chunksize = 1024 * 25,\n",
    "#     use_threads = True\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Writing\n",
    "\n",
    "### User ID\n",
    "\n",
    "To write to the **scratch** bucket the root of the key must be your AWS **User ID**.\n",
    "\n",
    "For a **project** bucket this restriction probably doesn't apply. Any root key conditions are managed by the bucket owner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "userid=`aws sts get-caller-identity --query 'UserId' | sed 's/[\"]//g'`\n",
    "echo $userid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "userid = boto3.client('sts').get_caller_identity()['UserId']\n",
    "print(userid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select a test file\n",
    "\n",
    "For use in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "testfile = '/home/jovyan/test-file.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$testfile\"\n",
    " \n",
    "testfile=$1\n",
    "touch $testfile\n",
    "ls -l $testfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$bucket\" \"$userid\" \"$testfile\"\n",
    "\n",
    "bucket=$1\n",
    "userid=$2\n",
    "testfile=$3\n",
    "\n",
    "aws s3 cp ${testfile} s3://${bucket}/${userid}/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target = testfile.split('/')[-1]\n",
    "try:\n",
    "    print(f'upload: {testfile} to s3://{bucket}/{userid}/{target}')\n",
    "    r = client.upload_file(testfile, bucket, f'{userid}/{target}')\n",
    "    print('Success.')\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "    print('Failed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading\n",
    "\n",
    "### List objects\n",
    "\n",
    "The `boto3.list_objects_v2` function will return at most 1000 keys. Two options are shown here.\n",
    "1. Basic use of `list_objects_v2`\n",
    "2. Paginated list objects, for potentially >1000 keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$bucket\" \"$userid\"\n",
    "\n",
    "bucket=$1\n",
    "userid=$2\n",
    "\n",
    "aws s3 ls s3://${bucket}/${userid}/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Basic use of list_objects_v2\n",
    "\n",
    "response = client.list_objects_v2(Bucket=bucket, Prefix=f'{userid}/')\n",
    "\n",
    "# from pprint import pprint\n",
    "# pprint(response)\n",
    "\n",
    "# List each key with its last modified time stamp\n",
    "if 'Contents' in response:\n",
    "    for c in response['Contents']:\n",
    "        key = c['Key']\n",
    "        lastmodified = c['LastModified'].strftime('%Y-%d-%m %H:%M:%S')\n",
    "        size = c['Size']\n",
    "        print(f'{lastmodified}\\t{size} {key}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Paginated list objects, for potentially >1000 keys\n",
    "\n",
    "paginator = client.get_paginator('list_objects_v2')\n",
    "page_iterator = paginator.paginate(Bucket=bucket, Prefix=f'{userid}/')\n",
    "\n",
    "for response in page_iterator:\n",
    "    if 'Contents' in response:\n",
    "        for c in response['Contents']:\n",
    "            key = c['Key']\n",
    "            lastmodified = c['LastModified'].strftime('%Y-%d-%m %H:%M:%S')\n",
    "            psize = c['Size']\n",
    "            print(f'{lastmodified}\\t{size} {key}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read a file directly\n",
    "\n",
    "Many data reading packages can read a file from an *s3://bucket/key* path into memory. Examples include:\n",
    "- `rasterio` and `rioxarray`\n",
    "- `gdal`\n",
    "\n",
    "For packages that can not read from an S3 path, first copy the file to your home directory or a temporary directory (e.g., dask workers). Then read the file with a normal file path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy a file to local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$bucket\" \"$userid\" \"$testfile\"\n",
    "\n",
    "bucket=$1\n",
    "userid=$2\n",
    "testfile=$3\n",
    "\n",
    "source=`basename $testfile`\n",
    "aws s3 cp s3://${bucket}/${userid}/${source} ${testfile}\n",
    "ls -l $testfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "source = testfile.split('/')[-1]\n",
    "try:\n",
    "    print(f'download: s3://{bucket}/{userid}/{source} to {testfile}')\n",
    "    r = client.download_file(bucket, f'{userid}/{source}', testfile)\n",
    "    print('Success.')\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "    print('Failed.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}