basketballcantho/remote-sensing
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

thêm chức năng train trên odc predict trên planetary

Browse Source
This commit is contained in:
Victor Phan
2026-03-04 23:03:04 +07:00
parent ebb8e6e4b3
commit 8a1e7bb22e
39 changed files with 15297 additions and 711 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ODC_COGNITO_GUIDE.md
+405
View File
@@ -0,0 +1,405 @@
# ODC with Cognito Authentication Guide
# Hướng Dẫn Sử Dụng ODC với Cognito Authentication
## 📦 Files Đã Tạo
### 1. Core Modules
- **`new_import_ODC_cognito.py`** - ODC module tích hợp Cognito authentication
- **`cognito_auth.py`** - Cognito authentication core module
### 2. Test Scripts
- **`test_cognito_s3.py`** - Test Cognito authentication + S3 access
- **`test_s3_datacube_access.py`** - Test S3 access với datacube pattern
- **`test_s3_list_all.py`** - Demo list nhiều objects từ S3
### 3. Notebooks
- **`train_files/01.train_ODC_DecisionTree.ipynb`** - ✨ Updated với Cognito auth
- **`train_files/test_cognito_odc.ipynb`** - Demo notebook test Cognito + ODC
### 4. Documentation
- **`COGNITO_GUIDE.md`** - Hướng dẫn chi tiết về Cognito
- **`S3_ACCESS_GUIDE.md`** - Hướng dẫn truy cập S3
- **`ODC_COGNITO_GUIDE.md`** - File này
---
## 🚀 Quick Start
### Option 1: Sử dụng trong Notebook
```python
# 1. Import module
import sys
sys.path.insert(0, '/media/x79/2A7D-FAA0/remote-sensing')
import new_import_ODC_cognito
from new_import_ODC_cognito import *
# 2. Setup Cognito authentication
auth = setup_cognito_auth('train_files/crediential.txt')
# 3. Initialize datacube (S3 đã được config)
dc = datacube.Datacube()
# 4. Load data như bình thường
data = load_data(
dc=dc,
date_range=("2023-01-01", "2023-01-31"),
longtitude_range=(105.5, 106.0),
latitude_range=(9.5, 10.0)
)
```
### Option 2: Test độc lập
```bash
# Test Cognito authentication + S3
python test_cognito_s3.py
# Test list nhiều objects
python test_s3_list_all.py
```
---
## 📚 So Sánh: Trước vs Sau
### ❌ Trước (Không có Cognito):
```python
import new_import_ODC
from new_import_ODC import *
# Khởi tạo Dask + Datacube
cluster, client = notebook_utils.initialize_dask(use_gateway=True)
dc = datacube.Datacube()
# Configure S3 (unsigned - public access only)
configure_s3_access(aws_unsigned=True)
```
**Hạn chế:**
- Chỉ truy cập public buckets
- Không có authentication
- Không biết ai đang truy cập
- Không có audit trail
### ✅ Sau (Có Cognito):
```python
import new_import_ODC_cognito
from new_import_ODC_cognito import *
# Setup Cognito authentication
auth = setup_cognito_auth('train_files/crediential.txt')
# Thông tin user tự động hiển thị:
# Username: hienm2523001
# Email: hienm2523001@gstudent.ctu.edu.vn
# Groups: CSIRO and Vietnam partners
# Khởi tạo Datacube (S3 đã được authenticated)
cluster, client = notebook_utils.initialize_dask(use_gateway=True)
dc = datacube.Datacube()
```
**Lợi ích:**
- ✅ Truy cập cả private buckets
- ✅ Identity-based authentication
- ✅ Biết user identity (name, email, groups)
- ✅ Token tự động expire (security)
- ✅ Audit trail đầy đủ
- ✅ Group-based permissions
---
## 🔐 Authentication Flow
```
┌─────────────────────────────────────────────────────────────┐
│ 1. User Login → EASI Hub │
└───────────────────────┬─────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 2. AWS Cognito Authentication │
│ - Verify username/password │
│ - Check group membership │
└───────────────────────┬─────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Cognito Returns Tokens │
│ - Access Token (for API authentication) │
│ - ID Token (user info: name, email, groups) │
└───────────────────────┬─────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 4. EASI Backend Exchanges Tokens → AWS Credentials │
│ - Access Key ID │
│ - Secret Access Key │
│ - Session Token │
└───────────────────────┬─────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 5. User Receives: │
│ ✓ Cognito Tokens (in crediential.txt) │
│ ✓ AWS Credentials (in crediential.txt) │
└───────────────────────┬─────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 6. In Your Code: │
│ setup_cognito_auth('crediential.txt') │
│ → Loads both tokens + credentials │
│ → Configures S3 access for datacube │
│ → Ready to use! │
└─────────────────────────────────────────────────────────────┘
```
---
## 📝 File Structure
```
remote-sensing/
├── cognito_auth.py # Core Cognito authentication
├── new_import_ODC_cognito.py # ODC module with Cognito
├── test_cognito_s3.py # Test script
├── test_s3_list_all.py # List S3 objects demo
├── COGNITO_GUIDE.md # Cognito documentation
├── S3_ACCESS_GUIDE.md # S3 access documentation
├── ODC_COGNITO_GUIDE.md # This file
│
└── train_files/
├── crediential.txt # ⚠️ PRIVATE - Credentials
├── 01.train_ODC_DecisionTree.ipynb # ✨ Updated notebook
└── test_cognito_odc.ipynb # Demo notebook
```
---
## 🔧 API Reference
### Core Functions
#### `setup_cognito_auth(credential_file, region='ap-southeast-1')`
Setup Cognito authentication cho S3/ODC access.
**Parameters:**
- `credential_file` (str): Path to credential file
- `region` (str): AWS region
**Returns:**
- `CognitoAuthenticator` instance hoặc `None` nếu failed
**Example:**
```python
auth = setup_cognito_auth('train_files/crediential.txt')
```
#### `get_cognito_auth()`
Lấy Cognito authenticator instance hiện tại.
**Returns:**
- Current `CognitoAuthenticator` instance
#### `print_auth_status()`
In trạng thái authentication hiện tại.
**Example:**
```python
print_auth_status()
# Output:
# ══════════════════════════════════════════════════════════
# AUTHENTICATION STATUS
# ══════════════════════════════════════════════════════════
# ✅ Cognito authentication is active
# ✅ AWS credentials loaded
# Access Key: ASIA4YF43ZWIXQ6HJIAY...
# ✅ Cognito tokens loaded
# User: hienm2523001
# Email: hienm2523001@gstudent.ctu.edu.vn
```
#### `auto_setup(credential_file='train_files/crediential.txt')`
Tự động setup nếu credential file tồn tại.
**Returns:**
- `CognitoAuthenticator` instance hoặc `None`
### Data Loading Functions
#### `load_data(dc, date_range, longtitude_range, latitude_range, measurements=None)`
Load Sentinel-2 L2A data từ datacube.
**Parameters:**
- `dc`: Datacube instance
- `date_range`: Tuple of (start_date, end_date)
- `longtitude_range`: Tuple of (min_lon, max_lon)
- `latitude_range`: Tuple of (min_lat, max_lat)
- `measurements`: List of bands (default: ['red', 'nir', 'scl'])
**Returns:**
- `xarray.Dataset`
#### `load_data_sen1(dc, date_range, longtitude_range, latitude_range)`
Load Sentinel-1 SAR data (VV, VH).
**Returns:**
- `xarray.Dataset` with VV, VH bands
#### `mask_clean(data)`
Apply cloud mask sử dụng SCL band.
**Returns:**
- Cleaned `xarray.Dataset`
#### `calculate_average(data, variables, resample='1MS')`
Calculate temporal average và resample.
**Returns:**
- Resampled `xarray.Dataset`
---
## ⚙️ Configuration
### Credential File Format
File `train_files/crediential.txt`:
```bash
export AWS_ACCESS_KEY_ID="ASIA4YF43ZWIXQ6HJIAY"
export AWS_SECRET_ACCESS_KEY="3N8KoV2ZBqQcFqRUVxQXW8K9sm90CNDV9aHUkNw0"
export AWS_SESSION_TOKEN="IQoJb3JpZ2luX2VjEO7//////////..."
Cognito: eyJraWQiOiIzejR4V0txYmd5Mlo4NXR3TFVvRGFSNmp4...
ID: eyJraWQiOiJOMmdRc1c0S3o1YUltR3hGZEVJVmUx...
```
### Environment Variables
Sau khi setup, các environment variables được set:
```bash
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
```
---
## 🐛 Troubleshooting
### Error: "Token đã hết hạn"
**Nguyên nhân:** Cognito tokens expire sau ~8 giờ
**Giải pháp:**
1. Login lại vào EASI Hub
2. Copy credentials mới
3. Update file `crediential.txt`
4. Restart notebook kernel
### Error: "cognito_auth module not found"
**Nguyên nhân:** Module chưa được import đúng path
**Giải pháp:**
```python
import sys
sys.path.insert(0, '/media/x79/2A7D-FAA0/remote-sensing')
import new_import_ODC_cognito
```
### Error: "No AWS credentials available"
**Nguyên nhân:** File credentials chưa đúng format hoặc thiếu
**Giải pháp:**
1. Check file `train_files/crediential.txt` exists
2. Verify format (có cả AWS credentials VÀ Cognito tokens)
3. Re-run `setup_cognito_auth()`
### Warning: "EASI tools not available"
**Tác động:** Module vẫn chạy nhưng dùng standard datacube functions
**Giải pháp:** (Optional)
```bash
pip install easi-tools
```
---
## 📊 Performance Notes
### Token Expiration
- **Cognito tokens**: ~8 hours
- **AWS session tokens**: ~12 hours
- **Best practice**: Refresh mỗi session
### S3 Access
- **Authenticated access**: Nhanh hơn (cached credentials)
- **Pagination**: Support listing unlimited objects
- **Concurrent requests**: Thread-safe
---
## 🔒 Security Best Practices
### DO ✅
- Store credentials trong file riêng biệt
- Add `crediential.txt` vào `.gitignore`
- Refresh tokens thường xuyên
- Use HTTPS cho mọi API calls
- Check token expiration trước khi dùng
### DON'T ❌
- KHÔNG commit credentials vào Git
- KHÔNG share credentials publicly
- KHÔNG hardcode credentials trong code
- KHÔNG dùng credentials đã expire
- KHÔNG skip authentication checks
---
## 📞 Support
### Documentation
- `COGNITO_GUIDE.md` - Chi tiết về Cognito
- `S3_ACCESS_GUIDE.md` - Chi tiết về S3
- `test_cognito_odc.ipynb` - Demo notebook
### Test Scripts
```bash
# Test full flow
python test_cognito_s3.py
# Test list objects
python test_s3_list_all.py
# Test trong notebook
jupyter notebook train_files/test_cognito_odc.ipynb
```
### Contact
- **Project**: R-19244: CSIRO and Vietnam partners
- **EASI Hub**: easi-asia-csiro
- **Region**: ap-southeast-1
---
## 📈 Migration Checklist
Nếu đang migrate từ code cũ (không có Cognito):
- [ ] Copy `cognito_auth.py` vào project
- [ ] Copy `new_import_ODC_cognito.py` vào project
- [ ] Update imports: `new_import_ODC` → `new_import_ODC_cognito`
- [ ] Thêm `setup_cognito_auth()` trước datacube initialization
- [ ] Remove `configure_s3_access(aws_unsigned=True)`
- [ ] Test với `test_cognito_odc.ipynb`
- [ ] Update notebooks khác tương tự
---
**Version:** 1.0
**Last Updated:** March 4, 2026
**Status:** ✅ Production Ready