basketballcantho/remote-sensing
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

update 01

Browse Source
This commit is contained in:
Victor Phan
2025-11-12 13:21:43 +07:00
parent e3777dd91c
commit e5e1ad3b01
4 changed files with 1283 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SIMPLIFIED_WORKFLOW.md
+365
View File
@@ -0,0 +1,365 @@
# 🎯 Simplified Workflow: Server Loads, Local Processes
## Overview
Workflow được đơn giản hóa để **tách rõ trách nhiệm**:
- **Server (Notebook 01):** Chỉ tải dữ liệu RAW từ S3, lưu NetCDF
- **Local (Notebook 02):** Tất cả xử lý + training model
## Why This Design?
### Lợi ích:
✅ **Server:** Tránh lãng phí tài nguyên cho xử lý → Tải nhanh, lưu ngay
✅ **Local:** Kiểm soát toàn bộ quy trình → Dễ debug, dễ thay đổi tham số
✅ **Tách biệt:** Server chỉ lo load, local chỉ lo xử lý
✅ **Linh hoạt:** Có thể reprocess dữ liệu mà không cần quay lại server
### So sánh:
**Cũ (All on Server):**
```
Server: Load → CloudMask → NDVI → Fill → Aggregation → Save → Transfer
Local: Unzip → Train
```
→ Server bị quá tải, chậm
**Mới (Simplified):**
```
Server: Load → Save RAW
Local: Load → CloudMask → NDVI → Fill → Aggregation → Train
```
→ Server chỉ làm việc nặng (loading), Local làm việc nhanh (processing)
---
## Workflow Chi Tiết
### Bước 1: Server - Tải Data Thô (Notebook 01)
**Thời gian:** 10-20 phút
**Tài nguyên:** Network (S3 download)
**Output:** 2 file NetCDF thô (~80-100 GB)
```
01.prepare_data_on_server.ipynb
├─ Cell 1: Intro
├─ Cell 2: Setup Dask + Datacube + S3
├─ Cell 3: Set coordinates
├─ Cell 4: Diagnostic (kiểm tra metadata)
├─ Cell 5: Load S2 (13 tháng) ← Monthly chunking
├─ Cell 6: Load S1 (raw)
├─ Cell 7: Save 2 file NetCDF
│ - sentinel2_raw.nc (S2 thô)
│ - sentinel1_raw.nc (S1 thô)
└─ Cell 8: Copy training shapefile + Close
```
**Output:**
```
data_for_training/
├─ sentinel2_raw.nc (~50 GB)
├─ sentinel1_raw.nc (~30 GB)
└─ train_data/
├─ *.shp, *.shx, *.dbf (training points)
```
### Bước 2: Local - Tải & Xử Lý Data (Notebook 02)
**Thời gian:** 30-60 phút (CPU) hoặc 10-15 phút (GPU)
**Tài nguyên:** CPU/GPU của máy local
**Output:** Trained PyTorch CNN model
```
02.process_and_train_local.ipynb
├─ Cell 1: Import libraries
├─ Cell 2: Load NetCDF files
├─ Cell 3: Cloud masking ← Processing starts here
├─ Cell 4: Calculate NDVI
├─ Cell 5: Fill NaN (seasonal interpolation)
├─ Cell 6: Monthly aggregation
├─ Cell 7: Load training data & extract features
├─ Cell 8: Split train/val/test
├─ Cell 9: Train PyTorch CNN
├─ Cell 10: Evaluate on test set
└─ Cell 11: Save model
```
**Output:**
```
model_cnn_pytorch_local.pth (Model weights)
model_cnn_pytorch_local_checkpoint.pth (Full checkpoint)
```
### Bước 3: Local - Dự Báo Toàn Bộ (Notebook 03)
**Thời gian:** 5-10 phút
**Input:** Trained model + processed data
**Output:** Classification maps (SHP, TIF)
```
03.predict_CNN_PyTorch_local.ipynb
├─ Load trained model
├─ Prepare full spatial data (cloud mask + NDVI + aggregation)
├─ Apply model to every pixel
└─ Save as shapefile/GeoTIFF
```
---
## File Structure
```
/home/x79/CSIROBoeingPhase5-Vietnam/
│
├─ 01.prepare_data_on_server.ipynb ← RUN ON SERVER
│ └─ Output: data_for_training/ (80-100 GB)
│
├─ 02.process_and_train_local.ipynb ← RUN LOCALLY
│ ├─ Input: data_for_training/ (from server)
│ └─ Output: model_cnn_pytorch_local.pth
│
├─ 03.predict_CNN_PyTorch_local.ipynb ← RUN LOCALLY
│ ├─ Input: model + processed data
│ └─ Output: prediction maps (SHP/TIF)
│
└─ new_import_ODC.py (helper functions)
```
---
## Timeline & Resource Usage
### Server Timeline:
```
Time Action Duration CPU Memory Network
────────────────────────────────────────────────────────────────────────────
00:00 Dask init 30 sec Low Moderate -
00:01 Set coordinates 1 sec - - -
00:02 Diagnostic check 30 sec Low Low High (query)
00:03 Load S2 monthly chunks (13×) 12 min Moderate High High (download)
00:15 Load S1 2 min Moderate High High
00:17 Save NetCDF 3 min Low Moderate -
00:20 Copy training data 1 min - - -
00:21 DONE ✅ Total: ~80-100 GB saved
```
### Local Timeline (CPU):
```
Time Action Duration
──────────────────────────────────────────────────
00:00 Load NetCDF 2 min
00:02 Cloud mask 3 min
00:05 NDVI + Fill 5 min
00:10 Aggregation 3 min
00:13 Load training data 1 min
00:14 Train CNN (50 epochs) 30-40 min
00:45 Evaluate 1 min
00:46 Save model 1 min
00:47 DONE ✅ Total: ~50 min
```
### Local Timeline (GPU):
```
Same as above but:
- Train CNN: 5-10 min instead of 30-40 min
- Total: ~20-30 min
```
---
## Data Flow Diagram
```
┌──────────────────────────────────────────────────────────┐
│ SERVER (Notebook 01) │
│ ──────────────────── │
│ │
│ [AWS S3] → [Datacube] → [NetCDF] → [Download] │
│ 396 scenes monthly 2 files 80-100 GB │
│ (Raw S2+S1) chunks raw data data_for_ │
│ (avoid OOM) training/ │
│ │
└──────────────────────────────────────────────────────────┘
⬇️ Transfer (SCP/FTP)
┌──────────────────────────────────────────────────────────┐
│ LOCAL MACHINE (Notebook 02) │
│ ────────────────────────────────── │
│ │
│ [NetCDF] → [CloudMask] → [NDVI] → [FillNaN] │
│ raw data SCL band red/nir seasonal │
│ interp │
│ ⬇️ │
│ [Aggregation] → [Train Data] → [CNN Training] │
│ monthly extract PyTorch │
│ averages features 50 epochs │
│ │
│ ⬇️ │
│ [Trained Model (100 MB)] │
│ │
└──────────────────────────────────────────────────────────┘
⬇️ (Notebook 03)
┌──────────────────────────────────────────────────────────┐
│ LOCAL MACHINE (Notebook 03) │
│ ────────────────────────────────── │
│ │
│ [Trained Model] + [Aggregated Data] → [Predict] │
│ 100 MB (monthly avg) All pixels │
│ │
│ ⬇️ │
│ [Classification Maps (SHP/TIF)] │
│ │
└──────────────────────────────────────────────────────────┘
```
---
## Processing Parameters
### Notebook 01 (Server):
```python
date_range = ("2022-09-01", "2023-10-01")
longtitude_range = (105.5, 106.4) # ~90 km
latitude_range = (9.2, 10.0) # ~90 km
resolution = (-10, 10) # 10 m/pixel
dask_chunks = {'x': 512, 'y': 512, 'time': 1}
```
### Notebook 02 (Local):
```python
# Cloud masking: Using SCL band
# NDVI calculation: (NIR - Red) / (NIR + Red)
# Fill NaN: Seasonal interpolation (4 seasons)
# Aggregation: Monthly averages (13 months)
# Training:
epochs = 50
batch_size = 32
learning_rate = 0.001
patience = 10 (early stopping)
split = 80% train, 10% val, 10% test
```
### Notebook 03 (Local):
```python
# Same processing as Notebook 02
# Apply model to every pixel
# Output: Classification map (8 classes)
```
---
## Data Quality Assurance
**Server (Notebook 01):**
- ✅ Diagnostic cell checks datacube metadata
- ✅ Monthly loading prevents OOM
- ✅ Error handling skips bad months
- ✅ File size validation before download
**Local (Notebook 02):**
- ✅ Data shape validation after loading
- ✅ NaN count reporting (before/after filling)
- ✅ Training progress monitoring (val loss, accuracy)
- ✅ Test accuracy + confusion matrix reporting
**Local (Notebook 03):**
- ✅ Prediction shape validation
- ✅ Class distribution analysis
- ✅ Output file size validation
---
## Troubleshooting
### Problem: "Server load too slow"
→ Check S3 bandwidth, Dask workers status
→ Reduce number of workers temporarily
### Problem: "Local processing uses too much RAM"
→ Cloud mask operation: Reduce `dask_chunks` size
→ NDVI calculation: Process month by month
→ Training: Reduce batch size (32 → 16)
### Problem: "Model accuracy too low"
→ Check training data quality
→ Verify cloud masking effectiveness
→ Increase training epochs
→ Use data augmentation in `new_import_ODC.py`
### Problem: "Prediction takes too long"
→ Use GPU if available
→ Batch predictions by month
→ Reduce output resolution if needed
---
## Success Criteria
### Notebook 01 ✅
- [ ] All 13 months loaded with ✓ marks
- [ ] Data shape correct (~10,000 × 10,000 pixels)
- [ ] Memory usage 15-20 GB (not 403 TB!)
- [ ] 2 NetCDF files saved (~80-100 GB)
- [ ] Training shapefile copied
### Notebook 02 ✅
- [ ] NetCDF files loaded successfully
- [ ] Cloud masking reduces NaN count
- [ ] NDVI values in expected range [-0.5, 1.0]
- [ ] Monthly aggregation produces 13 timesteps
- [ ] Training completes without OOM
- [ ] Test accuracy ≥ 0.70 (70%)
- [ ] Model saved as .pth file
### Notebook 03 ✅
- [ ] Model loads successfully
- [ ] Predictions on full extent complete
- [ ] Classification map generated
- [ ] All 8 classes represented
- [ ] Output files saved (SHP/TIF)
---
## Advantages of This Design
1. **Resource Efficiency:**
- Server: Only download/save (I/O bound)
- Local: Only compute (CPU/GPU bound)
2. **Flexibility:**
- Can reprocess locally without server
- Can experiment with hyperparameters
- Can apply to new regions easily
3. **Debugging:**
- Local processing is much faster to iterate
- Easy to visualize intermediate results
- Can save intermediate results for inspection
4. **Scalability:**
- Same pattern works for different regions
- Can train multiple models in parallel locally
- Server freed up for other tasks after initial load
5. **Reproducibility:**
- All processing code on local machine
- Easy to version control & document
- Results fully reproducible
---
## Next Steps
1. ✅ Run Notebook 01 on server (10-20 min)
2. ✅ Download data to local machine (size: 80-100 GB)
3. ✅ Run Notebook 02 on local (30-60 min)
4. ✅ Run Notebook 03 on local (5-10 min)
5. ✅ Evaluate results
**Total time:** ~1-2 hours (including transfer)
---
**Created:** November 12, 2025
**Status:** ✅ SIMPLIFIED WORKFLOW COMPLETE
**Design Pattern:** Server loads → Local processes