# 🎯 Simplified Workflow: Server Loads, Local Processes

## Overview

Workflow được đơn giản hóa để **tách rõ trách nhiệm**:
- **Server (Notebook 01):** Chỉ tải dữ liệu RAW từ S3, lưu NetCDF
- **Local (Notebook 02):** Tất cả xử lý + training model

## Why This Design?

### Lợi ích:
✅ **Server:** Tránh lãng phí tài nguyên cho xử lý → Tải nhanh, lưu ngay  
✅ **Local:** Kiểm soát toàn bộ quy trình → Dễ debug, dễ thay đổi tham số  
✅ **Tách biệt:** Server chỉ lo load, local chỉ lo xử lý  
✅ **Linh hoạt:** Có thể reprocess dữ liệu mà không cần quay lại server  

### So sánh:

**Cũ (All on Server):**
```
Server: Load → CloudMask → NDVI → Fill → Aggregation → Save → Transfer
Local: Unzip → Train
```
→ Server bị quá tải, chậm

**Mới (Simplified):**
```
Server: Load → Save RAW
Local: Load → CloudMask → NDVI → Fill → Aggregation → Train
```
→ Server chỉ làm việc nặng (loading), Local làm việc nhanh (processing)

---

## Workflow Chi Tiết

### Bước 1: Server - Tải Data Thô (Notebook 01)

**Thời gian:** 10-20 phút  
**Tài nguyên:** Network (S3 download)  
**Output:** 2 file NetCDF thô (~80-100 GB)

```
01.prepare_data_on_server.ipynb
├─ Cell 1: Intro
├─ Cell 2: Setup Dask + Datacube + S3
├─ Cell 3: Set coordinates
├─ Cell 4: Diagnostic (kiểm tra metadata)
├─ Cell 5: Load S2 (13 tháng) ← Monthly chunking
├─ Cell 6: Load S1 (raw)
├─ Cell 7: Save 2 file NetCDF
│   - sentinel2_raw.nc (S2 thô)
│   - sentinel1_raw.nc (S1 thô)
└─ Cell 8: Copy training shapefile + Close
```

**Output:**
```
data_for_training/
├─ sentinel2_raw.nc          (~50 GB)
├─ sentinel1_raw.nc          (~30 GB)
└─ train_data/
   ├─ *.shp, *.shx, *.dbf   (training points)
```

### Bước 2: Local - Tải & Xử Lý Data (Notebook 02)

**Thời gian:** 30-60 phút (CPU) hoặc 10-15 phút (GPU)  
**Tài nguyên:** CPU/GPU của máy local  
**Output:** Trained PyTorch CNN model

```
02.process_and_train_local.ipynb
├─ Cell 1: Import libraries
├─ Cell 2: Load NetCDF files
├─ Cell 3: Cloud masking ← Processing starts here
├─ Cell 4: Calculate NDVI
├─ Cell 5: Fill NaN (seasonal interpolation)
├─ Cell 6: Monthly aggregation
├─ Cell 7: Load training data & extract features
├─ Cell 8: Split train/val/test
├─ Cell 9: Train PyTorch CNN
├─ Cell 10: Evaluate on test set
└─ Cell 11: Save model
```

**Output:**
```
model_cnn_pytorch_local.pth          (Model weights)
model_cnn_pytorch_local_checkpoint.pth (Full checkpoint)
```

### Bước 3: Local - Dự Báo Toàn Bộ (Notebook 03)

**Thời gian:** 5-10 phút  
**Input:** Trained model + processed data  
**Output:** Classification maps (SHP, TIF)

```
03.predict_CNN_PyTorch_local.ipynb
├─ Load trained model
├─ Prepare full spatial data (cloud mask + NDVI + aggregation)
├─ Apply model to every pixel
└─ Save as shapefile/GeoTIFF
```

---

## File Structure

```
/home/x79/CSIROBoeingPhase5-Vietnam/
│
├─ 01.prepare_data_on_server.ipynb    ← RUN ON SERVER
│  └─ Output: data_for_training/ (80-100 GB)
│
├─ 02.process_and_train_local.ipynb   ← RUN LOCALLY
│  ├─ Input: data_for_training/ (from server)
│  └─ Output: model_cnn_pytorch_local.pth
│
├─ 03.predict_CNN_PyTorch_local.ipynb ← RUN LOCALLY
│  ├─ Input: model + processed data
│  └─ Output: prediction maps (SHP/TIF)
│
└─ new_import_ODC.py (helper functions)
```

---

## Timeline & Resource Usage

### Server Timeline:
```
Time    Action                          Duration    CPU   Memory   Network
────────────────────────────────────────────────────────────────────────────
00:00   Dask init                       30 sec      Low   Moderate  -
00:01   Set coordinates                 1 sec       -     -         -
00:02   Diagnostic check                30 sec      Low   Low       High (query)
00:03   Load S2 monthly chunks (13×)    12 min      Moderate High    High (download)
00:15   Load S1                         2 min       Moderate High    High
00:17   Save NetCDF                     3 min       Low   Moderate  -
00:20   Copy training data              1 min       -     -         -
00:21   DONE ✅                                      Total: ~80-100 GB saved
```

### Local Timeline (CPU):
```
Time    Action                          Duration
──────────────────────────────────────────────────
00:00   Load NetCDF                     2 min
00:02   Cloud mask                      3 min
00:05   NDVI + Fill                     5 min
00:10   Aggregation                     3 min
00:13   Load training data              1 min
00:14   Train CNN (50 epochs)           30-40 min
00:45   Evaluate                        1 min
00:46   Save model                      1 min
00:47   DONE ✅                         Total: ~50 min
```

### Local Timeline (GPU):
```
Same as above but:
  - Train CNN: 5-10 min instead of 30-40 min
  - Total: ~20-30 min
```

---

## Data Flow Diagram

```
┌──────────────────────────────────────────────────────────┐
│  SERVER (Notebook 01)                                    │
│  ────────────────────                                    │
│                                                          │
│  [AWS S3]  →  [Datacube]  →  [NetCDF]  →  [Download]   │
│  396 scenes    monthly       2 files       80-100 GB    │
│  (Raw S2+S1)   chunks        raw data      data_for_    │
│               (avoid OOM)                  training/    │
│                                                          │
└──────────────────────────────────────────────────────────┘
                         ⬇️ Transfer (SCP/FTP)
┌──────────────────────────────────────────────────────────┐
│  LOCAL MACHINE (Notebook 02)                             │
│  ──────────────────────────────────                      │
│                                                          │
│  [NetCDF]  →  [CloudMask]  →  [NDVI]  →  [FillNaN]    │
│  raw data      SCL band        red/nir   seasonal       │
│                                          interp         │
│     ⬇️                                                   │
│  [Aggregation]  →  [Train Data]  →  [CNN Training]     │
│  monthly            extract              PyTorch       │
│  averages           features             50 epochs     │
│                                                          │
│                                     ⬇️                   │
│                          [Trained Model (100 MB)]       │
│                                                          │
└──────────────────────────────────────────────────────────┘
                         ⬇️ (Notebook 03)
┌──────────────────────────────────────────────────────────┐
│  LOCAL MACHINE (Notebook 03)                             │
│  ──────────────────────────────────                      │
│                                                          │
│  [Trained Model]  +  [Aggregated Data]  →  [Predict]   │
│  100 MB               (monthly avg)         All pixels  │
│                                                          │
│                          ⬇️                              │
│              [Classification Maps (SHP/TIF)]            │
│                                                          │
└──────────────────────────────────────────────────────────┘
```

---

## Processing Parameters

### Notebook 01 (Server):
```python
date_range = ("2022-09-01", "2023-10-01")
longtitude_range = (105.5, 106.4)  # ~90 km
latitude_range = (9.2, 10.0)       # ~90 km
resolution = (-10, 10)              # 10 m/pixel
dask_chunks = {'x': 512, 'y': 512, 'time': 1}
```

### Notebook 02 (Local):
```python
# Cloud masking: Using SCL band
# NDVI calculation: (NIR - Red) / (NIR + Red)
# Fill NaN: Seasonal interpolation (4 seasons)
# Aggregation: Monthly averages (13 months)

# Training:
epochs = 50
batch_size = 32
learning_rate = 0.001
patience = 10 (early stopping)
split = 80% train, 10% val, 10% test
```

### Notebook 03 (Local):
```python
# Same processing as Notebook 02
# Apply model to every pixel
# Output: Classification map (8 classes)
```

---

## Data Quality Assurance

**Server (Notebook 01):**
- ✅ Diagnostic cell checks datacube metadata
- ✅ Monthly loading prevents OOM
- ✅ Error handling skips bad months
- ✅ File size validation before download

**Local (Notebook 02):**
- ✅ Data shape validation after loading
- ✅ NaN count reporting (before/after filling)
- ✅ Training progress monitoring (val loss, accuracy)
- ✅ Test accuracy + confusion matrix reporting

**Local (Notebook 03):**
- ✅ Prediction shape validation
- ✅ Class distribution analysis
- ✅ Output file size validation

---

## Troubleshooting

### Problem: "Server load too slow"
→ Check S3 bandwidth, Dask workers status  
→ Reduce number of workers temporarily

### Problem: "Local processing uses too much RAM"
→ Cloud mask operation: Reduce `dask_chunks` size
→ NDVI calculation: Process month by month
→ Training: Reduce batch size (32 → 16)

### Problem: "Model accuracy too low"
→ Check training data quality
→ Verify cloud masking effectiveness
→ Increase training epochs
→ Use data augmentation in `new_import_ODC.py`

### Problem: "Prediction takes too long"
→ Use GPU if available
→ Batch predictions by month
→ Reduce output resolution if needed

---

## Success Criteria

### Notebook 01 ✅
- [ ] All 13 months loaded with ✓ marks
- [ ] Data shape correct (~10,000 × 10,000 pixels)
- [ ] Memory usage 15-20 GB (not 403 TB!)
- [ ] 2 NetCDF files saved (~80-100 GB)
- [ ] Training shapefile copied

### Notebook 02 ✅
- [ ] NetCDF files loaded successfully
- [ ] Cloud masking reduces NaN count
- [ ] NDVI values in expected range [-0.5, 1.0]
- [ ] Monthly aggregation produces 13 timesteps
- [ ] Training completes without OOM
- [ ] Test accuracy ≥ 0.70 (70%)
- [ ] Model saved as .pth file

### Notebook 03 ✅
- [ ] Model loads successfully
- [ ] Predictions on full extent complete
- [ ] Classification map generated
- [ ] All 8 classes represented
- [ ] Output files saved (SHP/TIF)

---

## Advantages of This Design

1. **Resource Efficiency:**
   - Server: Only download/save (I/O bound)
   - Local: Only compute (CPU/GPU bound)

2. **Flexibility:**
   - Can reprocess locally without server
   - Can experiment with hyperparameters
   - Can apply to new regions easily

3. **Debugging:**
   - Local processing is much faster to iterate
   - Easy to visualize intermediate results
   - Can save intermediate results for inspection

4. **Scalability:**
   - Same pattern works for different regions
   - Can train multiple models in parallel locally
   - Server freed up for other tasks after initial load

5. **Reproducibility:**
   - All processing code on local machine
   - Easy to version control & document
   - Results fully reproducible

---

## Next Steps

1. ✅ Run Notebook 01 on server (10-20 min)
2. ✅ Download data to local machine (size: 80-100 GB)
3. ✅ Run Notebook 02 on local (30-60 min)
4. ✅ Run Notebook 03 on local (5-10 min)
5. ✅ Evaluate results

**Total time:** ~1-2 hours (including transfer)

---

**Created:** November 12, 2025  
**Status:** ✅ SIMPLIFIED WORKFLOW COMPLETE  
**Design Pattern:** Server loads → Local processes