remote-sensing/QUICK_REFERENCE.md

# ⚡ Quick Reference: 3-Step Workflow

## 3 Bước Đơn Giản

### Step 1️⃣: SERVER - Load Data Raw
```
Notebook: 01.prepare_data_on_server.ipynb
Location: Run on server
Time: 10-20 min
Output: 2 files NetCDF (~80-100 GB)
```

**What it does:**
- Tải S2 (red, nir, scl) từ S3
- Tải S1 (VH, VV) từ S3
- Lưu 2 file NetCDF thô (chưa xử lý)
- Chép file training shapefile

**How to run:**
```python
# Run cells in order (1-8)
# Watch for progress bars in cell 5: [01/13], [02/13], ..., [13/13]
# Expected: ✅ Success! Shape: {'time': 396, 'y': 10000, 'x': 10000}
```

**Output:**
```
data_for_training/
├─ sentinel2_raw.nc      (S2 thô, ~50 GB)
├─ sentinel1_raw.nc      (S1 thô, ~30 GB)
└─ train_data/
   ├─ *.shp, *.shx, *.dbf (training points)
```

---

### Step 2️⃣: LOCAL - Process & Train
```
Notebook: 02.process_and_train_local.ipynb
Location: Download data + run on local machine
Time: 30-60 min (CPU) or 10-15 min (GPU)
Output: Trained PyTorch CNN model (~100 MB)
```

**What it does:**
- Load NetCDF files
- Cloud mask (SCL band)
- Calculate NDVI
- Fill missing values
- Monthly aggregation
- Extract features at training points
- Train PyTorch CNN (50 epochs)
- Evaluate on test set
- Save model

**How to run:**
```python
# Make sure data_for_training/ folder exists locally
# Run cells in order (1-11)
# Watch training progress: epoch 1/50, epoch 2/50, ...
# Expected: ✅ Test Accuracy: 0.7-0.85
```

**Output:**
```
model_cnn_pytorch_local.pth         (Model weights)
model_cnn_pytorch_local_checkpoint.pth (Full checkpoint)
```

---

### Step 3️⃣: LOCAL - Make Predictions
```
Notebook: 03.predict_CNN_PyTorch_local.ipynb
Location: Run on local machine
Time: 5-10 min
Output: Classification maps (SHP/TIF)
```

**What it does:**
- Load trained model
- Process full spatial data
- Apply model to every pixel
- Generate classification map
- Save as SHP/TIF format

**How to run:**
```python
# Make sure model file exists locally
# Run cells in order
# Expected: ✅ Classification map generated with 8 classes
```

**Output:**
```
classification_map.shp          (Land use map)
classification_map.tif          (GeoTIFF format)
```

---

## File Structure

```
On Server:
────────────
/server/path/01.prepare_data_on_server.ipynb
→ Outputs to: data_for_training/ (80-100 GB)


On Local Machine:
─────────────────
/local/path/
├─ data_for_training/         ← Downloaded from server
│  ├─ sentinel2_raw.nc
│  ├─ sentinel1_raw.nc
│  └─ train_data/
│
├─ 02.process_and_train_local.ipynb
├─ 03.predict_CNN_PyTorch_local.ipynb
│
├─ model_cnn_pytorch_local.pth         ← Generated by step 2
├─ model_cnn_pytorch_local_checkpoint.pth
│
└─ classification_map.shp              ← Generated by step 3
```

---

## Key Differences from Old Workflow

| Aspect | Old | New |
|--------|-----|-----|
| **Processing** | Server does everything | Server loads, local processes |
| **Speed** | Slow (server overloaded) | Fast (parallel processing) |
| **Memory** | 403 TB attempt (crash!) | 20 GB (manageable) |
| **Flexibility** | Hard to debug | Easy to iterate locally |
| **Re-processing** | Must go back to server | Can redo locally anytime |

---

## Checklist

### Before Step 1:
- [ ] Server has Dask + Datacube + S3 access
- [ ] At least 500 GB free on server
- [ ] Network stable

### Before Step 2:
- [ ] Downloaded all data from server
- [ ] At least 100 GB free on local machine
- [ ] Local machine has Python + PyTorch installed
- [ ] GPU available (optional but faster)

### Before Step 3:
- [ ] Notebook 02 completed with accuracy ≥ 0.70
- [ ] Model file exists locally
- [ ] Processed data available

---

## Troubleshooting

| Problem | Solution |
|---------|----------|
| Server: "403 TB OOM" | Already fixed! Using monthly chunking in cell 5 |
| Server: "S3 access denied" | Check credentials in cell 2 |
| Local: "File not found" | Verify data_for_training/ folder location |
| Local: "Model accuracy too low" | Check cloud masking - increase training epochs |
| Local: "Out of memory" | Close other apps, reduce batch_size in training |

---

## Performance Expectations

| Step | Task | Time (CPU) | Time (GPU) |
|------|------|-----------|-----------|
| 1️⃣ Load (Server) | S2 + S1 download | 10-20 min | N/A |
| 🔄 Transfer | Download to local | 30-60 min | 30-60 min |
| 2️⃣ Process & Train (Local) | All preprocessing + CNN | 30-60 min | 10-15 min |
| 3️⃣ Predict (Local) | Full spatial predictions | 5-10 min | 2-5 min |
| **TOTAL** | **All steps** | **1-2 hours** | **1-1.5 hours** |

---

## Expected Results

### After Step 1:
```
✅ 2 NetCDF files (80-100 GB)
✅ Training shapefile (1130 points)
✅ Ready to download
```

### After Step 2:
```
✅ Model trained (50 epochs completed)
✅ Test accuracy: 70-85%
✅ All 8 classes learned
✅ Model saved (100 MB)
```

### After Step 3:
```
✅ Classification map generated
✅ 8 classes distributed
✅ Accuracy reasonable on test areas
✅ Output in SHP/TIF format
```

---

## Why This Design?

**Server chỉ load (không xử lý):**
- Tránh lãng phí tài nguyên server
- Tải nhanh, server sẵn cho task khác
- Monthly chunking giải quyết OOM

**Local chỉ xử lý (không load):**
- Toàn quyền kiểm soát quy trình
- Dễ debug & iterate
- GPU nếu có → nhanh

**Kết quả:**
- ✅ Không bao giờ OOM
- ✅ Tất cả hoạt động nhanh
- ✅ Dễ tái tạo & tùy chỉnh

---

## Next Steps

1. **Today:** Run Notebook 01 on server
2. **Tonight:** Download data (~30-60 min)
3. **Tomorrow:** Run Notebook 02 (train model)
4. **Tomorrow:** Run Notebook 03 (make predictions)
5. **Day after:** Analyze results

**Total timeline:** 2-3 days (with overnight download)

---

**Design:** Server loads → Local processes  
**Status:** ✅ Ready to use  
**Created:** November 12, 2025