CSIROBoeingPhase5-Vietnam/README_PYTORCH_WORKFLOW.md

Project Index - Land Use Classification using CNN PyTorch

📚 Document Structure

🚀 Getting Started

File	Purpose	Read Time
QUICKSTART_PYTORCH.md	Hướng dẫn nhanh gọn (TL;DR)	5 min
LOCAL_TRAINING_WORKFLOW.md	Chi tiết workflow 3 bước	15 min
PYTORCH_REQUIREMENTS.txt	Cài đặt dependencies	5 min
PYTORCH_INSTALLATION.md	Chi tiết cài PyTorch	10 min

---

📓 Jupyter Notebooks

Step 1️⃣: Data Preparation (Server)

01.prepare_data_on_server.ipynb
├── Kết nối Dask cluster
├── Tải Sentinel-2, Sentinel-1 từ S3
├── Xử lý mây, tính NDVI
├── Lưu NetCDF files
└── ⏱️ Thời gian: 1-3 giờ (phụ thuộc vào số scene)

Output: data_for_training/ (150-300 MB)

---

Step 2️⃣: Model Training (Local Machine)

02.train_CNN_PyTorch_local.ipynb
├── Load data từ NetCDF files
├── Trích xuất features từ 1130 training points
├── Xây dựng CNN model
├── Huấn luyện (100 epochs max)
├── Plot training curves
└── ⏱️ Thời gian: 30 min - 2 giờ (GPU/CPU)

Output:

• model_cnn_pytorch.pt (state dict)
• model_cnn_pytorch_full.pt (full model info)
• training_history.png

---

Step 3️⃣: Prediction (Local Machine)

03.predict_CNN_PyTorch_local.ipynb
├── Load trained model
├── Predict trên 11M pixels
├── Tạo classification map
├── Lưu NetCDF, GeoTIFF, PNG
└── ⏱️ Thời gian: 10 min - 30 min (GPU/CPU)

Output:

• land_use_prediction.nc
• land_use_prediction.tif
• prediction_map.png
• prediction_metadata.json

---

🔧 Source Code

Python Module

new_import_ODC.py
├── Load functions (Sentinel-1, 2, training data)
├── Data processing (masking, indices, resampling)
├── CNN PyTorch classes and functions
│   ├── CNNClassifier (model architecture)
│   ├── train_cnn_model()
│   ├── prepare_data_for_cnn()
│   └── save_cnn_model()
└── Utilities (normalization, evaluation)

Key Functions:

• prepare_data_for_cnn() - Chuẩn bị data format cho CNN
• CNNClassifier() - Model architecture
• train_cnn_model() - Training loop với early stopping
• plot_training_history() - Visualization

---

📊 Model Architecture

CNN Design

Input Layer: (N, 1, 35)  # 35 features (12 months × 3 bands - NDVI, VV, VH)
    ↓
Block 1: Conv1d(64) → Conv1d(64) → MaxPool → Dropout
    ↓
Block 2: Conv1d(128) → Conv1d(128) → MaxPool → Dropout
    ↓
Block 3: Conv1d(256) → Conv1d(256) → GlobalAvgPool → Dropout
    ↓
Dense 1: FC(256 → 128) → ReLU → Dropout
    ↓
Dense 2: FC(128 → 64) → ReLU → Dropout
    ↓
Output: FC(64 → 8)  # 8 land use classes

Total Parameters: ~500K Trainable Parameters: ~450K

---

🏷️ Land Use Classes

Index	Label	VN Name	English Name
0	Lua tom	Lúa Tôm	Rice-Shrimp
1	Lua	Lúa	Rice
2	CHN	Cây hằng năm	Perennial Crop
3	CLN	Cây lâu năm	Long-term Crop
4	TS	Thổ nhưỡng	Soil/Bare Land
5	Song	Sông	Water/River
6	Dat xay dung	Đất xây dựng	Built-up/Urban
7	Rung	Rừng	Forest

---

📈 Data Flow

┌─────────────────┐
│  S3 Cloud       │
│  (Sentinel-1,2) │
└────────┬────────┘
         │
         ↓
┌─────────────────────────────┐
│ Server (01.prepare_data)    │
├─────────────────────────────┤
│ • Download from S3          │
│ • Mask clouds               │
│ • Calculate NDVI            │
│ • Resample to 10m           │
│ • Save as NetCDF            │
└────────┬────────────────────┘
         │ (Download ~150-300 MB)
         ↓
┌─────────────────────────────┐
│ Local Machine               │
├─────────────────────────────┤
│ data_for_training/          │
│ ├── average_ndvi.nc         │
│ ├── average_vv.nc           │
│ ├── average_vh.nc           │
│ └── train_data/*.shp        │
└────────┬────────────────────┘
         │
         ├──────────────────────────────┐
         ↓                              ↓
┌──────────────────────────┐  ┌────────────────────┐
│ 02.train_CNN_PyTorch    │  │ 03.predict_CNN_    │
│                          │  │    PyTorch         │
├──────────────────────────┤  ├────────────────────┤
│ • Extract features       │  │ • Load trained     │
│ • Split data (60/20/20)  │  │   model            │
│ • Normalize              │  │ • Predict on pixels│
│ • Train CNN              │  │ • Create map       │
│ • Save model             │  │ • Export formats   │
└────────┬─────────────────┘  └────────┬───────────┘
         │                             │
         ↓                             ↓
┌──────────────────────┐  ┌─────────────────────────┐
│ model_cnn_pytorch    │  │ land_use_prediction     │
│ _full.pt (~100 MB)   │  │ .nc/.tif/.png (~100 MB) │
└──────────────────────┘  └─────────────────────────┘

---

💻 System Requirements

Minimum

• Python 3.8+
• 8 GB RAM
• 5 GB Disk space

Recommended

• Python 3.10+
• 16 GB RAM
• 10 GB Disk space
• GPU (NVIDIA/AMD/Apple Silicon)

---

📦 Dependencies

Core

• torch>=2.0 - Deep learning framework
• numpy>=1.21 - Numerical computing
• xarray>=0.20 - Multi-dimensional arrays
• geopandas>=0.10 - Geospatial operations

Optional

• rasterio>=1.2 - Raster I/O (GeoTIFF export)
• jupyter>=1.0 - Notebook environment
• matplotlib>=3.4 - Visualization

See PYTORCH_REQUIREMENTS.txt for full list

---

✨ Features

• ✅ End-to-End Pipeline: Từ S3 đến classification map
• ✅ GPU Accelerated: Hỗ trợ NVIDIA, AMD, Apple Silicon
• ✅ Memory Efficient: Batch processing, normalization
• ✅ Modular Design: Các notebook độc lập
• ✅ Visualization: Training curves, prediction maps
• ✅ Metadata Tracking: Model info, accuracy, label mapping
• ✅ Multiple Output Formats: NetCDF, GeoTIFF, PNG, JSON

---

🎯 Workflow Comparison

Old Workflow (Random Forest on Server)

Server: Load Data → Train → Predict → Save
        ❌ Chậm (CPU only)
        ❌ Không linh hoạt
        ❌ Phải code trên server

New Workflow (CNN PyTorch Local)

Server: Load Data → Save to Files
           ↓ (Download)
Local:  Load Data → Train → Predict → Save
        ✅ Nhanh (GPU)
        ✅ Linh hoạt
        ✅ Code trên máy cá nhân

---

📖 Quick Navigation

I want to...

• 🚀 Get started quickly → Read QUICKSTART_PYTORCH.md
• 📝 Understand the workflow → Read LOCAL_TRAINING_WORKFLOW.md
• 🔧 Set up environment → Read PYTORCH_REQUIREMENTS.txt
• 📓 See full code → Check notebooks (01, 02, 03)
• 🤖 Understand model → See new_import_ODC.py CNNClassifier class

---

📊 Expected Outputs

Training Phase

✅ Model Accuracy
   Train: 88.5%
   Val: 82.3%
   Test: 81.2%

✅ Training Time: 45 min (GPU) / 90 min (CPU)
✅ Model Size: ~100 MB

Prediction Phase

✅ Classification Map: 1080×1080 pixels
✅ Output Formats: NetCDF, GeoTIFF, PNG
✅ Prediction Time: 5 min (GPU) / 15 min (CPU)
✅ File Sizes: ~100 MB each

---

🐛 Troubleshooting

Problem	Solution	File
ModuleNotFoundError	Install dependencies	PYTORCH_REQUIREMENTS.txt
GPU not detected	Check CUDA/drivers	PYTORCH_INSTALLATION.md
Out of memory	Reduce batch_size	Notebook comments
Data not found	Run server notebook first	Step 1

---

🔗 Related Files

Original Notebooks (Reference)

• 01.train_ODC.ipynb - Old Random Forest workflow
• 02.predict_ODC.ipynb - Old prediction workflow
• 03.compare_ODC.ipynb - Old comparison

Documentation (Old)

• CNN_PYTORCH_README.md - Old CNN notes
• CNN_PYTORCH_SUMMARY.md - Old summary
• COMPARISON_RF_VS_CNN.md - RF vs CNN comparison
• PYTORCH_INSTALLATION.md - Original install guide

---

📋 File Checklist

Before running:

• data_for_training/ exists with NetCDF files
• PyTorch installed and GPU detected
• Jupyter or IDE ready
• Enough disk space (~500 MB total)

---

🎓 Learning Resources

• PyTorch Tutorial: https://pytorch.org/tutorials/
• CNN Basics: https://cs231n.github.io/convolutional-networks/
• xarray: https://docs.xarray.dev/
• GeoPandas: https://geopandas.org/

---

✅ Checklist for Success

Preparation Phase
□ Read QUICKSTART_PYTORCH.md
□ Set up Python environment
□ Install all dependencies

Data Phase
□ Run 01.prepare_data_on_server.ipynb
□ Download data_for_training/ folder

Training Phase
□ Run 02.train_CNN_PyTorch_local.ipynb
□ Check training curves
□ Model achieves >75% test accuracy

Prediction Phase
□ Run 03.predict_CNN_PyTorch_local.ipynb
□ Generate classification map
□ Export to multiple formats

Validation Phase
□ Visually inspect prediction map
□ Compare with ground truth
□ Calculate accuracy metrics

---

Last Updated: November 2025
Version: 1.0
Status: Ready for Use ✅

🚀 Start with: QUICKSTART_PYTORCH.md