hoàn thành chức năng remove cloud train

2026-01-26 13:44:55 +07:00
parent 61646de647
commit 5404f7393c
17 changed files with 4337 additions and 153 deletions
@@ -0,0 +1,227 @@
+# Cloud Removal Training với SEN12MS-CR Dataset
+
+Hướng dẫn train Deep Learning model để khử mây từ ảnh Sentinel-2 sử dụng dataset SEN12MS-CR.
+
+## 📂 Cấu trúc dữ liệu
+
+```
+winter_dataset/
+├── ROIs2017_winter_s1/          # Sentinel-1 SAR data (VV, VH)
+│   ├── s1_8/
+│   ├── s1_9/
+│   └── ...
+├── ROIs2017_winter_s2/          # Sentinel-2 CLEAN (ground truth)
+│   ├── s2_8/
+│   ├── s2_9/
+│   └── ...
+├── ROIs2017_winter_s2_cloudy/   # Sentinel-2 CLOUDY (input)
+│   ├── s2_cloudy_8/
+│   ├── s2_cloudy_9/
+│   └── ...
+└── sen12ms_cr_dataLoader.py     # Data loader
+```
+
+## 🚀 Quick Start
+
+### 1. Training Model
+
+```bash
+# Activate environment
+conda activate env_01
+
+# Train cloud removal model
+python train_cloud_removal.py
+```
+
+**Hyperparameters mặc định:**
+- Use S1: `True` (sử dụng radar data)
+- Batch size: `8`
+- Epochs: `50`
+- Learning rate: `1e-4`
+- Model: U-Net
+- Loss: MAE (L1 Loss)
+
+### 2. Test Training (Quick)
+
+```bash
+# Test với 5 epochs
+python test_cloud_training.py
+```
+
+### 3. Sử dụng Model đã train
+
+```python
+from cloud_removal import process_cloud_removal
+
+# Load Sentinel-2 data
+s2_data = load(...)  # Your S2 data with SCL band
+
+# Apply deep learning cloud removal
+cleaned_data, metadata = process_cloud_removal(
+    s2_data=s2_data,
+    method="deep",  # Use deep learning method
+    verbose=True
+)
+```
+
+## 🎯 Model Architecture
+
+**U-Net** với cấu trúc:
+- **Input:** S2 cloudy (4 bands: B02, B03, B04, B08) + S1 (2 bands: VV, VH) = 6 channels
+- **Output:** S2 clean (4 bands) = 4 channels
+- **Features:** [64, 128, 256, 512]
+- **Skip connections:** Encoder → Decoder
+- **Activation:** ReLU + BatchNorm
+
+## 📊 Dataset Info
+
+**SEN12MS-CR** (Sentinel-12 Multi-Seasonal Cloud Removal):
+- **Scenes:** ~2000+ patches
+- **Size:** 256x256 pixels
+- **Bands:**
+  - S1: VV, VH (2 channels)
+  - S2: 13 bands (chọn B02, B03, B04, B08 cho training)
+- **Seasons:** Spring, Summer, Fall, Winter
+- **Source:** [https://github.com/PatrickTUM/SEN12MS-CR](https://github.com/PatrickTUM/SEN12MS-CR)
+
+## 🔧 Customization
+
+### Thay đổi hyperparameters
+
+```python
+from train_cloud_removal import train_cloud_removal_model
+
+model, train_losses, val_losses = train_cloud_removal_model(
+    data_dir="winter_dataset",
+    use_s1=True,           # Có dùng S1 không
+    batch_size=16,         # Tăng nếu có GPU mạnh
+    num_epochs=100,        # Số epochs
+    learning_rate=5e-5,    # Learning rate
+    device="cuda",         # "cuda" hoặc "cpu"
+    save_dir="model_train" # Thư mục lưu model
+)
+```
+
+### Chỉ dùng S2 (không dùng S1)
+
+```python
+model, train_losses, val_losses = train_cloud_removal_model(
+    use_s1=False,  # Không dùng radar data
+    # ... other params
+)
+```
+
+### Thay đổi S2 bands
+
+Sửa trong `train_cloud_removal.py`:
+
+```python
+# Thay vì RGB + NIR
+s2_bands = [S2Bands.B02, S2Bands.B03, S2Bands.B04, S2Bands.B08]
+
+# Có thể dùng tất cả bands
+s2_bands = S2Bands.ALL
+```
+
+## 📈 Monitoring Training
+
+Model tự động lưu:
+- **Best model:** `model_train/cloud_removal_unet_best.pth`
+- **Training curves:** `model_train/training_curves.png`
+- **Visualizations:** `model_train/cloud_removal_epoch_*.png` (mỗi 10 epochs)
+
+## 🌐 Tích hợp vào API
+
+Model đã được tích hợp vào `cloud_removal.py`:
+
+```python
+# API endpoint
+GET /api/cloud-removal/methods
+
+# Response
+{
+  "methods": {
+    "deep": "Deep Learning U-Net inpainting (best quality, requires model)"
+  }
+}
+```
+
+Sử dụng trong prediction:
+
+```json
+{
+  "model_filename": "model_odc.joblib",
+  "cloud_removal_method": "deep",
+  "..."
+}
+```
+
+## 📝 Notes
+
+### GPU Requirements
+- **Recommended:** NVIDIA GPU với >= 6GB VRAM
+- **Minimum:** CPU (chậm hơn ~10x)
+
+### Training Time
+- **GPU (RTX 3060):** ~2-3 hours cho 50 epochs
+- **CPU:** ~20-30 hours cho 50 epochs
+
+### Data Download
+Nếu chưa có dữ liệu, download từ:
+```bash
+# Download SEN12MS-CR dataset
+wget https://mediatum.ub.tum.de/download/1554803/1554803.zip
+unzip 1554803.zip -d winter_dataset/
+```
+
+## 🐛 Troubleshooting
+
+### 1. CUDA out of memory
+```python
+# Giảm batch size
+batch_size=4  # hoặc 2
+```
+
+### 2. Import error
+```bash
+# Kiểm tra dependencies
+pip install torch torchvision tqdm matplotlib
+```
+
+### 3. Model không load được
+```python
+# Kiểm tra path
+model_path = "model_train/cloud_removal_unet_best.pth"
+assert Path(model_path).exists()
+```
+
+## 📚 References
+
+- **Paper:** SEN12MS-CR: A Dataset for Cloud Removal in Sentinel-2 Imagery
+- **GitHub:** https://github.com/PatrickTUM/SEN12MS-CR
+- **U-Net:** Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation"
+
+## ✅ Checklist
+
+- [x] Data loader cho SEN12MS-CR
+- [x] U-Net architecture
+- [x] Training script
+- [x] Visualization
+- [x] Model saving/loading
+- [x] Tích hợp vào cloud_removal.py
+- [x] API integration
+- [x] Test script
+- [x] Documentation
+
+## 🎓 Next Steps
+
+1. **Train model:** `python train_cloud_removal.py`
+2. **Evaluate:** Xem visualizations trong `model_train/`
+3. **Test inference:** Dùng `test_cloud_removal.py`
+4. **Deploy:** Model tự động được dùng khi chọn `cloud_removal_method="deep"`
+
+---
+
+**Tác giả:** AI Assistant  
+**Ngày tạo:** 2026-01-21  
+**Version:** 1.0