hoàn thành chức năng remove cloud train

CLOUD_PROCESSING.md

@@ -0,0 +1,346 @@
+# Xử lý mây (Cloud Processing) — Hệ thống Land Classification
+
+Tài liệu chi tiết về các phương pháp xử lý mây cho dữ liệu Sentinel-2. Module độc lập `cloud_removal.py` cung cấp nhiều chiến lược có thể chọn.
+
+## Tổng quan
+
+Hệ thống cung cấp **7 phương pháp xử lý mây** khác nhau, từ cổ điển đến hiện đại (ML/DL):
+
+1. **Classic** - 3 bước cổ điển (temporal → median → spatial) - mặc định
+2. **Temporal Only** - Chỉ temporal interpolation (nhanh nhất)
+3. **Median Composite** - Ưu tiên median composite (giảm nhiễu tốt nhất)
+4. **ML KNN** - Machine Learning K-Nearest Neighbors inpainting
+5. **ML RF** - Machine Learning Random Forest inpainting
+6. **Deep Inpainting** - Deep Learning CNN inpainting (yêu cầu model)
+7. **Hybrid** - Kết hợp classical + ML (cân bằng tốc độ và chất lượng)
+
+---
+
+## Cách sử dụng
+
+### API Endpoint
+
+Lấy danh sách các methods:
+```bash
+GET /api/cloud-removal/methods
+```
+
+Response:
+```json
+{
+  "success": true,
+  "methods": {
+    "classic": "3-step classical: temporal → median → spatial (default, balanced)",
+    "temporal_only": "Temporal interpolation only (fastest, needs many scenes)",
+    "median_composite": "Median composite priority (best noise reduction)",
+    "ml_knn": "ML K-Nearest Neighbors inpainting (good quality, medium speed)",
+    "ml_rf": "ML Random Forest inpainting (high quality, slower)",
+    "deep": "Deep Learning CNN inpainting (best quality, requires model)",
+    "hybrid": "Hybrid classical + ML (balanced speed & quality)"
+  },
+  "default": "classic"
+}
+```
+
+### Config trong Prediction
+
+Thêm `cloud_removal_method` vào config:
+
+```python
+config = {
+    "model_filename": "model_odc.joblib",
+    "min_lon": 105.5,
+    "max_lon": 105.6,
+    "min_lat": 10.0,
+    "max_lat": 10.1,
+    "start_date": "2024-01-01",
+    "end_date": "2024-12-31",
+    "max_scenes": 12,
+    "cloud_cover": 30,
+    "resolution": 20,
+    "use_gpu": false,
+    "cloud_removal_method": "hybrid"  # Chọn method tại đây
+}
+```
+
+### Programmatic Usage
+
+```python
+from cloud_removal import process_cloud_removal
+
+# Load Sentinel-2 data with SCL band
+s2_data = load(...) 
+
+# Process clouds with selected method
+cleaned_data, metadata = process_cloud_removal(
+    s2_data=s2_data,
+    method="hybrid",  # or "classic", "ml_knn", etc.
+    verbose=True
+)
+
+print(f"Cloud coverage: {metadata['cloud_coverage_percent']:.1f}%")
+print(f"Steps applied: {metadata['steps_applied']}")
+```
+
+---
+
+## Chi tiết các phương pháp
+
+### 1. Classic (Mặc định)
+
+**Mô tả:** 3 bước cổ điển kết hợp temporal, median, và spatial interpolation.
+
+**Quy trình:**
+1. Temporal interpolation (ffill + bfill)
+2. Median compositing (nếu >= 3 scenes)
+3. Spatial interpolation (nearest neighbor)
+4. Fallback fillna(0)
+
+**Ưu điểm:**
+- Cân bằng tốc độ và chất lượng
+- Đã được test kỹ, ổn định
+- Phù hợp hầu hết trường hợp
+
+**Nhược điểm:**
+- Không tối ưu cho các gaps lớn
+- Có thể tạo artifacts ở biên
+
+**Khi nào dùng:** Default choice, phù hợp cho production
+
+---
+
+### 2. Temporal Only
+
+**Mô tả:** Chỉ sử dụng temporal interpolation (ffill + bfill).
+
+**Ưu điểm:**
+- Nhanh nhất
+- Giữ xu hướng thời gian tốt
+- Ít tạo artifacts
+
+**Nhược điểm:**
+- Yêu cầu nhiều time steps
+- Không xử lý được gaps liên tục
+- Chất lượng kém nếu ít scenes
+
+**Khi nào dùng:** Khi có nhiều scenes (>10) và cần tốc độ
+
+---
+
+### 3. Median Composite
+
+**Mô tả:** Ưu tiên median composite, sau đó spatial interpolation.
+
+**Ưu điểm:**
+- Giảm nhiễu tốt nhất
+- Chống outliers hiệu quả
+- Tạo composite trơn
+
+**Nhược điểm:**
+- Mất thông tin temporal
+- Yêu cầu >= 3 scenes
+- Chậm hơn temporal only
+
+**Khi nào dùng:** Khi cần giảm nhiễu, không quan tâm temporal dynamics
+
+---
+
+### 4. ML KNN Inpainting
+
+**Mô tả:** Sử dụng K-Nearest Neighbors để học từ pixels hợp lệ và dự đoán pixels bị mây.
+
+**Quy trình:**
+1. Xác định valid pixels (không có mây)
+2. Train KNN model với spatial coordinates + spectral values
+3. Predict invalid pixels
+4. Fill predictions vào dataset
+
+**Ưu điểm:**
+- Chất lượng cao hơn classical
+- Học spatial patterns
+- Không cần pretrained model
+
+**Nhược điểm:**
+- Chậm hơn classical
+- Yêu cầu đủ valid pixels (>10)
+- Tốn RAM nếu ảnh lớn
+
+**Hyperparameters:**
+- n_neighbors: 5
+- weights: 'distance'
+
+**Khi nào dùng:** Khi cần chất lượng cao và có đủ valid pixels
+
+---
+
+### 5. ML Random Forest Inpainting
+
+**Mô tả:** Sử dụng Random Forest để inpainting, tương tự KNN nhưng phức tạp hơn.
+
+**Ưu điểm:**
+- Chất lượng cao nhất trong ML methods
+- Xử lý non-linear patterns tốt
+- Robust với outliers
+
+**Nhược điểm:**
+- Chậm nhất trong ML methods
+- Tốn nhiều RAM
+- Có thể overfit với ít data
+
+**Hyperparameters:**
+- n_estimators: 10
+- max_depth: 10
+- n_jobs: -1 (parallel)
+
+**Khi nào dùng:** Khi cần chất lượng tối đa và không quan tâm tốc độ
+
+---
+
+### 6. Deep Inpainting (CNN)
+
+**Mô tả:** Sử dụng CNN autoencoder để reconstruct pixels bị mây.
+
+**Trạng thái:** **Đang phát triển** - yêu cầu pretrained model
+
+**Quy trình (planned):**
+1. Stack bands thành multi-channel image
+2. Tạo binary mask (1=cloud, 0=valid)
+3. Run through CNN autoencoder
+4. Blend predictions với valid pixels
+
+**Ưu điểm (khi có model):**
+- Chất lượng tốt nhất
+- Xử lý large gaps hiệu quả
+- Học global context
+
+**Nhược điểm:**
+- Yêu cầu pretrained model
+- Chậm nhất (GPU recommended)
+- Phức tạp để deploy
+
+**Khi nào dùng:** Khi có GPU và pretrained model, cần chất lượng tối đa
+
+---
+
+### 7. Hybrid (Khuyến nghị)
+
+**Mô tả:** Kết hợp classical + ML để cân bằng tốc độ và chất lượng.
+
+**Quy trình:**
+1. Temporal interpolation (nhanh)
+2. Check remaining NaN percentage
+3. Nếu > 5%: Apply ML KNN inpainting
+4. Nếu <= 5%: Apply spatial interpolation
+5. Fallback fillna(0)
+
+**Ưu điểm:**
+- Cân bằng tốc độ và chất lượng
+- Adaptive - chỉ dùng ML khi cần
+- Hiệu quả với mọi cloud coverage
+
+**Nhược điểm:**
+- Phức tạp hơn classic
+- Khó debug
+
+**Khi nào dùng:** **Khuyến nghị cho production** - tự động chọn strategy phù hợp
+
+---
+
+## So sánh Performance
+
+| Method | Tốc độ | Chất lượng | RAM | Yêu cầu |
+|--------|--------|------------|-----|---------|
+| classic | ⭐⭐⭐⭐ | ⭐⭐⭐ | Thấp | Không |
+| temporal_only | ⭐⭐⭐⭐⭐ | ⭐⭐ | Thấp | Nhiều scenes |
+| median_composite | ⭐⭐⭐ | ⭐⭐⭐⭐ | Thấp | >= 3 scenes |
+| ml_knn | ⭐⭐ | ⭐⭐⭐⭐ | Trung bình | Đủ valid pixels |
+| ml_rf | ⭐ | ⭐⭐⭐⭐⭐ | Cao | Đủ valid pixels |
+| deep | ⭐ | ⭐⭐⭐⭐⭐ | Rất cao | Pretrained model + GPU |
+| hybrid | ⭐⭐⭐ | ⭐⭐⭐⭐ | Trung bình | Không |
+
+---
+
+## Phát hiện mây (SCL)
+
+Tất cả methods đều sử dụng SCL (Scene Classification Layer):
+
+```python
+# SCL values:
+# 0: No data, 1: Saturated/Defective, 2: Dark Area Pixels
+# 3: Cloud shadows, 4: Vegetation, 5: Not vegetated, 6: Water
+# 7: Unclassified, 8: Cloud medium probability, 9: Cloud high probability
+# 10: Thin cirrus, 11: Snow/Ice
+
+cloud_mask = (scl == 3) | (scl == 8) | (scl == 9) | (scl == 10) | (scl == 11)
+invalid_mask = (scl == 0) | (scl == 1)
+full_mask = cloud_mask | invalid_mask
+```
+
+---
+
+## Testing & Comparison
+
+So sánh nhiều methods trên cùng dữ liệu:
+
+```python
+from cloud_removal import compare_methods
+
+results = compare_methods(
+    s2_data=s2_data,
+    methods=["classic", "temporal_only", "ml_knn", "hybrid"]
+)
+
+for method, result in results.items():
+    print(f"{method}: {result['remaining_nan_percent']:.2f}% NaN remaining")
+```
+
+---
+
+## Khuyến nghị sử dụng
+
+### Production (General)
+```
+cloud_removal_method: "hybrid"
+```
+- Cân bằng tốc độ và chất lượng
+- Adaptive theo cloud coverage
+
+### High Quality (Research)
+```
+cloud_removal_method: "ml_rf"
+```
+- Chất lượng tối đa
+- Chấp nhận tốc độ chậm
+
+### Fast Processing (Monitoring)
+```
+cloud_removal_method: "temporal_only"
+```
+- Cần nhiều scenes (>10)
+- Ưu tiên tốc độ
+
+### Low Cloud Coverage (<10%)
+```
+cloud_removal_method: "classic"
+```
+- Đơn giản, hiệu quả
+- Ổn định, đã test kỹ
+
+---
+
+## Vị trí code
+
+- **Module:** `cloud_removal.py` - Standalone cloud removal module
+- **API Integration:** `api_server.py` - API endpoints và config
+- **Documentation:** `CLOUD_PROCESSING.md` - Tài liệu này
+
+---
+
+## Phát triển tiếp
+
+- [ ] Implement CNN autoencoder cho deep inpainting
+- [ ] Add quality scoring system
+- [ ] Optimize ML methods với Dask
+- [ ] Add weighted temporal interpolation
+- [ ] Support custom ML models
+

CLOUD_REMOVAL_UPLOAD_GUIDE.md

@@ -0,0 +1,200 @@
+# Cloud Removal Model Upload Feature
+
+## Overview
+Added functionality to upload and use custom deep learning cloud removal models (.pth files) during prediction.
+
+## Features Implemented
+
+### 1. API Endpoints
+
+#### Upload Cloud Removal Model
+```
+POST /api/cloud-removal/upload
+```
+- Upload `.pth` cloud removal model files
+- Validates file extension (.pth only)
+- Security checks for filename
+- Returns file info (name, size)
+
+**Example:**
+```bash
+curl -X POST -F "file=@cloud_removal_unet_best.pth" \
+  http://localhost:8000/api/cloud-removal/upload
+```
+
+#### List Cloud Removal Models
+```
+GET /api/cloud-removal/models
+```
+Already existing - lists all `.pth` models in `model_train/` directory
+
+#### Delete Cloud Removal Model
+```
+DELETE /api/cloud-removal/models/{filename}
+```
+Already existing - deletes a specific cloud removal model
+
+### 2. Prediction Configuration Updates
+
+#### PredictionConfig
+Added new optional field:
+```python
+cloud_removal_model: Optional[str] = None  # .pth filename
+```
+
+#### PredictionWithNDVIConfig  
+Added new optional field:
+```python
+cloud_removal_model: Optional[str] = None  # .pth filename
+```
+
+### 3. Prediction Function Integration
+
+The `run_prediction()` function now:
+1. Accepts `cloud_removal_model` parameter
+2. Passes model path to `process_cloud_removal()`
+3. Logs which model is being used
+
+**Code:**
+```python
+cloud_removal_method = config.cloud_removal_method
+cloud_removal_model = config.cloud_removal_model
+
+s2_data, cloud_metadata = process_cloud_removal(
+    s2_data=s2_data,
+    method=cloud_removal_method,
+    model_path=f"model_train/{cloud_removal_model}" if cloud_removal_model else None,
+    verbose=True
+)
+```
+
+### 4. Web Interface Updates
+
+#### Upload Button
+- Added file input in "Deep Learning" cloud removal section
+- Upload button appears when "Deep Learning" method is selected
+- Real-time upload status feedback
+- Auto-refreshes model list after successful upload
+
+#### Model Selection
+- Dropdown shows all available `.pth` models
+- Auto-selects newly uploaded model
+- Shows model metadata (epoch, loss)
+
+## Usage Guide
+
+### Step 1: Train or Obtain a Cloud Removal Model
+Train using the cloud training interface or obtain a pre-trained `.pth` model.
+
+### Step 2: Upload Model
+1. Go to Prediction Interface
+2. Scroll to "Cloud Removal Method" section
+3. Select "Deep Learning (U-Net)" from dropdown
+4. Model upload section appears
+5. Click "📤 Upload Cloud Removal Model (.pth)"
+6. Select your `.pth` file
+7. Wait for upload confirmation
+
+### Step 3: Use Model in Prediction
+1. The uploaded model is automatically selected
+2. Configure other prediction parameters (bbox, dates, etc.)
+3. Click "🚀 Start Prediction (với NDVI)"
+4. The system will use your custom model for cloud removal
+
+## File Structure
+```
+model_train/
+├── cloud_removal_unet_best.pth         # User uploaded
+├── cloud_removal_unet_epoch_10.pth     # User uploaded
+├── model_mobilenet-lraspp_*.joblib     # Land classification models
+└── ...
+```
+
+## API Request Example
+
+### Using Uploaded Model
+```json
+{
+  "model_filename": "model_mobilenet-lraspp_20260105_225459.joblib",
+  "min_lon": 105.80,
+  "min_lat": 10.00,
+  "max_lon": 105.82,
+  "max_lat": 10.02,
+  "start_date": "2024-01-15",
+  "end_date": "2024-01-17",
+  "max_scenes": 3,
+  "cloud_cover": 30,
+  "resolution": 20,
+  "use_gpu": true,
+  "export_ndvi": true,
+  "export_classification": true,
+  "cloud_removal_method": "deep",
+  "cloud_removal_model": "cloud_removal_unet_best.pth"
+}
+```
+
+### Without Custom Model (Classical Methods)
+```json
+{
+  ...
+  "cloud_removal_method": "hybrid",
+  "cloud_removal_model": null
+}
+```
+
+## Security Features
+- Filename validation (no path traversal)
+- File extension validation (.pth only)
+- File existence checks
+- Duplicate filename detection
+
+## Error Handling
+- Invalid file type → 400 Bad Request
+- Duplicate filename → 400 Bad Request  
+- Upload failure → 500 Internal Server Error
+- Missing model when "deep" selected → Falls back to "hybrid" method
+
+## Notes
+- Uploaded models are stored in `model_train/` directory
+- Models must be PyTorch `.pth` files
+- Compatible with `cloud_removal.py` module
+- Works with both `/api/prediction/start` and `/api/predict/with-ndvi` endpoints
+
+## Testing
+
+### Test Upload
+```bash
+# Upload a model
+curl -X POST -F "file=@my_cloud_model.pth" \
+  http://localhost:8000/api/cloud-removal/upload
+
+# List models
+curl http://localhost:8000/api/cloud-removal/models
+
+# Delete model
+curl -X DELETE \
+  http://localhost:8000/api/cloud-removal/models/my_cloud_model.pth
+```
+
+### Test Prediction
+```bash
+curl -X POST http://localhost:8000/api/predict/with-ndvi \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model_filename": "model_mobilenet-lraspp_20260105_225459.joblib",
+    "min_lon": 105.80, "min_lat": 10.00,
+    "max_lon": 105.82, "max_lat": 10.02,
+    "start_date": "2024-01-15", "end_date": "2024-01-17",
+    "max_scenes": 2, "cloud_cover": 30, "resolution": 20,
+    "use_gpu": false, "export_ndvi": true,
+    "cloud_removal_method": "deep",
+    "cloud_removal_model": "cloud_removal_unet_best.pth"
+  }'
+```
+
+## Future Enhancements
+- Model metadata display (architecture, training date)
+- Model validation on upload
+- Multiple model format support (.pt, .onnx)
+- Model performance metrics
+- Batch upload support

CLOUD_TRAINING_GUIDE.md

@@ -0,0 +1,227 @@
+# Cloud Removal Training với SEN12MS-CR Dataset
+
+Hướng dẫn train Deep Learning model để khử mây từ ảnh Sentinel-2 sử dụng dataset SEN12MS-CR.
+
+## 📂 Cấu trúc dữ liệu
+
+```
+winter_dataset/
+├── ROIs2017_winter_s1/          # Sentinel-1 SAR data (VV, VH)
+│   ├── s1_8/
+│   ├── s1_9/
+│   └── ...
+├── ROIs2017_winter_s2/          # Sentinel-2 CLEAN (ground truth)
+│   ├── s2_8/
+│   ├── s2_9/
+│   └── ...
+├── ROIs2017_winter_s2_cloudy/   # Sentinel-2 CLOUDY (input)
+│   ├── s2_cloudy_8/
+│   ├── s2_cloudy_9/
+│   └── ...
+└── sen12ms_cr_dataLoader.py     # Data loader
+```
+
+## 🚀 Quick Start
+
+### 1. Training Model
+
+```bash
+# Activate environment
+conda activate env_01
+
+# Train cloud removal model
+python train_cloud_removal.py
+```
+
+**Hyperparameters mặc định:**
+- Use S1: `True` (sử dụng radar data)
+- Batch size: `8`
+- Epochs: `50`
+- Learning rate: `1e-4`
+- Model: U-Net
+- Loss: MAE (L1 Loss)
+
+### 2. Test Training (Quick)
+
+```bash
+# Test với 5 epochs
+python test_cloud_training.py
+```
+
+### 3. Sử dụng Model đã train
+
+```python
+from cloud_removal import process_cloud_removal
+
+# Load Sentinel-2 data
+s2_data = load(...)  # Your S2 data with SCL band
+
+# Apply deep learning cloud removal
+cleaned_data, metadata = process_cloud_removal(
+    s2_data=s2_data,
+    method="deep",  # Use deep learning method
+    verbose=True
+)
+```
+
+## 🎯 Model Architecture
+
+**U-Net** với cấu trúc:
+- **Input:** S2 cloudy (4 bands: B02, B03, B04, B08) + S1 (2 bands: VV, VH) = 6 channels
+- **Output:** S2 clean (4 bands) = 4 channels
+- **Features:** [64, 128, 256, 512]
+- **Skip connections:** Encoder → Decoder
+- **Activation:** ReLU + BatchNorm
+
+## 📊 Dataset Info
+
+**SEN12MS-CR** (Sentinel-12 Multi-Seasonal Cloud Removal):
+- **Scenes:** ~2000+ patches
+- **Size:** 256x256 pixels
+- **Bands:**
+  - S1: VV, VH (2 channels)
+  - S2: 13 bands (chọn B02, B03, B04, B08 cho training)
+- **Seasons:** Spring, Summer, Fall, Winter
+- **Source:** [https://github.com/PatrickTUM/SEN12MS-CR](https://github.com/PatrickTUM/SEN12MS-CR)
+
+## 🔧 Customization
+
+### Thay đổi hyperparameters
+
+```python
+from train_cloud_removal import train_cloud_removal_model
+
+model, train_losses, val_losses = train_cloud_removal_model(
+    data_dir="winter_dataset",
+    use_s1=True,           # Có dùng S1 không
+    batch_size=16,         # Tăng nếu có GPU mạnh
+    num_epochs=100,        # Số epochs
+    learning_rate=5e-5,    # Learning rate
+    device="cuda",         # "cuda" hoặc "cpu"
+    save_dir="model_train" # Thư mục lưu model
+)
+```
+
+### Chỉ dùng S2 (không dùng S1)
+
+```python
+model, train_losses, val_losses = train_cloud_removal_model(
+    use_s1=False,  # Không dùng radar data
+    # ... other params
+)
+```
+
+### Thay đổi S2 bands
+
+Sửa trong `train_cloud_removal.py`:
+
+```python
+# Thay vì RGB + NIR
+s2_bands = [S2Bands.B02, S2Bands.B03, S2Bands.B04, S2Bands.B08]
+
+# Có thể dùng tất cả bands
+s2_bands = S2Bands.ALL
+```
+
+## 📈 Monitoring Training
+
+Model tự động lưu:
+- **Best model:** `model_train/cloud_removal_unet_best.pth`
+- **Training curves:** `model_train/training_curves.png`
+- **Visualizations:** `model_train/cloud_removal_epoch_*.png` (mỗi 10 epochs)
+
+## 🌐 Tích hợp vào API
+
+Model đã được tích hợp vào `cloud_removal.py`:
+
+```python
+# API endpoint
+GET /api/cloud-removal/methods
+
+# Response
+{
+  "methods": {
+    "deep": "Deep Learning U-Net inpainting (best quality, requires model)"
+  }
+}
+```
+
+Sử dụng trong prediction:
+
+```json
+{
+  "model_filename": "model_odc.joblib",
+  "cloud_removal_method": "deep",
+  "..."
+}
+```
+
+## 📝 Notes
+
+### GPU Requirements
+- **Recommended:** NVIDIA GPU với >= 6GB VRAM
+- **Minimum:** CPU (chậm hơn ~10x)
+
+### Training Time
+- **GPU (RTX 3060):** ~2-3 hours cho 50 epochs
+- **CPU:** ~20-30 hours cho 50 epochs
+
+### Data Download
+Nếu chưa có dữ liệu, download từ:
+```bash
+# Download SEN12MS-CR dataset
+wget https://mediatum.ub.tum.de/download/1554803/1554803.zip
+unzip 1554803.zip -d winter_dataset/
+```
+
+## 🐛 Troubleshooting
+
+### 1. CUDA out of memory
+```python
+# Giảm batch size
+batch_size=4  # hoặc 2
+```
+
+### 2. Import error
+```bash
+# Kiểm tra dependencies
+pip install torch torchvision tqdm matplotlib
+```
+
+### 3. Model không load được
+```python
+# Kiểm tra path
+model_path = "model_train/cloud_removal_unet_best.pth"
+assert Path(model_path).exists()
+```
+
+## 📚 References
+
+- **Paper:** SEN12MS-CR: A Dataset for Cloud Removal in Sentinel-2 Imagery
+- **GitHub:** https://github.com/PatrickTUM/SEN12MS-CR
+- **U-Net:** Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation"
+
+## ✅ Checklist
+
+- [x] Data loader cho SEN12MS-CR
+- [x] U-Net architecture
+- [x] Training script
+- [x] Visualization
+- [x] Model saving/loading
+- [x] Tích hợp vào cloud_removal.py
+- [x] API integration
+- [x] Test script
+- [x] Documentation
+
+## 🎓 Next Steps
+
+1. **Train model:** `python train_cloud_removal.py`
+2. **Evaluate:** Xem visualizations trong `model_train/`
+3. **Test inference:** Dùng `test_cloud_removal.py`
+4. **Deploy:** Model tự động được dùng khi chọn `cloud_removal_method="deep"`
+
+---
+
+**Tác giả:** AI Assistant  
+**Ngày tạo:** 2026-01-21  
+**Version:** 1.0

PLANETARY_COMPUTER_TIPS.md

@@ -0,0 +1,204 @@
+# Microsoft Planetary Computer - Giải pháp Timeout
+
+## ❌ Vấn đề
+```
+The request exceeded the maximum allowed time
+```
+
+## ✅ Giải pháp
+
+### 1. **Giảm Parameters** (Quan trọng nhất)
+
+**Thử theo thứ tự:**
+
+```python
+# ❌ QUÁ LỚN - Dễ timeout
+bbox = [105.48, 9.77, 106.14, 10.35]  # ~70km x 60km
+start_date = "2023-01-01"
+end_date = "2023-12-31"  # 12 months
+max_scenes = 12
+```
+
+```python
+# ✅ VỪA PHẢI - Tốt
+bbox = [105.8, 10.0, 105.9, 10.1]  # ~10km x 10km
+start_date = "2024-01-01"  
+end_date = "2024-01-31"  # 1 month
+max_scenes = 5
+```
+
+```python
+# ✅ RẤT NHỎ - Luôn work
+bbox = [105.85, 10.05, 105.87, 10.07]  # ~2km x 2km
+start_date = "2024-01-15"
+end_date = "2024-01-22"  # 1 week
+max_scenes = 3
+```
+
+### 2. **Chiến lược Progressive Loading**
+
+Thay vì load toàn bộ vùng lớn 1 lúc, chia nhỏ:
+
+```python
+# Ví dụ: Chia bbox lớn thành 4 phần nhỏ
+original_bbox = [105.48, 9.77, 106.14, 10.35]
+
+# Tính mid points
+min_lon, min_lat, max_lon, max_lat = original_bbox
+mid_lon = (min_lon + max_lon) / 2
+mid_lat = (min_lat + max_lat) / 2
+
+# 4 sub-regions
+sub_regions = [
+    [min_lon, min_lat, mid_lon, mid_lat],  # Bottom-left
+    [mid_lon, min_lat, max_lon, mid_lat],  # Bottom-right
+    [min_lon, mid_lat, mid_lon, max_lat],  # Top-left
+    [mid_lon, mid_lat, max_lon, max_lat],  # Top-right
+]
+
+# Load từng region riêng, sau đó merge
+```
+
+### 3. **Tăng Timeout trong Code**
+
+Sửa `fetch_sentinel_items_with_retry`:
+
+```python
+# Thử với timeout dài hơn và ít items hơn
+for attempt in range(max_retries):
+    try:
+        # Giảm target xuống còn 2-3 items cho lần đầu
+        target_items = min(3, max_scenes) if attempt == 0 else 2
+        
+        search = catalog.search(
+            collections=["sentinel-2-l2a"],
+            bbox=bbox,
+            datetime=time_range,
+            query={"eo:cloud_cover": {"lt": cloud_cover}},
+            limit=10  # Giảm từ 20-50 xuống 10
+        )
+        
+        # Set timeout cho iterator
+        items = []
+        import signal
+        
+        def timeout_handler(signum, frame):
+            raise TimeoutError("Item fetch timeout")
+        
+        signal.signal(signal.SIGALRM, timeout_handler)
+        signal.alarm(30)  # 30 giây timeout
+        
+        try:
+            for item in search.items():
+                items.append(item)
+                if len(items) >= target_items:
+                    break
+        finally:
+            signal.alarm(0)  # Cancel alarm
+```
+
+### 4. **Alternative: Dùng Dữ liệu Local**
+
+Nếu Planetary Computer liên tục timeout:
+
+#### **a) Download trước (Recommended)**
+
+```bash
+# Dùng sentinelsat để download
+pip install sentinelsat
+
+# Download Sentinel-2 về máy
+python download_sentinel2.py --bbox 105.8,10.0,105.9,10.1 \
+    --start 2024-01-01 --end 2024-01-31
+```
+
+#### **b) Dùng Google Earth Engine** (Nếu có account)
+
+```python
+import ee
+ee.Initialize()
+
+# Load Sentinel-2 từ GEE thay vì Planetary Computer
+image = ee.ImageCollection('COPERNICUS/S2_SR') \
+    .filterBounds(ee.Geometry.Rectangle(bbox)) \
+    .filterDate(start_date, end_date) \
+    .median()
+```
+
+### 5. **Cache Aggressive**
+
+Khi đã load được data, cache ngay:
+
+```python
+# Trong prediction interface, enable cache by default
+use_cache = True  # ALWAYS
+
+# Khi load thành công, lưu cache ngay
+if items and len(items) > 0:
+    cache_file = f"cache_{bbox_hash}_{date_hash}.joblib"
+    joblib.dump({
+        'items': items,
+        's2_data': s2_data,
+        'timestamp': datetime.now()
+    }, cache_file)
+```
+
+## 🎯 **Action Plan Ngay Bây Giờ**
+
+### **Bước 1: Test với bbox CỰC NHỎ**
+
+Web interface → Prediction:
+- Min Lon: **105.80**
+- Min Lat: **10.00**
+- Max Lon: **105.82** (chỉ 0.02 độ = ~2km)
+- Max Lat: **10.02**
+- Start: **2024-01-15**
+- End: **2024-01-17** (3 ngày)
+- Max Scenes: **2**
+- Cloud Cover: 50%
+
+→ Nếu vẫn timeout → Vấn đề là internet/firewall/server PC quá tải
+
+### **Bước 2: Nếu Step 1 OK → Tăng dần**
+
+- Tăng bbox lên 0.05 độ (~5km)
+- Tăng time range lên 1 tuần
+- Tăng max_scenes lên 5
+
+### **Bước 3: Dùng Batch Processing**
+
+Thay vì 1 query lớn:
+- Chia thành nhiều queries nhỏ
+- Dùng `/api/batch/start`
+- Mỗi job = 1 vùng nhỏ
+- Merge results sau
+
+## 🔧 **Debug Commands**
+
+```bash
+# Check internet
+ping -c 3 planetarycomputer.microsoft.com
+
+# Check DNS
+nslookup planetarycomputer.microsoft.com
+
+# Test với curl
+curl -I https://planetarycomputer.microsoft.com/api/stac/v1
+
+# Monitor network
+sudo tcpdump -i any host planetarycomputer.microsoft.com
+```
+
+## 📝 **Token Info** (FYI)
+
+Microsoft Planetary Computer **KHÔNG CẦN** manual token:
+- ✅ SAS tokens tự động gen bởi `planetary_computer.sign()`
+- ✅ Auto-refresh khi cần
+- ✅ Không cần API key/registration (public access)
+- ❌ KHÔNG có "hết token" - chỉ có timeout/rate limit
+
+Nếu thấy authentication error:
+```python
+# Cài lại thư viện
+pip install --upgrade planetary-computer pystac-client
+```

api_server.py

@@ -3,7 +3,7 @@ API Server for Land Classification Model Training
 Cho phép chọn dữ liệu và cấu hình training qua giao diện web
 """
 
-from fastapi import FastAPI, BackgroundTasks, HTTPException, UploadFile, File
+from fastapi import FastAPI, BackgroundTasks, HTTPException, UploadFile, File, Form
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.staticfiles import StaticFiles
 from fastapi.responses import HTMLResponse, FileResponse
@@ -38,6 +38,9 @@ from vietnam_provinces_merged import (
     search_province_32, get_merged_info, get_provinces_statistics
 )
 
+# Import cloud removal module
+from cloud_removal import process_cloud_removal, get_available_methods
+
 # Import planetary computer libraries (conditional)
 try:
     from pystac_client import Client
@@ -218,6 +221,10 @@ class PredictionConfig(BaseModel):
     # GPU support for deep learning models
     use_gpu: bool
     
+    # Cloud removal strategy
+    cloud_removal_method: str = "classic"
+    cloud_removal_model: Optional[str] = None  # Optional: .pth model filename for deep learning cloud removal
+
 
 class TrainingStatus(BaseModel):
     """Trạng thái training"""
@@ -236,6 +243,7 @@ class NDVIConfig(BaseModel):
     end_date: str
     max_cloud_cover: int = 30
     resolution: int = 20
+    cloud_removal_method: str = "classic"
 
 
 class ChangeDetectionWorkflowRequest(BaseModel):
@@ -258,6 +266,7 @@ class ComparePeriodsPredictionConfig(BaseModel):
     resolution: int = 20
     export_ndvi: bool = True
     export_classification: bool = True
+    cloud_removal_method: str = "classic"
 
 
 class PredictionWithNDVIConfig(BaseModel):
@@ -275,6 +284,19 @@ class PredictionWithNDVIConfig(BaseModel):
     use_gpu: bool = False  # Use GPU for deep learning models
     export_ndvi: bool = True  # Export NDVI raster
     export_classification: bool = True  # Export classification raster
+    cloud_removal_method: str = "classic"
+    cloud_removal_model: Optional[str] = None  # Optional: .pth model filename for deep learning cloud removal
+
+
+class CloudRemovalTrainingConfig(BaseModel):
+    """Cấu hình train cloud removal model"""
+    data_dir: str = "winter_dataset"
+    use_s1: bool = True  # Sử dụng Sentinel-1 radar data
+    batch_size: int = 8
+    num_epochs: int = 50
+    learning_rate: float = 1e-4
+    use_gpu: bool = True
+    model_name: str = "cloud_removal_unet"  # Tên model để lưu
 
 
 # Serve change detection interface page (moved here after app is defined)
@@ -356,6 +378,16 @@ async def training_page():
         raise HTTPException(status_code=404, detail="Training interface không tồn tại")
 
 
+@app.get("/cloud-training", response_class=HTMLResponse)
+async def cloud_training_page():
+    """Serve cloud removal training interface"""
+    html_file = Path(__file__).parent / "cloud_training_interface.html"
+    if html_file.exists():
+        return FileResponse(html_file)
+    else:
+        raise HTTPException(status_code=404, detail="Cloud training interface không tồn tại")
+
+
 @app.get("/prediction", response_class=HTMLResponse)
 async def prediction_page():
     """Serve prediction interface"""
@@ -463,6 +495,298 @@ async def validate_model(model_filename: str):
         }
 
 
+@app.get("/api/cloud-removal/methods")
+async def get_cloud_removal_methods():
+    """Lấy danh sách các phương pháp xử lý mây có sẵn"""
+    methods = get_available_methods()
+    return {
+        "success": True,
+        "methods": methods,
+        "default": "classic",
+        "description": "Cloud removal strategies for Sentinel-2 data processing"
+    }
+
+
+@app.get("/api/cloud-removal/models")
+async def list_cloud_removal_models():
+    """Liệt kê các cloud removal models đã train"""
+    model_dir = Path("model_train")
+    if not model_dir.exists():
+        return {"models": [], "count": 0}
+    
+    models = []
+    # Search for ALL .pth files in model_train and subdirectories
+    for model_file in model_dir.rglob("*.pth"):
+        # Skip non-cloud-removal models (keep land classification models separate)
+        if any(x in model_file.name.lower() for x in ['mobilenet', 'cnn_', 'swin', 'xgboost', 'random_forest']):
+            continue
+            
+        try:
+            import torch
+            import json
+            
+            # Try to load metadata from .json sidecar file first
+            metadata_file = model_file.with_suffix('.json')
+            if metadata_file.exists():
+                try:
+                    with open(metadata_file, 'r') as f:
+                        metadata = json.load(f)
+                    
+                    models.append({
+                        "filename": model_file.name,
+                        "path": str(model_file),
+                        "relative_path": str(model_file.relative_to(model_dir)),
+                        "epoch": metadata.get('epoch', 0),
+                        "train_loss": metadata.get('train_loss', 0),
+                        "val_loss": metadata.get('val_loss', 0),
+                        "use_s1": metadata.get('use_s1', True),
+                        "in_channels": metadata.get('in_channels', 6),
+                        "out_channels": metadata.get('out_channels', 4),
+                        "description": metadata.get('description', ''),
+                        "created": model_file.stat().st_mtime,
+                        "size_mb": model_file.stat().st_size / (1024 * 1024),
+                        "has_metadata": True
+                    })
+                    continue
+                except Exception as e:
+                    print(f"[Cloud Models] Failed to read metadata file {metadata_file}: {e}")
+            
+            # Try to load checkpoint metadata from .pth file
+            try:
+                checkpoint = torch.load(model_file, map_location='cpu')
+                epoch = checkpoint.get('epoch', 0) if isinstance(checkpoint, dict) else 0
+                train_loss = checkpoint.get('train_loss', 0) if isinstance(checkpoint, dict) else 0
+                val_loss = checkpoint.get('val_loss', 0) if isinstance(checkpoint, dict) else 0
+                use_s1 = checkpoint.get('use_s1', True) if isinstance(checkpoint, dict) else True
+                in_channels = checkpoint.get('in_channels', 6) if isinstance(checkpoint, dict) else 6
+                out_channels = checkpoint.get('out_channels', 4) if isinstance(checkpoint, dict) else 4
+            except:
+                # If checkpoint format is different or corrupted, use defaults
+                epoch = 0
+                train_loss = 0
+                val_loss = 0
+                use_s1 = True
+                in_channels = 3
+                out_channels = 3
+            
+            models.append({
+                "filename": model_file.name,
+                "path": str(model_file),
+                "relative_path": str(model_file.relative_to(model_dir)),
+                "epoch": epoch,
+                "train_loss": train_loss,
+                "val_loss": val_loss,
+                "use_s1": use_s1,
+                "in_channels": in_channels,
+                "out_channels": out_channels,
+                "description": "",
+                "created": model_file.stat().st_mtime,
+                "size_mb": model_file.stat().st_size / (1024 * 1024),
+                "has_metadata": False
+            })
+        except Exception as e:
+            print(f"[Cloud Models] Error loading {model_file}: {e}")
+            # Still add the file even if we can't load metadata
+            models.append({
+                "filename": model_file.name,
+                "path": str(model_file),
+                "relative_path": str(model_file.relative_to(model_dir)),
+                "epoch": 0,
+                "train_loss": 0,
+                "val_loss": 0,
+                "use_s1": False,
+                "in_channels": 3,
+                "out_channels": 3,
+                "description": "",
+                "created": model_file.stat().st_mtime,
+                "size_mb": model_file.stat().st_size / (1024 * 1024),
+                "has_metadata": False
+            })
+    
+    models.sort(key=lambda x: x['created'], reverse=True)
+    return {"models": models, "count": len(models)}
+
+
+@app.post("/api/cloud-removal/train")
+async def train_cloud_removal(config: CloudRemovalTrainingConfig, background_tasks: BackgroundTasks):
+    """Bắt đầu train cloud removal model"""
+    
+    # Check if data directory exists
+    data_dir = Path(config.data_dir)
+    if not data_dir.exists():
+        raise HTTPException(
+            status_code=404,
+            detail=f"Data directory not found: {config.data_dir}"
+        )
+    
+    # Create status tracking
+    training_id = datetime.now().strftime("%Y%m%d_%H%M%S")
+    
+    async def run_cloud_training():
+        try:
+            from train_cloud_removal import train_cloud_removal_model
+            
+            print(f"[CLOUD REMOVAL TRAINING] Starting training {training_id}")
+            
+            model, train_losses, val_losses = train_cloud_removal_model(
+                data_dir=config.data_dir,
+                use_s1=config.use_s1,
+                batch_size=config.batch_size,
+                num_epochs=config.num_epochs,
+                learning_rate=config.learning_rate,
+                device="cuda" if config.use_gpu else "cpu",
+                save_dir="model_train"
+            )
+            
+            print(f"[CLOUD REMOVAL TRAINING] Completed {training_id}")
+            
+            return {
+                "success": True,
+                "training_id": training_id,
+                "final_train_loss": train_losses[-1],
+                "final_val_loss": val_losses[-1],
+                "epochs": len(train_losses)
+            }
+            
+        except Exception as e:
+            print(f"[CLOUD REMOVAL TRAINING ERROR] {e}")
+            import traceback
+            traceback.print_exc()
+            return {
+                "success": False,
+                "error": str(e),
+                "training_id": training_id
+            }
+    
+    # Run in background
+    background_tasks.add_task(run_cloud_training)
+    
+    return {
+        "message": "Cloud removal training started",
+        "training_id": training_id,
+        "config": {
+            "data_dir": config.data_dir,
+            "use_s1": config.use_s1,
+            "batch_size": config.batch_size,
+            "num_epochs": config.num_epochs,
+            "learning_rate": config.learning_rate,
+            "use_gpu": config.use_gpu
+        }
+    }
+
+
+@app.post("/api/cloud-removal/upload")
+async def upload_cloud_removal_model(
+    file: UploadFile = File(...), 
+    epoch: int = Form(0),
+    train_loss: float = Form(0.0),
+    val_loss: float = Form(0.0),
+    in_channels: int = Form(6),
+    out_channels: int = Form(4),
+    use_s1: bool = Form(True),
+    description: str = Form("")
+):
+    """Upload cloud removal .pth model with optional metadata"""
+    
+    # Debug logging
+    print(f"[Upload] Received parameters:")
+    print(f"  File: {file.filename}")
+    print(f"  Epoch: {epoch} (type: {type(epoch)})")
+    print(f"  Train Loss: {train_loss} (type: {type(train_loss)})")
+    print(f"  Val Loss: {val_loss} (type: {type(val_loss)})")
+    print(f"  In Channels: {in_channels} (type: {type(in_channels)})")
+    print(f"  Out Channels: {out_channels} (type: {type(out_channels)})")
+    print(f"  Use S1: {use_s1} (type: {type(use_s1)})")
+    print(f"  Description: {description}")
+    
+    # Validate file extension
+    if not file.filename.endswith('.pth'):
+        raise HTTPException(status_code=400, detail="Only .pth files are allowed")
+    
+    # Security check
+    if ".." in file.filename or "/" in file.filename or "\\" in file.filename:
+        raise HTTPException(status_code=400, detail="Invalid filename")
+    
+    try:
+        model_dir = Path("model_train")
+        model_dir.mkdir(exist_ok=True)
+        
+        # Save uploaded file
+        file_path = model_dir / file.filename
+        
+        # Check if file already exists
+        if file_path.exists():
+            raise HTTPException(status_code=400, detail=f"Model {file.filename} already exists")
+        
+        # Write file
+        with open(file_path, "wb") as f:
+            content = await file.read()
+            f.write(content)
+        
+        file_size = file_path.stat().st_size
+        
+        # Save metadata as JSON sidecar file
+        import json
+        metadata_file = file_path.with_suffix('.json')
+        
+        metadata_dict = {
+            "filename": file.filename,
+            "epoch": epoch,
+            "train_loss": train_loss,
+            "val_loss": val_loss,
+            "in_channels": in_channels,
+            "out_channels": out_channels,
+            "use_s1": use_s1,
+            "description": description,
+            "uploaded_at": datetime.now().isoformat()
+        }
+        
+        with open(metadata_file, 'w') as f:
+            json.dump(metadata_dict, f, indent=2)
+        
+        print(f"[Upload] Saved model: {file_path}")
+        print(f"[Upload] Saved metadata: {metadata_file}")
+        print(f"[Upload] Metadata: {metadata_dict}")
+        
+        return {
+            "message": f"Successfully uploaded {file.filename}",
+            "filename": file.filename,
+            "size_mb": round(file_size / 1024 / 1024, 2),
+            "path": str(file_path),
+            "metadata": metadata_dict
+        }
+    except HTTPException:
+        raise
+    except Exception as e:
+        print(f"[Upload] Error: {e}")
+        import traceback
+        traceback.print_exc()
+        raise HTTPException(status_code=500, detail=f"Upload failed: {str(e)}")
+
+
+@app.delete("/api/cloud-removal/models/{filename}")
+async def delete_cloud_removal_model(filename: str):
+    """Xóa cloud removal model"""
+    model_dir = Path("model_train")
+    model_path = model_dir / filename
+    
+    # Security check
+    if ".." in filename or "/" in filename or "\\" in filename:
+        raise HTTPException(status_code=400, detail="Invalid filename")
+    
+    if not model_path.exists():
+        raise HTTPException(status_code=404, detail=f"Model not found: {filename}")
+    
+    try:
+        model_path.unlink()
+        return {
+            "success": True,
+            "message": f"Deleted cloud removal model: {filename}"
+        }
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=f"Failed to delete: {str(e)}")
+
+
 @app.delete("/api/models/{model_filename}")
 async def delete_model(model_filename: str):
     """Xóa model"""
@@ -1304,18 +1628,14 @@ async def run_prediction(config: PredictionConfig):
             )
             
             prediction_status["progress"] = "Đang tải dữ liệu Sentinel-2..."
-            s2_search = catalog.search(
+            s2_items = fetch_sentinel_items_with_retry(
-                collections=["sentinel-2-l2a"],
+                catalog=catalog,
                 bbox=bbox,
-                datetime=time_range,
+                time_range=time_range,
-                query={"eo:cloud_cover": {"lt": config.cloud_cover}}
+                cloud_cover=config.cloud_cover,
+                max_scenes=config.max_scenes,
+                max_retries=3
             )
-            s2_items = list(s2_search.items())
-            
-            if not s2_items:
-                raise ValueError("Không tìm thấy dữ liệu Sentinel-2 cho khu vực và thời gian này")
-            
-            s2_items = s2_items[:config.max_scenes]
         
         prediction_status["progress"] = f"Đang xử lý dữ liệu Sentinel-2..."
         
@@ -1417,87 +1737,27 @@ async def run_prediction(config: PredictionConfig):
         # ============ ADVANCED CLOUD MASKING & REMOVAL ============
         prediction_status["progress"] = "Đang xử lý mây nâng cao..."
         
-        cloud_coverage_percent = 0
+        # Use cloud removal module with user-selected method
-        if "SCL" in s2_data:
+        cloud_removal_method = config.cloud_removal_method if hasattr(config, 'cloud_removal_method') else "classic"
-            scl = s2_data["SCL"]
+        cloud_removal_model = config.cloud_removal_model if hasattr(config, 'cloud_removal_model') else None
         
-            # SCL classification values (Sentinel-2 Scene Classification):
+        print(f"[CLOUD REMOVAL] Using method: {cloud_removal_method}")
-            # 0: No data, 1: Saturated/Defective, 2: Dark Area Pixels
+        if cloud_removal_model:
-            # 3: Cloud shadows, 4: Vegetation, 5: Not vegetated, 6: Water
+            print(f"[CLOUD REMOVAL] Using custom model: {cloud_removal_model}")
-            # 7: Unclassified, 8: Cloud medium probability, 9: Cloud high probability
-            # 10: Thin cirrus, 11: Snow/Ice
         
-            # Comprehensive cloud mask (clouds, shadows, cirrus, snow)
+        s2_data, cloud_metadata = process_cloud_removal(
-            cloud_mask = (scl == 3) | (scl == 8) | (scl == 9) | (scl == 10) | (scl == 11)
+            s2_data=s2_data,
+            method=cloud_removal_method,
+            model_path=f"model_train/{cloud_removal_model}" if cloud_removal_model else None,
+            verbose=True
+        )
         
-            # Also mask no-data and saturated pixels
+        cloud_coverage_percent = cloud_metadata.get('cloud_coverage_percent', 0)
-            invalid_mask = (scl == 0) | (scl == 1)
-            full_mask = cloud_mask | invalid_mask
         
-            # Calculate cloud coverage percentage
+        # Quality warning if cloud coverage too high
-            total_pixels = full_mask.size
+        if cloud_coverage_percent > 30:
-            masked_pixels = int(full_mask.sum().values)
+            print(f"[WARNING] High cloud coverage ({cloud_coverage_percent:.1f}%) - prediction quality may be affected")
-            cloud_coverage_percent = (masked_pixels / total_pixels * 100) if total_pixels > 0 else 0
+            prediction_status["progress"] = f"⚠️ Cảnh báo: Độ phủ mây cao ({cloud_coverage_percent:.1f}%)"
-            
-            print(f"[CLOUD MASK] Cloud coverage: {cloud_coverage_percent:.1f}%")
-            print(f"[CLOUD MASK] Masked pixels: {masked_pixels}/{total_pixels}")
-            
-            # Apply mask to all bands
-            for band in s2_data.data_vars:
-                if band != "SCL":
-                    s2_data[band] = s2_data[band].where(~full_mask)
-            
-            # ============ CLOUD REMOVAL STRATEGIES ============
-            
-            # Strategy 1: Temporal Interpolation (fill gaps between time steps)
-            prediction_status["progress"] = "Đang khử mây bằng temporal interpolation..."
-            for band in s2_data.data_vars:
-                if band != "SCL":
-                    # Forward fill then backward fill along time dimension
-                    s2_data[band] = s2_data[band].ffill(dim='time').bfill(dim='time')
-            
-            print(f"[CLOUD REMOVAL] Applied temporal interpolation")
-            
-            # Strategy 2: Median Compositing (if multiple time steps available)
-            if len(s2_data.time) >= 3:
-                prediction_status["progress"] = "Đang tạo median composite để giảm nhiễu mây..."
-                
-                # Create median composite for each band
-                for band in s2_data.data_vars:
-                    if band != "SCL":
-                        # Median reduces cloud noise better than mean
-                        median_composite = s2_data[band].median(dim='time', skipna=True)
-                        
-                        # Fill remaining NaN with median
-                        s2_data[band] = s2_data[band].fillna(median_composite)
-                
-                print(f"[CLOUD REMOVAL] Applied median compositing from {len(s2_data.time)} scenes")
-            
-            # Strategy 3: Spatial Interpolation (fill small gaps)
-            prediction_status["progress"] = "Đang khử mây bằng spatial interpolation..."
-            for band in s2_data.data_vars:
-                if band != "SCL":
-                    # Use nearest neighbor interpolation for remaining small gaps
-                    s2_data[band] = s2_data[band].interpolate_na(dim='x', method='nearest', fill_value='extrapolate')
-                    s2_data[band] = s2_data[band].interpolate_na(dim='y', method='nearest', fill_value='extrapolate')
-            
-            print(f"[CLOUD REMOVAL] Applied spatial interpolation")
-            
-            # Final check: replace any remaining NaN with 0
-            for band in s2_data.data_vars:
-                if band != "SCL":
-                    s2_data[band] = s2_data[band].fillna(0)
-            
-            print(f"[CLOUD REMOVAL] Completed - all NaN values handled")
-            
-            # Quality warning if cloud coverage too high
-            if cloud_coverage_percent > 30:
-                print(f"[WARNING] High cloud coverage ({cloud_coverage_percent:.1f}%) - prediction quality may be affected")
-                prediction_status["progress"] = f"⚠️ Cảnh báo: Độ phủ mây cao ({cloud_coverage_percent:.1f}%)"
-            
-        else:
-            print("[WARNING] No SCL band available - skipping cloud masking")
-            prediction_status["progress"] = "⚠️ Không có SCL band - bỏ qua khử mây"
         
         # ============ EXTRACT FEATURES ============
         prediction_status["progress"] = f"Đang trích xuất features (mode={feature_mode})..."
@@ -2458,20 +2718,16 @@ def run_batch_prediction(job: dict, config: PredictionConfig):
         
         job["progress"] = 25
         
-        # Search Sentinel-2
+        # Search Sentinel-2 with retry
-        s2_search = catalog.search(
+        s2_items = fetch_sentinel_items_with_retry(
-            collections=["sentinel-2-l2a"],
+            catalog=catalog,
             bbox=bbox,
-            datetime=time_range,
+            time_range=time_range,
-            query={"eo:cloud_cover": {"lt": config.cloud_cover}}
+            cloud_cover=config.cloud_cover,
+            max_scenes=config.max_scenes,
+            max_retries=3
         )
         
-        s2_items = list(s2_search.items())
-        if not s2_items:
-            raise ValueError("Không tìm thấy dữ liệu Sentinel-2")
-        
-        s2_items = s2_items[:config.max_scenes]
-        
         job["progress"] = 35
         
         # Load Sentinel-2 data
@@ -2768,14 +3024,15 @@ async def change_detection_predict_workflow(
             modifier=planetary_computer.sign_inplace
         )
         
-        search = catalog.search(
+        # Use retry logic for fetching items
-            collections=["sentinel-2-l2a"],
+        items = fetch_sentinel_items_with_retry(
+            catalog=catalog,
             bbox=bbox,
-            datetime=time_range,
+            time_range=time_range,
-            query={"eo:cloud_cover": {"lt": cloud_cover}}
+            cloud_cover=cloud_cover,
+            max_scenes=max_scenes,
+            max_retries=3
         )
-        
-        items = list(search.items())[:max_scenes]
         print(f"[CHANGE DETECTION] Found {len(items)} Sentinel-2 scenes")
         
         if len(items) == 0:
@@ -3403,6 +3660,86 @@ async def change_detection_api(
 
 # ============ PREDICTION WITH NDVI API ============
 
+def fetch_sentinel_items_with_retry(catalog, bbox, time_range, cloud_cover, max_scenes, max_retries=3):
+    """
+    Fetch Sentinel-2 items with retry logic and exponential backoff
+    
+    Optimizations:
+    - Reduce page size on retry
+    - Use shorter timeouts for each attempt
+    - Fetch fewer items initially and expand if successful
+    """
+    import time
+    
+    for attempt in range(max_retries):
+        try:
+            # Reduce target items on each retry to minimize timeout risk
+            target_items = max_scenes if attempt == 0 else min(max_scenes, 20 // (attempt + 1) * 10)
+            page_limit = 50 if attempt == 0 else 20  # Smaller pages on retry
+            
+            print(f"[FETCH ATTEMPT {attempt + 1}/{max_retries}] Searching Sentinel-2...")
+            print(f"  → Target items: {target_items}, Page limit: {page_limit}")
+            
+            # Search with reduced limit on retries
+            search = catalog.search(
+                collections=["sentinel-2-l2a"],
+                bbox=bbox,
+                datetime=time_range,
+                query={"eo:cloud_cover": {"lt": cloud_cover}},
+                limit=page_limit
+            )
+            
+            # Try to get items with timeout protection
+            items = []
+            page_count = 0
+            max_pages = 3 if attempt > 0 else 5  # Fewer pages on retry
+            
+            for item in search.items():
+                items.append(item)
+                if len(items) >= target_items:
+                    print(f"[FETCH] Reached target ({target_items} items)")
+                    break
+                    
+                # Track pagination to prevent hanging
+                if len(items) % page_limit == 0:
+                    page_count += 1
+                    if page_count >= max_pages:
+                        print(f"[FETCH] Max pages reached ({max_pages}), got {len(items)} items")
+                        break
+            
+            if items:
+                print(f"[FETCH SUCCESS] Retrieved {len(items)} items")
+                # Return up to max_scenes, but accept fewer if that's all we got
+                return items[:min(len(items), max_scenes)]
+            else:
+                raise ValueError("No Sentinel-2 scenes found for the specified criteria")
+                
+        except Exception as e:
+            error_msg = str(e)
+            print(f"[FETCH ERROR] Attempt {attempt + 1} failed: {error_msg}")
+            
+            if attempt < max_retries - 1:
+                # Longer exponential backoff: 3, 6, 12 seconds
+                wait_time = 3 * (2 ** attempt)
+                print(f"[RETRY] Waiting {wait_time}s before retry...")
+                time.sleep(wait_time)
+            else:
+                # Final attempt failed
+                if "exceeded the maximum allowed time" in error_msg or "timeout" in error_msg.lower():
+                    raise HTTPException(
+                        status_code=504,
+                        detail=f"Microsoft Planetary Computer request timed out after {max_retries} attempts. "
+                               f"Please try: (1) Reduce date range (2) Reduce max_scenes to 5-10 (3) Use smaller bbox area"
+                    )
+                elif "no sentinel-2 scenes found" in error_msg.lower():
+                    raise HTTPException(
+                        status_code=404,
+                        detail="No Sentinel-2 data found. Try: (1) Different date range (2) Higher cloud_cover threshold (3) Different location"
+                    )
+                else:
+                    raise
+
+
 @app.post("/api/predict/with-ndvi")
 async def predict_with_ndvi(config: PredictionWithNDVIConfig, background_tasks: BackgroundTasks):
     """Predict land classification và NDVI cho một khu vực"""
@@ -3454,15 +3791,15 @@ async def predict_with_ndvi(config: PredictionWithNDVIConfig, background_tasks:
             
             time_range = f"{config.start_date}/{config.end_date}"
             
-            # Search for Sentinel-2 data
+            # Search for Sentinel-2 data with retry logic
-            search = catalog.search(
+            items = fetch_sentinel_items_with_retry(
-                collections=["sentinel-2-l2a"],
+                catalog=catalog,
                 bbox=bbox,
-                datetime=time_range,
+                time_range=time_range,
-                query={"eo:cloud_cover": {"lt": config.cloud_cover}}
+                cloud_cover=config.cloud_cover,
+                max_scenes=config.max_scenes,
+                max_retries=3
             )
-            
-            items = list(search.items())[:config.max_scenes]
             print(f"[PREDICT+NDVI] Found {len(items)} Sentinel-2 scenes")
             
             if len(items) == 0:
@@ -3546,13 +3883,17 @@ async def predict_with_ndvi(config: PredictionWithNDVIConfig, background_tasks:
                         modifier=planetary_computer.sign_inplace
                     )
                     time_range = f"{config.start_date}/{config.end_date}"
-                    search = catalog.search(
+                    
-                        collections=["sentinel-2-l2a"],
+                    # Use retry logic for B11 band
+                    b11_items = fetch_sentinel_items_with_retry(
+                        catalog=catalog,
                         bbox=bbox,
-                        datetime=time_range,
+                        time_range=time_range,
-                        query={"eo:cloud_cover": {"lt": config.cloud_cover}}
+                        cloud_cover=config.cloud_cover,
+                        max_scenes=config.max_scenes,
+                        max_retries=3
                     )
-                    signed_items = [planetary_computer.sign(item) for item in list(search.items())[:config.max_scenes]]
+                    signed_items = [planetary_computer.sign(item) for item in b11_items]
                     print(f"[PREDICT+NDVI] Fetched {len(signed_items)} scenes for B11")
                 
                 b11_data = odc.stac.load(
@@ -3912,6 +4253,9 @@ async def predict_with_ndvi(config: PredictionWithNDVIConfig, background_tasks:
             "change_detection": change_summary
         }
     
+    except HTTPException:
+        # Re-raise HTTPException with original status code (e.g., 504 for timeout)
+        raise
     except Exception as e:
         print(f"[PREDICT+NDVI ERROR] {str(e)}")
         import traceback
@@ -3942,15 +4286,15 @@ async def calculate_ndvi_timeseries(config: NDVIConfig):
         bbox = config.bbox
         time_range = f"{config.start_date}/{config.end_date}"
         
-        # Search for Sentinel-2 data
+        # Search for Sentinel-2 data with retry logic
-        search = catalog.search(
+        items = fetch_sentinel_items_with_retry(
-            collections=["sentinel-2-l2a"],
+            catalog=catalog,
             bbox=bbox,
-            datetime=time_range,
+            time_range=time_range,
-            query={"eo:cloud_cover": {"lt": config.max_cloud_cover}}
+            cloud_cover=config.max_cloud_cover,
+            max_scenes=1000,  # Get all available scenes for time series
+            max_retries=3
         )
-        
-        items = list(search.items())
         print(f"[NDVI] Found {len(items)} Sentinel-2 scenes")
         
         if len(items) == 0:
@@ -4137,20 +4481,15 @@ async def ndvi_predict_timeseries(config: NDVIPredictionConfig):
             modifier=planetary_computer.sign_inplace,
         )
         
-        s2_search = catalog.search(
+        # Search for Sentinel-2 data with retry
-            collections=["sentinel-2-l2a"],
+        s2_items = fetch_sentinel_items_with_retry(
+            catalog=catalog,
             bbox=bbox,
-            datetime=time_range,
+            time_range=time_range,
-            query={"eo:cloud_cover": {"lt": config.max_cloud_cover}}
+            cloud_cover=config.max_cloud_cover,
+            max_scenes=config.max_scenes,
+            max_retries=3
         )
-        s2_items = list(s2_search.items())
-        
-        if not s2_items:
-            raise HTTPException(status_code=404, detail="No Sentinel-2 data found")
-        
-        # Limit scenes to max_scenes
-        if len(s2_items) > config.max_scenes:
-            s2_items = s2_items[:config.max_scenes]
         
         print(f"✅ Found {len(s2_items)} Sentinel-2 scenes (limited to {config.max_scenes})")
         
@@ -4623,15 +4962,18 @@ async def ndvi_forecast(config: NDVIForecastConfig):
             modifier=planetary_computer.sign_inplace,
         )
         
-        s2_search = catalog.search(
+        # Search for historical Sentinel-2 data with retry
-            collections=["sentinel-2-l2a"],
+        try:
-            bbox=bbox,
+            s2_items = fetch_sentinel_items_with_retry(
-            datetime=time_range,
+                catalog=catalog,
-            query={"eo:cloud_cover": {"lt": config.max_cloud_cover}}
+                bbox=bbox,
-        )
+                time_range=time_range,
-        s2_items = list(s2_search.items())
+                cloud_cover=config.max_cloud_cover,
-        
+                max_scenes=config.max_scenes,
-        if not s2_items:
+                max_retries=3
+            )
+        except ValueError:
+            # No items found - provide helpful error message
             raise HTTPException(
                 status_code=404, 
                 detail=f"⚠️ Không tìm thấy dữ liệu Sentinel-2 cho khu vực này!\n\n"
@@ -4646,8 +4988,7 @@ async def ndvi_forecast(config: NDVIForecastConfig):
                        f"5. Đảm bảo forecast_start_date không quá xa trong tương lai"
             )
         
-        if len(s2_items) > config.max_scenes:
+        print(f"✅ Found {len(s2_items)} historical Sentinel-2 scenes")
-            s2_items = s2_items[:config.max_scenes]
         
         print(f"✅ Found {len(s2_items)} historical scenes")
         

cloud_removal.py

@@ -0,0 +1,628 @@
+"""
+Cloud Removal Module - Hệ thống xử lý mây độc lập
+Cung cấp nhiều phương pháp khử mây cho dữ liệu Sentinel-2
+"""
+
+import numpy as np
+import xarray as xr
+from typing import Tuple, Optional, Dict
+from sklearn.neighbors import KNeighborsRegressor
+from sklearn.ensemble import RandomForestRegressor
+import warnings
+warnings.filterwarnings('ignore')
+
+
+class CloudRemovalStrategy:
+    """Base class cho các chiến lược xử lý mây"""
+    
+    def __init__(self, name: str, description: str):
+        self.name = name
+        self.description = description
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        """
+        Xử lý mây và trả về dữ liệu đã được làm sạch
+        
+        Returns:
+            Tuple[xr.Dataset, Dict]: (cleaned_data, metadata)
+        """
+        raise NotImplementedError
+
+
+class ClassicStrategy(CloudRemovalStrategy):
+    """
+    Chiến lược cổ điển 3 bước:
+    1. Temporal interpolation (ffill + bfill)
+    2. Median compositing (nếu >= 3 scenes)
+    3. Spatial interpolation (nearest neighbor)
+    """
+    
+    def __init__(self):
+        super().__init__(
+            name="classic",
+            description="3-step classical approach: temporal → median → spatial interpolation"
+        )
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'steps_applied': []
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        # Step 1: Temporal Interpolation
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].ffill(dim='time').bfill(dim='time')
+        metadata['steps_applied'].append('temporal_interpolation')
+        
+        # Step 2: Median Compositing (if >= 3 time steps)
+        if len(s2_data.time) >= 3:
+            for band in s2_data.data_vars:
+                if band != "SCL":
+                    median_composite = s2_data[band].median(dim='time', skipna=True)
+                    s2_data[band] = s2_data[band].fillna(median_composite)
+            metadata['steps_applied'].append('median_compositing')
+        
+        # Step 3: Spatial Interpolation
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].interpolate_na(dim='x', method='nearest', fill_value='extrapolate')
+                s2_data[band] = s2_data[band].interpolate_na(dim='y', method='nearest', fill_value='extrapolate')
+        metadata['steps_applied'].append('spatial_interpolation')
+        
+        # Final fallback
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+class NoRemovalStrategy(CloudRemovalStrategy):
+    """Không xử lý mây - giữ nguyên dữ liệu gốc, chỉ fill NaN bằng 0"""
+    
+    def __init__(self):
+        super().__init__(
+            name="none",
+            description="No cloud removal - keep original data with NaN filled as 0"
+        )
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'steps_applied': ['none'],
+            'note': 'No cloud removal applied, only NaN filling'
+        }
+        
+        # Chỉ fill NaN bằng 0, không apply cloud mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+class TemporalOnlyStrategy(CloudRemovalStrategy):
+    """Chỉ sử dụng temporal interpolation - nhanh nhất, phù hợp khi có nhiều time steps"""
+    
+    def __init__(self):
+        super().__init__(
+            name="temporal_only",
+            description="Temporal interpolation only - fast, good for time series with many scenes"
+        )
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'steps_applied': ['temporal_interpolation']
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        # Temporal interpolation
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].ffill(dim='time').bfill(dim='time')
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+class MedianCompositeStrategy(CloudRemovalStrategy):
+    """Ưu tiên median composite - tốt nhất cho giảm noise"""
+    
+    def __init__(self):
+        super().__init__(
+            name="median_composite",
+            description="Median composite priority - best for noise reduction"
+        )
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'steps_applied': ['median_compositing', 'spatial_interpolation']
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        # Direct median composite
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                median_composite = s2_data[band].median(dim='time', skipna=True)
+                # Fill all NaN with median
+                s2_data[band] = s2_data[band].fillna(median_composite)
+        
+        # Spatial interpolation for remaining gaps
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].interpolate_na(dim='x', method='nearest')
+                s2_data[band] = s2_data[band].interpolate_na(dim='y', method='nearest')
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+class MLInpaintingStrategy(CloudRemovalStrategy):
+    """
+    Machine Learning Inpainting - sử dụng KNN hoặc Random Forest
+    Học từ pixels hợp lệ để dự đoán pixels bị mây
+    """
+    
+    def __init__(self, ml_model: str = "knn"):
+        """
+        Args:
+            ml_model: 'knn' hoặc 'rf' (random forest)
+        """
+        super().__init__(
+            name=f"ml_inpainting_{ml_model}",
+            description=f"ML-based cloud removal using {ml_model.upper()} - learns from valid pixels"
+        )
+        self.ml_model = ml_model
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'ml_model': self.ml_model,
+            'steps_applied': []
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        # ML inpainting cho từng time step
+        for time_idx in range(len(s2_data.time)):
+            # Get all bands for this time step
+            bands_data = []
+            band_names = []
+            
+            for band in s2_data.data_vars:
+                if band != "SCL":
+                    band_data = s2_data[band].isel(time=time_idx).values
+                    bands_data.append(band_data.flatten())
+                    band_names.append(band)
+            
+            if not bands_data:
+                continue
+            
+            # Stack bands: shape (n_pixels, n_bands)
+            X_all = np.column_stack(bands_data)
+            
+            # Find valid (non-NaN) and invalid (NaN) pixels
+            valid_mask = ~np.isnan(X_all).any(axis=1)
+            
+            if valid_mask.sum() < 10:  # Not enough training data
+                continue
+            
+            X_valid = X_all[valid_mask]
+            X_invalid_indices = np.where(~valid_mask)[0]
+            
+            if len(X_invalid_indices) == 0:  # No clouds
+                continue
+            
+            # Prepare features: use spatial coordinates + spectral values
+            y_coords, x_coords = np.meshgrid(
+                np.arange(s2_data.dims['y']),
+                np.arange(s2_data.dims['x']),
+                indexing='ij'
+            )
+            coords_flat = np.column_stack([y_coords.flatten(), x_coords.flatten()])
+            
+            # Train ML model on valid pixels
+            X_train = coords_flat[valid_mask]
+            y_train = X_valid
+            
+            try:
+                if self.ml_model == "knn":
+                    model = KNeighborsRegressor(n_neighbors=min(5, len(X_train)), weights='distance')
+                else:  # random forest
+                    model = RandomForestRegressor(n_estimators=10, max_depth=10, random_state=42, n_jobs=-1)
+                
+                model.fit(X_train, y_train)
+                
+                # Predict invalid pixels
+                X_test = coords_flat[X_invalid_indices]
+                predictions = model.predict(X_test)
+                
+                # Fill predictions back
+                X_all[X_invalid_indices] = predictions
+                
+                # Reshape and update dataset
+                for band_idx, band in enumerate(band_names):
+                    filled_data = X_all[:, band_idx].reshape(s2_data.dims['y'], s2_data.dims['x'])
+                    s2_data[band].values[time_idx] = filled_data
+                
+                metadata['steps_applied'].append(f'ml_inpainting_time_{time_idx}')
+                
+            except Exception as e:
+                print(f"[ML INPAINTING] Error at time {time_idx}: {e}")
+                continue
+        
+        # Final cleanup
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+class DeepInpaintingStrategy(CloudRemovalStrategy):
+    """
+    Deep Learning Inpainting - sử dụng U-Net CNN
+    Phức tạp hơn nhưng cho kết quả tốt nhất với large cloud gaps
+    
+    Note: Yêu cầu pretrained model (train bằng train_cloud_removal.py)
+    """
+    
+    def __init__(self, model_path: Optional[str] = None):
+        super().__init__(
+            name="deep_inpainting",
+            description="Deep Learning U-Net based cloud removal - best quality for large gaps"
+        )
+        self.model_path = model_path or "model_train/cloud_removal_unet_best.pth"
+        self.model = None
+        self.device = None
+        
+        # Try to load model if provided
+        if model_path or Path(self.model_path).exists():
+            try:
+                import torch
+                import torch.nn as nn
+                
+                # Load checkpoint
+                checkpoint = torch.load(self.model_path, map_location='cpu')
+                
+                # Recreate U-Net architecture
+                from train_cloud_removal import UNet
+                self.model = UNet(
+                    in_channels=checkpoint.get('in_channels', 4),
+                    out_channels=checkpoint.get('out_channels', 4)
+                )
+                self.model.load_state_dict(checkpoint['model_state_dict'])
+                
+                # Set device
+                self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+                self.model = self.model.to(self.device)
+                self.model.eval()
+                
+                print(f"[DEEP INPAINTING] Loaded U-Net model from {self.model_path}")
+                print(f"[DEEP INPAINTING] Using device: {self.device}")
+            except Exception as e:
+                print(f"[DEEP INPAINTING] Could not load model: {e}")
+                self.model = None
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'has_model': self.model is not None,
+            'steps_applied': []
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        if self.model is None:
+            # Fallback to classical method
+            print("[DEEP INPAINTING] No model available, falling back to median composite")
+            for band in s2_data.data_vars:
+                if band != "SCL":
+                    median_composite = s2_data[band].median(dim='time', skipna=True)
+                    s2_data[band] = s2_data[band].fillna(median_composite)
+                    s2_data[band] = s2_data[band].interpolate_na(dim='x', method='nearest')
+                    s2_data[band] = s2_data[band].interpolate_na(dim='y', method='nearest')
+                    s2_data[band] = s2_data[band].fillna(0)
+            metadata['steps_applied'].append('fallback_median')
+        else:
+            # Use U-Net for cloud removal
+            print("[DEEP INPAINTING] Applying U-Net cloud removal...")
+            import torch
+            
+            try:
+                # Process each time step
+                for time_idx in range(len(s2_data.time)):
+                    # Get bands for this time step (B02, B03, B04, B08)
+                    bands_to_process = ['B02', 'B03', 'B04', 'B08']
+                    available_bands = [b for b in bands_to_process if b in s2_data.data_vars]
+                    
+                    if len(available_bands) < 4:
+                        print(f"[DEEP INPAINTING] Warning: Not all required bands available, skipping time {time_idx}")
+                        continue
+                    
+                    # Stack bands [C, H, W]
+                    input_bands = []
+                    for band in available_bands:
+                        band_data = s2_data[band].isel(time=time_idx).values.astype(np.float32)
+                        # Normalize to [0, 1] (S2 values are typically 0-10000)
+                        band_data = np.clip(band_data / 10000.0, 0, 1)
+                        input_bands.append(band_data)
+                    
+                    input_array = np.stack(input_bands, axis=0)  # [C, H, W]
+                    
+                    # Convert to tensor and add batch dimension
+                    input_tensor = torch.from_numpy(input_array).unsqueeze(0).to(self.device)
+                    
+                    # Run through U-Net
+                    with torch.no_grad():
+                        output_tensor = self.model(input_tensor)
+                    
+                    # Convert back to numpy
+                    output_array = output_tensor[0].cpu().numpy()  # [C, H, W]
+                    
+                    # Denormalize back to original scale
+                    output_array = output_array * 10000.0
+                    
+                    # Update dataset with cleaned data
+                    for i, band in enumerate(available_bands):
+                        s2_data[band].values[time_idx] = output_array[i]
+                    
+                    metadata['steps_applied'].append(f'unet_time_{time_idx}')
+                
+                print(f"[DEEP INPAINTING] Processed {len(s2_data.time)} time steps with U-Net")
+                
+            except Exception as e:
+                print(f"[DEEP INPAINTING] Error during inference: {e}")
+                # Fallback to classical method
+                for band in s2_data.data_vars:
+                    if band != "SCL":
+                        s2_data[band] = s2_data[band].ffill(dim='time').bfill(dim='time')
+                        s2_data[band] = s2_data[band].fillna(0)
+                metadata['steps_applied'].append('unet_error_fallback')
+        
+        return s2_data, metadata
+
+
+class HybridStrategy(CloudRemovalStrategy):
+    """
+    Hybrid Strategy - kết hợp Classical + ML
+    1. Classical temporal interpolation (nhanh)
+    2. ML inpainting cho gaps còn lại (chất lượng cao)
+    3. Spatial interpolation (cleanup)
+    """
+    
+    def __init__(self):
+        super().__init__(
+            name="hybrid",
+            description="Hybrid classical + ML - balanced speed and quality"
+        )
+    
+    def remove_clouds(self, s2_data: xr.Dataset, cloud_mask: xr.DataArray) -> Tuple[xr.Dataset, Dict]:
+        metadata = {
+            'method': self.name,
+            'steps_applied': []
+        }
+        
+        # Apply mask
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].where(~cloud_mask)
+        
+        # Step 1: Temporal interpolation (fast)
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].ffill(dim='time').bfill(dim='time')
+        metadata['steps_applied'].append('temporal_interpolation')
+        
+        # Step 2: Check remaining NaN percentage
+        nan_count = 0
+        total_count = 0
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                nan_count += np.isnan(s2_data[band].values).sum()
+                total_count += s2_data[band].values.size
+        
+        nan_percentage = (nan_count / total_count * 100) if total_count > 0 else 0
+        
+        # Step 3: ML inpainting if still significant gaps (>5%)
+        if nan_percentage > 5.0:
+            print(f"[HYBRID] {nan_percentage:.1f}% NaN remaining, applying ML inpainting...")
+            ml_strategy = MLInpaintingStrategy(ml_model="knn")
+            s2_data, ml_meta = ml_strategy.remove_clouds(s2_data, cloud_mask)
+            metadata['steps_applied'].extend(['ml_inpainting_knn'])
+            metadata['nan_before_ml'] = nan_percentage
+        else:
+            # Step 4: Spatial interpolation for small gaps
+            for band in s2_data.data_vars:
+                if band != "SCL":
+                    s2_data[band] = s2_data[band].interpolate_na(dim='x', method='nearest')
+                    s2_data[band] = s2_data[band].interpolate_na(dim='y', method='nearest')
+            metadata['steps_applied'].append('spatial_interpolation')
+        
+        # Final cleanup
+        for band in s2_data.data_vars:
+            if band != "SCL":
+                s2_data[band] = s2_data[band].fillna(0)
+        
+        return s2_data, metadata
+
+
+# ============ FACTORY & UTILITIES ============
+
+def get_available_methods() -> Dict[str, str]:
+    """Trả về dictionary của tất cả methods có sẵn"""
+    return {
+        "none": "No cloud removal - keep original data (fastest, may have cloud artifacts)",
+        "classic": "3-step classical: temporal → median → spatial (default, balanced)",
+        "temporal_only": "Temporal interpolation only (fast, needs many scenes)",
+        "median_composite": "Median composite priority (best noise reduction)",
+        "ml_knn": "ML K-Nearest Neighbors inpainting (good quality, medium speed)",
+        "ml_rf": "ML Random Forest inpainting (high quality, slower)",
+        "deep": "Deep Learning CNN inpainting (best quality, requires model)",
+        "hybrid": "Hybrid classical + ML (balanced speed & quality)"
+    }
+
+
+def create_cloud_removal_strategy(method: str = "classic", **kwargs) -> CloudRemovalStrategy:
+    """
+    Factory function để tạo strategy từ tên method
+    
+    Args:
+        method: Tên method ("classic", "temporal_only", "median_composite", 
+                "ml_knn", "ml_rf", "deep", "hybrid")
+        **kwargs: Additional parameters cho specific strategies
+    
+    Returns:
+        CloudRemovalStrategy instance
+    """
+    method = method.lower()
+    
+    if method == "none":
+        return NoRemovalStrategy()
+    elif method == "classic":
+        return ClassicStrategy()
+    elif method == "temporal_only":
+        return TemporalOnlyStrategy()
+    elif method == "median_composite":
+        return MedianCompositeStrategy()
+    elif method == "ml_knn":
+        return MLInpaintingStrategy(ml_model="knn")
+    elif method == "ml_rf":
+        return MLInpaintingStrategy(ml_model="rf")
+    elif method == "deep":
+        model_path = kwargs.get('model_path', None)
+        return DeepInpaintingStrategy(model_path=model_path)
+    elif method == "hybrid":
+        return HybridStrategy()
+    else:
+        print(f"[CLOUD REMOVAL] Unknown method '{method}', using 'classic'")
+        return ClassicStrategy()
+
+
+def process_cloud_removal(
+    s2_data: xr.Dataset,
+    method: str = "classic",
+    verbose: bool = True,
+    **kwargs
+) -> Tuple[xr.Dataset, Dict]:
+    """
+    Main entry point cho cloud removal
+    
+    Args:
+        s2_data: Sentinel-2 dataset với SCL band
+        method: Cloud removal method name
+        verbose: Print progress messages
+        **kwargs: Additional parameters
+    
+    Returns:
+        Tuple[xr.Dataset, Dict]: (cleaned_data, metadata)
+    """
+    if verbose:
+        print(f"[CLOUD REMOVAL] Using method: {method}")
+    
+    # Detect clouds from SCL
+    if "SCL" not in s2_data:
+        if verbose:
+            print("[CLOUD REMOVAL] Warning: No SCL band, cannot mask clouds")
+        return s2_data, {'method': 'none', 'warning': 'no_scl_band'}
+    
+    scl = s2_data["SCL"]
+    
+    # Create comprehensive cloud mask
+    cloud_mask = (scl == 3) | (scl == 8) | (scl == 9) | (scl == 10) | (scl == 11)
+    invalid_mask = (scl == 0) | (scl == 1)
+    full_mask = cloud_mask | invalid_mask
+    
+    # Calculate coverage
+    total_pixels = full_mask.size
+    masked_pixels = int(full_mask.sum().values)
+    cloud_coverage_percent = (masked_pixels / total_pixels * 100) if total_pixels > 0 else 0
+    
+    if verbose:
+        print(f"[CLOUD REMOVAL] Cloud coverage: {cloud_coverage_percent:.1f}%")
+        print(f"[CLOUD REMOVAL] Masked pixels: {masked_pixels:,}/{total_pixels:,}")
+    
+    # Create strategy and process
+    strategy = create_cloud_removal_strategy(method, **kwargs)
+    cleaned_data, metadata = strategy.remove_clouds(s2_data.copy(deep=True), full_mask)
+    
+    # Add coverage info to metadata
+    metadata['cloud_coverage_percent'] = float(cloud_coverage_percent)
+    metadata['masked_pixels'] = masked_pixels
+    metadata['total_pixels'] = total_pixels
+    
+    if verbose:
+        print(f"[CLOUD REMOVAL] Completed using {metadata['method']}")
+        print(f"[CLOUD REMOVAL] Steps: {', '.join(metadata['steps_applied'])}")
+    
+    return cleaned_data, metadata
+
+
+# ============ TESTING & COMPARISON ============
+
+def compare_methods(s2_data: xr.Dataset, methods: list = None) -> Dict:
+    """
+    So sánh các methods khác nhau trên cùng dữ liệu
+    
+    Args:
+        s2_data: Sentinel-2 dataset
+        methods: List of method names to compare (default: all)
+    
+    Returns:
+        Dict: Comparison results
+    """
+    if methods is None:
+        methods = ["classic", "temporal_only", "median_composite", "ml_knn", "hybrid"]
+    
+    results = {}
+    
+    for method in methods:
+        try:
+            print(f"\n{'='*60}")
+            print(f"Testing: {method}")
+            print(f"{'='*60}")
+            
+            cleaned_data, metadata = process_cloud_removal(s2_data, method=method, verbose=True)
+            
+            # Calculate remaining NaN
+            nan_count = sum(np.isnan(cleaned_data[band].values).sum() 
+                          for band in cleaned_data.data_vars if band != "SCL")
+            total_count = sum(cleaned_data[band].values.size 
+                            for band in cleaned_data.data_vars if band != "SCL")
+            
+            results[method] = {
+                'metadata': metadata,
+                'remaining_nan_percent': (nan_count / total_count * 100) if total_count > 0 else 0,
+                'success': True
+            }
+            
+        except Exception as e:
+            results[method] = {
+                'error': str(e),
+                'success': False
+            }
+            print(f"[ERROR] {method}: {e}")
+    
+    return results

cloud_removal_train.ipynb

@@ -0,0 +1,10 @@
+{
+ "cells": [],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

cloud_training_interface.html

@@ -0,0 +1,605 @@
+<!DOCTYPE html>
+<html lang="vi">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Cloud Removal Training - Deep Learning</title>
+    <style>
+        * {
+            margin: 0;
+            padding: 0;
+            box-sizing: border-box;
+        }
+
+        body {
+            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
+            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+            min-height: 100vh;
+            padding: 20px;
+        }
+
+        .container {
+            max-width: 1200px;
+            margin: 0 auto;
+            background: white;
+            border-radius: 15px;
+            box-shadow: 0 20px 60px rgba(0,0,0,0.3);
+            overflow: hidden;
+        }
+
+        .header {
+            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+            color: white;
+            padding: 30px;
+            text-align: center;
+        }
+
+        .header h1 {
+            font-size: 2.5em;
+            margin-bottom: 10px;
+        }
+
+        .header p {
+            font-size: 1.1em;
+            opacity: 0.9;
+        }
+
+        .nav {
+            background: #f8f9fa;
+            padding: 15px 30px;
+            border-bottom: 2px solid #e9ecef;
+        }
+
+        .nav a {
+            color: #667eea;
+            text-decoration: none;
+            margin-right: 20px;
+            font-weight: 500;
+            transition: color 0.3s;
+        }
+
+        .nav a:hover {
+            color: #764ba2;
+        }
+
+        .content {
+            padding: 30px;
+        }
+
+        .section {
+            margin-bottom: 30px;
+        }
+
+        .section-title {
+            font-size: 1.5em;
+            color: #333;
+            margin-bottom: 15px;
+            padding-bottom: 10px;
+            border-bottom: 3px solid #667eea;
+        }
+
+        .card {
+            background: #f8f9fa;
+            border-radius: 10px;
+            padding: 20px;
+            margin-bottom: 20px;
+            border-left: 4px solid #667eea;
+        }
+
+        .form-group {
+            margin-bottom: 20px;
+        }
+
+        label {
+            display: block;
+            font-weight: 600;
+            margin-bottom: 8px;
+            color: #333;
+        }
+
+        input[type="text"],
+        input[type="number"],
+        select {
+            width: 100%;
+            padding: 12px;
+            border: 2px solid #e9ecef;
+            border-radius: 8px;
+            font-size: 1em;
+            transition: border-color 0.3s;
+        }
+
+        input[type="text"]:focus,
+        input[type="number"]:focus,
+        select:focus {
+            outline: none;
+            border-color: #667eea;
+        }
+
+        .checkbox-group {
+            display: flex;
+            align-items: center;
+            gap: 10px;
+        }
+
+        input[type="checkbox"] {
+            width: 20px;
+            height: 20px;
+            cursor: pointer;
+        }
+
+        .btn {
+            padding: 12px 30px;
+            border: none;
+            border-radius: 8px;
+            font-size: 1em;
+            font-weight: 600;
+            cursor: pointer;
+            transition: all 0.3s;
+            margin-right: 10px;
+        }
+
+        .btn-primary {
+            background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+            color: white;
+        }
+
+        .btn-primary:hover {
+            transform: translateY(-2px);
+            box-shadow: 0 5px 15px rgba(102, 126, 234, 0.4);
+        }
+
+        .btn-secondary {
+            background: #6c757d;
+            color: white;
+        }
+
+        .btn-danger {
+            background: #dc3545;
+            color: white;
+        }
+
+        .btn-success {
+            background: #28a745;
+            color: white;
+        }
+
+        .model-list {
+            display: grid;
+            grid-template-columns: repeat(auto-fill, minmax(300px, 1fr));
+            gap: 20px;
+        }
+
+        .model-card {
+            background: white;
+            border: 2px solid #e9ecef;
+            border-radius: 10px;
+            padding: 20px;
+            transition: all 0.3s;
+        }
+
+        .model-card:hover {
+            border-color: #667eea;
+            box-shadow: 0 5px 15px rgba(0,0,0,0.1);
+        }
+
+        .model-card h3 {
+            color: #667eea;
+            margin-bottom: 10px;
+        }
+
+        .model-info {
+            font-size: 0.9em;
+            color: #6c757d;
+            margin: 5px 0;
+        }
+
+        .status-badge {
+            display: inline-block;
+            padding: 5px 15px;
+            border-radius: 20px;
+            font-size: 0.9em;
+            font-weight: 600;
+            margin-top: 10px;
+        }
+
+        .status-success {
+            background: #d4edda;
+            color: #155724;
+        }
+
+        .status-training {
+            background: #fff3cd;
+            color: #856404;
+        }
+
+        .status-error {
+            background: #f8d7da;
+            color: #721c24;
+        }
+
+        .progress-bar {
+            width: 100%;
+            height: 30px;
+            background: #e9ecef;
+            border-radius: 15px;
+            overflow: hidden;
+            margin: 20px 0;
+        }
+
+        .progress-fill {
+            height: 100%;
+            background: linear-gradient(90deg, #667eea 0%, #764ba2 100%);
+            transition: width 0.3s;
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            color: white;
+            font-weight: 600;
+        }
+
+        .info-box {
+            background: #e7f3ff;
+            border-left: 4px solid #2196F3;
+            padding: 15px;
+            border-radius: 5px;
+            margin-bottom: 20px;
+        }
+
+        .warning-box {
+            background: #fff3cd;
+            border-left: 4px solid #ffc107;
+            padding: 15px;
+            border-radius: 5px;
+            margin-bottom: 20px;
+        }
+
+        .grid-2 {
+            display: grid;
+            grid-template-columns: 1fr 1fr;
+            gap: 20px;
+        }
+
+        @media (max-width: 768px) {
+            .grid-2 {
+                grid-template-columns: 1fr;
+            }
+
+            .model-list {
+                grid-template-columns: 1fr;
+            }
+        }
+
+        .logs {
+            background: #1e1e1e;
+            color: #d4d4d4;
+            padding: 20px;
+            border-radius: 8px;
+            font-family: 'Courier New', monospace;
+            font-size: 0.9em;
+            max-height: 400px;
+            overflow-y: auto;
+            margin-top: 20px;
+        }
+
+        .logs .log-entry {
+            margin: 5px 0;
+        }
+
+        .logs .log-info {
+            color: #4ec9b0;
+        }
+
+        .logs .log-warning {
+            color: #dcdcaa;
+        }
+
+        .logs .log-error {
+            color: #f48771;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <div class="header">
+            <h1>🌥️ Cloud Removal Training</h1>
+            <p>Train Deep Learning Models để khử mây từ ảnh Sentinel-2</p>
+        </div>
+
+        <div class="nav">
+            <a href="/">← Trang chủ</a>
+            <a href="/training">Land Classification</a>
+            <a href="/prediction">Prediction</a>
+            <a href="#models">Models đã train</a>
+        </div>
+
+        <div class="content">
+            <!-- Info Section -->
+            <div class="section">
+                <div class="info-box">
+                    <strong>📚 Dataset:</strong> SEN12MS-CR (Sentinel-12 Multi-Seasonal Cloud Removal)<br>
+                    <strong>🏗️ Architecture:</strong> U-Net với skip connections<br>
+                    <strong>📊 Input:</strong> S2 cloudy (4 bands) + S1 radar (2 bands) = 6 channels<br>
+                    <strong>🎯 Output:</strong> S2 clean (4 bands)<br>
+                    <strong>⏱️ Training time:</strong> ~2-3 hours (GPU) / ~20-30 hours (CPU)
+                </div>
+            </div>
+
+            <!-- Training Configuration -->
+            <div class="section">
+                <h2 class="section-title">⚙️ Cấu hình Training</h2>
+                
+                <div class="card">
+                    <form id="trainingForm">
+                        <div class="grid-2">
+                            <div class="form-group">
+                                <label>📂 Data Directory</label>
+                                <input type="text" id="dataDir" value="winter_dataset" required>
+                                <small style="color: #6c757d;">Thư mục chứa dữ liệu SEN12MS-CR</small>
+                            </div>
+
+                            <div class="form-group">
+                                <label>🏷️ Model Name</label>
+                                <input type="text" id="modelName" value="cloud_removal_unet" required>
+                                <small style="color: #6c757d;">Tên model để lưu</small>
+                            </div>
+
+                            <div class="form-group">
+                                <label>📦 Batch Size</label>
+                                <input type="number" id="batchSize" value="8" min="1" max="32" required>
+                                <small style="color: #6c757d;">Giảm xuống 4 hoặc 2 nếu GPU hết RAM</small>
+                            </div>
+
+                            <div class="form-group">
+                                <label>🔄 Number of Epochs</label>
+                                <input type="number" id="numEpochs" value="50" min="1" max="200" required>
+                                <small style="color: #6c757d;">Số lượng epochs training</small>
+                            </div>
+
+                            <div class="form-group">
+                                <label>📈 Learning Rate</label>
+                                <input type="number" id="learningRate" value="0.0001" step="0.00001" min="0.00001" max="0.01" required>
+                                <small style="color: #6c757d;">Learning rate (default: 1e-4)</small>
+                            </div>
+
+                            <div class="form-group">
+                                <div class="checkbox-group">
+                                    <input type="checkbox" id="useS1" checked>
+                                    <label for="useS1">📡 Use Sentinel-1 (Radar Data)</label>
+                                </div>
+                                <small style="color: #6c757d;">Sử dụng dữ liệu radar (VV, VH) để cải thiện kết quả</small>
+                            </div>
+
+                            <div class="form-group">
+                                <div class="checkbox-group">
+                                    <input type="checkbox" id="useGPU" checked>
+                                    <label for="useGPU">🚀 Use GPU</label>
+                                </div>
+                                <small style="color: #6c757d;">Sử dụng GPU để training nhanh hơn</small>
+                            </div>
+                        </div>
+
+                        <div class="form-group" style="margin-top: 20px;">
+                            <button type="submit" class="btn btn-primary">🚀 Start Training</button>
+                            <button type="button" class="btn btn-secondary" onclick="refreshModels()">🔄 Refresh Models</button>
+                        </div>
+                    </form>
+                </div>
+            </div>
+
+            <!-- Training Status -->
+            <div class="section" id="trainingStatus" style="display: none;">
+                <h2 class="section-title">📊 Training Status</h2>
+                <div class="card">
+                    <div id="statusMessage"></div>
+                    <div class="progress-bar">
+                        <div class="progress-fill" id="progressBar" style="width: 0%;">0%</div>
+                    </div>
+                    <div class="logs" id="trainingLogs">
+                        <div class="log-entry log-info">Training logs will appear here...</div>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Models List -->
+            <div class="section" id="models">
+                <h2 class="section-title">🤖 Cloud Removal Models</h2>
+                <div class="model-list" id="modelsList">
+                    <div class="model-card">
+                        <p style="text-align: center; color: #6c757d;">Loading models...</p>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Methods Info -->
+            <div class="section">
+                <h2 class="section-title">📖 Cloud Removal Methods</h2>
+                <div class="grid-2">
+                    <div class="card">
+                        <h3>🔹 Classic (Default)</h3>
+                        <p>3-step approach: temporal → median → spatial interpolation</p>
+                        <div class="status-badge status-success">Fast</div>
+                    </div>
+
+                    <div class="card">
+                        <h3>🔹 Hybrid</h3>
+                        <p>Classical + ML KNN - balanced speed & quality</p>
+                        <div class="status-badge status-success">Recommended</div>
+                    </div>
+
+                    <div class="card">
+                        <h3>🔹 ML KNN</h3>
+                        <p>K-Nearest Neighbors inpainting - good quality</p>
+                        <div class="status-badge status-training">Medium Speed</div>
+                    </div>
+
+                    <div class="card">
+                        <h3>🔹 Deep Learning</h3>
+                        <p>U-Net CNN - best quality for large gaps</p>
+                        <div class="status-badge status-error">Requires Model</div>
+                    </div>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    <script>
+        // Load models on page load
+        window.addEventListener('load', () => {
+            refreshModels();
+            loadCloudRemovalMethods();
+        });
+
+        // Handle training form submission
+        document.getElementById('trainingForm').addEventListener('submit', async (e) => {
+            e.preventDefault();
+
+            const config = {
+                data_dir: document.getElementById('dataDir').value,
+                model_name: document.getElementById('modelName').value,
+                use_s1: document.getElementById('useS1').checked,
+                batch_size: parseInt(document.getElementById('batchSize').value),
+                num_epochs: parseInt(document.getElementById('numEpochs').value),
+                learning_rate: parseFloat(document.getElementById('learningRate').value),
+                use_gpu: document.getElementById('useGPU').checked
+            };
+
+            try {
+                const response = await fetch('/api/cloud-removal/train', {
+                    method: 'POST',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify(config)
+                });
+
+                const result = await response.json();
+
+                if (response.ok) {
+                    // Show training status section
+                    document.getElementById('trainingStatus').style.display = 'block';
+                    document.getElementById('statusMessage').innerHTML = `
+                        <div class="status-badge status-training">Training Started: ${result.training_id}</div>
+                        <p style="margin-top: 10px;">Model training has started in background. This may take several hours.</p>
+                    `;
+
+                    addLog('info', `Training started: ${result.training_id}`);
+                    addLog('info', `Config: ${JSON.stringify(config, null, 2)}`);
+
+                    // Simulate progress (actual progress would come from websocket)
+                    simulateProgress();
+                } else {
+                    alert('Error starting training: ' + (result.detail || result.error));
+                }
+            } catch (error) {
+                alert('Error: ' + error.message);
+            }
+        });
+
+        // Refresh models list
+        async function refreshModels() {
+            try {
+                const response = await fetch('/api/cloud-removal/models');
+                const data = await response.json();
+
+                const modelsList = document.getElementById('modelsList');
+
+                if (data.models && data.models.length > 0) {
+                    modelsList.innerHTML = data.models.map(model => `
+                        <div class="model-card">
+                            <h3>📦 ${model.filename}</h3>
+                            <div class="model-info">📊 Epoch: ${model.epoch}</div>
+                            <div class="model-info">📉 Train Loss: ${model.train_loss.toFixed(6)}</div>
+                            <div class="model-info">📉 Val Loss: ${model.val_loss.toFixed(6)}</div>
+                            <div class="model-info">📡 Use S1: ${model.use_s1 ? 'Yes' : 'No'}</div>
+                            <div class="model-info">💾 Size: ${model.size_mb.toFixed(2)} MB</div>
+                            <div class="model-info">📅 Created: ${new Date(model.created * 1000).toLocaleString()}</div>
+                            <div style="margin-top: 15px;">
+                                <button class="btn btn-danger" onclick="deleteModel('${model.filename}')">
+                                    🗑️ Delete
+                                </button>
+                            </div>
+                        </div>
+                    `).join('');
+                } else {
+                    modelsList.innerHTML = `
+                        <div class="model-card">
+                            <p style="text-align: center; color: #6c757d;">
+                                No cloud removal models found.<br>
+                                Train your first model above!
+                            </p>
+                        </div>
+                    `;
+                }
+            } catch (error) {
+                console.error('Error loading models:', error);
+            }
+        }
+
+        // Delete model
+        async function deleteModel(filename) {
+            if (!confirm(`Delete model ${filename}?`)) return;
+
+            try {
+                const response = await fetch(`/api/cloud-removal/models/${filename}`, {
+                    method: 'DELETE'
+                });
+
+                if (response.ok) {
+                    alert('Model deleted successfully');
+                    refreshModels();
+                } else {
+                    const error = await response.json();
+                    alert('Error deleting model: ' + error.detail);
+                }
+            } catch (error) {
+                alert('Error: ' + error.message);
+            }
+        }
+
+        // Load cloud removal methods
+        async function loadCloudRemovalMethods() {
+            try {
+                const response = await fetch('/api/cloud-removal/methods');
+                const data = await response.json();
+                console.log('Available cloud removal methods:', data.methods);
+            } catch (error) {
+                console.error('Error loading methods:', error);
+            }
+        }
+
+        // Add log entry
+        function addLog(type, message) {
+            const logs = document.getElementById('trainingLogs');
+            const timestamp = new Date().toLocaleTimeString();
+            const logClass = type === 'error' ? 'log-error' : (type === 'warning' ? 'log-warning' : 'log-info');
+            
+            const entry = document.createElement('div');
+            entry.className = `log-entry ${logClass}`;
+            entry.textContent = `[${timestamp}] ${message}`;
+            
+            logs.appendChild(entry);
+            logs.scrollTop = logs.scrollHeight;
+        }
+
+        // Simulate progress (replace with real progress tracking)
+        function simulateProgress() {
+            let progress = 0;
+            const interval = setInterval(() => {
+                progress += Math.random() * 5;
+                if (progress >= 100) {
+                    progress = 100;
+                    clearInterval(interval);
+                    addLog('info', 'Training completed! Check models list below.');
+                    setTimeout(refreshModels, 2000);
+                }
+                
+                const progressBar = document.getElementById('progressBar');
+                progressBar.style.width = progress + '%';
+                progressBar.textContent = Math.floor(progress) + '%';
+
+                if (progress % 10 < 5) {
+                    addLog('info', `Training progress: ${Math.floor(progress)}%`);
+                }
+            }, 3000);
+        }
+    </script>
+</body>
+</html>

index.html

@@ -430,6 +430,7 @@
         <div style="background: white; padding: 15px; display: flex; gap: 10px; flex-wrap: wrap; justify-content: center; border-bottom: 2px solid #e0e0e0;">
             <a href="/" style="padding: 10px 20px; background: #667eea; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🏠 Trang Chủ (Active)</a>
             <a href="/training" style="padding: 10px 20px; background: #f093fb; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🎓 Training</a>
+            <a href="/cloud-training" style="padding: 10px 20px; background: #00bcd4; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌥️ Cloud Removal</a>
             <a href="/prediction" style="padding: 10px 20px; background: #4facfe; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🗺️ Prediction</a>
             <a href="/batch" style="padding: 10px 20px; background: #764ba2; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🚀 Batch Processing</a>
             <a href="/ndvi" style="padding: 10px 20px; background: #2ecc71; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌿 NDVI Analysis</a>

prediction_interface.html

@@ -295,6 +295,7 @@
         <div style="background: white; padding: 15px; display: flex; gap: 10px; flex-wrap: wrap; justify-content: center; border-bottom: 2px solid #e0e0e0;">
             <a href="/" style="padding: 10px 20px; background: #667eea; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🏠 Trang Chủ</a>
             <a href="/training" style="padding: 10px 20px; background: #f093fb; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🎓 Training</a>
+            <a href="/cloud-training" style="padding: 10px 20px; background: #00bcd4; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌥️ Cloud Removal</a>
             <a href="/prediction" style="padding: 10px 20px; background: #4facfe; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🗺️ Prediction (Active)</a>
             <a href="/batch" style="padding: 10px 20px; background: #764ba2; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🚀 Batch Processing</a>
             <a href="/ndvi" style="padding: 10px 20px; background: #2ecc71; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌿 NDVI Analysis</a>
@@ -427,6 +428,97 @@
                     </select>
                 </div>
                 
+                <!-- Cloud Removal Configuration -->
+                <div class="form-group" style="margin-top: 15px; padding: 15px; background: #e3f2fd; border-radius: 8px; border-left: 4px solid #2196F3;">
+                    <label for="cloudRemovalMethod" style="font-weight: 600; color: #1976d2; margin-bottom: 10px; display: block;">
+                        🌥️ Cloud Removal Method
+                    </label>
+                    <select id="cloudRemovalMethod" onchange="handleCloudMethodChange()" style="padding: 10px; width: 100%; border: 2px solid #2196F3; border-radius: 6px; font-size: 14px; cursor: pointer; margin-bottom: 10px;">
+                        <option value="none">🚫 No Cloud Removal - Keep Original Data</option>
+                        <option value="classic">Classic (3-step: temporal + median + spatial)</option>
+                        <option value="hybrid" selected>Hybrid (Classical + ML KNN) - Recommended</option>
+                        <option value="temporal_only">Temporal Only (Fast)</option>
+                        <option value="median_composite">Median Composite</option>
+                        <option value="ml_knn">ML KNN (K-Nearest Neighbors)</option>
+                        <option value="ml_rf">ML Random Forest</option>
+                        <option value="deep">Deep Learning (U-Net)</option>
+                    </select>
+                    
+                    <!-- Deep Learning Model Selection (only shown when method is 'deep') -->
+                    <div id="cloudModelSelection" style="display: none; margin-top: 10px;">
+                        <label for="cloudModelSelect" style="font-weight: 500; color: #1565c0; margin-bottom: 5px; display: block;">
+                            📦 Select Trained Model:
+                        </label>
+                        <div style="display: flex; gap: 10px; align-items: center; margin-bottom: 10px;">
+                            <select id="cloudModelSelect" style="flex: 1; padding: 10px; border: 2px solid #64b5f6; border-radius: 6px; font-size: 14px; cursor: pointer;">
+                                <option value="">Loading models...</option>
+                            </select>
+                            <button onclick="loadCloudRemovalModels(); return false;" style="padding: 10px 15px; background: #2196F3; color: white; border: none; border-radius: 4px; cursor: pointer; font-size: 13px; white-space: nowrap;">
+                                🔄 Refresh
+                            </button>
+                        </div>
+                        
+                        <!-- Upload Model Button -->
+                        <div style="margin-top: 10px; padding: 10px; background: #f5f5f5; border-radius: 6px;">
+                            <input type="file" id="cloudModelUpload" accept=".pth" style="display: none;" onchange="showMetadataForm()">
+                            <button onclick="document.getElementById('cloudModelUpload').click()" style="padding: 8px 15px; background: #4CAF50; color: white; border: none; border-radius: 4px; cursor: pointer; font-size: 13px;">
+                                📤 Upload Cloud Removal Model (.pth)
+                            </button>
+                            <span id="uploadCloudStatus" style="margin-left: 10px; font-size: 12px; color: #666;"></span>
+                            
+                            <!-- Metadata Form (shown after file selection) -->
+                            <div id="cloudMetadataForm" style="display: none; margin-top: 15px; padding: 15px; background: white; border: 2px solid #4CAF50; border-radius: 6px;">
+                                <h4 style="margin: 0 0 10px 0; color: #2e7d32;">📝 Model Metadata</h4>
+                                <div style="display: grid; grid-template-columns: 1fr 1fr; gap: 10px; margin-bottom: 10px;">
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Epoch:</label>
+                                        <input type="number" id="uploadEpoch" min="0" placeholder="e.g., 50" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                    </div>
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Validation Loss:</label>
+                                        <input type="number" id="uploadValLoss" step="0.0001" min="0" placeholder="e.g., 0.0134" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                    </div>
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Train Loss:</label>
+                                        <input type="number" id="uploadTrainLoss" step="0.0001" min="0" placeholder="e.g., 0.0142" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                    </div>
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Input Channels:</label>
+                                        <input type="number" id="uploadInChannels" min="1" placeholder="e.g., 6" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                    </div>
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Output Channels:</label>
+                                        <input type="number" id="uploadOutChannels" min="1" placeholder="e.g., 4" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                    </div>
+                                    <div>
+                                        <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Use Sentinel-1:</label>
+                                        <select id="uploadUseS1" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                            <option value="true">Yes</option>
+                                            <option value="false">No</option>
+                                        </select>
+                                    </div>
+                                </div>
+                                <div style="margin-bottom: 10px;">
+                                    <label style="font-size: 12px; color: #666; display: block; margin-bottom: 3px;">Description (optional):</label>
+                                    <input type="text" id="uploadDescription" placeholder="e.g., Trained on winter dataset, 50 epochs" style="width: 100%; padding: 6px; border: 1px solid #ddd; border-radius: 4px;">
+                                </div>
+                                <div style="display: flex; gap: 10px;">
+                                    <button onclick="uploadCloudModelWithMetadata()" style="flex: 1; padding: 8px; background: #4CAF50; color: white; border: none; border-radius: 4px; cursor: pointer; font-weight: 600;">
+                                        ✅ Upload with Metadata
+                                    </button>
+                                    <button onclick="cancelUpload()" style="padding: 8px 15px; background: #f44336; color: white; border: none; border-radius: 4px; cursor: pointer;">
+                                        ❌ Cancel
+                                    </button>
+                                </div>
+                            </div>
+                        </div>
+                    </div>
+                    
+                    <div style="font-size: 12px; color: #1976d2; margin-top: 8px;">
+                        💡 Hybrid method balances speed and quality. Deep learning provides best results but requires trained model.
+                    </div>
+                </div>
+                
                 <div class="form-group" style="margin-top: 15px; padding: 15px; background: #fff3e0; border-radius: 8px; border-left: 4px solid #ff9800;">
                     <label style="display: flex; align-items: center; cursor: pointer; margin: 0;">
                         <input type="checkbox" id="useGpuPred" checked style="width: 18px; height: 18px; margin-right: 10px;">
@@ -872,6 +964,243 @@
             } catch (error) {
                 console.error('Error loading models:', error);
             }
+            
+            // Load cloud removal models as well
+            await loadCloudRemovalModels();
+        }
+        
+        // Load cloud removal models
+        async function loadCloudRemovalModels() {
+            try {
+                console.log('[Cloud Models] Loading cloud removal models...');
+                const response = await fetch('/api/cloud-removal/models');
+                const data = await response.json();
+                
+                console.log('[Cloud Models] Received data:', data);
+                
+                const select = document.getElementById('cloudModelSelect');
+                select.innerHTML = '<option value="">No model (will use classical methods)</option>';
+                
+                if (data.models && data.models.length > 0) {
+                    console.log(`[Cloud Models] Found ${data.models.length} models`);
+                    data.models.forEach(model => {
+                        const option = document.createElement('option');
+                        option.value = model.filename;
+                        
+                        // Handle missing metadata gracefully
+                        const epoch = model.epoch || 'N/A';
+                        const loss = model.val_loss ? model.val_loss.toFixed(4) : 'N/A';
+                        option.textContent = `${model.filename} (Epoch ${epoch}, Loss: ${loss})`;
+                        
+                        select.appendChild(option);
+                        console.log(`[Cloud Models] Added: ${model.filename}`);
+                    });
+                    
+                    // Auto-select first model
+                    select.value = data.models[0].filename;
+                    console.log('[Cloud Models] Auto-selected:', data.models[0].filename);
+                } else {
+                    console.log('[Cloud Models] No models found');
+                }
+            } catch (error) {
+                console.error('[Cloud Models] Error loading cloud removal models:', error);
+                // Show user-friendly error
+                const select = document.getElementById('cloudModelSelect');
+                select.innerHTML = '<option value="">Error loading models - check console</option>';
+            }
+        }
+        
+        // Handle cloud removal method change
+        function handleCloudMethodChange() {
+            const method = document.getElementById('cloudRemovalMethod').value;
+            const modelSelection = document.getElementById('cloudModelSelection');
+            
+            if (method === 'deep') {
+                modelSelection.style.display = 'block';
+            } else {
+                modelSelection.style.display = 'none';
+            }
+        }
+        
+        // Upload cloud removal model
+        async function uploadCloudModel() {
+            const fileInput = document.getElementById('cloudModelUpload');
+            const file = fileInput.files[0];
+            const statusSpan = document.getElementById('uploadCloudStatus');
+            
+            if (!file) {
+                return;
+            }
+            
+            if (!file.name.endsWith('.pth')) {
+                statusSpan.textContent = '❌ Only .pth files allowed';
+                statusSpan.style.color = 'red';
+                return;
+            }
+            
+            statusSpan.textContent = '⏳ Uploading...';
+            statusSpan.style.color = '#2196F3';
+            
+            try {
+                const formData = new FormData();
+                formData.append('file', file);
+                
+                const response = await fetch('/api/cloud-removal/upload', {
+                    method: 'POST',
+                    body: formData
+                });
+                
+                const result = await response.json();
+                
+                if (response.ok) {
+                    statusSpan.textContent = `✅ Uploaded: ${result.filename} (${result.size_mb} MB)`;
+                    statusSpan.style.color = '#4CAF50';
+                    
+                    // Reload cloud removal models list
+                    await loadCloudRemovalModels();
+                    
+                    // Auto-select the newly uploaded model
+                    document.getElementById('cloudModelSelect').value = result.filename;
+                } else {
+                    statusSpan.textContent = `❌ ${result.detail || 'Upload failed'}`;
+                    statusSpan.style.color = 'red';
+                }
+            } catch (error) {
+                statusSpan.textContent = `❌ Error: ${error.message}`;
+                statusSpan.style.color = 'red';
+            }
+            
+            // Clear file input
+            fileInput.value = '';
+        }
+        
+        // Show metadata form after file selection
+        function showMetadataForm() {
+            const fileInput = document.getElementById('cloudModelUpload');
+            const file = fileInput.files[0];
+            const statusSpan = document.getElementById('uploadCloudStatus');
+            const metadataForm = document.getElementById('cloudMetadataForm');
+            
+            if (!file) {
+                return;
+            }
+            
+            if (!file.name.endsWith('.pth')) {
+                statusSpan.textContent = '❌ Only .pth files allowed';
+                statusSpan.style.color = 'red';
+                fileInput.value = '';
+                return;
+            }
+            
+            // Show form
+            metadataForm.style.display = 'block';
+            statusSpan.textContent = `📁 Selected: ${file.name} (${(file.size / (1024 * 1024)).toFixed(2)} MB)`;
+            statusSpan.style.color = '#2196F3';
+        }
+        
+        // Upload with metadata
+        async function uploadCloudModelWithMetadata() {
+            const fileInput = document.getElementById('cloudModelUpload');
+            const file = fileInput.files[0];
+            const statusSpan = document.getElementById('uploadCloudStatus');
+            const metadataForm = document.getElementById('cloudMetadataForm');
+            
+            if (!file) {
+                alert('No file selected');
+                return;
+            }
+            
+            // Get values from form with defaults for empty fields
+            const epoch = document.getElementById('uploadEpoch').value || '0';
+            const valLoss = document.getElementById('uploadValLoss').value || '0';
+            const trainLoss = document.getElementById('uploadTrainLoss').value || '0';
+            const inChannels = document.getElementById('uploadInChannels').value || '6';
+            const outChannels = document.getElementById('uploadOutChannels').value || '4';
+            const useS1 = document.getElementById('uploadUseS1').value || 'true';
+            const description = document.getElementById('uploadDescription').value || '';
+            
+            // Debug log
+            console.log('[Upload] Form values:', {
+                epoch, valLoss, trainLoss, inChannels, outChannels, useS1, description
+            });
+            
+            // Confirm upload
+            if (!confirm(`Upload ${file.name} with metadata?\nEpoch: ${epoch}\nVal Loss: ${valLoss}\nTrain Loss: ${trainLoss}`)) {
+                return;
+            }
+            
+            statusSpan.textContent = '⏳ Uploading with metadata...';
+            statusSpan.style.color = '#2196F3';
+            
+            try {
+                const formData = new FormData();
+                formData.append('file', file);
+                formData.append('epoch', epoch);
+                formData.append('val_loss', valLoss);
+                formData.append('train_loss', trainLoss);
+                formData.append('in_channels', inChannels);
+                formData.append('out_channels', outChannels);
+                formData.append('use_s1', useS1);
+                formData.append('description', description);
+                
+                console.log('[Upload] Sending FormData...');
+                
+                const response = await fetch('/api/cloud-removal/upload', {
+                    method: 'POST',
+                    body: formData
+                });
+                
+                const result = await response.json();
+                
+                console.log('[Upload] Response:', result);
+                
+                if (response.ok) {
+                    statusSpan.textContent = `✅ Uploaded: ${result.filename} (${result.size_mb} MB) - Epoch ${result.metadata?.epoch || 0}`;
+                    statusSpan.style.color = '#4CAF50';
+                    
+                    // Hide form
+                    metadataForm.style.display = 'none';
+                    
+                    // Clear form
+                    document.getElementById('uploadEpoch').value = '';
+                    document.getElementById('uploadValLoss').value = '';
+                    document.getElementById('uploadTrainLoss').value = '';
+                    document.getElementById('uploadInChannels').value = '';
+                    document.getElementById('uploadOutChannels').value = '';
+                    document.getElementById('uploadUseS1').value = 'true';
+                    document.getElementById('uploadDescription').value = '';
+                    
+                    // Reload cloud removal models list
+                    await loadCloudRemovalModels();
+                    
+                    // Auto-select the newly uploaded model
+                    document.getElementById('cloudModelSelect').value = result.filename;
+                    
+                    console.log('[Upload] Success:', result);
+                } else {
+                    statusSpan.textContent = `❌ ${result.detail || 'Upload failed'}`;
+                    statusSpan.style.color = 'red';
+                }
+            } catch (error) {
+                statusSpan.textContent = `❌ Error: ${error.message}`;
+                statusSpan.style.color = 'red';
+                console.error('[Upload] Error:', error);
+            }
+            
+            // Clear file input
+            fileInput.value = '';
+        }
+        
+        // Cancel upload
+        function cancelUpload() {
+            const fileInput = document.getElementById('cloudModelUpload');
+            const statusSpan = document.getElementById('uploadCloudStatus');
+            const metadataForm = document.getElementById('cloudMetadataForm');
+            
+            // Clear and hide
+            fileInput.value = '';
+            metadataForm.style.display = 'none';
+            statusSpan.textContent = '';
         }
         
         // Update model info display
@@ -927,6 +1256,23 @@
             const exportNDVI = document.getElementById('exportNDVI').checked;
             const useGpu = document.getElementById('useGpuPred').checked;
             
+            // Get cloud removal configuration
+            const cloudRemovalMethod = document.getElementById('cloudRemovalMethod').value;
+            const cloudRemovalConfig = {
+                method: cloudRemovalMethod
+            };
+            
+            // If deep learning method is selected, include model filename
+            if (cloudRemovalMethod === 'deep') {
+                const cloudModelFilename = document.getElementById('cloudModelSelect').value;
+                if (cloudModelFilename) {
+                    cloudRemovalConfig.model_filename = cloudModelFilename;
+                } else {
+                    alert('⚠️ Deep learning method selected but no model chosen. Will fall back to hybrid method.');
+                    cloudRemovalConfig.method = 'hybrid';
+                }
+            }
+            
             const config = {
                 model_filename: modelFilename,
                 min_lon: selectedBbox.min_lon,
@@ -940,7 +1286,9 @@
                 resolution: parseInt(document.getElementById('predResolution').value),
                 use_gpu: useGpu,
                 export_ndvi: exportNDVI,
-                export_classification: true
+                export_classification: true,
+                cloud_removal_method: cloudRemovalConfig.method,
+                cloud_removal_model: cloudRemovalConfig.model_filename || null
             };
             
             try {

test_cloud_removal.py

@@ -0,0 +1,194 @@
+"""
+Test script for cloud_removal module
+Kiểm tra các phương pháp xử lý mây
+"""
+
+import numpy as np
+import xarray as xr
+from cloud_removal import (
+    process_cloud_removal,
+    get_available_methods,
+    compare_methods
+)
+
+
+def create_mock_s2_data():
+    """Tạo mock Sentinel-2 data để test"""
+    # Create synthetic data: 5 time steps, 100x100 pixels
+    np.random.seed(42)
+    
+    time_steps = 5
+    y_size = 100
+    x_size = 100
+    
+    # Create bands
+    bands = {}
+    for band in ["B02", "B03", "B04", "B08", "B11"]:
+        # Random reflectance values
+        data = np.random.rand(time_steps, y_size, x_size) * 0.3 + 0.1
+        bands[band] = (["time", "y", "x"], data)
+    
+    # Create SCL (Scene Classification Layer)
+    # Mostly vegetation (4), with some clouds
+    scl_data = np.full((time_steps, y_size, x_size), 4, dtype=np.uint8)
+    
+    # Add clouds (class 9) in random locations
+    for t in range(time_steps):
+        # Random cloud patches
+        n_clouds = np.random.randint(5, 15)
+        for _ in range(n_clouds):
+            y_start = np.random.randint(0, y_size - 20)
+            x_start = np.random.randint(0, x_size - 20)
+            cloud_height = np.random.randint(10, 20)
+            cloud_width = np.random.randint(10, 20)
+            scl_data[t, y_start:y_start+cloud_height, x_start:x_start+cloud_width] = 9
+    
+    bands["SCL"] = (["time", "y", "x"], scl_data)
+    
+    # Create xarray Dataset
+    ds = xr.Dataset(
+        bands,
+        coords={
+            "time": np.arange(time_steps),
+            "y": np.arange(y_size),
+            "x": np.arange(x_size)
+        }
+    )
+    
+    return ds
+
+
+def test_available_methods():
+    """Test lấy danh sách methods"""
+    print("=" * 60)
+    print("TEST: Get Available Methods")
+    print("=" * 60)
+    
+    methods = get_available_methods()
+    print(f"\nFound {len(methods)} methods:")
+    for method, description in methods.items():
+        print(f"  - {method:20s}: {description}")
+    
+    print("\n✅ Test passed!")
+
+
+def test_single_method(method_name="classic"):
+    """Test một method cụ thể"""
+    print("\n" + "=" * 60)
+    print(f"TEST: Cloud Removal Method '{method_name}'")
+    print("=" * 60)
+    
+    # Create mock data
+    s2_data = create_mock_s2_data()
+    print(f"\nMock data created: {dict(s2_data.dims)}")
+    
+    # Process clouds
+    cleaned_data, metadata = process_cloud_removal(
+        s2_data=s2_data,
+        method=method_name,
+        verbose=True
+    )
+    
+    # Check results
+    print(f"\nMetadata:")
+    print(f"  - Method: {metadata['method']}")
+    print(f"  - Cloud coverage: {metadata['cloud_coverage_percent']:.1f}%")
+    print(f"  - Masked pixels: {metadata['masked_pixels']:,}/{metadata['total_pixels']:,}")
+    print(f"  - Steps applied: {', '.join(metadata['steps_applied'])}")
+    
+    # Verify no NaN remaining
+    nan_count = 0
+    for band in cleaned_data.data_vars:
+        if band != "SCL":
+            nan_count += np.isnan(cleaned_data[band].values).sum()
+    
+    print(f"\nRemaining NaN pixels: {nan_count}")
+    
+    if nan_count == 0:
+        print("✅ Test passed - no NaN remaining!")
+    else:
+        print(f"⚠️  Warning - {nan_count} NaN pixels remaining")
+
+
+def test_comparison():
+    """Test so sánh nhiều methods"""
+    print("\n" + "=" * 60)
+    print("TEST: Compare Multiple Methods")
+    print("=" * 60)
+    
+    # Create mock data
+    s2_data = create_mock_s2_data()
+    
+    # Compare methods
+    methods_to_test = ["classic", "temporal_only", "median_composite", "ml_knn"]
+    
+    print(f"\nComparing {len(methods_to_test)} methods...")
+    results = compare_methods(s2_data, methods=methods_to_test)
+    
+    # Print summary
+    print("\n" + "-" * 60)
+    print(f"{'Method':<20} {'Success':<10} {'NaN %':<10} {'Steps'}")
+    print("-" * 60)
+    
+    for method, result in results.items():
+        if result['success']:
+            nan_pct = result['remaining_nan_percent']
+            steps = ', '.join(result['metadata']['steps_applied'][:2])  # First 2 steps
+            print(f"{method:<20} {'✅':<10} {nan_pct:>6.2f}%   {steps}")
+        else:
+            print(f"{method:<20} {'❌':<10} {'ERROR':<10} {result['error']}")
+    
+    print("-" * 60)
+    print("\n✅ Comparison test completed!")
+
+
+def test_edge_cases():
+    """Test các trường hợp đặc biệt"""
+    print("\n" + "=" * 60)
+    print("TEST: Edge Cases")
+    print("=" * 60)
+    
+    # Case 1: No SCL band
+    print("\n1. Testing without SCL band...")
+    s2_data = create_mock_s2_data()
+    s2_data_no_scl = s2_data.drop_vars("SCL")
+    
+    cleaned, meta = process_cloud_removal(s2_data_no_scl, method="classic", verbose=False)
+    print(f"   Result: {meta.get('warning', 'OK')}")
+    
+    # Case 2: 100% cloud coverage
+    print("\n2. Testing with 100% cloud coverage...")
+    s2_data_full_cloud = create_mock_s2_data()
+    s2_data_full_cloud["SCL"][:] = 9  # All clouds
+    
+    cleaned, meta = process_cloud_removal(s2_data_full_cloud, method="classic", verbose=False)
+    print(f"   Cloud coverage: {meta['cloud_coverage_percent']:.1f}%")
+    
+    # Case 3: No clouds
+    print("\n3. Testing with no clouds...")
+    s2_data_clear = create_mock_s2_data()
+    s2_data_clear["SCL"][:] = 4  # All vegetation
+    
+    cleaned, meta = process_cloud_removal(s2_data_clear, method="classic", verbose=False)
+    print(f"   Cloud coverage: {meta['cloud_coverage_percent']:.1f}%")
+    
+    print("\n✅ Edge case tests passed!")
+
+
+if __name__ == "__main__":
+    print("\n" + "🌥️  CLOUD REMOVAL MODULE TESTS 🌥️ ".center(60, "="))
+    print()
+    
+    # Run tests
+    test_available_methods()
+    test_single_method("classic")
+    test_single_method("hybrid")
+    test_comparison()
+    test_edge_cases()
+    
+    print("\n" + "=" * 60)
+    print("ALL TESTS COMPLETED!")
+    print("=" * 60)
+    print("\nModule is ready to use. Available methods:")
+    for method, desc in get_available_methods().items():
+        print(f"  • {method}")

test_cloud_training.py

@@ -0,0 +1,28 @@
+"""
+Script test nhanh cho cloud removal training
+"""
+
+import sys
+from pathlib import Path
+
+# Add winter_dataset to path
+sys.path.insert(0, str(Path(__file__).parent / "winter_dataset"))
+
+from train_cloud_removal import train_cloud_removal_model
+
+if __name__ == "__main__":
+    print("\n🌥️  Starting Cloud Removal Training Test")
+    print("=" * 70)
+    
+    # Test with small dataset
+    model, train_losses, val_losses = train_cloud_removal_model(
+        data_dir="winter_dataset",
+        use_s1=True,  # Use S1 radar data
+        batch_size=4,  # Small batch for testing
+        num_epochs=5,  # Few epochs for quick test
+        learning_rate=1e-4
+    )
+    
+    print("\n✅ Training test completed!")
+    print(f"Final train loss: {train_losses[-1]:.6f}")
+    print(f"Final val loss: {val_losses[-1]:.6f}")

test_cloud_upload.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""
+Test Cloud Removal Model Upload Feature
+"""
+
+import requests
+import json
+from pathlib import Path
+
+# API base URL
+BASE_URL = "http://localhost:8000"
+
+def test_upload_cloud_model(file_path):
+    """Test uploading a cloud removal model"""
+    print(f"\n{'='*60}")
+    print("TEST 1: Upload Cloud Removal Model")
+    print(f"{'='*60}")
+    
+    if not Path(file_path).exists():
+        print(f"❌ File not found: {file_path}")
+        print("   Create a dummy .pth file for testing:")
+        print(f"   touch {file_path}")
+        return None
+    
+    with open(file_path, 'rb') as f:
+        files = {'file': (Path(file_path).name, f, 'application/octet-stream')}
+        
+        print(f"📤 Uploading: {file_path}")
+        response = requests.post(f"{BASE_URL}/api/cloud-removal/upload", files=files)
+        
+        if response.status_code == 200:
+            result = response.json()
+            print(f"✅ Upload successful!")
+            print(f"   Filename: {result['filename']}")
+            print(f"   Size: {result['size_mb']} MB")
+            print(f"   Path: {result['path']}")
+            return result['filename']
+        else:
+            print(f"❌ Upload failed: {response.status_code}")
+            print(f"   {response.json().get('detail', 'Unknown error')}")
+            return None
+
+def test_list_cloud_models():
+    """Test listing cloud removal models"""
+    print(f"\n{'='*60}")
+    print("TEST 2: List Cloud Removal Models")
+    print(f"{'='*60}")
+    
+    response = requests.get(f"{BASE_URL}/api/cloud-removal/models")
+    
+    if response.status_code == 200:
+        data = response.json()
+        print(f"✅ Found {data['count']} models:")
+        for i, model in enumerate(data['models'], 1):
+            print(f"\n   {i}. {model['filename']}")
+            print(f"      Size: {model['size_mb']} MB")
+            print(f"      Created: {model['created']}")
+            if 'epoch' in model:
+                print(f"      Epoch: {model['epoch']}, Val Loss: {model['val_loss']:.4f}")
+        return data['models']
+    else:
+        print(f"❌ Failed to list models: {response.status_code}")
+        return []
+
+def test_prediction_with_cloud_model(model_filename, cloud_model_filename):
+    """Test prediction using uploaded cloud removal model"""
+    print(f"\n{'='*60}")
+    print("TEST 3: Prediction with Custom Cloud Removal Model")
+    print(f"{'='*60}")
+    
+    config = {
+        "model_filename": model_filename,
+        "min_lon": 105.80,
+        "min_lat": 10.00,
+        "max_lon": 105.82,
+        "max_lat": 10.02,
+        "start_date": "2024-01-15",
+        "end_date": "2024-01-17",
+        "max_scenes": 2,
+        "cloud_cover": 30,
+        "resolution": 20,
+        "use_gpu": False,
+        "export_ndvi": True,
+        "export_classification": True,
+        "cloud_removal_method": "deep",
+        "cloud_removal_model": cloud_model_filename
+    }
+    
+    print("📊 Prediction Config:")
+    print(json.dumps(config, indent=2))
+    
+    print(f"\n🚀 Starting prediction with cloud removal model: {cloud_model_filename}")
+    response = requests.post(
+        f"{BASE_URL}/api/predict/with-ndvi",
+        json=config,
+        headers={'Content-Type': 'application/json'}
+    )
+    
+    if response.status_code == 200:
+        result = response.json()
+        print(f"✅ Prediction started!")
+        print(f"   Message: {result.get('message')}")
+        return result
+    else:
+        print(f"❌ Prediction failed: {response.status_code}")
+        print(f"   {response.json().get('detail', 'Unknown error')}")
+        return None
+
+def test_delete_cloud_model(filename):
+    """Test deleting a cloud removal model"""
+    print(f"\n{'='*60}")
+    print("TEST 4: Delete Cloud Removal Model")
+    print(f"{'='*60}")
+    
+    print(f"🗑️  Deleting: {filename}")
+    response = requests.delete(f"{BASE_URL}/api/cloud-removal/models/{filename}")
+    
+    if response.status_code == 200:
+        result = response.json()
+        print(f"✅ {result['message']}")
+        return True
+    else:
+        print(f"❌ Delete failed: {response.status_code}")
+        return False
+
+def main():
+    print("="*60)
+    print("CLOUD REMOVAL MODEL UPLOAD - FEATURE TEST")
+    print("="*60)
+    
+    # Test file path (create a dummy file for testing)
+    test_file = "test_cloud_removal_model.pth"
+    
+    # Create dummy file if it doesn't exist
+    if not Path(test_file).exists():
+        print(f"\n📝 Creating dummy test file: {test_file}")
+        Path(test_file).write_bytes(b"dummy_pytorch_model_data")
+    
+    # Run tests
+    uploaded_filename = test_upload_cloud_model(test_file)
+    
+    if uploaded_filename:
+        models = test_list_cloud_models()
+        
+        # Test prediction (requires a real land classification model)
+        print(f"\n{'='*60}")
+        print("NOTE: Prediction test requires a trained land classification model")
+        print("      Skipping prediction test in this demo")
+        print(f"{'='*60}")
+        
+        # Cleanup - delete test model
+        if input("\nDelete test model? (y/n): ").lower() == 'y':
+            test_delete_cloud_model(uploaded_filename)
+    
+    # Cleanup dummy file
+    if Path(test_file).exists():
+        Path(test_file).unlink()
+        print(f"\n🗑️  Cleaned up dummy file: {test_file}")
+    
+    print(f"\n{'='*60}")
+    print("TESTS COMPLETED")
+    print(f"{'='*60}")
+
+if __name__ == "__main__":
+    main()

test_planetary_computer.py

@@ -0,0 +1,88 @@
+"""
+Test Microsoft Planetary Computer connectivity và token
+"""
+import planetary_computer
+from pystac_client import Client
+from datetime import datetime, timedelta
+
+print("=" * 70)
+print("🧪 TESTING MICROSOFT PLANETARY COMPUTER CONNECTION")
+print("=" * 70)
+
+# Test 1: Basic connection
+print("\n1️⃣ Testing basic connection...")
+try:
+    catalog = Client.open(
+        "https://planetarycomputer.microsoft.com/api/stac/v1",
+        modifier=planetary_computer.sign_inplace,
+    )
+    print("✅ Successfully connected to Planetary Computer")
+    print(f"   Catalog ID: {catalog.id}")
+    print(f"   Title: {catalog.title}")
+except Exception as e:
+    print(f"❌ Connection failed: {e}")
+    exit(1)
+
+# Test 2: List collections
+print("\n2️⃣ Testing collections access...")
+try:
+    collections = list(catalog.get_collections())
+    print(f"✅ Found {len(collections)} collections")
+    sentinel_2 = [c for c in collections if 'sentinel-2' in c.id.lower()]
+    print(f"   Sentinel-2 collections: {[c.id for c in sentinel_2]}")
+except Exception as e:
+    print(f"❌ Collections access failed: {e}")
+
+# Test 3: Small search query (very conservative)
+print("\n3️⃣ Testing small search query...")
+try:
+    # Tiny bbox in Vietnam
+    bbox = [105.8, 10.0, 105.9, 10.1]  # ~10km x 10km area
+    end_date = datetime.now()
+    start_date = end_date - timedelta(days=7)  # Last 7 days only
+    
+    time_range = f"{start_date.strftime('%Y-%m-%d')}/{end_date.strftime('%Y-%m-%d')}"
+    
+    print(f"   Bbox: {bbox}")
+    print(f"   Time: {time_range}")
+    print(f"   Searching...")
+    
+    search = catalog.search(
+        collections=["sentinel-2-l2a"],
+        bbox=bbox,
+        datetime=time_range,
+        limit=5  # Only 5 items
+    )
+    
+    items = []
+    for i, item in enumerate(search.items()):
+        items.append(item)
+        if i >= 4:  # Stop at 5
+            break
+    
+    print(f"✅ Search successful! Found {len(items)} items")
+    if items:
+        first_item = items[0]
+        print(f"   First item: {first_item.id}")
+        print(f"   Date: {first_item.datetime}")
+        
+        # Test token signing
+        signed_item = planetary_computer.sign(first_item)
+        print(f"✅ SAS token signing works")
+        print(f"   Asset keys: {list(signed_item.assets.keys())[:5]}")
+        
+except Exception as e:
+    print(f"❌ Search failed: {e}")
+    import traceback
+    traceback.print_exc()
+
+print("\n" + "=" * 70)
+print("🏁 Test completed!")
+print("=" * 70)
+print("\n💡 Nếu test này PASS:")
+print("   → Planetary Computer hoạt động bình thường")
+print("   → Vấn đề là query quá lớn (bbox/time range/max_scenes)")
+print("\n💡 Nếu test này FAIL:")
+print("   → Kiểm tra internet connection")
+print("   → Thử lại sau (server có thể bị quá tải)")
+print("   → Xem xét dùng dữ liệu local")

train_cloud_removal.py

@@ -0,0 +1,512 @@
+"""
+Train Cloud Removal Model using SEN12MS-CR Dataset
+Huấn luyện model Deep Learning để khử mây từ ảnh Sentinel-2
+
+Dataset: SEN12MS-CR (Sentinel-12 Multi-Seasonal Cloud Removal)
+- Input: S2 cloudy images (ảnh Sentinel-2 bị mây)
+- Target: S2 clean images (ảnh Sentinel-2 sạch)
+- Optional: S1 SAR data (radar data không bị ảnh hưởng bởi mây)
+"""
+
+import os
+import sys
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.optim as optim
+from torch.utils.data import Dataset, DataLoader
+from pathlib import Path
+import matplotlib.pyplot as plt
+from tqdm import tqdm
+
+# Add winter_dataset to path
+sys.path.insert(0, str(Path(__file__).parent / "winter_dataset"))
+from sen12ms_cr_dataLoader import SEN12MSCRDataset, Seasons, S1Bands, S2Bands
+
+
+# ============ DATASET WRAPPER ============
+
+class CloudRemovalDataset(Dataset):
+    """
+    PyTorch Dataset wrapper cho SEN12MS-CR
+    Input: S2 cloudy + S1 (optional)
+    Target: S2 clean
+    """
+    
+    def __init__(self, base_dir, season=Seasons.WINTER, use_s1=True, 
+                 s2_bands=S2Bands.ALL, normalize=True):
+        """
+        Args:
+            base_dir: Đường dẫn đến thư mục chứa dữ liệu
+            season: Mùa (SPRING, SUMMER, FALL, WINTER)
+            use_s1: Có sử dụng dữ liệu S1 (radar) không
+            s2_bands: Các band S2 cần dùng
+            normalize: Normalize dữ liệu về [0, 1]
+        """
+        self.dataset = SEN12MSCRDataset(base_dir)
+        self.season = season
+        self.use_s1 = use_s1
+        self.s2_bands = s2_bands
+        self.normalize = normalize
+        
+        # Lấy tất cả scene và patch IDs
+        season_ids = self.dataset.get_season_ids(season)
+        
+        # Tạo list of (scene_id, patch_id) pairs
+        self.samples = []
+        for scene_id, patch_ids in season_ids.items():
+            for patch_id in patch_ids:
+                self.samples.append((scene_id, patch_id))
+        
+        # Get band count
+        n_s2_bands = len(s2_bands.value) if hasattr(s2_bands, 'value') else len(s2_bands)
+        
+        print(f"[DATASET] Loaded {len(self.samples)} samples from {season.value}")
+        print(f"[DATASET] Use S1: {use_s1}, S2 bands: {n_s2_bands}")
+    
+    def __len__(self):
+        return len(self.samples)
+    
+    def __getitem__(self, idx):
+        scene_id, patch_id = self.samples[idx]
+        
+        # Load triplet: S1, S2 clean, S2 cloudy
+        s1, s2_clean, s2_cloudy, bounds = self.dataset.get_s1s2s2cloudy_triplet(
+            self.season,
+            scene_id,
+            patch_id,
+            s1_bands=S1Bands.ALL if self.use_s1 else S1Bands.NONE,
+            s2_bands=self.s2_bands,
+            s2cloudy_bands=self.s2_bands
+        )
+        
+        # Normalize to [0, 1] if needed
+        if self.normalize:
+            s2_clean = s2_clean.astype(np.float32) / 10000.0  # S2 values are in [0, 10000]
+            s2_cloudy = s2_cloudy.astype(np.float32) / 10000.0
+            if self.use_s1:
+                # S1 values need different normalization (dB scale)
+                s1 = (s1.astype(np.float32) + 30) / 50.0  # Normalize from [-30, 20] to [0, 1]
+                s1 = np.clip(s1, 0, 1)
+        
+        # Convert to torch tensors
+        s2_clean = torch.from_numpy(s2_clean).float()
+        s2_cloudy = torch.from_numpy(s2_cloudy).float()
+        
+        # Input: S2 cloudy + S1 (if enabled)
+        if self.use_s1:
+            s1 = torch.from_numpy(s1).float()
+            input_data = torch.cat([s2_cloudy, s1], dim=0)
+        else:
+            input_data = s2_cloudy
+        
+        return input_data, s2_clean
+
+
+# ============ U-NET ARCHITECTURE ============
+
+class DoubleConv(nn.Module):
+    """(Conv2d -> BatchNorm -> ReLU) x 2"""
+    
+    def __init__(self, in_channels, out_channels):
+        super().__init__()
+        self.double_conv = nn.Sequential(
+            nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
+            nn.BatchNorm2d(out_channels),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1),
+            nn.BatchNorm2d(out_channels),
+            nn.ReLU(inplace=True)
+        )
+    
+    def forward(self, x):
+        return self.double_conv(x)
+
+
+class UNet(nn.Module):
+    """
+    U-Net architecture cho cloud removal
+    Input: S2 cloudy (+ S1 optional) [B, C_in, H, W]
+    Output: S2 clean [B, C_out, H, W]
+    """
+    
+    def __init__(self, in_channels, out_channels, features=[64, 128, 256, 512]):
+        super().__init__()
+        self.encoder = nn.ModuleList()
+        self.decoder = nn.ModuleList()
+        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
+        
+        # Encoder (downsampling)
+        for feature in features:
+            self.encoder.append(DoubleConv(in_channels, feature))
+            in_channels = feature
+        
+        # Bottleneck
+        self.bottleneck = DoubleConv(features[-1], features[-1] * 2)
+        
+        # Decoder (upsampling)
+        for feature in reversed(features):
+            self.decoder.append(
+                nn.ConvTranspose2d(feature * 2, feature, kernel_size=2, stride=2)
+            )
+            self.decoder.append(DoubleConv(feature * 2, feature))
+        
+        # Final output layer
+        self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)
+    
+    def forward(self, x):
+        skip_connections = []
+        
+        # Encoder
+        for encode in self.encoder:
+            x = encode(x)
+            skip_connections.append(x)
+            x = self.pool(x)
+        
+        # Bottleneck
+        x = self.bottleneck(x)
+        
+        # Decoder
+        skip_connections = skip_connections[::-1]
+        
+        for idx in range(0, len(self.decoder), 2):
+            x = self.decoder[idx](x)  # Upsample
+            skip_connection = skip_connections[idx // 2]
+            
+            # Handle size mismatch
+            if x.shape != skip_connection.shape:
+                x = nn.functional.interpolate(x, size=skip_connection.shape[2:])
+            
+            concat_skip = torch.cat((skip_connection, x), dim=1)
+            x = self.decoder[idx + 1](concat_skip)  # Double conv
+        
+        return self.final_conv(x)
+
+
+# ============ TRAINING FUNCTIONS ============
+
+def train_epoch(model, dataloader, criterion, optimizer, device):
+    """Train for one epoch"""
+    model.train()
+    total_loss = 0
+    
+    pbar = tqdm(dataloader, desc="Training")
+    for batch_idx, (inputs, targets) in enumerate(pbar):
+        inputs = inputs.to(device)
+        targets = targets.to(device)
+        
+        # Forward pass
+        optimizer.zero_grad()
+        outputs = model(inputs)
+        loss = criterion(outputs, targets)
+        
+        # Backward pass
+        loss.backward()
+        optimizer.step()
+        
+        total_loss += loss.item()
+        pbar.set_postfix({'loss': loss.item()})
+    
+    return total_loss / len(dataloader)
+
+
+def validate(model, dataloader, criterion, device):
+    """Validate model"""
+    model.eval()
+    total_loss = 0
+    
+    with torch.no_grad():
+        for inputs, targets in tqdm(dataloader, desc="Validation"):
+            inputs = inputs.to(device)
+            targets = targets.to(device)
+            
+            outputs = model(inputs)
+            loss = criterion(outputs, targets)
+            total_loss += loss.item()
+    
+    return total_loss / len(dataloader)
+
+
+def visualize_results(model, dataset, device, num_samples=3):
+    """Visualize cloud removal results"""
+    model.eval()
+    
+    fig, axes = plt.subplots(num_samples, 3, figsize=(15, 5 * num_samples))
+    
+    with torch.no_grad():
+        for i in range(num_samples):
+            idx = np.random.randint(0, len(dataset))
+            input_data, target = dataset[idx]
+            
+            input_data = input_data.unsqueeze(0).to(device)
+            output = model(input_data)
+            
+            # Convert to numpy
+            input_rgb = input_data[0, :3, :, :].cpu().numpy().transpose(1, 2, 0)
+            target_rgb = target[:3, :, :].cpu().numpy().transpose(1, 2, 0)
+            output_rgb = output[0, :3, :, :].cpu().numpy().transpose(1, 2, 0)
+            
+            # Clip to [0, 1]
+            input_rgb = np.clip(input_rgb * 3, 0, 1)  # Enhance for visualization
+            target_rgb = np.clip(target_rgb * 3, 0, 1)
+            output_rgb = np.clip(output_rgb * 3, 0, 1)
+            
+            if num_samples == 1:
+                axes[0].imshow(input_rgb)
+                axes[0].set_title("Input (Cloudy)")
+                axes[0].axis('off')
+                
+                axes[1].imshow(output_rgb)
+                axes[1].set_title("Output (Predicted)")
+                axes[1].axis('off')
+                
+                axes[2].imshow(target_rgb)
+                axes[2].set_title("Target (Clean)")
+                axes[2].axis('off')
+            else:
+                axes[i, 0].imshow(input_rgb)
+                axes[i, 0].set_title(f"Sample {i+1}: Input (Cloudy)")
+                axes[i, 0].axis('off')
+                
+                axes[i, 1].imshow(output_rgb)
+                axes[i, 1].set_title(f"Sample {i+1}: Output (Predicted)")
+                axes[i, 1].axis('off')
+                
+                axes[i, 2].imshow(target_rgb)
+                axes[i, 2].set_title(f"Sample {i+1}: Target (Clean)")
+                axes[i, 2].axis('off')
+    
+    plt.tight_layout()
+    return fig
+
+
+# ============ MAIN TRAINING SCRIPT ============
+
+def train_cloud_removal_model(
+    data_dir="winter_dataset",
+    use_s1=True,
+    batch_size=8,
+    num_epochs=50,
+    learning_rate=1e-4,
+    device="cuda" if torch.cuda.is_available() else "cpu",
+    save_dir="model_train"
+):
+    """
+    Train cloud removal model
+    
+    Args:
+        data_dir: Thư mục chứa dữ liệu SEN12MS-CR
+        use_s1: Có sử dụng S1 radar data không
+        batch_size: Batch size
+        num_epochs: Số epochs
+        learning_rate: Learning rate
+        device: 'cuda' hoặc 'cpu'
+        save_dir: Thư mục lưu model
+    """
+    
+    print("=" * 70)
+    print("🌥️  CLOUD REMOVAL MODEL TRAINING")
+    print("=" * 70)
+    print(f"Data directory: {data_dir}")
+    print(f"Use S1 (SAR): {use_s1}")
+    print(f"Device: {device}")
+    print(f"Batch size: {batch_size}")
+    print(f"Epochs: {num_epochs}")
+    print(f"Learning rate: {learning_rate}")
+    print("=" * 70)
+    
+    # Create dataset
+    print("\n📂 Loading dataset...")
+    
+    # Use RGB + NIR bands for training (B02, B03, B04, B08)
+    s2_bands = [S2Bands.B02, S2Bands.B03, S2Bands.B04, S2Bands.B08]
+    
+    dataset = CloudRemovalDataset(
+        base_dir=data_dir,
+        season=Seasons.WINTER,
+        use_s1=use_s1,
+        s2_bands=s2_bands,
+        normalize=True
+    )
+    
+    # Split train/val
+    train_size = int(0.8 * len(dataset))
+    val_size = len(dataset) - train_size
+    train_dataset, val_dataset = torch.utils.data.random_split(
+        dataset, [train_size, val_size]
+    )
+    
+    print(f"Train samples: {len(train_dataset)}")
+    print(f"Val samples: {len(val_dataset)}")
+    
+    # Create dataloaders
+    train_loader = DataLoader(
+        train_dataset, 
+        batch_size=batch_size, 
+        shuffle=True,
+        num_workers=4,
+        pin_memory=True if device == "cuda" else False
+    )
+    
+    val_loader = DataLoader(
+        val_dataset,
+        batch_size=batch_size,
+        shuffle=False,
+        num_workers=4,
+        pin_memory=True if device == "cuda" else False
+    )
+    
+    # Create model
+    print("\n🏗️  Creating U-Net model...")
+    in_channels = len(s2_bands) + (2 if use_s1 else 0)  # S2 + S1 (VV, VH)
+    out_channels = len(s2_bands)
+    
+    model = UNet(in_channels=in_channels, out_channels=out_channels)
+    model = model.to(device)
+    
+    print(f"Input channels: {in_channels}")
+    print(f"Output channels: {out_channels}")
+    print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
+    
+    # Loss and optimizer
+    criterion = nn.L1Loss()  # MAE loss
+    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
+    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
+        optimizer, mode='min', factor=0.5, patience=5
+    )
+    
+    # Training loop
+    print("\n🚀 Starting training...")
+    best_val_loss = float('inf')
+    train_losses = []
+    val_losses = []
+    
+    for epoch in range(num_epochs):
+        print(f"\n{'='*70}")
+        print(f"Epoch {epoch + 1}/{num_epochs}")
+        print(f"{'='*70}")
+        
+        # Train
+        train_loss = train_epoch(model, train_loader, criterion, optimizer, device)
+        train_losses.append(train_loss)
+        
+        # Validate
+        val_loss = validate(model, val_loader, criterion, device)
+        val_losses.append(val_loss)
+        
+        # Update learning rate
+        scheduler.step(val_loss)
+        
+        print(f"\nEpoch {epoch + 1} Summary:")
+        print(f"  Train Loss: {train_loss:.6f}")
+        print(f"  Val Loss:   {val_loss:.6f}")
+        
+        # Save best model
+        if val_loss < best_val_loss:
+            best_val_loss = val_loss
+            save_path = Path(save_dir) / "cloud_removal_unet_best.pth"
+            save_path.parent.mkdir(exist_ok=True)
+            
+            torch.save({
+                'epoch': epoch,
+                'model_state_dict': model.state_dict(),
+                'optimizer_state_dict': optimizer.state_dict(),
+                'train_loss': train_loss,
+                'val_loss': val_loss,
+                'use_s1': use_s1,
+                'in_channels': in_channels,
+                'out_channels': out_channels
+            }, save_path)
+            
+            print(f"  💾 Saved best model: {save_path}")
+        
+        # Visualize every 10 epochs
+        if (epoch + 1) % 10 == 0:
+            print("\n📊 Generating visualizations...")
+            fig = visualize_results(model, val_dataset, device, num_samples=3)
+            
+            viz_path = Path(save_dir) / f"cloud_removal_epoch_{epoch+1}.png"
+            fig.savefig(viz_path, dpi=150, bbox_inches='tight')
+            plt.close(fig)
+            
+            print(f"  💾 Saved visualization: {viz_path}")
+    
+    # Plot training curves
+    print("\n📈 Plotting training curves...")
+    fig, ax = plt.subplots(figsize=(10, 6))
+    ax.plot(train_losses, label='Train Loss')
+    ax.plot(val_losses, label='Val Loss')
+    ax.set_xlabel('Epoch')
+    ax.set_ylabel('Loss (MAE)')
+    ax.set_title('Cloud Removal Training Progress')
+    ax.legend()
+    ax.grid(True)
+    
+    curve_path = Path(save_dir) / "training_curves.png"
+    fig.savefig(curve_path, dpi=150, bbox_inches='tight')
+    plt.close(fig)
+    
+    print(f"  💾 Saved training curves: {curve_path}")
+    
+    # Final summary
+    print("\n" + "=" * 70)
+    print("✅ TRAINING COMPLETED!")
+    print("=" * 70)
+    print(f"Best validation loss: {best_val_loss:.6f}")
+    print(f"Model saved to: {Path(save_dir) / 'cloud_removal_unet_best.pth'}")
+    print("=" * 70)
+    
+    return model, train_losses, val_losses
+
+
+# ============ INFERENCE FUNCTION ============
+
+def apply_cloud_removal(model_path, cloudy_image, s1_data=None, device="cuda"):
+    """
+    Áp dụng model để khử mây cho một ảnh
+    
+    Args:
+        model_path: Đường dẫn đến model đã train
+        cloudy_image: Ảnh S2 bị mây [C, H, W]
+        s1_data: Dữ liệu S1 (optional) [2, H, W]
+        device: 'cuda' hoặc 'cpu'
+    
+    Returns:
+        cleaned_image: Ảnh đã khử mây [C, H, W]
+    """
+    # Load model
+    checkpoint = torch.load(model_path, map_location=device)
+    
+    model = UNet(
+        in_channels=checkpoint['in_channels'],
+        out_channels=checkpoint['out_channels']
+    )
+    model.load_state_dict(checkpoint['model_state_dict'])
+    model = model.to(device)
+    model.eval()
+    
+    # Prepare input
+    input_tensor = torch.from_numpy(cloudy_image).float().unsqueeze(0).to(device)
+    
+    if checkpoint['use_s1'] and s1_data is not None:
+        s1_tensor = torch.from_numpy(s1_data).float().unsqueeze(0).to(device)
+        input_tensor = torch.cat([input_tensor, s1_tensor], dim=1)
+    
+    # Inference
+    with torch.no_grad():
+        output = model(input_tensor)
+    
+    cleaned_image = output[0].cpu().numpy()
+    
+    return cleaned_image
+
+
+if __name__ == "__main__":
+    # Train model
+    model, train_losses, val_losses = train_cloud_removal_model(
+        data_dir="winter_dataset",
+        use_s1=True,
+        batch_size=8,
+        num_epochs=50,
+        learning_rate=1e-4
+    )

training_interface.html

@@ -293,6 +293,7 @@
         <div style="background: white; padding: 15px; display: flex; gap: 10px; flex-wrap: wrap; justify-content: center; border-bottom: 2px solid #e0e0e0;">
             <a href="/" style="padding: 10px 20px; background: #667eea; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🏠 Trang Chủ</a>
             <a href="/training" style="padding: 10px 20px; background: #f093fb; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🎓 Training (Active)</a>
+            <a href="/cloud-training" style="padding: 10px 20px; background: #00bcd4; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌥️ Cloud Removal</a>
             <a href="/prediction" style="padding: 10px 20px; background: #4facfe; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🗺️ Prediction</a>
             <a href="/batch" style="padding: 10px 20px; background: #764ba2; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🚀 Batch Processing</a>
             <a href="/ndvi" style="padding: 10px 20px; background: #2ecc71; color: white; border-radius: 8px; text-decoration: none; font-weight: 600;">🌿 NDVI Analysis</a>

winter_dataset/sen12ms_cr_dataLoader.py

@@ -0,0 +1,286 @@
+"""
+    Generic data loading routines for the SEN12MS-CR dataset of corresponding Sentinel 1,
+    Sentinel 2 and cloudy Sentinel 2 data.
+
+    The SEN12MS-CR class is meant to provide a set of helper routines for loading individual
+    image patches as well as triplets of patches from the dataset. These routines can easily
+    be wrapped or extended for use with many deep learning frameworks or as standalone helper 
+    methods. For an example use case please see the "main" routine at the end of this file.
+
+    NOTE: Some folder/file existence and validity checks are implemented but it is 
+          by no means complete.
+
+    Authors: Patrick Ebel (patrick.ebel@tum.de), Lloyd Hughes (lloyd.hughes@tum.de),
+    based on the exemplary data loader code of https://mediatum.ub.tum.de/1474000, with minimal modifications applied.
+"""
+
+import os
+import rasterio
+
+import numpy as np
+
+from enum import Enum
+from glob import glob
+
+
+class S1Bands(Enum):
+    VV = 1
+    VH = 2
+    ALL = [VV, VH]
+    NONE = []
+
+
+class S2Bands(Enum):
+    B01 = aerosol = 1
+    B02 = blue = 2
+    B03 = green = 3
+    B04 = red = 4
+    B05 = re1 = 5
+    B06 = re2 = 6
+    B07 = re3 = 7
+    B08 = nir1 = 8
+    B08A = nir2 = 9
+    B09 = vapor = 10
+    B10 = cirrus = 11
+    B11 = swir1 = 12
+    B12 = swir2 = 13
+    ALL = [B01, B02, B03, B04, B05, B06, B07, B08, B08A, B09, B10, B11, B12]
+    RGB = [B04, B03, B02]
+    NONE = []
+
+
+class Seasons(Enum):
+    SPRING = "ROIs1158_spring"
+    SUMMER = "ROIs1868_summer"
+    FALL = "ROIs1970_fall"
+    WINTER = "ROIs2017_winter"
+    ALL = [SPRING, SUMMER, FALL, WINTER]
+
+
+class Sensor(Enum):
+    s1 = "s1"
+    s2 = "s2"
+    s2cloudy = "s2cloudy"
+
+# Note: The order in which you request the bands is the same order they will be returned in.
+
+
+class SEN12MSCRDataset:
+    def __init__(self, base_dir):
+        self.base_dir = base_dir
+
+        if not os.path.exists(self.base_dir):
+            raise Exception(
+                "The specified base_dir for SEN12MS-CR dataset does not exist")
+
+    """
+        Returns a list of scene ids for a specific season.
+    """
+
+    def get_scene_ids(self, season):
+        season = Seasons(season).value
+        
+        # Check if season folder exists directly
+        path = os.path.join(self.base_dir, season)
+        
+        # If season folder doesn't exist, try with _s1 suffix (alternative structure)
+        if not os.path.exists(path):
+            path = os.path.join(self.base_dir, season + "_s1")
+        
+        if not os.path.exists(path):
+            raise NameError("Could not find season {} in base directory {}".format(
+                season, self.base_dir))
+
+        # add all dirs except "s2_cloudy" (which messes with subsequent string splits)
+        scene_list = [os.path.basename(s)
+                      for s in glob(os.path.join(path, "*")) if "s2_cloudy" not in s]
+        scene_list = [int(s.split("_")[1]) for s in scene_list]
+        return set(scene_list)
+
+    """
+        Returns a list of patch ids for a specific scene within a specific season
+    """
+
+    def get_patch_ids(self, season, scene_id):
+        season = Seasons(season).value
+        path = os.path.join(self.base_dir, season, f"s1_{scene_id}")
+        
+        # If path doesn't exist, try with _s1 suffix
+        if not os.path.exists(path):
+            path = os.path.join(self.base_dir, season + "_s1", f"s1_{scene_id}")
+
+        if not os.path.exists(path):
+            raise NameError(
+                "Could not find scene {} within season {}".format(scene_id, season))
+
+        patch_ids = [os.path.splitext(os.path.basename(p))[0]
+                     for p in glob(os.path.join(path, "*"))]
+        patch_ids = [int(p.rsplit("_", 1)[1].split("p")[1]) for p in patch_ids]
+
+        return patch_ids
+
+    """
+        Return a dict of scene ids and their corresponding patch ids.
+        key => scene_ids, value => list of patch_ids
+    """
+
+    def get_season_ids(self, season):
+        season = Seasons(season).value
+        ids = {}
+        scene_ids = self.get_scene_ids(season)
+
+        for sid in scene_ids:
+            ids[sid] = self.get_patch_ids(season, sid)
+
+        return ids
+
+    """
+        Returns raster data and image bounds for the defined bands of a specific patch
+        This method only loads a sinlge patch from a single sensor as defined by the bands specified
+    """
+
+    def get_patch(self, season, scene_id, patch_id, bands, is_cloudy=False):
+        season = Seasons(season).value
+        sensor = None
+
+        if isinstance(bands, (list, tuple)):
+            b = bands[0]
+        else:
+            b = bands
+        
+        if isinstance(b, S1Bands):
+            sensor = Sensor.s1.value
+            bandEnum = S1Bands
+        elif isinstance(b, S2Bands):
+            sensor = Sensor.s2.value if not is_cloudy else "s2_cloudy"
+            bandEnum = S2Bands
+        else:
+            raise Exception("Invalid bands specified")
+
+        if isinstance(bands, (list, tuple)):
+            bands = [b.value for b in bands]
+        else:
+            bands = bands.value
+
+        scene = "{}_{}".format("s2_cloudy" if is_cloudy else sensor.replace("_cloudy", ""), scene_id)
+        filename = "{}_{}_p{}.tif".format(season, scene, patch_id)
+        
+        # Try standard structure first: base_dir/season/scene/filename
+        patch_path = os.path.join(self.base_dir, season, scene, filename)
+        
+        # If not found, try alternative structure: base_dir/season_sensor/scene/filename
+        if not os.path.exists(patch_path):
+            patch_path = os.path.join(self.base_dir, season + "_" + sensor, scene, filename)
+
+        with rasterio.open(patch_path) as patch:
+            data = patch.read(bands)
+            bounds = patch.bounds
+
+        if len(data.shape) == 2:
+            data = np.expand_dims(data, axis=0)
+
+        return data, bounds
+
+    """
+        Returns a triplet of patches. S1, S2 and cloudy S2 as well as the geo-bounds of the patch
+    """
+
+    def get_s1s2s2cloudy_triplet(self, season, scene_id, patch_id, s1_bands=S1Bands.ALL, s2_bands=S2Bands.ALL, s2cloudy_bands=S2Bands.ALL):
+        s1, bounds = self.get_patch(season, scene_id, patch_id, s1_bands, is_cloudy=False)
+        s2, _ = self.get_patch(season, scene_id, patch_id, s2_bands, is_cloudy=False)
+        s2cloudy, _ = self.get_patch(season, scene_id, patch_id, s2cloudy_bands, is_cloudy=True)
+
+        return s1, s2, s2cloudy, bounds
+
+    """
+        Returns a triplet of numpy arrays with dimensions D, B, W, H where D is the number of patches specified
+        using scene_ids and patch_ids and B is the number of bands for S1, S2 or cloudy S2
+    """
+
+    def get_triplets(self, season, scene_ids=None, patch_ids=None, s1_bands=S1Bands.ALL, s2_bands=S2Bands.ALL, s2cloudy_bands=S2Bands.ALL):
+        season = Seasons(season)
+        scene_list = []
+        patch_list = []
+        bounds = []
+        s1_data = []
+        s2_data = []
+        s2cloudy_data = []
+
+        # This is due to the fact that not all patch ids are available in all scenes
+        # And not all scenes exist in all seasons
+        if isinstance(scene_ids, list) and isinstance(patch_ids, list):
+            raise Exception("Only scene_ids or patch_ids can be a list, not both.")
+
+        if scene_ids is None:
+            scene_list = self.get_scene_ids(season)
+        else:
+            try:
+                scene_list.extend(scene_ids)
+            except TypeError:
+                scene_list.append(scene_ids)
+
+        if patch_ids is not None:
+            try:
+                patch_list.extend(patch_ids)
+            except TypeError:
+                patch_list.append(patch_ids)
+
+        for sid in scene_list:
+            if patch_ids is None:
+                patch_list = self.get_patch_ids(season, sid)
+
+            for pid in patch_list:
+                s1, s2, s2cloudy, bound = self.get_s1s2s2cloudy_triplet(
+                    season, sid, pid, s1_bands, s2_bands, s2cloudy_bands)
+                s1_data.append(s1)
+                s2_data.append(s2)
+                s2cloudy_data.append(s2cloudy)
+                bounds.append(bound)
+
+        return np.stack(s1_data, axis=0), np.stack(s2_data, axis=0), np.stack(s2cloudy_data, axis=0), bounds
+
+
+if __name__ == "__main__":
+    import time
+    # Load the dataset specifying the base directory
+    sen12mscr = SEN12MSCRDataset(".")
+
+    spring_ids = sen12mscr.get_season_ids(Seasons.SPRING)
+    cnt_patches = sum([len(pids) for pids in spring_ids.values()])
+    print("Spring: {} scenes with a total of {} patches".format(
+        len(spring_ids), cnt_patches))
+
+    start = time.time()
+    # Load the RGB bands of the first S2 patch in scene 8
+    SCENE_ID = 8
+    s2_rgb_patch, bounds = sen12mscr.get_patch(Seasons.SPRING, SCENE_ID, 
+                                            spring_ids[SCENE_ID][0], bands=S2Bands.RGB)
+    print("Time Taken {}s".format(time.time() - start))
+                                            
+    print("S2 RGB: {} Bounds: {}".format(s2_rgb_patch.shape, bounds))
+
+    print("\n")
+
+    # Load a triplet of patches from the first three scenes of Spring - all S1 bands, NDVI S2 bands, and NDVI S2 cloudy bands
+    i = 0
+    start = time.time()
+    for scene_id, patch_ids in spring_ids.items():
+        if i >= 3:
+            break
+
+        s1, s2, s2cloudy, bounds = sen12mscr.get_s1s2s2cloudy_triplet(Seasons.SPRING, scene_id, patch_ids[0], s1_bands=S1Bands.ALL,
+                                                        s2_bands=[S2Bands.red, S2Bands.nir1], s2cloudy_bands=[S2Bands.red, S2Bands.nir1])
+        print(
+            f"Scene: {scene_id}, S1: {s1.shape}, S2: {s2.shape}, cloudy S2: {s2cloudy.shape}, Bounds: {bounds}")
+        i += 1
+
+    print("Time Taken {}s".format(time.time() - start))
+    print("\n")
+
+    start = time.time()
+    # Load all bands of all patches in a specified scene (scene 106)
+    s1, s2, s2cloudy, _ = sen12mscr.get_triplets(Seasons.SPRING, 106, s1_bands=S1Bands.ALL, 
+                                        s2_bands=S2Bands.ALL, s2cloudy_bands=S2Bands.ALL)
+
+    print(f"Scene: 106, S1: {s1.shape}, S2: {s2.shape}, cloudy S2: {s2cloudy.shape}")
+    print("Time Taken {}s".format(time.time() - start))