remote-sensing/GEMINI_PROJECT_CONTEXT.md

# GEMINI PROJECT CONTEXT - Land Classification & Remote Sensing System

**Last Updated**: March 26, 2026  
**Project Location**: `/home/x79/remote-sensing`  
**Purpose**: Complete land classification and environmental monitoring system using satellite remote sensing for Vietnam

---

## 📋 PROJECT OVERVIEW

### High-Level Purpose & Problem Domain
- **Core Task**: Classify land use/land cover (8 land classes) in Vietnam using multispectral Sentinel-2 and radar Sentinel-1 data from Microsoft Planetary Computer
- **Geographic Focus**: Vietnam provinces/regions with bounding-box (bbox) based Area-of-Interest (AOI) selection
- **Key Capabilities**:
  - Dynamic training with user-selected regions and time periods
  - Pixel-wise inference (prediction) on new regions
  - Cloud removal using 7 different strategies
  - NDVI time-series forecasting and change detection workflows
  - Auto-generated HTML reports with visualizations
  - Batch processing of multiple regions
  - Model lifecycle management (save, load, validate, delete)

### Data Pipeline
```
Sentinel-2 (optical) + Sentinel-1 (SAR) 
    ↓
[Feature Extraction: 4 modes - simple (3) / temporal (39) / extended (15) / odc (8)]
    ↓
[Model Training: XGBoost, RF, SVM, CNN, Swin-UNet, MobileNet-LRASPP]
    ↓
[Prediction: Pixel-wise classification]
    ↓
[Output: GeoTIFF + PNG preview + HTML report + JSON metadata]
```

---

## 🏗️ SYSTEM ARCHITECTURE

### Core Technology Stack
- **Backend**: FastAPI (~4200 lines in `api_server.py`)
- **ML Training**: scikit-learn (XGBoost, RF, SVM, DT) + PyTorch (CNN, Swin-UNet, MobileNet)
- **Geospatial**: rasterio, rioxarray, geopandas, xarray, odc.stac
- **Data Access**: Microsoft Planetary Computer STAC API (Sentinel-2 L2A, Sentinel-1 RTC)
- **Frontend**: HTML + Leaflet.js (map drawing) + Fetch API + Chart.js
- **GPU Support**: PyTorch with CUDA 12.x (optional fallback to CPU)

### Folder Structure
```
remote-sensing/
├── Core Backend
│   ├── api_server.py                 # FastAPI app (~4200 LOC, 70+ endpoints)
│   ├── train_module.py               # Training pipeline engine
│   ├── feature_extractor.py          # Unified feature extraction (4 modes)
│   ├── model_manager.py              # Model lifecycle management
│   ├── cloud_removal.py              # 7 cloud removal strategies
│   ├── report_generator.py           # Auto HTML/PNG report generation
│   ├── generate_previews.py          # GeoTIFF → PNG conversion
│   │
├── Utilities & Lookup
│   ├── vietnam_provinces.py          # Province bboxes & metadata
│   ├── vietnam_provinces_merged.py   # 32-province variant
│   ├── utils.py                      # Geospatial helper functions
│   ├── create_odc_metadata.py        # Metadata generator utility
│   │
├── Frontend Pages (HTML)
│   ├── index.html                    # Main dashboard hub
│   ├── training_interface.html       # Training UI
│   ├── prediction_interface.html     # Prediction UI
│   ├── batch_interface.html          # Batch processing UI
│   ├── ndvi_interface.html           # NDVI time-series UI
│   ├── dashboard.html                # Analytics dashboard
│   ├── reports_interface.html        # Reports management
│   ├── change_detection_interface.html # Change detection UI
│   ├── cloud_training_interface.html # Cloud removal training UI
│   │
├── Tests & Notebooks
│   ├── test_*.py                     # Unit & integration tests
│   ├── 01.train_ODC*.ipynb           # Training notebooks
│   ├── 02.predict_ODC.ipynb          # Prediction notebooks
│   ├── cloud_removal_train.ipynb     # Cloud removal training
│   │
├── Model Storage & Caches
│   ├── model_train/                  # Trained models (*.joblib, *.pth)
│   │   ├── model_odc.joblib          # Legacy GridSearchCV model
│   │   ├── model_*_info.json         # Metadata sidecar files
│   ├── cloud_removal_model/          # Cloud removal U-Net models (.pth)
│   ├── predictions/                  # Prediction output (GeoTIFF + PNG)
│   ├── reports/                      # Generated HTML reports
│   ├── dataset_cache/                # Cached Sentinel data (optional)
│   │
├── Config & Documentation
│   ├── requirement.txt               # Python dependencies
│   ├── requirements_api.txt          # API-specific deps
│   ├── IMPLEMENTATION_SUMMARY.md     # Model manager summary
│   ├── MODEL_MANAGER_GUIDE.md        # Full model management guide
│   ├── NDVI_FORECAST_METHODOLOGY.md  # NDVI algorithm docs
│   ├── CLOUD_TRAINING_GUIDE.md       # Cloud removal training guide
│   └── [Other guides & docs]
```

---

## 🔧 MAIN MODULES & RESPONSIBILITIES

| **Module** | **File(s)** | **Key Responsibility** |
|---|---|---|
| **API Server** | `api_server.py` | FastAPI app with 70+ endpoints; routes all training, prediction, batch, cloud removal, dashboard, reports, model management tasks |
| **Training Engine** | `train_module.py` | Complete training pipeline: fetch data → feature extraction → train/test split → model training → evaluation → save with metadata |
| **Feature Extraction** | `feature_extractor.py` | Standardized feature extraction with 4 modes: simple, temporal, extended, odc; used by both training and prediction |
| **Model Manager** | `model_manager.py` | Lifecycle management: list, load, save, validate, delete models; handles metadata JSON; auto-detects CNN/PyTorch models |
| **Cloud Removal** | `cloud_removal.py` | 7 cloud removal strategies: classic (3-step), temporal_only, median_composite, none, speckle filter, ML inpainting, deep learning U-Net |
| **Report Generator** | `report_generator.py` | Auto-generates HTML/PNG reports with confusion matrices, class distributions, accuracy trends |
| **Preview Generator** | `generate_previews.py` | Converts GeoTIFF outputs to PNG previews (NDVI or classification rasters) |
| **Province Lookup** | `vietnam_provinces*.py` | Lookup tables for 32+ Vietnamese provinces with bboxes and region grouping |
| **Utilities** | `utils.py` | Geospatial helper functions (load GeoDataFrames, etc.) |

---

## 📊 END-TO-END WORKFLOWS

### 1. TRAINING WORKFLOW
```
User Input → Training Configuration
    ↓
API Endpoint: POST /api/training/start
    ↓
train_module.py: train_model()
    1. Fetch Sentinel-2 & Sentinel-1 from Planetary Computer STAC
    2. Apply cloud mask (SCL band: clouds, shadows, cirrus masked)
    3. Extract features via FeatureExtractor (mode: simple/temporal/extended/odc)
    4. Train/test split (default 0.2)
    5. Train selected model type (XGBoost, RF, CNN, Swin-UNet, MobileNet)
    6. Evaluate: accuracy, precision, recall, F1, confusion matrix
    ↓
model_manager.py: Save model + JSON metadata
    ↓
report_generator.py: Auto-generate HTML training report
    ↓
Return: {model_filename, accuracy_metrics, training_time}
```

**Key Metadata Saved**:
```json
{
  "timestamp": "2026-03-26T14:30:00",
  "model_type": "xgboost",
  "feature_mode": "temporal",
  "n_features": 39,
  "n_classes": 8,
  "features": ["NDVI_t1", "NDVI_t2", ..., "NDWI_t1", ...],
  "test_accuracy": 0.85,
  "train_accuracy": 0.92,
  "bbox": [105.6, 9.3, 106.2, 9.8],
  "time_range": "2023-03-01/2023-05-31",
  "resolution": 20,
  "data_source": "Microsoft Planetary Computer STAC"
}
```

### 2. PREDICTION WORKFLOW
```
User Input → Prediction Configuration (model_filename, bbox, time_range, cloud_strategy)
    ↓
API Endpoint: POST /api/predict or POST /api/predict/with-ndvi
    ↓
run_prediction() function:
    1. Load model via model_manager.py (retrieves metadata, feature requirements)
    2. Fetch Sentinel-2 & Sentinel-1 for new region
    3. Apply chosen cloud_removal_method (classic/temporal_only/median_composite/none/deep_learning)
    4. Extract features matching model's metadata requirements
    5. Auto-adjust if feature count mismatch (pad/trim)
    6. Predict land class for each pixel
    7. (Optional) Calculate NDVI: (NIR - Red) / (NIR + Red)
    8. Save outputs: GeoTIFF + PNG preview
    ↓
generate_previews.py: Create PNG from GeoTIFF
    ↓
report_generator.py: Generate prediction report
    ↓
Return: {prediction_file, ndvi_file, class_distribution, statistics}
```

### 3. BATCH PROCESSING WORKFLOW
```
User uploads CSV with multiple regions:
  (name, min_lon, min_lat, max_lon, max_lat, start_date, end_date, max_scenes, cloud_cover, resolution)
    ↓
API Endpoint: POST /api/batch/start
    ↓
Enqueue all regions; process sequentially
    ↓
For each region: Run same prediction workflow
    ↓
Track status per region: Queued → Running → Completed/Failed
    ↓
UI shows progress bar, auto-retry on failure (max 3 retries)
    ↓
Return: Bulk results with per-region status & output files
```

### 4. CLOUD REMOVAL WORKFLOW
```
User selects cloud_removal_method in prediction config:
    ↓
cloud_removal.py: process_cloud_removal()
    ↓
Strategy Selection:
  • 'classic': temporal interpolation → median composite → spatial interpolation (3-step)
  • 'temporal_only': ffill + bfill across time dimension (fast, good for many scenes)
  • 'median_composite': Prioritize median across scenes (best for noise reduction)
  • 'none': Keep original, just fill NaN with 0
  • 'deep': Use trained U-Net model (S2 cloudy + S1 → clean S2)
  • 'ml_inpainting': KNN or Random Forest based inpainting
  • 'speckle_filter': Reduce radar noise
    ↓
Return cleaned Sentinel-2 data for subsequent feature extraction
```

### 5. NDVI TIME-SERIES WORKFLOW
```
User requests NDVI calculation (bbox + time_range + aggregation)
    ↓
API Endpoint: POST /api/ndvi/timeseries or /api/ndvi/predict-timeseries
    ↓
Load Sentinel-2 (B04 Red, B08 NIR)
    ↓
Calculate NDVI = (NIR - Red) / (NIR + Red + 0.00001)
    ↓
Resample to monthly or user-defined aggregation
    ↓
Export as GeoTIFF + PNG visualization
    ↓
Show time-series graph & statistics (mean, min, max, std, trend)
```

### 6. CHANGE DETECTION WORKFLOW
```
User selects: model + current_period + prediction_period
    ↓
API Endpoint: POST /api/change-detection/compare-periods
    ↓
Run prediction for both time periods
    ↓
Compute difference map (current - prediction)
    ↓
Classify changes: increased vegetation, decreased vegetation, stable
    ↓
Generate change map GeoTIFF + report with statistics
```

---

## 🌐 API ENDPOINTS SUMMARY (70+ endpoints)

### Model Management
- `GET /api/models/list` - List all trained models with metadata
- `GET /api/models/{filename}/info` - Get model details
- `GET /api/models/{filename}/validate` - Validate model integrity
- `DELETE /api/models/{filename}` - Delete model file

### Training APIs
- `POST /api/training/start` - Start land classification training
- `GET /api/training/status` - Get training progress
- `POST /api/training/stop` - Cancel ongoing training
- `POST /api/cloud-removal/train` - Train cloud removal U-Net

### Prediction APIs
- `POST /api/predict` - Standard prediction (classification only)
- `POST /api/predict/with-ndvi` - Prediction with NDVI export
- `POST /api/change-detection/compare-periods` - Change detection
- `GET /api/prediction/status` - Check prediction progress
- `GET /api/predictions/list` - List prediction outputs
- `GET /api/predictions/download/{filename}` - Download prediction file
- `GET /api/predictions/preview/{filename}` - View PNG preview

### Batch Processing
- `POST /api/batch/start` - Enqueue multiple predictions from CSV
- `GET /api/batch/status` - Check batch queue
- `GET /api/batch/results/{batch_id}` - Retrieve batch results
- `POST /api/batch/cancel/{batch_id}` - Cancel batch job

### Cloud Removal
- `GET /api/cloud-removal/methods` - List available strategies
- `GET /api/cloud-removal/models` - List trained .pth models
- `POST /api/cloud-removal/upload` - Upload .pth cloud removal model
- `DELETE /api/cloud-removal/models/{filename}` - Delete cloud removal model

### Dashboard & Reports
- `GET /api/dashboard/statistics` - Overall system stats
- `GET /api/dashboard/accuracy-trends` - Accuracy over time
- `GET /api/dashboard/class-distribution/{model_filename}` - Class distribution
- `GET /api/reports/list` - List generated reports
- `GET /api/reports/view/{filename}` - View HTML report
- `GET /api/reports/download/{filename}` - Download report
- `DELETE /api/reports/delete/{filename}` - Delete report

### Provinces & Utilities
- `GET /api/provinces/list` - List all Vietnamese provinces
- `GET /api/provinces/by-region` - Group provinces by region
- `GET /api/provinces/{province_name}/bbox` - Get province bbox
- `GET /api/provinces/search/{query}` - Search province by name
- `GET /api/provinces-32/*` - Alternative 32-province variant
- `GET /api/network/check` - Check connectivity to Planetary Computer
- `GET /api/cache/info` - Show cache statistics
- `POST /api/cache/clear` - Clear local cache

### NDVI & Time-Series
- `POST /api/ndvi/timeseries` - Calculate NDVI time-series
- `POST /api/ndvi/predict-timeseries` - NDVI prediction/forecast
- `POST /api/ndvi/forecast` - NDVI forecasting

### File Management
- `GET /api/training/files` - List training files
- `GET /api/overlay/shapefiles` - List available shapefiles
- `GET /api/training/shapefile/{filename}/labels` - Get shapefile labels
- `POST /api/land-classification/upload` - Upload custom model
- `POST /api/cloud-removal/upload` - Upload cloud removal model

### Frontend Routes (Serve HTML)
- `GET /` - Main dashboard
- `GET /training` - Training interface
- `GET /prediction` - Prediction interface
- `GET /dashboard` - Analytics dashboard
- `GET /batch` - Batch processing UI
- `GET /ndvi` - NDVI time-series UI
- `GET /reports` - Reports management
- `GET /cloud-training` - Cloud removal training
- `GET /change-detection` - Change detection UI

---

## 💾 DATA INPUTS / OUTPUTS & FOLDER CONVENTIONS

### Input Data Sources
- **Sentinel-2 L2A** from Microsoft Planetary Computer STAC API
  - Bands: B02 (blue), B03 (green), B04 (red), B08 (NIR), B11 (SWIR), SCL (cloud mask)
  - Resolution: 10m or 20m (user selectable)
  - Collection: `sentinel-2-l2a`

- **Sentinel-1 RTC** from Planetary Computer
  - Bands: VH, VV (radar polarizations)
  - Converted to dB scale: `10 * log10(intensity)`
  - Collection: `sentinel-1-rtc`

- **Training Labels**: User-provided shapefiles with pixel-level class labels

### Output File Structure
```
predictions/
  ├── prediction_YYYYMMDD_HHMMSS.tif         # Classification GeoTIFF
  ├── prediction_YYYYMMDD_HHMMSS.png         # PNG preview
  ├── ndvi_YYYYMMDD_HHMMSS.tif              # NDVI raster
  ├── ndvi_YYYYMMDD_HHMMSS.png              # NDVI preview

reports/
  ├── training_report_*.html                 # Auto training reports
  ├── prediction_report_*.html                # Auto prediction reports

model_train/
  ├── model_odc.joblib                      # Legacy model
  ├── model_odc_info.json                    # Metadata
  ├── model_xgboost_*.joblib                 # XGBoost models
  ├── model_xgboost_*_info.json              # Metadata
  ├── model_cnn_*.joblib                     # CNN models
  ├── model_cnn_*_info.json                  # Metadata

cloud_removal_model/
  ├── cloud_removal_unet_best.pth            # Trained U-Net
  ├── *.pth                                  # Custom models
  ├── *.json                                 # Model metadata
```

---

## 🔌 EXTERNAL DEPENDENCIES & PLATFORMS

### Critical External Services
- **Microsoft Planetary Computer** (STAC API)
  - Hosts Sentinel-2 L2A and Sentinel-1 RTC archives
  - URL: `https://planetarycomputer.microsoft.com/api/stac/v1`
  - Auto-signed access tokens via `planetary_computer.sign_inplace`
  - Network connectivity check: `GET /api/network/check`

### Key Python Libraries
- **Geospatial**: rasterio, rioxarray, geopandas, shapely, Cartopy, folium, ipyleaflet
- **Data Processing**: numpy, pandas, xarray, dask
- **ML**: scikit-learn, xgboost
- **Deep Learning**: torch, torchvision
- **Web**: fastapi, uvicorn, pydantic
- **Visualization**: matplotlib, Pillow (PIL)
- **Document Gen**: markdown, Pillow

### GPU Support
- PyTorch with CUDA 12.x (optional; falls back to CPU)
- Benefits Swin-UNet and CNN models (10-100x speedup)
- CPU training for XGBoost/RF typically <1 hour; deep models need GPU for reasonable speed

---

## ⚙️ FEATURE EXTRACTION MODES (CRITICAL)

Train and prediction **MUST** use same feature mode and dimension; metadata auto-detects this.

| Mode | # Features | Description | Best For | Training Time |
|---|---|---|---|---|
| **simple** | 3 | NDVI_mean, VH_db_mean, VV_db_mean | Fast iteration, baseline | ~5-10 min |
| **temporal** | 39 | NDVI/NDWI/NDBI across 13 months + radar stats | High accuracy (~85%+) | ~30-60 min |
| **extended** | 15 | NDVI/NDWI/NDBI stats (mean/std/min/max) + radar | Balanced speed/accuracy | ~15-30 min |
| **odc** | 8 | NDVI stats + NDWI/NDBI/EVI mean (legacy ODC mode) | Legacy compatibility | ~10-20 min |

**Critical**: If feature mode = "temporal" (39 features) at training, prediction MUST extract 39 features. System auto-detects from metadata but will fail if mismatched.

---

## 🎯 OPERATIONAL NOTES & CONSTRAINTS

### Performance Limits
1. **Planetary Computer Timeout Issues**
   - Large bbox (>10km × 10km) + long time range (>1 month) + high max_scenes → timeouts
   - **Solution**: Progressive loading (subdivide bbox), reduce time window, reduce max_scenes
   - **Safe Settings**: bbox ≤ 10km × 10km, time ≤ 1 month, max_scenes ≤ 12

2. **Memory Usage**
   - Temporal mode (39 features) requires ~2-3x RAM vs simple mode
   - Large regions: reduce resolution (10m → 20m) or split into sub-tiles
   - Batch processing: sequential (one region at a time due to API limits)

3. **GPU Training**
   - Swin-UNet: ~15-60 min on GPU vs ~2-4 hours on CPU
   - CNN: ~10-30 min on GPU vs ~1-2 hours on CPU
   - XGBoost/RF: CPU-bound; GPU not beneficial

### Data Quality Issues
1. **Cloud Cover**
   - SCL band values: 3=cloud shadow, 8=cloud medium, 9=cloud high, 10=cirrus
   - Recommend multiple scenes (≥5) for temporal aggregation
   - Cloud removal strategy critical—test different approaches

2. **Radar Data (Sentinel-1)**
   - Not always available for all regions/dates
   - System gracefully falls back to zeros if unavailable
   - Safe for "extended" & "odc" modes that have radar fallback

3. **Feature Mode Mismatch**
   - Model trained with "temporal" (39 features) needs 39-dim input
   - System auto-adjusts (pads/trims) from metadata but may degrade accuracy
   - **Best Practice**: Align feature mode explicitly; don't mix

### Known Caveats
1. **Legacy Model (model_odc.joblib)**: Hardcoded 39 temporal features; auto-detected via `model_odc_info.json`
2. **Metadata Consistency**: Old models may lack `.json` sidecar; system generates default (may be incorrect)
3. **Batch Processing**: Sequential only; large batches (100+ regions) take hours
4. **Change Detection**: Simple differencing approach; requires same model & feature mode for both periods
5. **Rate Limiting**: Planetary Computer may rate-limit if too many concurrent requests

### Recommended Best Practices
- Test model on small bbox first (2km × 2km, 1 week, 3 scenes)
- Use "simple" mode for fast iteration, "temporal" for best accuracy (85%+)
- Store metadata JSON alongside model file (sidecar pattern)
- Version control: record feature_mode & n_features in every training
- Monitor training accuracy; retrain if <70% accuracy
- Cache Sentinel data locally to avoid repeated downloads
- Use "median_composite" cloud strategy if >5 scenes; "temporal_only" if 3-4 scenes

---

## 🔍 TEST COVERAGE MAP

| Test File | Coverage | Status |
|---|---|---|
| `test_model_manager.py` | ModelManager lifecycle (list, load, validate) | ✅ Well-tested |
| `test_feature_extractor.py` | All 4 feature extraction modes | ✅ Well-tested |
| `test_training_api.py` | Training API endpoints | ✅ Partial |
| `test_cloud_removal.py` | 7 cloud removal strategies | ✅ Well-tested |
| `test_cloud_training.py` | U-Net cloud removal training | ✅ Partial |
| `test_shapefile_api.py` | Shapefile overlay feature | ✅ Partial |
| `test_planetary_computer.py` | Planetary Computer STAC access | ✅ Well-tested |
| `test_new_features.py` | Recent feature releases | ✅ Partial |
| Jupyter Notebooks | Training & prediction workflows | ✅ Mix of unit/integration/notebooks |

**Coverage Notes**: Model management, feature extraction, and cloud removal well-tested; Dashboard UI, change detection, NDVI time-series mostly tested via notebooks.

---

## 📚 FILE REFERENCE MAP

### Core Execution
- `api_server.py` — Main FastAPI application (~4200 LOC)
- `train_module.py` — Training logic (data fetch → feature extraction → training)
- `run_prediction_new.py` — Prediction execution function
- `feature_extractor.py` — Unified feature extraction (4 modes)
- `model_manager.py` — Model lifecycle (load/save/validate/list)
- `cloud_removal.py` — Cloud removal strategies (7 methods)
- `report_generator.py` — HTML/PNG report auto-generation
- `generate_previews.py` — GeoTIFF → PNG conversion

### Data & Config
- `vietnam_provinces.py` — 32+ province lookup tables & bboxes
- `vietnam_provinces_merged.py` — Alternative 32-province variant
- `utils.py` — Geospatial utility functions
- `create_odc_metadata.py` — Legacy metadata generator

### Frontend
- `index.html` — Main dashboard hub (tab navigation)
- `training_interface.html` — Training configuration UI
- `prediction_interface.html` — Prediction configuration UI
- `batch_interface.html` — Batch processing (CSV upload)
- `ndvi_interface.html` — NDVI time-series visualization
- `dashboard.html` — Analytics & model performance dashboard
- `reports_interface.html` — Report management & viewing
- `change_detection_interface.html` — Change detection visualization
- `cloud_training_interface.html` — Cloud removal U-Net training

### Documentation
- `IMPLEMENTATION_SUMMARY.md` — Model manager & system overview
- `MODEL_MANAGER_GUIDE.md` — Complete model management guide
- `NDVI_FORECAST_METHODOLOGY.md` — NDVI algorithm documentation
- `CLOUD_TRAINING_GUIDE.md` — Cloud removal training guide
- `NDVI_PREDICTION_GUIDE.md` — NDVI prediction workflow
- `CLOUD_PROCESSING.md` — Cloud processing notes
- `UPDATE_SUMMARY.md` — Recent updates & features

---

## 🚀 BOOTSTRAP PROMPT FOR GEMINI

### System Context (Copy & Paste for Gemini)

```
You are assisting a remote-sensing land-classification project for Vietnam.

## ARCHITECTURE SNAPSHOT
- **Backend**: FastAPI (~4200 LOC, 70+ endpoints) for orchestrating training, prediction, batch, cloud removal, reporting
- **Data Source**: Microsoft Planetary Computer STAC API (Sentinel-2 L2A + Sentinel-1 RTC)
- **Training**: scikit-learn (XGBoost/RF/SVM/DT) + PyTorch (CNN/Swin-UNet/MobileNet)
- **Feature Extraction**: 4 modes (simple 3-feat / temporal 39-feat / extended 15-feat / odc 8-feat)
- **Cloud Removal**: 7 strategies (classic, temporal_only, median_composite, none, ML inpainting, deep U-Net)
- **Output**: GeoTIFF + PNG + HTML report + JSON metadata

## CORE FILES TO UNDERSTAND (Priority Order)
1. api_server.py — Main API server (training, prediction, batch, models, reports)
2. train_module.py — Training pipeline (data fetch → feature extraction → train → save)
3. feature_extractor.py — Unified feature extraction with auto mode detection
4. model_manager.py — Model lifecycle (load/save/validate/list)
5. cloud_removal.py — Cloud removal strategies (7 methods)
6. report_generator.py — Auto-generate HTML reports
7. run_prediction_new.py — Prediction execution
8. vietnam_provinces.py — Province lookup & bbox tables

## CRITICAL CONSTRAINTS & GOTCHAS
1. **Feature Mode Consistency**: Training & prediction MUST use same mode (simple/temporal/extended/odc)
   → Auto-detected from metadata JSON
   → Mismatch causes dimension error or accuracy degradation

2. **Planetary Computer Limits**:
   → Timeout if bbox >10km×10km OR time range >1 month OR max_scenes >12
   → Solution: subdivide bbox, reduce time window, limit scenes

3. **Cloud Strategy Selection**:
   → ≥5 scenes → use "median_composite" (best noise reduction)
   → 3-4 scenes → use "temporal_only" (fast temporal interp)
   → <3 scenes → use "none" (skip cloud removal)

4. **Radar Data Fallback**:
   → Sentinel-1 may be unavailable for some regions
   → System gracefully falls back to zeros (safe for all modes)

5. **Model Metadata**:
   → Always stored as `model_name_info.json` sidecar file
   → Contains: n_features, feature_mode, features list, accuracy, bbox, time_range
   → Missing metadata → system uses defaults (may be incorrect)

6. **Legacy Model (model_odc.joblib)**:
   → Hardcoded 39 temporal features
   → Metadata in model_odc_info.json

## REASONING CHECKLIST (before answering)
□ Is feature_mode consistent between train and prediction?
□ Is metadata.json present and correct?
□ Does bbox exceed 10km×10km? (Planetary Computer timeout risk)
□ Is cloud_removal_strategy appropriate for # of scenes?
□ Is Sentinel-1 data available for this region/date?
□ Is model a joblib (scikit-learn) or .pth (PyTorch) file?
□ Is GPU available for deep models (CNN, Swin-UNet)?
□ Does memory allow temporal feature extraction (39-feat)?

## RESPONSE FORMAT
- Always cite api_server.py endpoint, function name, or module being discussed
- Verify feature_mode & n_features from metadata JSON
- Suggest cloud_removal_strategy based on # of scenes available
- For unknown issues: offer alternative approaches (reduce bbox, cache results, use simpler model)
- Explain reasoning using checklist above

## DATA FLOW SUMMARY
Sentinel-2/S1 → [Cloud Remove] → [Feature Extract] → [Train/Predict] → [GeoTIFF + PNG + Report]
```

---

## 📞 QUICK REFERENCE CHECKLIST

### Before Troubleshooting Any Issue
- [ ] Check feature_mode consistency (metadata JSON)
- [ ] Verify metadata.json exists for the model
- [ ] Check Planetary Computer connectivity (`GET /api/network/check`)
- [ ] Review cloud_removal_method choice (≥5 scenes = median_composite)
- [ ] Confirm Sentinel-1 availability (or fallback to zeros if missing)
- [ ] Validate bbox size (≤10km×10km for safety)
- [ ] Check memory usage for temporal feature mode
- [ ] Verify GPU if using CNN/Swin-UNet models

### Common Issues & Solutions
| Issue | Likely Cause | Solution |
|---|---|---|
| Training timeout | Large bbox / long time / many scenes | Subdivide bbox, reduce time window, max_scenes ≤ 12 |
| Feature dimension mismatch | Different feature_mode between train & predict | Check metadata.json, ensure same mode |
| Low prediction accuracy | Cloud cover, poor training data, feature mode too simple | Use "temporal" mode, increase training data, try cloud removal |
| Out of memory | Temporal features + large region | Reduce resolution (20m), split into sub-tiles, increase RAM |
| Model not found | Wrong filename or model_train/ path issue | `GET /api/models/list` to verify, check file path |
| Planetary Computer error | Network issue or API rate limit | Check DNS, retry later, reduce concurrent requests |
| Cloud removal failing | Strategy not suitable for scene count | Try "none" or "median_composite" depending on scenes |

---

## 🎓 LEARNING RESOURCES IN REPO

- **Notebooks**: `01.train_ODC.ipynb`, `02.predict_ODC.ipynb`, `cloud_removal_train.ipynb`
- **Tests**: `test_*.py` files for unit test patterns
- **Docs**: All `*.md` files for detailed guides and methodology
- **Code Comments**: API server and modules heavily commented

---

**Generated**: March 26, 2026  
**For Use By**: Gemini, Claude, GPT, or any AI system needing project context  
**Maintainer**: Remote-Sensing Project Team