Victor Phan
285 lines
7.1 KiB
Markdown
285 lines
7.1 KiB
Markdown
# Model Upload Guide
|
|
|
|
## Overview
|
|
This system now supports uploading custom models for both **Cloud Removal** and **Land Classification** tasks with full metadata tracking.
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
remote-sensing/
|
|
├── cloud_removal_model/ # Cloud removal models (U-Net, GAN, etc.)
|
|
│ ├── *.pth # PyTorch model files
|
|
│ └── *.json # Metadata sidecar files
|
|
├── land_classification_model/ # Land use classification models
|
|
│ ├── *.pth, *.pkl, *.joblib # Model files (various formats)
|
|
│ ├── *.h5, *.keras # TensorFlow/Keras models
|
|
│ └── *.json # Metadata sidecar files
|
|
└── model_train/ # Legacy training outputs (other models)
|
|
```
|
|
|
|
## Cloud Removal Model Upload
|
|
|
|
### Supported Format
|
|
- **File Extension**: `.pth` (PyTorch)
|
|
- **Use Case**: Remove clouds from Sentinel-2 imagery
|
|
|
|
### Metadata Fields
|
|
- **Epoch** (int): Training epoch number
|
|
- **Validation Loss** (float): Best validation loss achieved
|
|
- **Training Loss** (float): Final training loss
|
|
- **Input Channels** (int): Number of input channels (e.g., 6 for S2+S1)
|
|
- **Output Channels** (int): Number of output channels (e.g., 4 for RGBN)
|
|
- **Use Sentinel-1** (bool): Whether model uses SAR data
|
|
- **Description** (string): Optional notes about the model
|
|
|
|
### API Endpoint
|
|
```http
|
|
POST /api/cloud-removal/upload
|
|
Content-Type: multipart/form-data
|
|
|
|
{
|
|
"file": <binary>,
|
|
"epoch": 50,
|
|
"val_loss": 0.0134,
|
|
"train_loss": 0.0142,
|
|
"in_channels": 6,
|
|
"out_channels": 4,
|
|
"use_s1": true,
|
|
"description": "Trained on winter dataset"
|
|
}
|
|
```
|
|
|
|
### Example Metadata File
|
|
`cloud_removal_unet_winter.pth.json`:
|
|
```json
|
|
{
|
|
"filename": "cloud_removal_unet_winter.pth",
|
|
"epoch": 50,
|
|
"train_loss": 0.0142,
|
|
"val_loss": 0.0134,
|
|
"in_channels": 6,
|
|
"out_channels": 4,
|
|
"use_s1": true,
|
|
"description": "Trained on winter dataset, 50 epochs",
|
|
"uploaded_at": "2026-01-26T15:30:00"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Land Classification Model Upload
|
|
|
|
### Supported Formats
|
|
- **PyTorch**: `.pth`
|
|
- **Scikit-learn**: `.pkl`, `.joblib`
|
|
- **TensorFlow/Keras**: `.h5`, `.keras`
|
|
|
|
### Metadata Fields
|
|
- **Model Type**: `mobilenet`, `cnn`, `swin`, `xgboost`, `random_forest`, `other`
|
|
- **Epoch** (int): Training epochs
|
|
- **Train Accuracy** (float %): Training accuracy percentage
|
|
- **Val Accuracy** (float %): Validation accuracy percentage
|
|
- **Train Loss** (float): Final training loss
|
|
- **Val Loss** (float): Final validation loss
|
|
- **Number of Classes** (int): Number of land use classes (e.g., 10)
|
|
- **Input Size** (int): Input image dimension (e.g., 64x64)
|
|
- **Description** (string): Optional notes
|
|
|
|
### API Endpoint
|
|
```http
|
|
POST /api/land-classification/upload
|
|
Content-Type: multipart/form-data
|
|
|
|
{
|
|
"file": <binary>,
|
|
"model_type": "mobilenet",
|
|
"epoch": 100,
|
|
"train_accuracy": 95.5,
|
|
"val_accuracy": 93.2,
|
|
"train_loss": 0.12,
|
|
"val_loss": 0.18,
|
|
"num_classes": 10,
|
|
"input_size": 64,
|
|
"description": "MobileNetV2 trained on Mekong Delta"
|
|
}
|
|
```
|
|
|
|
### Example Metadata File
|
|
`mobilenet_mekong_v2.pth.json`:
|
|
```json
|
|
{
|
|
"filename": "mobilenet_mekong_v2.pth",
|
|
"model_type": "mobilenet",
|
|
"epoch": 100,
|
|
"train_accuracy": 95.5,
|
|
"val_accuracy": 93.2,
|
|
"train_loss": 0.12,
|
|
"val_loss": 0.18,
|
|
"num_classes": 10,
|
|
"input_size": 64,
|
|
"description": "MobileNetV2 trained on Mekong Delta dataset",
|
|
"uploaded_at": "2026-01-26T15:45:00"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Usage in Web Interface
|
|
|
|
### Cloud Removal Models
|
|
1. Navigate to **Prediction Interface**
|
|
2. Select **Cloud Removal Method** → "Deep Learning (U-Net)"
|
|
3. Click **📤 Upload Cloud Removal Model (.pth)**
|
|
4. Fill in metadata form
|
|
5. Click **✅ Upload with Metadata**
|
|
6. Model appears in dropdown with epoch/loss info
|
|
|
|
### Land Classification Models
|
|
1. Navigate to **Prediction Interface**
|
|
2. In **Model Selection** section
|
|
3. Click **📤 Upload Land Classification Model**
|
|
4. Fill in metadata form (model type, accuracy, etc.)
|
|
5. Click **✅ Upload with Metadata**
|
|
6. Model appears in main model dropdown
|
|
|
|
---
|
|
|
|
## API Reference
|
|
|
|
### List Models
|
|
|
|
**Cloud Removal:**
|
|
```http
|
|
GET /api/cloud-removal/models
|
|
```
|
|
|
|
**Land Classification:**
|
|
```http
|
|
GET /api/land-classification/models
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"models": [
|
|
{
|
|
"filename": "model.pth",
|
|
"epoch": 50,
|
|
"val_loss": 0.0134,
|
|
"size_mb": 356.2,
|
|
"has_metadata": true,
|
|
"created": 1706284800
|
|
}
|
|
],
|
|
"count": 1
|
|
}
|
|
```
|
|
|
|
### Delete Model
|
|
|
|
**Cloud Removal:**
|
|
```http
|
|
DELETE /api/cloud-removal/models/{filename}
|
|
```
|
|
|
|
**Land Classification:**
|
|
```http
|
|
DELETE /api/land-classification/models/{filename}
|
|
```
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
1. **Naming Convention**: Use descriptive names
|
|
- ✅ `cloud_removal_unet_winter_50ep.pth`
|
|
- ✅ `mobilenet_v2_mekong_acc93.pth`
|
|
- ❌ `model1.pth`
|
|
|
|
2. **Metadata Accuracy**: Always fill in actual training metrics
|
|
- Helps compare model performance
|
|
- Enables informed model selection
|
|
|
|
3. **Version Control**: Include version/date in description
|
|
- "v2.0 - Improved augmentation"
|
|
- "2026-01-15 - Fixed class imbalance"
|
|
|
|
4. **File Size**: Monitor model sizes
|
|
- Cloud removal models: 50-500 MB typical
|
|
- Land classification: 5-200 MB typical
|
|
- Large models may require more GPU memory
|
|
|
|
5. **Testing**: Always test uploaded model on small region first
|
|
- Verify predictions are reasonable
|
|
- Check for errors/crashes
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Upload Fails with "Already Exists"
|
|
- Model filename is duplicate
|
|
- Delete old model first or rename new one
|
|
|
|
### Model Shows Default Values (0, 0, 0)
|
|
- Server needs restart to load `Form(...)` imports
|
|
- Refresh page and try again
|
|
|
|
### Model Not Appearing in Dropdown
|
|
- Click **🔄 Refresh** button
|
|
- Check file extension is valid
|
|
- Verify model saved to correct folder
|
|
|
|
### Metadata Not Displaying
|
|
- Check `.json` file exists alongside model
|
|
- Verify JSON format is valid
|
|
- Look for server errors in terminal
|
|
|
|
---
|
|
|
|
## Migration from Old System
|
|
|
|
If you have models in `model_train/`:
|
|
|
|
1. **Cloud Removal Models**: Move to `cloud_removal_model/`
|
|
```bash
|
|
mv model_train/cloud_removal_*.pth cloud_removal_model/
|
|
mv model_train/*_unet*.pth cloud_removal_model/
|
|
mv model_train/*GAN*.pth cloud_removal_model/
|
|
```
|
|
|
|
2. **Land Classification Models**: Move to `land_classification_model/`
|
|
```bash
|
|
mv model_train/mobilenet*.pth land_classification_model/
|
|
mv model_train/cnn*.pth land_classification_model/
|
|
mv model_train/swin*.pth land_classification_model/
|
|
mv model_train/*.pkl land_classification_model/
|
|
```
|
|
|
|
3. **Create metadata files** by re-uploading through web interface
|
|
|
|
---
|
|
|
|
## Security Features
|
|
|
|
✅ **File Extension Validation**: Only allowed formats accepted
|
|
✅ **Path Traversal Prevention**: No `../` or `/` in filenames
|
|
✅ **Duplicate Detection**: Prevents overwriting existing models
|
|
✅ **Size Limits**: Prevents extremely large uploads
|
|
✅ **JSON Sanitization**: Metadata stored safely
|
|
|
|
---
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Batch model upload
|
|
- [ ] Model versioning system
|
|
- [ ] Automated benchmarking
|
|
- [ ] Model comparison tool
|
|
- [ ] Export/import model configs
|
|
- [ ] Cloud storage integration
|
|
|
|
---
|
|
|
|
**Last Updated**: January 26, 2026
|