change to NAS
This commit is contained in:
Victor Phan
@@ -0,0 +1,284 @@
|
||||
# Model Upload Guide
|
||||
|
||||
## Overview
|
||||
This system now supports uploading custom models for both **Cloud Removal** and **Land Classification** tasks with full metadata tracking.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
remote-sensing/
|
||||
├── cloud_removal_model/ # Cloud removal models (U-Net, GAN, etc.)
|
||||
│ ├── *.pth # PyTorch model files
|
||||
│ └── *.json # Metadata sidecar files
|
||||
├── land_classification_model/ # Land use classification models
|
||||
│ ├── *.pth, *.pkl, *.joblib # Model files (various formats)
|
||||
│ ├── *.h5, *.keras # TensorFlow/Keras models
|
||||
│ └── *.json # Metadata sidecar files
|
||||
└── model_train/ # Legacy training outputs (other models)
|
||||
```
|
||||
|
||||
## Cloud Removal Model Upload
|
||||
|
||||
### Supported Format
|
||||
- **File Extension**: `.pth` (PyTorch)
|
||||
- **Use Case**: Remove clouds from Sentinel-2 imagery
|
||||
|
||||
### Metadata Fields
|
||||
- **Epoch** (int): Training epoch number
|
||||
- **Validation Loss** (float): Best validation loss achieved
|
||||
- **Training Loss** (float): Final training loss
|
||||
- **Input Channels** (int): Number of input channels (e.g., 6 for S2+S1)
|
||||
- **Output Channels** (int): Number of output channels (e.g., 4 for RGBN)
|
||||
- **Use Sentinel-1** (bool): Whether model uses SAR data
|
||||
- **Description** (string): Optional notes about the model
|
||||
|
||||
### API Endpoint
|
||||
```http
|
||||
POST /api/cloud-removal/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
{
|
||||
"file": <binary>,
|
||||
"epoch": 50,
|
||||
"val_loss": 0.0134,
|
||||
"train_loss": 0.0142,
|
||||
"in_channels": 6,
|
||||
"out_channels": 4,
|
||||
"use_s1": true,
|
||||
"description": "Trained on winter dataset"
|
||||
}
|
||||
```
|
||||
|
||||
### Example Metadata File
|
||||
`cloud_removal_unet_winter.pth.json`:
|
||||
```json
|
||||
{
|
||||
"filename": "cloud_removal_unet_winter.pth",
|
||||
"epoch": 50,
|
||||
"train_loss": 0.0142,
|
||||
"val_loss": 0.0134,
|
||||
"in_channels": 6,
|
||||
"out_channels": 4,
|
||||
"use_s1": true,
|
||||
"description": "Trained on winter dataset, 50 epochs",
|
||||
"uploaded_at": "2026-01-26T15:30:00"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Land Classification Model Upload
|
||||
|
||||
### Supported Formats
|
||||
- **PyTorch**: `.pth`
|
||||
- **Scikit-learn**: `.pkl`, `.joblib`
|
||||
- **TensorFlow/Keras**: `.h5`, `.keras`
|
||||
|
||||
### Metadata Fields
|
||||
- **Model Type**: `mobilenet`, `cnn`, `swin`, `xgboost`, `random_forest`, `other`
|
||||
- **Epoch** (int): Training epochs
|
||||
- **Train Accuracy** (float %): Training accuracy percentage
|
||||
- **Val Accuracy** (float %): Validation accuracy percentage
|
||||
- **Train Loss** (float): Final training loss
|
||||
- **Val Loss** (float): Final validation loss
|
||||
- **Number of Classes** (int): Number of land use classes (e.g., 10)
|
||||
- **Input Size** (int): Input image dimension (e.g., 64x64)
|
||||
- **Description** (string): Optional notes
|
||||
|
||||
### API Endpoint
|
||||
```http
|
||||
POST /api/land-classification/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
{
|
||||
"file": <binary>,
|
||||
"model_type": "mobilenet",
|
||||
"epoch": 100,
|
||||
"train_accuracy": 95.5,
|
||||
"val_accuracy": 93.2,
|
||||
"train_loss": 0.12,
|
||||
"val_loss": 0.18,
|
||||
"num_classes": 10,
|
||||
"input_size": 64,
|
||||
"description": "MobileNetV2 trained on Mekong Delta"
|
||||
}
|
||||
```
|
||||
|
||||
### Example Metadata File
|
||||
`mobilenet_mekong_v2.pth.json`:
|
||||
```json
|
||||
{
|
||||
"filename": "mobilenet_mekong_v2.pth",
|
||||
"model_type": "mobilenet",
|
||||
"epoch": 100,
|
||||
"train_accuracy": 95.5,
|
||||
"val_accuracy": 93.2,
|
||||
"train_loss": 0.12,
|
||||
"val_loss": 0.18,
|
||||
"num_classes": 10,
|
||||
"input_size": 64,
|
||||
"description": "MobileNetV2 trained on Mekong Delta dataset",
|
||||
"uploaded_at": "2026-01-26T15:45:00"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage in Web Interface
|
||||
|
||||
### Cloud Removal Models
|
||||
1. Navigate to **Prediction Interface**
|
||||
2. Select **Cloud Removal Method** → "Deep Learning (U-Net)"
|
||||
3. Click **📤 Upload Cloud Removal Model (.pth)**
|
||||
4. Fill in metadata form
|
||||
5. Click **✅ Upload with Metadata**
|
||||
6. Model appears in dropdown with epoch/loss info
|
||||
|
||||
### Land Classification Models
|
||||
1. Navigate to **Prediction Interface**
|
||||
2. In **Model Selection** section
|
||||
3. Click **📤 Upload Land Classification Model**
|
||||
4. Fill in metadata form (model type, accuracy, etc.)
|
||||
5. Click **✅ Upload with Metadata**
|
||||
6. Model appears in main model dropdown
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### List Models
|
||||
|
||||
**Cloud Removal:**
|
||||
```http
|
||||
GET /api/cloud-removal/models
|
||||
```
|
||||
|
||||
**Land Classification:**
|
||||
```http
|
||||
GET /api/land-classification/models
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"models": [
|
||||
{
|
||||
"filename": "model.pth",
|
||||
"epoch": 50,
|
||||
"val_loss": 0.0134,
|
||||
"size_mb": 356.2,
|
||||
"has_metadata": true,
|
||||
"created": 1706284800
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
### Delete Model
|
||||
|
||||
**Cloud Removal:**
|
||||
```http
|
||||
DELETE /api/cloud-removal/models/{filename}
|
||||
```
|
||||
|
||||
**Land Classification:**
|
||||
```http
|
||||
DELETE /api/land-classification/models/{filename}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Naming Convention**: Use descriptive names
|
||||
- ✅ `cloud_removal_unet_winter_50ep.pth`
|
||||
- ✅ `mobilenet_v2_mekong_acc93.pth`
|
||||
- ❌ `model1.pth`
|
||||
|
||||
2. **Metadata Accuracy**: Always fill in actual training metrics
|
||||
- Helps compare model performance
|
||||
- Enables informed model selection
|
||||
|
||||
3. **Version Control**: Include version/date in description
|
||||
- "v2.0 - Improved augmentation"
|
||||
- "2026-01-15 - Fixed class imbalance"
|
||||
|
||||
4. **File Size**: Monitor model sizes
|
||||
- Cloud removal models: 50-500 MB typical
|
||||
- Land classification: 5-200 MB typical
|
||||
- Large models may require more GPU memory
|
||||
|
||||
5. **Testing**: Always test uploaded model on small region first
|
||||
- Verify predictions are reasonable
|
||||
- Check for errors/crashes
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Upload Fails with "Already Exists"
|
||||
- Model filename is duplicate
|
||||
- Delete old model first or rename new one
|
||||
|
||||
### Model Shows Default Values (0, 0, 0)
|
||||
- Server needs restart to load `Form(...)` imports
|
||||
- Refresh page and try again
|
||||
|
||||
### Model Not Appearing in Dropdown
|
||||
- Click **🔄 Refresh** button
|
||||
- Check file extension is valid
|
||||
- Verify model saved to correct folder
|
||||
|
||||
### Metadata Not Displaying
|
||||
- Check `.json` file exists alongside model
|
||||
- Verify JSON format is valid
|
||||
- Look for server errors in terminal
|
||||
|
||||
---
|
||||
|
||||
## Migration from Old System
|
||||
|
||||
If you have models in `model_train/`:
|
||||
|
||||
1. **Cloud Removal Models**: Move to `cloud_removal_model/`
|
||||
```bash
|
||||
mv model_train/cloud_removal_*.pth cloud_removal_model/
|
||||
mv model_train/*_unet*.pth cloud_removal_model/
|
||||
mv model_train/*GAN*.pth cloud_removal_model/
|
||||
```
|
||||
|
||||
2. **Land Classification Models**: Move to `land_classification_model/`
|
||||
```bash
|
||||
mv model_train/mobilenet*.pth land_classification_model/
|
||||
mv model_train/cnn*.pth land_classification_model/
|
||||
mv model_train/swin*.pth land_classification_model/
|
||||
mv model_train/*.pkl land_classification_model/
|
||||
```
|
||||
|
||||
3. **Create metadata files** by re-uploading through web interface
|
||||
|
||||
---
|
||||
|
||||
## Security Features
|
||||
|
||||
✅ **File Extension Validation**: Only allowed formats accepted
|
||||
✅ **Path Traversal Prevention**: No `../` or `/` in filenames
|
||||
✅ **Duplicate Detection**: Prevents overwriting existing models
|
||||
✅ **Size Limits**: Prevents extremely large uploads
|
||||
✅ **JSON Sanitization**: Metadata stored safely
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Batch model upload
|
||||
- [ ] Model versioning system
|
||||
- [ ] Automated benchmarking
|
||||
- [ ] Model comparison tool
|
||||
- [ ] Export/import model configs
|
||||
- [ ] Cloud storage integration
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: January 26, 2026
|
||||
Reference in New Issue
Block a user