update memory issues
This commit is contained in:
Victor Phan
+325
@@ -0,0 +1,325 @@
|
||||
# 🎯 Start Here - Complete Guide
|
||||
|
||||
## Welcome! 👋
|
||||
|
||||
Tôi đã tạo một **workflow hoàn chỉnh** cho bạn. Bắt đầu từ đây!
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick 30-Second Summary
|
||||
|
||||
**Vấn đề**: Phải code trên server, kéo data lớn, train chậm
|
||||
**Giải pháp**: Code trên local, kéo data nhỏ (~300MB), train trên GPU
|
||||
|
||||
**Workflow**:
|
||||
1. Server: Tải S3 → Lưu NetCDF → Bạn download
|
||||
2. Local: Load data → Train model → Predict
|
||||
3. Local: Lưu kết quả (4 format)
|
||||
|
||||
**Total Time**: 3-7 hours (depends on GPU)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Step-by-Step Guide
|
||||
|
||||
### 📖 Step 1: Read Intro Docs (10 minutes)
|
||||
Pick ONE to start:
|
||||
- **Very quick** (5 min): `QUICKSTART_PYTORCH.md`
|
||||
- **Quick** (10 min): `PYTORCH_WORKFLOW_SUMMARY.md`
|
||||
- **Complete** (20 min): `README_PYTORCH_WORKFLOW.md`
|
||||
|
||||
### 🔧 Step 2: Setup Python (15 minutes)
|
||||
Follow: `PYTORCH_REQUIREMENTS.txt`
|
||||
|
||||
```bash
|
||||
# Create env
|
||||
python -m venv pytorch_env
|
||||
source pytorch_env/bin/activate
|
||||
|
||||
# Install PyTorch
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
|
||||
|
||||
# Install dependencies
|
||||
pip install numpy xarray netcdf4 geopandas scikit-learn matplotlib
|
||||
```
|
||||
|
||||
### 🖥️ Step 3: Server - Prepare Data (1-3 hours)
|
||||
Run notebook on SERVER:
|
||||
```
|
||||
01.prepare_data_on_server.ipynb
|
||||
```
|
||||
Output: data_for_training/ folder (~300 MB)
|
||||
|
||||
### ⬇️ Step 4: Download Data (30 minutes)
|
||||
```bash
|
||||
scp -r user@server:path/data_for_training ./
|
||||
```
|
||||
|
||||
### 💻 Step 5: Local - Train Model (30 min - 2 hours)
|
||||
Run notebook on LOCAL:
|
||||
```
|
||||
02.train_CNN_PyTorch_local.ipynb
|
||||
```
|
||||
Output: model_cnn_pytorch_full.pt
|
||||
|
||||
### 🎯 Step 6: Local - Predict (10-30 minutes)
|
||||
Run notebook on LOCAL:
|
||||
```
|
||||
03.predict_CNN_PyTorch_local.ipynb
|
||||
```
|
||||
Output: land_use_prediction.{nc, tif, png, json}
|
||||
|
||||
---
|
||||
|
||||
## 📁 What You Get
|
||||
|
||||
### After Step 3 (Server):
|
||||
```
|
||||
data_for_training/
|
||||
├── average_ndvi.nc (100-150 MB)
|
||||
├── average_vv.nc (50-100 MB)
|
||||
├── average_vh.nc (50-100 MB)
|
||||
└── train_data/ (training points)
|
||||
```
|
||||
|
||||
### After Step 5 (Local):
|
||||
```
|
||||
model_cnn_pytorch_full.pt (100 MB - trained model)
|
||||
training_history.png (training curves)
|
||||
```
|
||||
|
||||
### After Step 6 (Local):
|
||||
```
|
||||
land_use_prediction.nc (classification map - NetCDF)
|
||||
land_use_prediction.tif (classification map - GeoTIFF)
|
||||
prediction_map.png (visualization)
|
||||
prediction_metadata.json (model info + accuracy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⏱️ Time Estimates
|
||||
|
||||
| Phase | GPU | CPU |
|
||||
|-------|-----|-----|
|
||||
| Data prep (server) | 1-2h | 2-4h |
|
||||
| Training | 30-60m | 90-150m |
|
||||
| Prediction | 5-10m | 15-30m |
|
||||
| **Total** | **~2-3h** | **~4-6h** |
|
||||
|
||||
---
|
||||
|
||||
## 📖 Documentation Files
|
||||
|
||||
All documentation organized:
|
||||
|
||||
| File | Purpose | Time | When to Read |
|
||||
|------|---------|------|--------------|
|
||||
| **THIS FILE** | Overview | 5 min | Start here! |
|
||||
| `QUICKSTART_PYTORCH.md` | Quick guide | 5 min | First |
|
||||
| `PYTORCH_REQUIREMENTS.txt` | Setup | 5 min | Before starting |
|
||||
| `PYTORCH_INSTALLATION.md` | GPU setup | 10 min | If GPU issues |
|
||||
| `LOCAL_TRAINING_WORKFLOW.md` | Full workflow | 15 min | For details |
|
||||
| `README_PYTORCH_WORKFLOW.md` | Project index | 20 min | Reference |
|
||||
| `PYTORCH_WORKFLOW_SUMMARY.md` | Summary | 10 min | Overview |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Choose Your Path
|
||||
|
||||
### 🏃 I'm in a hurry (15 min read)
|
||||
1. Read: `QUICKSTART_PYTORCH.md`
|
||||
2. Read: `PYTORCH_REQUIREMENTS.txt`
|
||||
3. Start: Notebook 01 on server
|
||||
|
||||
### 🚶 I want to understand everything (1 hour read)
|
||||
1. Read: `PYTORCH_WORKFLOW_SUMMARY.md`
|
||||
2. Read: `LOCAL_TRAINING_WORKFLOW.md`
|
||||
3. Read: `README_PYTORCH_WORKFLOW.md`
|
||||
4. Start: Notebook 01 on server
|
||||
|
||||
### 🤔 I have specific questions
|
||||
- **GPU issues**: Check `PYTORCH_INSTALLATION.md`
|
||||
- **Setup issues**: Check `PYTORCH_REQUIREMENTS.txt`
|
||||
- **Workflow questions**: Check `LOCAL_TRAINING_WORKFLOW.md`
|
||||
- **Model architecture**: Check `README_PYTORCH_WORKFLOW.md`
|
||||
|
||||
---
|
||||
|
||||
## ✅ Checklist Before Starting
|
||||
|
||||
- [ ] Read at least `QUICKSTART_PYTORCH.md`
|
||||
- [ ] Python 3.8+ installed
|
||||
- [ ] Virtual environment ready
|
||||
- [ ] ~500 MB free disk space
|
||||
- [ ] Understand the 3-step workflow
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Key Concepts
|
||||
|
||||
### What is this workflow?
|
||||
|
||||
**Phase 1: Server** (1-3 hours)
|
||||
- Download satellite images from S3
|
||||
- Process: remove clouds, calculate vegetation indices
|
||||
- Save as compact NetCDF files (300 MB)
|
||||
|
||||
**Phase 2: Local Machine** (30 min - 2 hours)
|
||||
- Load NetCDF files
|
||||
- Train CNN model using PyTorch
|
||||
- Use GPU for 10-100x speedup
|
||||
|
||||
**Phase 3: Local Machine** (10-30 minutes)
|
||||
- Use trained model to classify entire region
|
||||
- Create classification map (1080×1080 pixels)
|
||||
- Export to multiple formats
|
||||
|
||||
### Why this approach?
|
||||
|
||||
| Aspect | Benefit |
|
||||
|--------|---------|
|
||||
| **Data** | Download once (~300 MB), use many times |
|
||||
| **Training** | GPU on your machine (faster + no server wait) |
|
||||
| **Development** | Code locally (easier debugging) |
|
||||
| **Flexibility** | Tune hyperparameters quickly |
|
||||
| **Privacy** | Data stays mostly local |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Start Command
|
||||
|
||||
### If on server now:
|
||||
```bash
|
||||
jupyter notebook 01.prepare_data_on_server.ipynb
|
||||
```
|
||||
|
||||
### If on local machine now:
|
||||
```bash
|
||||
# After downloading data_for_training/ folder
|
||||
jupyter notebook 02.train_CNN_PyTorch_local.ipynb
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎁 Bonus Features
|
||||
|
||||
- ✅ **GPU Auto-detection** - Automatically uses GPU if available
|
||||
- ✅ **Early Stopping** - Prevents overfitting
|
||||
- ✅ **Learning Rate Scheduling** - Automatic adjustment
|
||||
- ✅ **Visualization** - Training curves + maps
|
||||
- ✅ **Metadata** - Saves model info + accuracy
|
||||
- ✅ **Multiple Formats** - NetCDF, GeoTIFF, PNG, JSON
|
||||
|
||||
---
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Save bandwidth**: Run server notebook once, download data, use for multiple experiments
|
||||
2. **Experiment locally**: Change hyperparameters easily, retrain quickly
|
||||
3. **Batch predictions**: Handle large regions by batch processing
|
||||
4. **GPU matters**: 10-100x faster than CPU for training
|
||||
|
||||
---
|
||||
|
||||
## ❓ FAQ
|
||||
|
||||
**Q: Can I run everything on server?**
|
||||
A: Yes, but slower (CPU-only). Better to follow 3-step workflow.
|
||||
|
||||
**Q: Can I run everything local?**
|
||||
A: Notebooks 02-03 yes, notebook 01 needs server (data on S3).
|
||||
|
||||
**Q: How much data will I download?**
|
||||
A: ~300 MB for data_for_training folder.
|
||||
|
||||
**Q: Do I need GPU?**
|
||||
A: Not required, but 10-100x faster with GPU.
|
||||
|
||||
**Q: Can I use CPU only?**
|
||||
A: Yes, will take 2-3x longer.
|
||||
|
||||
**Q: What if I have NVIDIA GPU?**
|
||||
A: Install with CUDA 11.8 or 12.1 for best performance.
|
||||
|
||||
---
|
||||
|
||||
## 📞 Need Help?
|
||||
|
||||
1. **Installation issues**: → `PYTORCH_INSTALLATION.md`
|
||||
2. **Dependencies**: → `PYTORCH_REQUIREMENTS.txt`
|
||||
3. **Workflow questions**: → `LOCAL_TRAINING_WORKFLOW.md`
|
||||
4. **GPU setup**: → `PYTORCH_INSTALLATION.md`
|
||||
5. **Model details**: → `README_PYTORCH_WORKFLOW.md`
|
||||
|
||||
---
|
||||
|
||||
## 🎉 Ready to Start?
|
||||
|
||||
Pick ONE action now:
|
||||
|
||||
### Option A: Fast Track (5 min)
|
||||
```
|
||||
1. Read: QUICKSTART_PYTORCH.md
|
||||
2. Go to: Step 3 (run notebook 01)
|
||||
```
|
||||
|
||||
### Option B: Full Understanding (1 hour)
|
||||
```
|
||||
1. Read: PYTORCH_WORKFLOW_SUMMARY.md
|
||||
2. Read: LOCAL_TRAINING_WORKFLOW.md
|
||||
3. Read: README_PYTORCH_WORKFLOW.md
|
||||
4. Setup: PYTORCH_REQUIREMENTS.txt
|
||||
5. Go to: Step 3 (run notebook 01)
|
||||
```
|
||||
|
||||
### Option C: Hands-On Learning (1-2 hours)
|
||||
```
|
||||
1. Setup environment
|
||||
2. Run all 3 notebooks in order
|
||||
3. Read docs as you go
|
||||
4. Experiment with hyperparameters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Action
|
||||
|
||||
**You are here!** ← You've read this file
|
||||
|
||||
**Next**: Choose your path above and pick ONE file to read next
|
||||
|
||||
**Then**: Follow the steps
|
||||
|
||||
**Finally**: Enjoy your trained CNN model! 🎉
|
||||
|
||||
---
|
||||
|
||||
## 📊 Success Looks Like
|
||||
|
||||
After completing all steps:
|
||||
- ✅ Trained CNN model on local machine
|
||||
- ✅ Model accuracy: 75-85%
|
||||
- ✅ Classification map: 1080×1080 pixels
|
||||
- ✅ Results exported in 4 formats
|
||||
- ✅ Everything completed locally (no server wait)
|
||||
|
||||
---
|
||||
|
||||
**Status**: Ready to Go 🟢
|
||||
**Version**: 1.0
|
||||
**Last Updated**: November 2025
|
||||
|
||||
---
|
||||
|
||||
## 🎯 One Last Thing
|
||||
|
||||
The best part? **No more uploading code to server!** 🎉
|
||||
|
||||
- 💻 Code on your machine
|
||||
- 📊 Use server data
|
||||
- ⚡ Train with GPU
|
||||
- 🎨 Visualize instantly
|
||||
- 🚀 Iterate quickly
|
||||
|
||||
Enjoy! 🚀
|
||||
Reference in New Issue
Block a user