basketballcantho/remote-sensing
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Files
new-feature
remote-sensing/START_HERE.md
T
Victor Phan e3777dd91c update memory issues
2025-11-11 15:27:54 +07:00

7.8 KiB
Raw Permalink Blame History

🎯 Start Here - Complete Guide

Welcome! 👋

Tôi đã tạo một workflow hoàn chỉnh cho bạn. Bắt đầu từ đây!

🚀 Quick 30-Second Summary

Vấn đề: Phải code trên server, kéo data lớn, train chậm
Giải pháp: Code trên local, kéo data nhỏ (~300MB), train trên GPU

Workflow:

  1. Server: Tải S3 → Lưu NetCDF → Bạn download
  2. Local: Load data → Train model → Predict
  3. Local: Lưu kết quả (4 format)

Total Time: 3-7 hours (depends on GPU)

📚 Step-by-Step Guide

📖 Step 1: Read Intro Docs (10 minutes)

Pick ONE to start:

  • Very quick (5 min): QUICKSTART_PYTORCH.md
  • Quick (10 min): PYTORCH_WORKFLOW_SUMMARY.md
  • Complete (20 min): README_PYTORCH_WORKFLOW.md

🔧 Step 2: Setup Python (15 minutes)

Follow: PYTORCH_REQUIREMENTS.txt

# Create env
python -m venv pytorch_env
source pytorch_env/bin/activate

# Install PyTorch
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install dependencies
pip install numpy xarray netcdf4 geopandas scikit-learn matplotlib

🖥️ Step 3: Server - Prepare Data (1-3 hours)

Run notebook on SERVER:

01.prepare_data_on_server.ipynb

Output: data_for_training/ folder (~300 MB)

⬇️ Step 4: Download Data (30 minutes)

scp -r user@server:path/data_for_training ./

💻 Step 5: Local - Train Model (30 min - 2 hours)

Run notebook on LOCAL:

02.train_CNN_PyTorch_local.ipynb

Output: model_cnn_pytorch_full.pt

🎯 Step 6: Local - Predict (10-30 minutes)

Run notebook on LOCAL:

03.predict_CNN_PyTorch_local.ipynb

Output: land_use_prediction.{nc, tif, png, json}

📁 What You Get

After Step 3 (Server):

data_for_training/
├── average_ndvi.nc      (100-150 MB)
├── average_vv.nc        (50-100 MB)
├── average_vh.nc        (50-100 MB)
└── train_data/          (training points)

After Step 5 (Local):

model_cnn_pytorch_full.pt      (100 MB - trained model)
training_history.png            (training curves)

After Step 6 (Local):

land_use_prediction.nc          (classification map - NetCDF)
land_use_prediction.tif         (classification map - GeoTIFF)
prediction_map.png              (visualization)
prediction_metadata.json        (model info + accuracy)

⏱️ Time Estimates

Phase GPU CPU
Data prep (server) 1-2h 2-4h
Training 30-60m 90-150m
Prediction 5-10m 15-30m
Total ~2-3h ~4-6h

📖 Documentation Files

All documentation organized:

File Purpose Time When to Read
THIS FILE Overview 5 min Start here!
QUICKSTART_PYTORCH.md Quick guide 5 min First
PYTORCH_REQUIREMENTS.txt Setup 5 min Before starting
PYTORCH_INSTALLATION.md GPU setup 10 min If GPU issues
LOCAL_TRAINING_WORKFLOW.md Full workflow 15 min For details
README_PYTORCH_WORKFLOW.md Project index 20 min Reference
PYTORCH_WORKFLOW_SUMMARY.md Summary 10 min Overview

🎯 Choose Your Path

🏃 I'm in a hurry (15 min read)

  1. Read: QUICKSTART_PYTORCH.md
  2. Read: PYTORCH_REQUIREMENTS.txt
  3. Start: Notebook 01 on server

🚶 I want to understand everything (1 hour read)

  1. Read: PYTORCH_WORKFLOW_SUMMARY.md
  2. Read: LOCAL_TRAINING_WORKFLOW.md
  3. Read: README_PYTORCH_WORKFLOW.md
  4. Start: Notebook 01 on server

🤔 I have specific questions

  • GPU issues: Check PYTORCH_INSTALLATION.md
  • Setup issues: Check PYTORCH_REQUIREMENTS.txt
  • Workflow questions: Check LOCAL_TRAINING_WORKFLOW.md
  • Model architecture: Check README_PYTORCH_WORKFLOW.md

✅ Checklist Before Starting

  • Read at least QUICKSTART_PYTORCH.md
  • Python 3.8+ installed
  • Virtual environment ready
  • ~500 MB free disk space
  • Understand the 3-step workflow

🎓 Key Concepts

What is this workflow?

Phase 1: Server (1-3 hours)

  • Download satellite images from S3
  • Process: remove clouds, calculate vegetation indices
  • Save as compact NetCDF files (300 MB)

Phase 2: Local Machine (30 min - 2 hours)

  • Load NetCDF files
  • Train CNN model using PyTorch
  • Use GPU for 10-100x speedup

Phase 3: Local Machine (10-30 minutes)

  • Use trained model to classify entire region
  • Create classification map (1080×1080 pixels)
  • Export to multiple formats

Why this approach?

Aspect Benefit
Data Download once (~300 MB), use many times
Training GPU on your machine (faster + no server wait)
Development Code locally (easier debugging)
Flexibility Tune hyperparameters quickly
Privacy Data stays mostly local

🚀 Start Command

If on server now:

jupyter notebook 01.prepare_data_on_server.ipynb

If on local machine now:

# After downloading data_for_training/ folder
jupyter notebook 02.train_CNN_PyTorch_local.ipynb

🎁 Bonus Features

  • ✅ GPU Auto-detection - Automatically uses GPU if available
  • ✅ Early Stopping - Prevents overfitting
  • ✅ Learning Rate Scheduling - Automatic adjustment
  • ✅ Visualization - Training curves + maps
  • ✅ Metadata - Saves model info + accuracy
  • ✅ Multiple Formats - NetCDF, GeoTIFF, PNG, JSON

💡 Pro Tips

  1. Save bandwidth: Run server notebook once, download data, use for multiple experiments
  2. Experiment locally: Change hyperparameters easily, retrain quickly
  3. Batch predictions: Handle large regions by batch processing
  4. GPU matters: 10-100x faster than CPU for training

❓ FAQ

Q: Can I run everything on server?
A: Yes, but slower (CPU-only). Better to follow 3-step workflow.

Q: Can I run everything local?
A: Notebooks 02-03 yes, notebook 01 needs server (data on S3).

Q: How much data will I download?
A: ~300 MB for data_for_training folder.

Q: Do I need GPU?
A: Not required, but 10-100x faster with GPU.

Q: Can I use CPU only?
A: Yes, will take 2-3x longer.

Q: What if I have NVIDIA GPU?
A: Install with CUDA 11.8 or 12.1 for best performance.

📞 Need Help?

  1. Installation issues: → PYTORCH_INSTALLATION.md
  2. Dependencies: → PYTORCH_REQUIREMENTS.txt
  3. Workflow questions: → LOCAL_TRAINING_WORKFLOW.md
  4. GPU setup: → PYTORCH_INSTALLATION.md
  5. Model details: → README_PYTORCH_WORKFLOW.md

🎉 Ready to Start?

Pick ONE action now:

Option A: Fast Track (5 min)

1. Read: QUICKSTART_PYTORCH.md
2. Go to: Step 3 (run notebook 01)

Option B: Full Understanding (1 hour)

1. Read: PYTORCH_WORKFLOW_SUMMARY.md
2. Read: LOCAL_TRAINING_WORKFLOW.md
3. Read: README_PYTORCH_WORKFLOW.md
4. Setup: PYTORCH_REQUIREMENTS.txt
5. Go to: Step 3 (run notebook 01)

Option C: Hands-On Learning (1-2 hours)

1. Setup environment
2. Run all 3 notebooks in order
3. Read docs as you go
4. Experiment with hyperparameters

🚀 Next Action

You are here! ← You've read this file

Next: Choose your path above and pick ONE file to read next

Then: Follow the steps

Finally: Enjoy your trained CNN model! 🎉

📊 Success Looks Like

After completing all steps:

  • ✅ Trained CNN model on local machine
  • ✅ Model accuracy: 75-85%
  • ✅ Classification map: 1080×1080 pixels
  • ✅ Results exported in 4 formats
  • ✅ Everything completed locally (no server wait)

Status: Ready to Go 🟢
Version: 1.0
Last Updated: November 2025

🎯 One Last Thing

The best part? No more uploading code to server! 🎉

  • 💻 Code on your machine
  • 📊 Use server data
  • ⚡ Train with GPU
  • 🎨 Visualize instantly
  • 🚀 Iterate quickly

Enjoy! 🚀

Reference in New Issue View Git Blame Copy Permalink