remote-sensing/MEMORY_FIX_EXPLAINED.md

🔧 Memory Overflow Fix: Sentinel-2 Data Loading

Problem

When attempting to load Sentinel-2 data, you got:

❌ Error loading data: Unable to allocate 403. TiB for an array with shape 
(396, 563539, 992108) and data type uint16

Root Cause Analysis

The datacube's load_s2l2a_with_offset() function was loading an entire massive tile instead of clipping to your AOI bounds.

Math:

• Shape: 396 scenes × 563,539 pixels × 992,108 pixels × 2 bytes (uint16)
• Memory needed: ~403 Terabytes 😱
• Reality: Your server has maybe 100-500 GB

Expected behavior:

• Your AOI: 105.5-106.4°E × 9.2-10.0°N (~100×100 km)
• At 10m resolution: ~10,000 × 10,000 pixels
• For 396 scenes: 396 × 10,000 × 10,000 × 2 bytes = 20 GB ✓ (reasonable!)

Solution Implemented

Strategy: Monthly Chunking

Instead of loading all 396 scenes at once, split into 13 monthly chunks:

date_ranges = [
    ("2022-09-01", "2022-10-01"),   # ~30 scenes
    ("2022-10-01", "2022-11-01"),   # ~30 scenes
    ...
    ("2023-09-01", "2023-10-01"),   # ~30 scenes
]

for month in date_ranges:
    monthly_data = load_s2l2a_with_offset(dc, query_for_month)
    data_list.append(monthly_data)

data = xr.concat(data_list, dim='time')  # Combine after loading

Advantages:

• ✅ Each monthly load: ~30 scenes = 600 GB → manageable chunks
• ✅ Processing happens per-month, memory freed after each
• ✅ If one month fails, others still complete
• ✅ Dask can distribute work across multiple workers

Dask Chunk Optimization

'dask_chunks': {'x': 512, 'y': 512, 'time': 1}

• x/y (512×512): Spatial chunks (100×100 km tile → 4×4 chunks)
• time (1): Each scene is separate, allows parallel processing
• Result: Dask workers process ~512²×1 = 262K pixels per chunk

Error Handling

try:
    monthly_data = load_s2l2a_with_offset(...)
    data_list.append(monthly_data)
except Exception as e:
    print(f"Month {month} failed: {e}")
    continue  # Skip failed month, continue with others

Testing the Fix

Cell 4: Diagnostic Check ✨ NEW

Runs first to inspect datacube metadata without loading data:

• Lists available S2 products
• Checks how many scenes are available for Jan 2023
• Shows bounds/CRS of first scene
• Do NOT modify - helps diagnose issues

Cell 5: Sentinel-2 Loading ✅ UPDATED

Now uses monthly chunking with progress bars:

[01/13] 2022-09-01 → 2022-10-01  ✓ 32 scenes
[02/13] 2022-10-01 → 2022-11-01  ✓ 28 scenes
...
[13/13] 2023-09-01 → 2023-10-01  ✓ 31 scenes

🔗 Combining 13 monthly chunks...
✅ Success! Shape: {'time': 396, 'y': 10000, 'x': 10000}
   Memory: 64.5 GB

Expected Output

After loading, data should have:

• Shape: (396 time steps, ~10000 y pixels, ~10000 x pixels)
• Bands: red, nir, scl
• CRS: EPSG:32648 (UTM Zone 48N)
• Memory: ~15-20 GB (reasonable for distributed processing)

If Issues Persist

Issue 1: "Still getting huge spatial dimensions"

→ The datacube function has a bug with spatial subsetting → Add explicit clipping with rasterio before concat:

import rasterio.mask
# Clip each monthly dataset to exact AOI bounds

Issue 2: "One month loads but others fail"

→ Those S3 objects may be corrupted/missing → Check logs - the code continues anyway (fallback behavior)

Issue 3: "Dask workers running out of memory"

→ Reduce chunk size: 'dask_chunks': {'x': 256, 'y': 256, 'time': 1} → Or reduce workers: change workers=(1, 10) to workers=(1, 5)

Next Steps (After successful data load)

Cells 6-10 process the data:

1. Cell 6: Cloud masking (SCL band)
2. Cell 7: NDVI calculation
3. Cell 8: Fill missing values (interpolation)
4. Cell 9: Monthly aggregation
5. Cell 10: Sentinel-1 loading (VH, VV)

Files Modified

• 01.prepare_data_on_server.ipynb
  • Cell 4: NEW diagnostic check
  • Cell 5: UPDATED monthly chunking strategy
  • Cell 6-14: Unchanged (process data as before)

---

Created: 2025-11-11
Status: Ready to test ✅