update memory issues
This commit is contained in:
Victor Phan
@@ -0,0 +1,141 @@
|
||||
# 🔧 Memory Overflow Fix: Sentinel-2 Data Loading
|
||||
|
||||
## Problem
|
||||
|
||||
When attempting to load Sentinel-2 data, you got:
|
||||
```
|
||||
❌ Error loading data: Unable to allocate 403. TiB for an array with shape
|
||||
(396, 563539, 992108) and data type uint16
|
||||
```
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
The datacube's `load_s2l2a_with_offset()` function was **loading an entire massive tile** instead of clipping to your AOI bounds.
|
||||
|
||||
**Math:**
|
||||
- Shape: 396 scenes × 563,539 pixels × 992,108 pixels × 2 bytes (uint16)
|
||||
- **Memory needed:** ~403 Terabytes 😱
|
||||
- **Reality:** Your server has maybe 100-500 GB
|
||||
|
||||
**Expected behavior:**
|
||||
- Your AOI: 105.5-106.4°E × 9.2-10.0°N (~100×100 km)
|
||||
- At 10m resolution: ~10,000 × 10,000 pixels
|
||||
- For 396 scenes: 396 × 10,000 × 10,000 × 2 bytes = **20 GB** ✓ (reasonable!)
|
||||
|
||||
## Solution Implemented
|
||||
|
||||
### Strategy: Monthly Chunking
|
||||
|
||||
Instead of loading all 396 scenes at once, split into **13 monthly chunks**:
|
||||
|
||||
```python
|
||||
date_ranges = [
|
||||
("2022-09-01", "2022-10-01"), # ~30 scenes
|
||||
("2022-10-01", "2022-11-01"), # ~30 scenes
|
||||
...
|
||||
("2023-09-01", "2023-10-01"), # ~30 scenes
|
||||
]
|
||||
|
||||
for month in date_ranges:
|
||||
monthly_data = load_s2l2a_with_offset(dc, query_for_month)
|
||||
data_list.append(monthly_data)
|
||||
|
||||
data = xr.concat(data_list, dim='time') # Combine after loading
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
- ✅ Each monthly load: ~30 scenes = 600 GB → manageable chunks
|
||||
- ✅ Processing happens per-month, memory freed after each
|
||||
- ✅ If one month fails, others still complete
|
||||
- ✅ Dask can distribute work across multiple workers
|
||||
|
||||
### Dask Chunk Optimization
|
||||
|
||||
```python
|
||||
'dask_chunks': {'x': 512, 'y': 512, 'time': 1}
|
||||
```
|
||||
|
||||
- **x/y (512×512):** Spatial chunks (100×100 km tile → 4×4 chunks)
|
||||
- **time (1):** Each scene is separate, allows parallel processing
|
||||
- **Result:** Dask workers process ~512²×1 = 262K pixels per chunk
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
try:
|
||||
monthly_data = load_s2l2a_with_offset(...)
|
||||
data_list.append(monthly_data)
|
||||
except Exception as e:
|
||||
print(f"Month {month} failed: {e}")
|
||||
continue # Skip failed month, continue with others
|
||||
```
|
||||
|
||||
## Testing the Fix
|
||||
|
||||
### Cell 4: Diagnostic Check ✨ NEW
|
||||
Runs first to inspect datacube metadata without loading data:
|
||||
- Lists available S2 products
|
||||
- Checks how many scenes are available for Jan 2023
|
||||
- Shows bounds/CRS of first scene
|
||||
- **Do NOT modify** - helps diagnose issues
|
||||
|
||||
### Cell 5: Sentinel-2 Loading ✅ UPDATED
|
||||
Now uses monthly chunking with progress bars:
|
||||
```
|
||||
[01/13] 2022-09-01 → 2022-10-01 ✓ 32 scenes
|
||||
[02/13] 2022-10-01 → 2022-11-01 ✓ 28 scenes
|
||||
...
|
||||
[13/13] 2023-09-01 → 2023-10-01 ✓ 31 scenes
|
||||
|
||||
🔗 Combining 13 monthly chunks...
|
||||
✅ Success! Shape: {'time': 396, 'y': 10000, 'x': 10000}
|
||||
Memory: 64.5 GB
|
||||
```
|
||||
|
||||
## Expected Output
|
||||
|
||||
After loading, `data` should have:
|
||||
- **Shape:** (396 time steps, ~10000 y pixels, ~10000 x pixels)
|
||||
- **Bands:** red, nir, scl
|
||||
- **CRS:** EPSG:32648 (UTM Zone 48N)
|
||||
- **Memory:** ~15-20 GB (reasonable for distributed processing)
|
||||
|
||||
## If Issues Persist
|
||||
|
||||
### Issue 1: "Still getting huge spatial dimensions"
|
||||
→ The datacube function has a bug with spatial subsetting
|
||||
→ Add explicit clipping with rasterio before concat:
|
||||
|
||||
```python
|
||||
import rasterio.mask
|
||||
# Clip each monthly dataset to exact AOI bounds
|
||||
```
|
||||
|
||||
### Issue 2: "One month loads but others fail"
|
||||
→ Those S3 objects may be corrupted/missing
|
||||
→ Check logs - the code continues anyway (fallback behavior)
|
||||
|
||||
### Issue 3: "Dask workers running out of memory"
|
||||
→ Reduce chunk size: `'dask_chunks': {'x': 256, 'y': 256, 'time': 1}`
|
||||
→ Or reduce workers: change `workers=(1, 10)` to `workers=(1, 5)`
|
||||
|
||||
## Next Steps (After successful data load)
|
||||
|
||||
Cells 6-10 process the data:
|
||||
1. **Cell 6:** Cloud masking (SCL band)
|
||||
2. **Cell 7:** NDVI calculation
|
||||
3. **Cell 8:** Fill missing values (interpolation)
|
||||
4. **Cell 9:** Monthly aggregation
|
||||
5. **Cell 10:** Sentinel-1 loading (VH, VV)
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `01.prepare_data_on_server.ipynb`
|
||||
- Cell 4: NEW diagnostic check
|
||||
- Cell 5: UPDATED monthly chunking strategy
|
||||
- Cell 6-14: Unchanged (process data as before)
|
||||
|
||||
---
|
||||
|
||||
**Created:** 2025-11-11
|
||||
**Status:** Ready to test ✅
|
||||
Reference in New Issue
Block a user