Traditional Linear Approach
Global ERA5 NetCDF File
(e.g., 50GB+ per archive)
(e.g., 50GB+ per archive)
↓
xarray.open_dataset()
Attempts to map entire global grid
Attempts to map entire global grid
↓
RAM Limit Exceeded
Out Of Memory (OOM) Killed
Out Of Memory (OOM) Killed
regional-era5-chunks
Define Bounding Box & Time
Pre-calculate regional indices
Pre-calculate regional indices
↓
Worker 1
(Day 1-5)
(Day 1-5)
Worker 2
(Day 6-10)
(Day 6-10)
Worker N
...
...
↓
Save as Regional Chunks
Zero RAM Overload (O(1) Memory)
Zero RAM Overload (O(1) Memory)
Memory Profiling