Large regions and resuming

mapcv is built so that a city-sized job behaves like a small one: it estimates first, works in chunks, saves progress as it goes, and resumes with the same command.

1. Plan first

mapcv plan costs nothing and tells you whether the job is reasonable:

$ mapcv plan big.yaml
╭─ Plan for big.yaml ──────────────────────────────────────────────────────────────────────────────╮
│ Region   4.7, 52.25 → 5.05, 52.45  (≈ 23.8 × 22.1 km)                                            │
│ Imagery  esri_satellite · zoom 19 (≈ 0.18 m/px)                                                  │
│ Raster   130,816 × 122,368 px · 244,258 tiles (≈ 6.1 GB to download)                             │
│ Labels   none (image-only dataset)                                                               │
│ Patches  ≈ 61,184 × 512 px (grid, stride 512)                                                    │
│ Output   big · png (≈ 26.5 GB)                                                                   │
│ Memory   ≈ 1.6 GB per chunk                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

Things to check:

Tiles and download size. A quarter of a million tiles takes a while and may run into the provider’s rate limits or quota. One zoom level lower means four times fewer tiles.
Output size. Estimates use typical compression; real PNGs of dense urban imagery can be larger.
Memory per chunk. See below.

mapcv generate --dry-run prints the same plan and exits.

Large-job confirmation

If a job has more than 20,000 tiles, more than 5 GB of estimated download plus output, or more than 1,000,000 patches, mapcv generate asks before starting:

This is a large job. Start it? [y/n] (n):

In a script or CI (no terminal), it refuses with exit code 2 instead:

This is a large job. Re-run with --yes to confirm.

A job with more than 3,000,000 patches is not a large job but a refused one: every patch position is held in memory during a run, so generate stops before it starts, and plan lists the same problem under its warnings (and ends with exit code 1, without the Looks right? line):

This config cannot be built: the grid has 8,960 x 5,888 = 52,756,480 patches, more than the limit of 3,000,000 for one run. Raise sampler.stride (or patch_size), shrink the region or split it into several smaller ones.

A stride of 1 on a region of a few square kilometres gives tens of millions of positions; there is one position per stride in each direction, so doubling the stride divides the count by four. plan itself counts patches per axis and never builds them, so it answers at once for any region.

Pass --yes (-y) once you have read the plan. In a log, progress is a plain line per tenth of the chunks (Reading imagery and writing patches: 40/400 chunks (10%), 3m 12s) instead of a bar; add -q to drop it too: mapcv -q generate big.yaml --yes.

2. Keep memory bounded

mapcv never holds the whole region in memory. Patch positions are computed once for the whole raster, then the raster is processed in chunks of rows, each spanning the full raster width:

Source	Chunk height	Set with
XYZ	`strip_rows` × 256 px (default 4 → 1,024 px)	`imagery.strip_rows`
Sentinel-2	`chunk_rows` px (default 1,024)	`imagery.chunk_rows`
GeoTIFF	`chunk_rows` px (default 1,024)	`imagery.chunk_rows`

Each chunk reads a window one patch taller than the chunk (so patches that cross a chunk boundary are cut once, whole), rasterizes the labels for that window and writes the patches; the manifest is saved every 10 seconds and when the run stops. XYZ tiles are fetched per chunk and dropped once the window has moved past them.

Peak memory therefore grows with the width of the region and the chunk height, not with its area. A chunk whose window would pass 256 MB (a 10,980-column, 12-band float32 Sentinel-2 chunk is 540 MB) is read as several column windows instead, so memory stops growing with the width there; the patches are the same. The plan’s Memory line estimates it. If it is too high:

lower imagery.strip_rows (minimum 1, i.e. 256 px) or imagery.chunk_rows, or
split a very wide region into several configs, each with its own writer.staging_dir.

3. Survive failures

Flaky tile servers. Every tile gets up to four attempts with backoff (respecting Retry-After). What happens to a tile that still fails depends on imagery.policy:

lenient (default) leaves it black and carries on, unless more than max_failed_ratio (5 %) of a chunk’s tiles fail. Then the run stops:

Generation failed: Too many failed tiles: 2/2 (100.0% exceeds 5.0% threshold): 2 x HTTP 503 (e.g.
HTTP 503 for URL: https://tiles.example.com/…/18/134567/86123). If the provider is busy or
rate-limiting, try again later or lower imagery.max_connections; raise imagery.max_failed_ratio to
accept gaps.
Fix the cause and run the same command again: finished chunks are kept and the run resumes.

strict stops on the first failed tile, naming its URL (without query string).
ignore fills failed tiles with black and never stops. Combine it with sampler.max_empty_ratio to drop the resulting black patches, and watch the Tiles … failed count in the summary. If more than half of the tiles fail (under any policy that lets the run finish), mapcv warns:
```
⚠  2 of 2 tiles failed; check the tile URL, your network and imagery.policy (failed tiles are
left empty or black).
```

Sentinel-2 read errors. A remote read that times out is reported as Generation failed: … with the server’s message, and a band that comes back empty while the other bands have data fails the chunk instead of writing patches full of NaN. In both cases, run the same command again to continue from that chunk.

Interruptions. Ctrl-C, an error or a lost connection lose at most the chunk in progress: the manifest is written atomically when the run stops (and every 10 seconds while it runs), and any files left over from the unfinished chunk are overwritten on resume. A run killed outright (kill -9, a power cut) loses at most the last 10 seconds of chunks.

Until a run has finished, splits and annotation files included, the manifest records "complete": false. mapcv info shows such a dataset as incomplete and mapcv verify fails on it, so a copy of a half-made dataset is not mistaken for the whole one; the same mapcv generate command finishes it.

One run per folder. generate and split lock the dataset folder (.mapcv.lock, an operating-system lock that ends with the process, however it ends). A second run into the same folder stops at once with another mapcv command is writing to dataset. generate also refuses a folder that holds files of its own and no manifest.json, so none of them is overwritten or removed: generate into a new or empty folder. Hidden entries (.git, .gitkeep, .gitattributes, …) and a top-level README.md do not count, so a cloned Hugging Face dataset repository or a freshly initialised version-control folder is fine.

4. Resume

Run the same command again. Under the plan, mapcv prints Resuming: <n> patches already written, <m> to go. and continues.

If everything is already written, mapcv says so and only refreshes the splits and the summary:

Nothing left to do: all 77 patches are already written.

mapcv loads manifest.json from writer.staging_dir, skips every patch already recorded, and only fetches imagery for chunks that still have patches to write. Splits are recomputed at the end over all patches. A patch the manifest lists whose file has gone missing or is empty (a power cut can leave empty files) is written again, with the rest of its chunk and everything after it, so the files match an uninterrupted run.

XYZ tiles that were already downloaded come from the on-disk tile cache, also when you start over in a new staging_dir with other settings; the summary says how many (Tiles 0 fetched · 96 from the cache · 0 failed). mapcv cache shows how much space it takes and mapcv cache --clear --expired frees the expired part.

What must stay the same

A resume is refused, before any imagery is read or written, if the existing manifest was produced with a different:

Recorded field (named in the error)	Changes when you edit…
`task`	`task`
`target type`, `mask dtype`	`task`, `labels.type`
`task options`	the `detection`, `instance`, `classification` or `change` block
`class_map`	`labels.label_field`, `labels.classes`, or the set of label values
`ignore_index`	`labels.ignore_index`
`labels`	any other `labels` setting, the contents of the label file (a SHA-256 hash is recorded), or adding or removing `labels`
`source_type`, `product_id`, `bands`, `dtype`	the imagery type, `source`, the product, or `bands`
`imagery file, URL template or read settings`	the imagery file itself (a GeoTIFF’s size and a hash of its first and last 64 KiB, or a remote file’s `ETag`, size and first 64 KiB; every file of a mosaic), a custom `url_template` (another layer, year, style or key on the same host), `overview`, `nodata`, the STAC item, `scl_mask`, or the Earth Engine image and rendering
`imagery file modification time`	a local GeoTIFF’s modification time: the file was edited in place (which can keep its size and both ends), re-exported, or copied without its times. Copy with `cp -p` or `rsync -a` to keep them. When the content is unchanged, the message names the recorded time and the command that restores it (`touch -d '2026-10-08T09:14:03.123456789Z' file.tif` on Linux and macOS, a `LastWriteTimeUtc` assignment on Windows); run it and generate again. For a mosaic, only the files whose time changed are named
`sources`	the names of the imagery sources, or their number
`patch_shape`	`sampler.patch_size`, `bands`, `image_format` between NPY/TIF and PNG/JPG
`crs`, `transform`	`region` (if it snaps to a different tile/pixel grid), `zoom`, `resolution`
`region`	the region’s extent (the raster size it covers, also on the same grid), the polygons of `region.path` (a hash of the file is recorded), `region.name_field` or `region.layer`
`sampler`	any `sampler` field
`writer`	`image_format` (including `png` ↔ `jpg`), `mask_format`, `world_files`, `stack_sources`, `jpg_quality` or `jpg_subsampling` (`footprints` may change)

In the manifest these are task, the target block, the sources block, region, sampler and writer. With several sources, a source’s fields are named after it (before: dtype).

Moving the label file is fine as long as its contents stay the same. A refused resume looks like this, here after a single byte was added to the label file:

Cannot resume: dataset/manifest.json was generated with a different configuration
(labels); use a new writer.staging_dir or remove the old dataset

Re-running does not help: point writer.staging_dir at a new folder, or undo the change.

Settings that you can change between runs: split (it is recomputed), max_connections, policy, max_failed_ratio, and strip_rows/chunk_rows.

Datasets from mapcv 0.1

0.1 manifests (version 1) used per-strip row numbers, so they cannot be resumed:

dataset/manifest.json is a version-1 manifest from mapcv 0.1.x and cannot be resumed;
generate into a new writer.staging_dir ('mapcv split' still reads it)

mapcv split and mapcv info still work on them. See Migrating.

Datasets from mapcv 0.2

mapcv 0.3 resumes a 0.2 dataset (manifest version 2) when its settings match, with labels.ignore_index: null in the config: 0.2 wrote background where there is no imagery, and the rest of the dataset must do the same. Without it the resume is refused with a message that says so. A finished 0.2 dataset is left as it is; one that gains patches is saved as manifest version 3. 0.2 datasets made with sampler.mode: random cannot be resumed, because 0.3 draws random positions differently. See Migrating to 0.3.

In the other direction, a manifest written by a newer mapcv (a higher manifest version) is refused with “manifest version N was written by a newer mapcv; upgrade mapcv to read it”.

Checklist for a big run

Generate a tiny version first (a few hundred metres) and look at the patches.
mapcv plan the full config; check tiles, output size, memory and warnings.
Run mapcv generate --yes under nohup, tmux or a job scheduler.
If it stops, fix the cause and run the same command again.
mapcv info the result.