Migrating

Note

The sections below reproduce MIGRATION.md from the repository. If the two ever differ, MIGRATION.md is authoritative.

Migrating to 0.3

1. Update code that reads manifest.json directly to manifest version 3: files, summary, sources[0] and target. Code that uses mapcv.Manifest.load keeps working for old and new datasets.
2. To resume a dataset made by 0.2, add labels.ignore_index: null to its config, and for JPEG patches also writer.jpg_subsampling: "4:4:4" (why).
3. Nothing to do for task: it defaults to segmentation.

Manifest version 3

New datasets are written with manifest version 3. It records what the dataset is for (task), the imagery as a list of sources, the annotation as a target, and for each patch the paths of its files and a summary of its annotation, so that every task (segmentation, detection, instance, classification, change, regression) and output layout fits one format. The folder layout and the split lists’ format are unchanged. What changed in images and masks is listed under Datasets made by mapcv 0.2 and Pixels without imagery.

Version 2 (mapcv 0.2)	Version 3 (mapcv 0.3)
patches[i].filename	patches[i].files.image, a path relative to the dataset folder: "Images/patch_0000000.png"
patches[i].mask_filename	patches[i].files.mask ("Masks/patch_0000000.png"); absent without labels instead of null
patches[i].per_class_pixel_counts	patches[i].summary.class_pixels (absent without labels)
patches[i].empty_ratio	patches[i].summary.empty_ratio
patches[i].strip_index	patches[i].chunk
class_map	target.class_map
labels (settings and sha256)	target.labels; labels.ignore_index is now target.ignore_index
source_type, product_id, bands, dtype, crs, transform, patch_shape	the same keys in sources[0] (named "image")
writer	writer, plus layout: files, mask_format: png and jpg_subsampling
(none)	task, mapcv_version, complete, target.type, target.dtype, target.options, and in each source width, height and fingerprint

An image-only dataset has "target": null. Split lists are unchanged: one image file name per line. Each patch is one line of manifest.json, which makes large manifests about a third smaller. The full schema is in the dataset format reference.

Training code that reads manifest.json directly needs the new keys:

import json
from pathlib import Path

root = Path("dataset")
manifest = json.loads((root / "manifest.json").read_text())
target = manifest["target"] or {}  # null for an image-only dataset
class_map = target.get("class_map", {})  # was: manifest["class_map"]
ignore_index = target.get("ignore_index")  # was: manifest["labels"]["ignore_index"]
bands = manifest["sources"][0]["bands"]  # was: manifest["bands"]
for patch in manifest["patches"]:
    image = root / patch["files"]["image"]  # was: root / "Images" / patch["filename"]
    mask = root / patch["files"]["mask"]  # was: root / "Masks" / patch["mask_filename"]
    counts = patch["summary"]["class_pixels"]  # was: patch["per_class_pixel_counts"]

mapcv.Manifest.load(path) reads versions 1, 2 and 3 and returns version 3 without touching the file. To convert an existing manifest on disk (for example so the new loader above can read a 0.2 dataset), run mapcv.Manifest.load(path).save(path). 0.2 code cannot read a version-3 manifest.

Python API changes: Manifest has task, sources, target (SourceRecord, TargetRecord), writer, sampler and patches, with the read-only shortcuts manifest.source, manifest.class_map and manifest.ignore_index, and patch_transform(entry) / patch_bounds(entry) for georeferencing. ManifestEntry has row, col, padded, chunk, files and summary. load_or_create_manifest(path, expected) takes the expected Manifest instead of one keyword per field, write_patches(..., chunk_index=0) replaces strip_index, and the manifest classes live in mapcv.manifest (still importable from mapcv and mapcv.writer). For custom targets, Target.fingerprint() became Target.record(), which returns a TargetRecord, and targets and writers have a type and layout with Writer.supports(target_type).

Datasets made by mapcv 0.2

mapcv info, mapcv split and Manifest.load read 0.2 datasets as before and never rewrite their manifest.

mapcv generate resumes a 0.2 dataset when the imagery, labels, sampler and writer settings match, with one addition: set labels.ignore_index: null. mapcv 0.2 wrote background (0) where there was no imagery; 0.3 writes labels.ignore_index (default 255) there, and a resumed run must keep the dataset consistent. Without it, generation stops with a message that says so. A finished 0.2 dataset resumes as a no-op and is left as it is; one that gains patches is saved as version 3. New masks come from 0.3’s rasterizer, which follows GDAL’s rules exactly, so pixels on polygon edges can differ slightly from masks 0.2 wrote.

JPEG patches are encoded with 4:2:0 chroma subsampling by default (writer.jpg_subsampling: "4:2:0"), as Pillow and libjpeg do, which makes them up to half the size of 0.2’s 4:4:4 patches at the same quality. A 0.2 dataset written as JPEG resumes only with writer.jpg_subsampling: "4:4:4", so that old and new patches match; without it, generation stops and names the writer settings. New datasets need no change.

Two kinds of datasets cannot be resumed and need a new writer.staging_dir: 0.1 datasets (as in 0.2), and 0.2 datasets made with sampler.mode: random, because 0.3 draws random positions without repeats and would mix two different samples. Both can still be split and inspected.

Pixels without imagery

Masks now mark pixels without imagery (padding, failed tiles, NoData, the area outside a GeoTIFF) with labels.ignore_index, 255 by default, instead of background 0. Pass the same value to your loss (ignore_index=255 in PyTorch’s CrossEntropyLoss), or set labels.ignore_index: null to keep 0.2’s behaviour. The manifest records it as target.ignore_index.

Imagery inputs

Imagery settings are checked more strictly. imagery.max_connections is at most 64 and sampler.patch_size at most 16384 (larger values exhausted sockets or memory). An imagery.url_template needs a host, a valid port and each of {x}, {y} and {z} written once, without doubled braces. A GeoTIFF whose CRS is geocentric or vertical is refused, and an imagery.nodata its data type cannot hold now warns. An imagery.path that exists is read as that file even when its name has [, * or ? in it (0.2 read it as a glob pattern, and could pick up a different file); a pattern is used only when no such path exists. XYZ tiles that are fully transparent now count as empty (their pixels are black), and 16-bit grayscale PNG tiles are scaled to 8 bits instead of clipped to white; datasets made from such tiles can differ from 0.2’s.

Removed 0.1 config keys

tiles: and region.zoom, deprecated in 0.2, are no longer accepted: a config that still has them fails validation and says what to change. Move them into the imagery block as described under Imagery configuration below.

Patch footprints

Every dataset now also gets patches.geojson: one WGS-84 footprint polygon per patch with its file name, split, row and column, to open in QGIS. It is rewritten by mapcv split. Set writer.footprints: false to leave it out.

The task setting

Configs take an optional top-level task: segmentation (the default, so existing configs need no change), detection, instance, classification, change or regression. The manifest records the task, and a resumed run must use the same one.

Library output

run_generate no longer draws a progress bar or prints resume messages, and download_region/stitch_region no longer print: the library writes nothing to stdout. Messages go to the mapcv logger (logging.basicConfig(level=logging.INFO) shows them), and progress is reported through progress(done, total) callbacks (on_chunk for run_generate). The CLI shows its spinner and progress bars as before. New: mapcv.generate, mapcv.split and mapcv.iter_patches, which yields the patches a config describes without writing them (Python API).

Saving progress

mapcv generate saves manifest.json every 10 seconds and whenever a run stops (Ctrl-C, an error, a progress callback that raises) instead of after every chunk, which cost more than the chunk itself on large datasets. An interrupted run still resumes after its last finished chunk. A process killed outright (kill -9, a power cut) can lose up to the last 10 seconds of chunks; the next run writes them again, and the dataset ends up byte for byte the same.

Label files read differently

A few label files give other class IDs or masks than with 0.2. mapcv plan prints the class map; pin IDs with labels.classes to keep the ones a model was trained with.

• Numbers sort by value. When the labels are not all integers in 1–255 (for example 0, 2, 10), they are numbered in sorted order, and numbers now sort by value: 0 → 1, 2 → 2, 10 → 3. 0.2 sorted them as text (0 → 1, 10 → 2, 2 → 3). Only the digits 0–9 make a number: a full-width １ or a superscript ² is a name of its own, not the class 1.
• A label_field that no feature has is an error in GeoJSON and KML too, as it was for GeoPackage and Shapefile, instead of a dataset whose masks are all background. KML labels come from <ExtendedData> fields, not from the placemark’s <name>.
• A label raster’s NoData tag no longer overrides labels.classes. A value the file tags as NoData but labels.classes maps to a class is that class (with a warning), not labels.ignore_index.
• Polygons that reach the poles are cut at latitude ±85.0511° on XYZ imagery instead of being left out of the masks, and a feature with a coordinate that is not a number (NaN, infinite) or a latitude beyond ±90° is skipped and counted in a warning.

Patches without imagery

A patch without a single pixel of imagery (all its tiles failed, or it lies wholly in a GeoTIFF’s NoData or outside the file) is no longer written, whatever sampler.max_empty_ratio is; generate says how many were left out. Its image was all black and its mask all labels.ignore_index, so it taught a model nothing. A dataset made by 0.3 can therefore have fewer patches than one made by 0.2 from the same config, and a 0.2 dataset resumed by 0.3 keeps the empty patches it already has. Partly empty patches still follow max_empty_ratio.

Dataset folders

writer.staging_dir must be new, empty or a mapcv dataset: generate refuses a folder that holds other files and no manifest.json (for example the folder of the config, with staging_dir: .), so none of them is overwritten or removed. Hidden entries (.git, .gitkeep, …) and a top-level README.md do not count. generate and split lock the folder while they write, and a second run into it stops at once. An interrupted generate records "complete": false in the manifest until a run finishes it, and mapcv verify fails on such a dataset. Resuming also refuses a custom url_template other than the dataset’s (also on the same host), another region on the same grid, and a local GeoTIFF whose modification time changed: copy imagery with cp -p or rsync -a to resume with the copy.

Runs that write nothing, and YAML booleans

A generate whose plan holds no patch stops before it starts (exit code 1), and one that finds every patch dropped leaves no manifest.json, split lists or patches.geojson: mapcv 0.2 wrote an empty dataset that verify, stats and export accepted. A number field now refuses a YAML boolean (patch_size: true, zoom: yes), which mapcv 0.2 read as 1; write the number.

Repeated config keys

A key written twice in the same mapping of a config (for example stride twice under sampler) is now an error that names the key and both lines; YAML used to keep the last one silently. Delete the one you do not mean. An empty imagery.path for a Sentinel-2 product is rejected as well.

Links and redirects

A dataset folder that holds a symbolic link pointing outside it is refused by generate, split, stats, card, verify and export (they would write or read through it); replace the link with the file itself. A manifest whose patch files are absolute paths or contain .. no longer loads. Tile and GeoTIFF requests send no Referer header, and a redirect is checked like the configured URL: a url_template with a query string follows redirects only within its own host, so if your tile server sends tiles from another host, put that final URL in url_template. The query values of labels.osm.overpass_url are recorded in the manifest as ***.

The host a URL in your config names is still connected wherever it resolves (a docker-compose service or an on-prem server works as before). Another host, reached through a redirect or named by a STAC catalog, may not be on this machine or a private network (10.x, 192.168.x, 127.x, link-local, …) by its address or by the addresses its name resolves to; neither may another port of your host that resolves there. Put the final URL in the config if a server sends you on to such a host.

Tile helpers check their input

mapcv.bounds and mapcv.snap_bbox (and tile, xy, xy_bounds, tile_transform of mapcv._mapcv_rs) used to clamp invalid input and return a plausible answer: a NaN position became a real tile, zoom 64 became 32, and tile column 8 at zoom 3 got the bounds of column 7. They raise ValueError now: for a coordinate that is not a number, for a zoom above 32, and for a tile column or row that does not exist at its zoom. Valid input gives what it gave before, and a position beyond the world (longitude 200) is still moved onto the edge.

Migrating to 0.2

Checklist

1. Run mapcv validate on each 0.1 config. Errors tell you what to change, including unknown keys that 0.2 now rejects.
2. If a config is not in the same folder as its data, check its relative paths: they now resolve against the config file’s folder.
3. Move region.zoom and tiles into an imagery: {type: xyz, zoom: …} block. Configs that still use them stop validating in 0.3.0.
4. Replace google_satellite or osm sources (Providers & licensing).
5. If you rely on 0.1 class IDs, pin them with labels.classes.
6. If you used sampler.mode: random, raise random_count: it is now a total, not per strip.
7. Generate into a new writer.staging_dir; 0.1 datasets can be split but not resumed.
8. Decide on splits: the default is now spatial. Set strategy: stratified to keep a patch-level split.

Imagery configuration

The imagery source is now a discriminated imagery block. Move the XYZ zoom out of region and replace tiles with imagery:

region:
  west: 4.883
  south: 52.371
  east: 4.896
  north: 52.378

imagery:
  type: xyz
  zoom: 17
  source: esri_satellite

mapcv 0.2 still accepted region.zoom plus tiles with a deprecation notice. That compatibility path was removed in 0.3.0: a config with tiles: or region.zoom now fails validation with an error that points here. Move tiles and region.zoom into the imagery block as shown above, and delete region.zoom. imagery.type is required.

Stricter configs

Unknown keys are now errors: a typo such as stide: or an option from a newer mapcv stops validation instead of being ignored. Remove keys that 0.2 does not know; mapcv validate names each one. An XYZ imagery block takes either source or url_template, not both.

Relative paths (labels.path, writer.staging_dir, a local EOPF imagery.path) now resolve against the folder that contains the config file, not the directory you run mapcv from. Configs kept next to their data are unaffected; otherwise use absolute paths.

mapcv init writes mapcv.yaml unless you give it a path. Use mapcv init --stdout to print the config instead.

Google imagery preset

The built-in google_satellite preset has been removed, and configs that still use it fail validation with a pointer to this guide. mapcv cannot determine whether a use of Google imagery is licensed for downloading, caching, or machine-learning dataset creation. Authorized custom XYZ templates remain supported through imagery.url_template; do not put credentials or signed query strings in committed YAML.

OpenStreetMap preset

The built-in osm preset has also been removed. OpenStreetMap data is open (ODbL), but the OpenStreetMap Foundation’s tile servers run on donated capacity and their tile usage policy forbids bulk downloading, which is what dataset generation does. To use OSM-styled tiles, point imagery.url_template at a commercial OSM tile service whose terms allow it (with your API key kept out of committed YAML) or at a tile server you host. To use OSM features as labels, export them (for example with Overpass or QuackOSM) to GeoJSON and pass them as labels.path, with ODbL attribution.

EOPF output and manifests

EOPF Sentinel-2 L2A input requires pip install "mapcv[zarr]" and writer.image_format: npy (or tif). Each image patch has shape (bands, height, width) and decoded float32 values. Band order is exactly the order configured in imagery.bands.

New datasets use manifest version 2. It records the source type, product identifier, bands, dtype, patch shape, CRS, and dataset affine transform. Patch rows and columns are global raster coordinates. Version 1 manifests remain readable by mapcv split, but 0.1.x datasets cannot be resumed by mapcv generate: their row coordinates were per strip, while 0.2 uses global coordinates. Generate into a new writer.staging_dir instead.

Resuming any dataset now requires the same imagery, label file, label settings, sampler and writer settings that created it. A mismatch stops generation with an error instead of mixing incompatible patches. The manifest is saved after every chunk, so an interrupted run resumes from the last completed chunk.

Sampling changes

Anchors are now computed once over the whole raster instead of per strip:

• sampler.mode: random draws random_count patches in total. In 0.1 it drew random_count per strip, so the same config now yields fewer patches; raise random_count to compensate.
• Grid patches can cross the old strip seams, and edge_strategy applies only at the raster’s outer edges. Patch positions and counts can therefore differ from a 0.1 run over the same region.
• For XYZ, all-black pixels (including tiles that failed under policy: lenient or ignore) count as empty for max_empty_ratio, as in 0.1.

Splitting

The default split.strategy is now spatial: whole blocks of the raster (4 × patch_size by default, set with split.block_size) go to one split, and train/val patches that overlap a held-out patch are left out. Random patch-level splits overstate accuracy when patches overlap or neighbour each other. Set strategy: stratified or random to keep a patch-level split; mapcv warns when that leaks overlapping pixels. Manifests from 0.1 do not record the patch size, so spatial falls back to stratified unless block_size is set.

Class IDs

With labels.label_field, class IDs used to follow the order labels first appeared in the file, so the same classes could get different IDs in different files. Now integer labels in 1..255 are used as their own IDs and other labels are numbered in sorted order. Set labels.classes (for example {building: 1, road: 2}) to pin IDs explicitly, for instance to match a dataset created with 0.1.

Removed Python APIs

download_region_strips, iter_tile_strips, TilesConfig and the hello() stub are removed; mapcv generate reads imagery in chunks without them. download_region, stitch_region and sample_patches remain for in-memory workflows.