Skip to content
mapcvmapcv

Spatial leakage

A test set is supposed to tell you how a model does on data it has never seen. With map imagery, a random split of patches quietly breaks that promise.

Why random splits leak

Section titled “Why random splits leak”

Overlapping patches share pixels. With stride smaller than patch_size, or with random sampling, two patches can contain the same pixels. If one lands in train and the other in test, the model is tested on pixels it was trained on.

Neighbouring patches share almost everything else. Even without overlap, adjacent patches show the same buildings cut in half, the same field, the same roof material, the same acquisition date, sun angle and haze. Nearby things are more alike than distant things (spatial autocorrelation), so a test patch next to a train patch is far easier than a patch from a new area.

A model can score well on such a split by recognising local context instead of learning the concept, and the score drops when it is deployed somewhere else. Kattenborn et al. showed this effect for convolutional neural networks on remote-sensing imagery: validation samples that are spatially autocorrelated with the training samples inflate the measured performance.

Kattenborn, T., Schiefer, F., Frey, J., Feilhauer, H., Mahecha, M. D., & Dormann, C. F. (2022). Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks. ISPRS Open Journal of Photogrammetry and Remote Sensing, 5, 100018. doi:10.1016/j.ophoto.2022.100018

What mapcv does

Section titled “What mapcv does”

Map of 256-pixel patches over Amsterdam outlined in green (train), yellow (val) and pink (test), with val and test in whole blocks

The quickstart’s patches by split: val and test patches come in whole blocks, so no test patch has a training neighbour that shows the same buildings.

The default split.strategy: spatial:

  1. Splits by area, not by patch. The raster is divided into square blocks (block_size: 4 × patch_size by default, smaller on a small raster so that it has at least 10 blocks) and each block goes entirely to train, val or test. Most neighbours of a test patch are therefore test patches too.
  2. Removes overlap across splits. Train or val patches whose footprint overlaps a test patch, and train patches overlapping a val patch, are left out of every list. No pixel appears in two splits.
  3. Warns about leaky choices. If you pick stratified or random for a dataset whose patches overlap or repeat, mapcv writes the split but warns that it leaks pixels.
✓ Splits written to overlap/splits: train 137 (67%) · val 10 (5%) · test 57 (28%) · 69
overlapping patches left out

The cost: some patches are discarded when patches overlap, and split ratios are approximate because blocks move as a whole. Splits without leakage shows how to tune this.

What it cannot do

Section titled “What it cannot do”
  • Patches along block borders still touch. A test block’s edge patches sit next to train patches. Larger blocks make this a smaller share of the test set.
  • One region is one distribution. All blocks come from the same city or scene, date and sensor. A spatial split measures generalisation to nearby unseen areas. To claim more (another city, country or season), hold out a separate region. With an area of interest of several polygons (region.path), split.strategy: region sends whole regions to one split each; or generate a second dataset with its own config and use it as the test set.
  • Labels can leak too. If labels were drawn by a model trained on the same area, or if a region’s labels are systematically shifted, no split will reveal it.