Problem#Link to this section
The GEO-AI challenge asked teams to label points in two regions, Fergana in Uzbekistan and Orenburg in Russia, as cropland or not, using Sentinel-1 radar and Sentinel-2 optical satellite data. Maps like this help plan food security and land use where field surveys are expensive.
Constraints#Link to this section
- Few labels. 1,000 labelled points, about 30% cropland.
- Large, messy imagery. About 1.75 million radar rows and 5.6 million optical rows, with clouds and gaps.
- Two very different regions. A model that learns one landscape can fail on the other.
- A team and a deadline. Five contributors over a few weeks.
Approach#Link to this section
I led a team of five. We aggregated the satellite rows by location and matched each labelled point to its nearest pixels with a BallTree search, falling back to rounded coordinates where coverage was thin. On top of the raw bands we computed vegetation and water indices (NDVI, NDWI, EVI, SAVI and others) and a radar ratio, then selected the 40 most informative features.
We tuned RandomForest, XGBoost, LightGBM and CatBoost with Optuna, tried oversampling for the class imbalance, and stacked the models into an ensemble with a small neural network on top.
Evaluation#Link to this section
The final comparison holds out a stratified random 20% of the labelled points, 200 of them, and scores every model on that same split. The figure that matters most is the baseline: labelling every point "non-crop" is already right 70.3% of the time.
| Metric | Value | Measured on | Caveat |
|---|---|---|---|
| Accuracy, RandomForest | 0.745 | 200 held-out labelled points (stratified random 20% of 1,000 across Fergana and Orenburg) | Guessing "non-crop" for every point scores 0.703 on this data, and a random split lets nearby points leak between train and test. |
| ROC-AUC, RandomForest | 0.727 | The same 200 held-out points | One split of 200 points; no confidence interval was computed. |
| Accuracy, stacked ensemble | 0.725 | The same 200 held-out points | The ensemble scored below the single RandomForest, so it didn't earn its complexity. |
Results#Link to this section
RandomForest reached 0.745 accuracy on the held-out points, about 4 points above always guessing "non-crop". The stacked ensemble scored 0.725, below the single model it was built from.
Security#Link to this section
The data is public satellite imagery and labelled coordinates, with no personal data. The project ships trained models as joblib files; anyone reusing them should load them only from a trusted copy, because joblib files can run code when loaded.
Limits#Link to this section
- Only 1,000 labelled points, 297 of them cropland, so every figure moves with a handful of points.
- Splits were random, not spatial. The project's own notes call spatial cross-validation essential; it wasn't done.
- Earlier runs oversampled with SMOTE before splitting, which put synthetic copies of test points into training. Their higher scores (around 0.76) aren't quoted here.
- Satellite matches covered 81.6% of the labelled points; the rest relied on a coarser coordinate fallback.
- The competition leaderboard score isn't recorded in the repository.
What I'd do next#Link to this section
- Evaluate with spatial blocks or region holdout, so the score reflects new land.
- Oversample inside each training fold only.
- Add seasonal features from the time series, which the notes flagged and we didn't build.
- Report intervals, not single numbers, on a split this small.