Drone NDVI Crop Health

Drone Multispectral NDVI Mapping & Crop Health Engine

Turn raw 5‑band drone multispectral GeoTIFFs into NDVI / NDRE Cloud‑Optimized GeoTIFFs and ISO 11783 Variable Rate Application prescriptions — with an async raster pipeline, spatial persistence, and a map‑tile API.

---

📖 What This Is

A precision‑agriculture backend that ingests drone survey imagery and produces decision‑ready agronomic output rather than raw index numbers:

The full path runs locally against Docker containers — generate a synthetic flight, process it, and export a prescription without any cloud service.

---

✨ Key Features

• 🧮 Vectorized index engine — NDVI, NDRE, EVI and GNDVI computed with NumPy, not per‑pixel loops. Every result is range‑asserted to [-1, 1], and a zero denominator resolves to NoData instead of a NaN that poisons the zone statistics.
• 🛰️ Radiometric calibration — Raw digital counts are converted to surface reflectance using downwelling‑light‑sensor or reflectance‑panel coefficients, per band, with defensive clipping.
• 🗺️ Cloud‑Optimized GeoTIFF output — Written through GDAL's COG driver: 256×256 internal tiles, pyramid overviews at 2/4/8/16 with average resampling, DEFLATE compression, and BigTIFF=IF_SAFER.
• 🌾 Agronomic vigour zonation — Continuous NDVI is sliced into four classes (NON_VEGETATED, HIGH_STRESS, MODERATE, VIGOROUS) at the precision‑agriculture thresholds of 0.1 / 0.3 / 0.6, then polygonized.
• 🧹 Topology‑clean vector output — Sliver polygons below a minimum mapping unit are dropped, boundaries are simplified with topology‑preserving Douglas–Peucker, dissolved per zone, repaired with a zero‑width buffer, and re‑oriented CCW exterior / CW interior per OGC & ISO 11783.
• ⚡ Async pipeline — POST /surveys returns 202 immediately. A Celery worker does the raster work with acks_late + prefetch_multiplier=1, so a survey is redelivered rather than lost mid‑flight.
• 🗺️ Map tile endpoint — GET /surveys/{id}/tiles/{z}/{x}/{y}.png renders 256 px XYZ tiles straight from the stored COG, with a local disk cache.
• 🎲 Synthetic flight simulator — Generates realistic 1024×1024 five‑band scenes with dense canopy, drought foci, a bare‑soil track, a water body and a nodata corner, so the pipeline can be exercised end‑to‑end without hardware.
• 🧪 72 tests across unit, integration and end‑to‑end layers.

---

🏗️ System Architecture

The system separates a synchronous HTTP tier from an asynchronous raster pipeline, so a multi‑gigabyte raster never blocks an HTTP request.

Component breakdown

Trust & boundary notes

• No authentication layer. The API is designed to sit behind a gateway; it does not implement auth itself, and CORS is currently wide open.
• agri-raw is treated as immutable input. The pipeline never writes back into the raw bucket.
• The worker is the only writer of raster state. The API writes survey metadata only. This keeps status transitions authoritative in one place.

---

🔄 Core Workflow

The full request lifecycle for POST /api/v1/surveys, including the asynchronous hand‑off:

Step by step

1. Submit — POST /api/v1/surveys with a field_id, flight_date, sensor_model and the raw_multispectral_key. Unknown fields return 404; a dead broker returns 503.
2. Persist & enqueue — The survey row is inserted as PENDING, then the task is published by name, so the API never imports the worker module.
3. Calibrate — The worker downloads the raw cube, replaces nodata sentinels with NaN, and applies per‑band calibration coefficients.
4. Compute — NDVI = (NIR − Red) / (NIR + Red) and NDRE = (NIR − RedEdge) / (NIR + RedEdge), both clipped and range‑asserted.
5. Write COGs — Each index is written as a Cloud‑Optimized GeoTIFF and streamed to agri-processed; per‑survey statistics are stored alongside.
6. Zonate — The NDVI grid is classified into four vigour classes, polygonized, simplified, dissolved and re‑oriented.
7. Persist zones — Zone geometries are inserted as MultiPolygon with their mean index value and recommended nitrogen rate.
8. Complete or fail — The survey moves to COMPLETED, or to FAILED with the error message stored. Either way the temp working directory is removed.

Vigour zonation

NaN (nodata) is classified as NON_VEGETATED, so missing data never inflates a vigour average.

---

🛠️ Tech Stack

---

📂 Project Structure

.
├── cmd/
│   ├── api/main.py                 # FastAPI app: routers, CORS, /health
│   └── worker/main.py              # Celery worker entry point
├── internal/
│   ├── config.py                   # Env-driven settings (pydantic-settings)
│   ├── domain/                     # Pure logic — no I/O, fully unit testable
│   │   ├── survey.py               # Survey, Field, Band, CalibrationModel, DTOs
│   │   ├── indices.py              # NDVI/NDRE/EVI/GNDVI formulas + registry
│   │   └── zonation.py             # VigourZone thresholds + N prescriptions
│   ├── raster/
│   │   ├── io.py                   # GridSpec, read_cube, band alignment guards
│   │   ├── calibrator.py           # DLS / reflectance-panel calibration
│   │   ├── calculator.py           # Vectorized index engine + range assertion
│   │   └── cog_writer.py           # GDAL COG driver, 256px tiles, overviews
│   ├── vector/
│   │   ├── polygonizer.py          # Classified raster → Shapely geometries
│   │   └── smoothing.py            # MMU filter, Douglas–Peucker, CCW orient
│   ├── storage/
│   │   ├── s3.py                   # MinIO/S3 client, multipart, key parsing
│   │   └── local.py                # StorageClient protocol + local adapter
│   ├── repository/
│   │   ├── db.py                   # asyncpg pool + json/jsonb codecs
│   │   ├── field_repo.py           # Field queries (ST_Contains lookup)
│   │   └── survey_repo.py          # Surveys, indices, zones, status
│   └── transport/
│       ├── api/                    # survey_routes · tile_routes · export_routes
│       └── queue/                  # celery_app · tasks
├── migrations/
│   └── 001_init_agriculture_schema.sql
├── simulator/
│   └── generate_multispectral_tiff.py  # Synthetic 5-band flight generator
├── tests/                          # 72 tests: unit + integration + e2e
├── docker-compose.yml
├── Dockerfile
├── Makefile
└── pyproject.toml

The layering is strict: domain knows nothing about I/O, raster and vector know nothing about the database, and only transport talks to FastAPI or Celery.

---

🚀 Quick Start

Prerequisites

• Python 3.11+
• Docker + Docker Compose (for PostGIS, Redis and MinIO)

1. Clone and install

git clone https://github.com/<your-username>/agri-ndvi-engine.git
cd agri-ndvi-engine

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

2. Start infrastructure

make up

Brings up PostGIS, Redis and MinIO with health-checked waits. minio-init creates the agri-raw and agri-processed buckets, and migrations/ is mounted into the PostGIS entrypoint so the schema is applied on first boot.

Published host ports (from docker-compose.yml): PostGIS 55432, Redis 56379, MinIO 59000, MinIO console 59001.

3. Configure the environment

internal/config.py defaults assume standard ports, so for the containerized stack above create a .env that points at the published ports:

cat > .env <<'EOF'
DATABASE_URL=postgresql://agri:agri@localhost:55432/agri
REDIS_URL=redis://localhost:56379/0
CELERY_BROKER_URL=redis://localhost:56379/1
CELERY_RESULT_BACKEND=redis://localhost:56379/2
S3_ENDPOINT_URL=http://localhost:59000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
RAW_BUCKET=agri-raw
PROCESSED_BUCKET=agri-processed
MIN_POLYGON_AREA_M2=5.0
SIMPLIFY_TOLERANCE_M=1.0
EOF

All settings

4. Generate a synthetic flight

make generate-raster

Writes data/raw/synthetic_multispectral.tif — a 1024×1024 five-band scene at 8 cm/pixel containing dense canopy, stress foci, bare soil, water, and a nodata corner that exercises the masking path.

5. Load the raster and a field polygon

The pipeline reads the survey from object storage and requires the field to exist:

# upload the synthetic flight to agri-raw
python -c "from internal.storage import s3 as storage; storage.upload_file('agri-raw', 'synthetic_multispectral.tif', 'data/raw/synthetic_multispectral.tif'); print('uploaded')"

# seed one field (prints its UUID)
docker compose exec -T db psql -U agri -d agri -c "INSERT INTO farm_fields (name, crop_type, hectares, boundary) VALUES ('North Paddock', 'winter wheat', 82.5, ST_GeomFromText('POLYGON((-74.01 40.71, -74.00 40.71, -74.00 40.72, -74.01 40.72, -74.01 40.71))', 4326)) RETURNING id;"

6. Run the API and the worker

make worker     # terminal 1 — consumes process_survey tasks
make api        # terminal 2 — http://localhost:8000/docs

7. Submit a survey

curl -X POST http://localhost:8000/api/v1/surveys \
  -H "Content-Type: application/json" \
  -d '{
        "field_id": "<id printed in step 5>",
        "flight_date": "2026-05-17",
        "sensor_model": "MicaSense RedEdge-P",
        "raw_multispectral_key": "agri-raw/synthetic_multispectral.tif"
      }'

Expect 202 Accepted with status: PENDING. Poll, then export the prescription:

curl http://localhost:8000/api/v1/surveys/<survey_id>/status
curl "http://localhost:8000/api/v1/surveys/<survey_id>/prescription/geojson?index_type=NDVI"

Point any slippy map at http://localhost:8000/api/v1/surveys/<survey_id>/tiles/0/0/0.png to watch the NDVI mosaic stream in as it completes.

Notes

• The bundled Makefile targets .venv/Scripts/python.exe (Windows). On Linux/macOS pass the interpreter: make api PYTHON=.venv/bin/python.
• The api service in docker-compose.yml does not receive CELERY_BROKER_URL, so container-to-container submission would enqueue against localhost. Run the API and worker on the host as shown above, or add that one env line to the service.

---

🔌 API Reference

Interactive docs are served at /docs and /redoc.

---

🧪 Testing

make test         # everything (integration tests need the containers up)
make test-unit    # skip tests marked `integration`

---

🧭 Design Decisions

• Why Postgres and not a raster store? Index statistics and zone polygons are small and highly relational to a survey. The bulk pixels stay in object storage as COGs; the database holds only what needs querying.
• Why send_task by name instead of importing the task? The API process does not need GDAL or NumPy loaded to accept a survey. Publishing by string keeps that dependency out of the HTTP tier.
• Why clip indices to [-1, 1]? Normalized differences are mathematically bounded. Clipping converts sensor noise into a bounded error rather than an out‑of‑range zone assignment.
• Why a StorageClient protocol? internal/storage/local.py implements the same surface as the S3 adapter, so the pipeline runs in tests and offline runs without MinIO.

---

📌 Status & Next Steps

• ✅ Five‑phase build complete — all components implemented and tested.
• ⏳ Authentication is not implemented; put the API behind a gateway before exposing it.
• ⏳ Field ingestion (farm_fields) is repository‑level only — there is no public write endpoint yet.
• ⏳ EVI and GNDVI are implemented and tested in the domain layer but are not yet emitted by the default pipeline (NDVI, NDRE).
• ⏳ calibration_for() currently returns a fixed coefficient profile and ignores sensor_model; wire real per-sensor panel / DLS values before trusting absolute reflectance on production flights.

---

Built with FastAPI, Celery, GDAL and PostGIS · Python 3.11+