Fouzan Ahmed.
All projects
Full-Stack DevelopmentGeospatial DataClimate AnalyticsData Engineering

Weather, Together

130+ years of climate data and live BOM feeds, combined into suburb-level flood, bushfire and heat risk scores.

Team project, Monash UniversityAug 2024 – Nov 2024
Repository not public yetJump to architecture

Problem

  • If you want to know how exposed a specific Melbourne suburb actually is to flooding, bushfire or heat, the underlying data exists, but it's spread across 1-in-100-year flood shapefiles, well over a century of station-level weather records, historical bushfire records going back to 1903, and separate live BOM forecasts and fire danger index projections. None of it is suburb-level, and none of it is combined.
  • Weather, Together pulls all five sources into one pipeline and turns them into a single, interpretable suburb rating, so residents can see relative risk without needing to interpret a flood shapefile themselves.

Architecture

  1. Step 01

    5 raw sources

    Flood, SILO, fire, BOM, FFDI

  2. Step 02

    EPSG:3857 + GeoJSON

    SHP conversion & standardization

  3. Step 03

    R-tree indexing

    Fast spatial intersection

  4. Step 04

    3-tier PostgreSQL

    History, cache, projections

  5. Step 05

    Risk scoring

    60% flood / 40% fire weighting

  6. Step 06

    FastAPI + Mapbox GL

    Suburb drill-down choropleth

Five raw geospatial sources get standardized, indexed and scored before ever reaching the map.

  • Five data sources feed the pipeline: 1-in-100-year flood shapefiles, 130+ years of SILO weather data from 3,000+ stations, bushfire history back to 1903, live BOM forecasts, and forward-looking FFDI (fire danger index) projections.
  • Every source gets standardized to EPSG:3857, with SHP files converted to GeoJSON and indexed with an R-tree structure for fast spatial intersection queries across Victoria.
  • Three PostgreSQL storage tiers separate concerns by how the data behaves: permanent history that never changes, a time-limited cache for real-time BOM data, and a separately versioned table for future projections, so comparisons over time stay possible without the database growing without bound.
  • The risk score itself is a weighted spatial calculation, 60% flood zone intersection and 40% historical fire frequency, producing a Low-to-Very-High rating per suburb.
  • A FastAPI backend serves the processed GeoJSON to a React frontend, where Mapbox GL renders interactive choropleth maps with suburb-level drill-down.

Engineering decisions

  • Three separate storage tiers instead of one table. Permanent history, a live cache, and versioned projections behave completely differently: history is append-only and never shrinks, the BOM cache needs to expire, and projections need versioning so an old forecast doesn't silently get overwritten. Splitting them by tier kept each one simple instead of building one table with fields that only made sense for some rows.
  • R-tree indexing over a naive bounding-box scan. With flood shapefiles and suburb boundaries both being complex polygons, brute-force intersection checks across all of Victoria would have been far too slow for interactive map queries. R-tree indexing made suburb-level spatial queries fast enough to feel live.
  • A weighted, explainable score over a black-box model. 60/40 flood-to-fire weighting is simple enough that a resident (or a reviewer) can understand exactly why a suburb got the rating it did, which mattered more here than squeezing out marginal accuracy from a more opaque approach.
  • Standardizing every source to EPSG:3857 up front. Mixing coordinate reference systems across five independently sourced datasets is a classic way to get spatial queries that are silently wrong. Standardizing once at ingestion meant every downstream query could assume a single consistent projection.

Implementation

  • Built the climate risk pipeline across all 5 data sources: flood shapefiles, SILO station data, bushfire history, BOM forecasts and FFDI projections.
  • Designed the spatial risk score combining 60% flood zone intersection and 40% historical fire frequency into Low-to-Very-High suburb ratings.
  • Standardized coordinates to EPSG:3857, converted SHP files to GeoJSON, and added R-tree spatial indexing for fast intersection queries across Victoria.
  • Designed the three-tier PostgreSQL schema (permanent history, time-limited real-time cache, versioned future projections) so historical comparisons remain possible without uncontrolled growth.
  • Served the processed GeoJSON through FastAPI REST endpoints to a React frontend, using Mapbox GL for interactive choropleth maps and suburb drill-downs.

Challenges

  • Reconciling five independently sourced datasets, each with its own native format, resolution and coordinate system, into one consistent, queryable spatial model.
  • Keeping the real-time BOM cache genuinely fresh without hammering the upstream API, which shaped the time-limited cache design in the storage tier.
  • Tuning the R-tree indexing and query patterns so suburb drill-down felt interactive rather than waiting on a multi-second spatial join.

Results

  • Suburb-level flood, bushfire and heat risk became a single interpretable rating instead of five separate raw datasets a resident would have to interpret themselves.
  • R-tree indexing made suburb-level spatial queries fast enough for an interactive map experience across the whole of Victoria.
  • The three-tier storage design kept historical, real-time and projected data comparable over time without the database growing without bound.

Technology

Geospatial

PostGISGeoJSONR-tree IndexingEPSG:3857Mapbox GL

Backend

PythonFastAPIPostgreSQLREST API

Data Sources

SILO APIBOM APIFlood ShapefilesFFDI Projections

Frontend

ReactInteractive Choropleth

More projects

APRA Data Engineering & Analytics Platform

Personal / Portfolio project

Problem

APRA publishes superannuation performance data as unversioned Excel filings, with no API and no easy way to compare funds over time.

Key result

Weekly pipeline runs unattended end to end, gated by dbt tests and CI, feeding a live 3-page Power BI dashboard.

PythonApache AirflowdbtAWS S3AWS RDSAWS EC2+8 more
ArchitectureGitHubProject details

Market Data Warehouse on Snowflake

Personal / Portfolio project

Problem

Backtesting a forecasting model on financial time series is easy to get subtly wrong: without point-in-time discipline, the model ends up training on information it wouldn't have had yet.

Key result

A clustering key fix cut partitions scanned on date filters by roughly an order of magnitude, and Time Travel snapshots kept lookahead bias out of every backtest.

SnowflakeSQLPythonAWS S3Streams & TasksWindow Functions+6 more
ArchitectureGitHubProject details