Fouzan Ahmed.
All projects
Big DataReal-Time StreamingMachine LearningData Visualization

StreamPredict Real-Time ML Pipeline

Kafka streams into PySpark MLlib to predict delivery times and assign drivers as orders arrive.

Individual project, Monash University FIT5202 Big Data ProcessingFeb 2025 – Apr 2025
View repositoryJump to architecture

Problem

  • Food delivery platforms need a delivery-time estimate and a driver assignment before the order is even fully placed, not five minutes later once a batch job gets around to it. That rules out the usual train-a-model-then-run-it-nightly pattern and forces the whole thing to happen inside the stream, on data that's still arriving.
  • Built for FIT5202 (Big Data Processing), the goal was to simulate that constraint honestly: real streaming ingestion, real in-stream inference, and a live visualization that reflects what's actually happening as orders come in.

Architecture

  1. Step 01

    Kafka producers

    Orders + driver availability

  2. Step 02

    Structured Streaming

    Schema, checkpoints, joins

  3. Step 03

    MLlib regression

    In-stream delivery time scoring

  4. Step 04

    Driver assignment

    Fastest driver, 1 per batch

  5. Step 05

    Windowed aggregation

    15s revenue, 30s suburb counts

  6. Step 06

    Live choropleth

    Downstream Kafka consumer

Two Kafka topics join in-stream, get scored by MLlib, and drive both a driver assignment and a live map.

  • Two Kafka topics feed the system: simulated order events and a parallel driver availability feed, each produced on its own cadence to mimic a live system.
  • Spark Structured Streaming consumes both topics with explicit schema enforcement, checkpointing and timezone handling, then joins them with restaurant and geolocation reference datasets.
  • A PySpark MLlib regression model, trained beforehand on historical delivery data, runs inference inside the stream itself as each order arrives, rather than in a separate offline batch step.
  • For every order, five available drivers are sampled, each scored by the model, and the fastest is assigned, with a constraint enforcing one assignment per driver per micro-batch so the same driver can't be double-booked in the same window.
  • Windowed aggregations compute 15-second revenue windows by order type and 30-second suburb-level order counts, split by whether the predicted delivery time is above or below a 15-minute threshold.
  • A downstream Kafka consumer feeds live choropleth and bubble map visualizations that refresh suburb-level delivery performance every 15 seconds.

Engineering decisions

  • In-stream inference, not a lookup against a pre-computed batch table. Scoring the model live inside Structured Streaming, instead of joining against nightly batch predictions, is what makes the driver assignment actually reflect current conditions rather than yesterday's traffic pattern.
  • Sample-then-score over scoring every available driver. Scoring five sampled drivers per order rather than the entire available pool keeps per-order latency bounded and predictable as the number of active drivers grows, at a small, acceptable cost in optimality.
  • One assignment per driver per micro-batch, enforced explicitly. Without this constraint, two orders arriving in the same micro-batch could both get assigned to the same fastest-looking driver, which doesn't reflect how real dispatch has to work.
  • Separate windows for revenue and suburb counts (15s versus 30s). Revenue by order type needed a tighter window to feel 'live'; suburb-level counts are naturally noisier at small windows, so a wider 30-second window gave a more readable signal without losing responsiveness.

Implementation

  • Built Kafka producers simulating live order ingestion, sending batches of 20 to 50 orders every 5 seconds with evenly distributed timestamps, alongside a parallel driver availability feed.
  • Set up Spark Structured Streaming with schema enforcement, checkpoints and timezone handling to consume both Kafka topics and join them against restaurant and geolocation datasets.
  • Ran inference inside the stream: sampled 5 available drivers per order, scored each with the trained MLlib regression model, and assigned the fastest while enforcing one assignment per driver per batch.
  • Computed 15-second revenue windows by order type and 30-second suburb order counts, split by a predicted 15-minute delivery threshold.
  • Built live choropleth and bubble map visualizations from a downstream Kafka consumer, refreshing suburb-level delivery performance every 15 seconds.

Challenges

  • Getting checkpointing and timezone handling right in Structured Streaming took real care, a subtle timezone mismatch between the two Kafka topics would have silently broken the join.
  • Balancing latency against assignment quality when sampling drivers, scoring every available driver would have been more accurate but didn't scale, so the sampling approach was a deliberate tradeoff.
  • Keeping the downstream visualization genuinely live rather than just fast, the 15-second refresh needed the whole path (stream, aggregation, consumer, map) to stay within that budget consistently.

Results

  • Delivery time predictions and driver assignment happen inside the stream as orders arrive, not in a separate batch step after the fact.
  • The one-assignment-per-driver-per-batch constraint kept driver assignment realistic under concurrent order arrivals.
  • Live choropleth and bubble maps refresh every 15 seconds, giving a genuinely real-time view of suburb-level delivery performance.

Technology

Streaming

Apache KafkaSpark Structured StreamingCheckpointing

ML

PySpark MLlibRegressionFeature EngineeringIn-Stream Inference

Data & Viz

ParquetPandasChoropleth MapsWindowed Aggregation

More projects

APRA Data Engineering & Analytics Platform

Personal / Portfolio project

Problem

APRA publishes superannuation performance data as unversioned Excel filings, with no API and no easy way to compare funds over time.

Key result

Weekly pipeline runs unattended end to end, gated by dbt tests and CI, feeding a live 3-page Power BI dashboard.

PythonApache AirflowdbtAWS S3AWS RDSAWS EC2+8 more
ArchitectureGitHubProject details

Market Data Warehouse on Snowflake

Personal / Portfolio project

Problem

Backtesting a forecasting model on financial time series is easy to get subtly wrong: without point-in-time discipline, the model ends up training on information it wouldn't have had yet.

Key result

A clustering key fix cut partitions scanned on date filters by roughly an order of magnitude, and Time Travel snapshots kept lookahead bias out of every backtest.

SnowflakeSQLPythonAWS S3Streams & TasksWindow Functions+6 more
ArchitectureGitHubProject details