Real-Time E-commerce Streaming with Flink and Kafka
A stream-processing lab where orders created through a Node.js API flow into Kafka and are analysed live by an Apache Flink job with event-time windows, enrichment and fraud-style pattern detection.
Overview
I built this project during a university lab on stream processing. A Node.js/Express API saves each order in MongoDB and publishes one payment event per product to a Kafka topic (served by Redpanda). An Apache Flink DataStream job written in Java consumes the topic, enriches the events with product categories and computes rolling statistics and alerts. The whole stack runs locally with Docker Compose.
The problem
E-commerce platforms need answers while events are still arriving: how much each product is selling right now, how often a user's payments succeed, and whether a pattern of failed payments looks suspicious. Batch reports computed later cannot answer these questions in time.
What I built
I wrote an Express endpoint that stores orders in MongoDB with Mongoose and sends payment events to Kafka with KafkaJS. On the Flink side, the job reads the topic with the Kafka connector, parses JSON events with Jackson and assigns event-time timestamps with a 5-second bounded out-of-orderness watermark. Payments are enriched with their product category through broadcast state, then aggregated in a 1-minute tumbling window per product, a 1-minute tumbling window per category and a 5-minute sliding window per user (success rate). Flink CEP detects three or more consecutive failed payments followed by a success within 10 minutes and raises an alert.
Results
- Working end-to-end pipeline: REST API → MongoDB + Kafka → Flink job
- Three window aggregations (per product, per category, per user) on event time
- CEP alert for repeated failed payments followed by a success
- Docker Compose stack with a Flink JobManager and TaskManager, Redpanda and the backend API
Highlights
- Event-time processing with bounded out-of-orderness watermarks
- Broadcast state to enrich payments with product categories
- Tumbling and sliding windows with ProcessWindowFunction
- Complex event processing with Flink CEP patterns
- Order API that writes to MongoDB and produces to Kafka in one request
- Fat JAR built with the Maven Shade plugin and submitted to the cluster
