Hello

Fan-Out Engine

Backend / Systems

Overview

A high-throughput distributed data-processing engine in Java that streams very large files (100GB+) and fans records out to multiple sink systems in parallel — each with its own transform, rate limit, and fault handling.

Tech Stack

JavaMavenMultithreading

Key Features

  • Streams files line-by-line via BufferedReader — processes 100GB+ inputs on a 512MB heap, never loading the whole file
  • Pluggable transformers per sink: JSON, Protobuf, XML, and Avro
  • Per-sink token-bucket rate limiting that allows bursts while holding an average rate
  • Fault tolerance with up to 3 retries, exponential backoff, and a dead-letter queue for failed records
  • Backpressure via a bounded ArrayBlockingQueue so a slow sink can't exhaust memory
  • Graceful degradation — one sink failing doesn't stall the others