The system around the model

The classifier was only one part of this project. The more interesting question was what it takes to turn model inference into a service that behaves predictably under real traffic.

I built the API with FastAPI and DistilBERT, then used Redis to keep repeated work off the inference path. The resulting service reached sub-100ms response times with a 95% cache hit rate. Async request handling and horizontal scaling made the surrounding system as important as the model itself.

What I was exploring

This was an exercise in treating machine learning as a systems problem: latency, cache behavior, health checks, and deployment all shape whether a capable model becomes a useful product.

The project also became a practical reference for the boundaries between application code and model-serving infrastructure: what should be optimized in the model, what belongs in the API, and what can be avoided altogether through a good cache.

Built with

FastAPI, Redis, DistilBERT, and Kubernetes.