AI engineering practice

Building with LLMs requires rethinking infrastructure choices, prioritizing flexibility and quality over simplistic cost optimization. Martin's approach centers on hybrid architectures that reserve commercial APIs for high-stakes reasoning while self-hosting commodity components—a philosophy rooted in direct experience with the constraints and tradeoffs of different deployment models.

Flexibility over API restrictions

Commercial LLM APIs impose architectural constraints that become limiting as systems grow more sophisticated. The key limitation isn't performance or cost—it's control over data flow. ▶ 6:36 Martin describes the friction point: you can't influence how APIs receive or send data, forcing your system into predetermined pipelines. This inflexibility pushed him toward self-hosting despite the added operational burden.

The cost narrative—that rising API bills drove migration to self-hosted models—misses the actual driver. ▶ 2:17 Renting GPUs by the hour proves economically indistinguishable from buying hardware, but offers the flexibility to tune data transformation at each layer.

Architecture: tiered quality, not uniform standards

Not all model invocations warrant the same capability. The reasoning step—critical for final output quality—demands state-of-the-art performance. ▶ 14:03 Martin refuses to compromise on reasoning intelligence, using GPT-4 as the foundation layer even within an otherwise self-hosted stack. ▶ 10:53 Other capabilities—like speech-to-text—can reliably delegate to open-source alternatives as those models close the capability gap.

This inverts the cost-driven thinking that leads to uniform model selection. Instead, each stage of a pipeline gets matched to the minimum capability that preserves quality for that stage's role.

Infrastructure: the learning curve and middle grounds

Self-hosting comes with real friction. Martin, not trained as a full-stack developer, spent weeks learning Docker and cloud deployment through frustration. ▶ 10:01 This investment was necessary to achieve the data-flow control he needed.

For teams unable to absorb that operational load, serverless GPU rental offers a hybrid path. Services like RunPod let you deploy custom Docker images with your own models, but only spin up on demand and sleep after 5 seconds of inactivity. ▶ 23:57 You pay per second of actual use rather than per-hour reservations, recovering some cost efficiency while maintaining architectural control.

Latency: where it matters and where it doesn't

Network transport latency—a few extra milliseconds—is genuinely negligible against the model's own computation time, typically hundreds of milliseconds. ▶ 17:41 The hidden cost appears when requests cascade through a pipeline: many chained models compound latency into a meaningful delay. Pipeline design becomes the real optimization frontier, not reducing hops.