Six months after GPT-4’s launch, the demo cycle has quieted and the deployment cycle has started in earnest. The inboxes we see in Q3 are no longer about what’s possible — they’re about what’s still unreliable in production. If the first half of 2023 was about capability, the second half is about engineering discipline.
The teams that have shipped useful LLM features this year have done so by borrowing from a well-understood toolkit: contracts, observability, retries, evals. The teams that haven’t shipped have usually skipped straight from prototype to production, and are now living with the consequences.
Five patterns separating the LLM applications that work from the ones that don’t:
- Structured output with schema validation. Don’t accept raw LLM text as input to downstream systems. Validate with JSON Schema, Zod or Pydantic — and retry on failure, because the model will eventually return something malformed. This one pattern eliminates a whole class of production bugs.
- RAG is load-bearing — treat it that way. Your retrieval layer is part of your product surface, not a hidden implementation detail. Index freshness, chunking strategy, reranker quality and evaluation matter more than clever prompt engineering, and they fail silently if you don’t measure them.
- Fall back to the cheaper model by default. Most requests don’t need GPT-4. Route based on request shape, not organisational ambition. A cheap model with a careful prompt often beats an expensive model with a sloppy one, and your invoice will thank you.
- Observability at the prompt level. Log every request, every response, every token count, every latency. The first production bug you cannot reproduce will teach you, in a single painful week, why this matters. Add it before launch, not after.
- Eval harness as a product, not an afterthought. The fastest teams we’ve seen ship this year have a regression suite of 50 to 200 cases per feature, scored with a mix of exact-match rules and LLM-as-judge checks, and they run it on every prompt change. You can deploy without it, but you can’t iterate without it.
The pattern that keeps emerging is unglamorous: the teams treating LLM features like regular software — with contracts, tests and monitoring — are the ones shipping reliably. The teams chasing the next capability release are still in prototype mode. You don’t need to pick between them, but you should know which one you’re doing today, and why.
