LLM Integration Patterns

Connecting a large language model to a production system is an engineering problem, not a deployment configuration. Successful LLM integration requires deliberate decisions about prompt architecture, retrieval design, evaluation pipelines, failure handling, and operational cost control — all of which must be made before a system running large language models can be considered production-ready.

The prototype illusion is the characteristic failure mode of early LLM integration efforts. A developer sends a prompt, receives an impressive response, and declares the integration done. The product ships. The problem surfaces at scale: the output that looked excellent in twenty manual tests degrades unpredictably across ten thousand real requests. The model behaved consistently in the conditions tested; it behaved inconsistently in all the conditions that were not tested. LLMs are probabilistic systems. Production engineering for probabilistic systems requires evaluation infrastructure proportional to the operational risk of output degradation.

Why LLM integration is harder than it looks

Three structural properties of large language models create engineering complexity that does not exist in conventional API integrations:

Non-determinism. The same input can produce substantially different outputs across different model versions, temperature settings, context window compositions, and in some implementations, across individual requests. An integration that looks reliable in a small test set may be systematically unreliable at production scale. Temperature alone is not the controlling variable — model version, token budget, context position, and prompt phrasing all affect output consistency in ways that require empirical measurement rather than reasoning from first principles.

Context requirements. Applications that need to incorporate user-specific data, organisational knowledge, or real-time information cannot rely on the base model's training data. Retrieval infrastructure becomes a first-class engineering concern. The quality of the retrieval pipeline determines the quality of the generated output more consistently than prompt engineering at the margin. A well-tuned prompt cannot compensate for systematically poor retrieval results.

Evaluation gap. The standard quality signal for non-AI API integrations is: does the response conform to the schema? For LLM integrations, schema conformance is necessary but not sufficient. The model can return well-formed JSON containing factually incorrect, incomplete, or policy-violating content. Evaluation infrastructure that detects these failure modes must be built deliberately and maintained as an engineering artefact, not treated as a manual QA step.

The core integration patterns

Four primary patterns cover the majority of production LLM integration scenarios. The appropriate choice depends on context availability, latency requirements, and the acceptable engineering overhead for the use case:

Direct prompting. A structured prompt is constructed programmatically and sent to the model. The model returns a completion that is parsed, validated, and passed downstream. Appropriate for classification, structured extraction, summarisation, and single-turn generation tasks where inputs are deterministic and controlled. The engineering surface area is manageable; the primary risks are prompt drift and model version regression. Direct prompting is the correct starting point before reaching for more complex patterns.

Retrieval-Augmented Generation (RAG). External documents, database records, knowledge chunks, or structured data are retrieved at inference time and injected into the prompt. The model generates its response based on retrieved context rather than — or in addition to — training knowledge. The dominant pattern for knowledge-intensive applications. Requires retrieval infrastructure (embedding model, vector store, retrieval logic) that is distinct from the LLM infrastructure and must be engineered, evaluated, and maintained independently.

Tool-calling and function-calling. The model is provided with a schema of available functions and can request their execution as part of its reasoning process. The executing system handles tool calls and returns structured results for the next model step. Foundation for agent architectures. Appropriate when the model needs to access real-time data, perform calculations, or take actions that cannot be embedded in the prompt directly.

Fine-tuning. The base model is further trained on domain-specific data to adjust output style, increase domain knowledge density, or improve instruction-following behaviour for a specific task. High engineering overhead; the most warranted when direct prompting, RAG, and structured prompting have reached their quality ceiling for a specific task. Fine-tuning is often proposed prematurely — before the retrieval and prompt engineering approaches have been genuinely optimised.

Retrieval-Augmented Generation (RAG) in production

RAG is the most widely applicable pattern for knowledge-intensive LLM applications and the most consistently underscoped. The basic prototype — embed documents, retrieve top-k, inject into prompt — is straightforward to build and systematically insufficient for production at any meaningful scale.

Retrieval quality is the primary variable. The model generates based on what it receives. Poor retrieval — wrong chunks, semantically irrelevant results, incomplete passages — produces confidently wrong answers. The model does not flag retrieved context as questionable; it synthesises from it. Poor retrieval quality produces not uncertainty but confident inaccuracy. Retrieval pipeline quality controls output quality more reliably than prompt tuning at the margin.

Chunking strategy determines retrievability. Documents must be segmented at boundaries that preserve semantic coherence. Chunks that begin and end mid-sentence, split tables across segment boundaries, or strip contextual labels — "the above applies to..." — produce unreliable retrieval regardless of embedding model quality. Chunking is a document structure analysis problem, not a parameter to be set once and forgotten.

Reranking improves precision. Top-k vector search retrieves semantically similar chunks, not necessarily the most relevant ones for a specific query. A reranker model — evaluating retrieved candidates against the query in a second pass — improves precision materially, particularly for long-tail or ambiguous queries. The latency cost of reranking is typically justified when retrieval precision below 80% produces unacceptable output quality.

Metadata filtering scopes retrieval. In large knowledge bases, pure semantic search cannot scope retrieval to relevant document categories, time ranges, access levels, or content types. Structured metadata filtering in combination with vector search is necessary for production accuracy in multi-domain or access-controlled knowledge bases.

Context window management is a system design concern. Retrieved chunks must fit within the model's context window alongside system instructions, user message history, and response capacity. Context budget management — deciding what to include, what to compress, and what to exclude when the budget is constrained — is a system design constraint that should be addressed in architecture, not handled as a runtime exception.

Evaluation pipelines and output validation

An LLM integration without a systematic evaluation framework is not production-ready. The evaluation infrastructure must address multiple failure categories independently, because failure modes do not correlate — a system can pass factual accuracy checks and fail format compliance, or pass content policy checks and fail completeness.

Evaluation dimensions that must be addressed independently:

Regression testing upon model version changes is non-negotiable. Prompt behaviour that is stable under one model version commonly degrades under a successor, including in ways that are not immediately obvious from surface-level inspection. Version-locking, staged migration, and evaluation comparison between versions before traffic migration are standard practice in production LLM platforms.

Operability and cost management

LLM API costs are not hardware costs — they scale with token consumption in ways that must be modelled before launch. Applications with unmanaged LLM call volumes can generate costs that exceed revenue in early growth phases. Cost engineering is not a post-launch optimisation; it is an architecture concern.

Caching. Exact-match key caching for deterministic requests; semantic caching via vector similarity for near-identical requests where the same response is appropriate. Cache hit rates of 20–40% are achievable in many knowledge-intensive applications and reduce API spend proportionally. Cache staleness management requires explicit TTL policy aligned with knowledge base update frequency.

Model routing. Request characteristics — complexity, expected output length, urgency, quality tolerance — can be used to route requests to lower-cost models without material quality degradation for the request category. A tiered routing policy — small model for structured extraction and classification, large model for complex generation and reasoning — is standard cost management practice in systems with heterogeneous request types.

Context compression. Long prompts disproportionately increase cost. Context summarisation at session boundaries, relevant chunk selection rather than broad retrieval, and prompt compression techniques reduce cost-per-call at acceptable quality levels. The trade-off between context completeness and cost is a system design decision, not a tuning parameter.

Observability. Cost anomaly detection, latency tracking across the full request pipeline, error rate monitoring by failure category, and output quality flagging must be implemented before traffic scales. Reactive cost investigation after budget overrun is a worse outcome than the engineering investment in structured monitoring before launch.

Logic Grid Studio's Software Development and AI Systems services address LLM integration as a complete engineering problem — from pattern selection and architecture through retrieval design, evaluation frameworks, and production readiness. The Services page covers how this fits within the broader AI delivery offering.