Spring AI and LangChain4j: Enterprise Java AI Applications and AI Agent Architecture

Verification and reading baseline: This article was verified on 2026-05-15. The Java baseline is JDK 26 GA / 26.0.1 update line, JDK 25 LTS, and JDK 27 EA. Spring AI, LangChain4j, GraalVM Native Image, cloud model services, vector databases, and model gateways are fast-moving technologies. Exact package names, overloads, dependency coordinates, configuration properties, default behavior, and provider features must be checked against the project version used by the reader. This article separates “official API direction”, “sample wrapper”, and “conceptual pseudo-code”. Official API direction means the concept exists in current documentation. Sample wrapper means application glue code written for explanation. Conceptual pseudo-code means an architecture strategy, not production API.

Abstract

This article answers one question: when an enterprise already has Java services, Spring Boot applications, identity systems, audit trails, transaction workflows, knowledge bases, ticketing platforms, CRM systems, and operations discipline, how should it introduce large model capabilities without turning a chat demo into an unsafe “AI platform”? Java’s value in the AI era is usually not training foundation models or owning every GPU inference path. Its stronger role is to own enterprise boundaries: identity, tenant isolation, permissions, transactions, auditability, data classification, cost budgets, SLOs, rollout control, rollback paths, compliance evidence, and reliable integration with existing systems.

Spring AI and LangChain4j both help Java applications call models, use RAG, register tools, and manage conversational context, but they operate at different levels. Spring AI is close to Spring Boot configuration, auto-configuration, ChatClient, Advisors, Tool Calling, VectorStore, Observability, Micrometer, Actuator, and the surrounding production model. It fits teams whose production estate is already Spring. LangChain4j is closer to a framework-neutral AI application toolkit. It emphasizes AI Services, Tools, Memory, retrieval, RAG, and agent-style orchestration. DJL, ONNX Runtime, local inference services, Python/C++/Rust model servers, and enterprise model gateways are not merely competitors. They belong to model execution, governance, or specialized inference layers.

API snippets are not the main source of architecture value. Readers should first understand system layers, boundaries, failure modes, and governance paths. Code exists only to clarify a boundary. Even if every code block remains collapsed or unread, the reader should still understand the complete engineering argument.

Keywords: Spring AI, LangChain4j, RAG, Tool Calling, AI Agent, Chat Memory, VectorStore, enterprise AI, Prompt Injection, model gateway, observability, cost governance

1. Java’s Role in the AI Era: Not Training Models, but Owning Enterprise Governance Boundaries

This section answers why enterprise Java teams should not understand AI adoption as “replace one SDK with another SDK and call a model.” The reader should be able to decide which AI responsibilities belong in Java services, and which should be delegated to model gateways, dedicated inference services, data platforms, or Python/C++/Rust ecosystems. The production boundary is clear: Java is strong at governance, integration, and business orchestration. That does not mean it should train foundation models or host every high-performance inference workload directly.

An enterprise AI application crosses at least seven boundaries. The identity boundary answers who the user is, which tenant they belong to, which roles they hold, which knowledge base they may read, and which tools they may trigger. The data boundary answers what sensitivity level is present in the question, retrieved context, tool parameters, and model output, and whether that data may enter an external model, an internal model, or no model at all. The execution boundary answers whether the model only generates advice or may call order, refund, approval, ticketing, code, deployment, or notification tools. The reliability boundary answers what happens when the model provider times out, the vector store becomes slow, a tool fails, or a supplier is rate limited. The cost boundary answers who pays for token usage, which business line owns the budget, when to route to a cheaper model, and when to reject a long-context request. The compliance boundary answers what must be retained, redacted, approved, or audited. The evolution boundary answers whether model providers, framework APIs, vector stores, and cloud services can change without rewriting business logic.

Java’s advantage is exactly in those boundaries. Most enterprises already use Java for identity, account systems, order systems, payments, CRM, ERP, ticketing, logging, tracing, configuration, and release control. If an AI system bypasses those boundaries, it may demo quickly, but it will become hard to audit, hard to roll back, expensive to run, and unsafe to trust. A better architecture is not “let the model take over the business system.” It is “let the model operate inside boundaries that the Java system defines, observes, and can roll back.”

Layer	Main responsibility	Java fit	Production judgment
Application entry	Auth, tenant, input validation, rate limit, idempotency key	High	Usually belongs in Java/Spring gateway or service code
Orchestration	Prompt templates, tools, RAG, memory, model routing	High	Java is a good place for business context and transaction boundaries
Tool layer	Business APIs, order lookup, ticket updates, approval workflows	High	Must be wrapped by permissions, idempotency, and audit
Retrieval layer	Document governance, chunking, embeddings, ACL filtering, citations	High	Vector databases are components; governance must remain visible
Model execution	Remote LLM, local inference, specialized model serving	Medium	Java can call this layer, but large inference may need specialized services
Training and fine-tuning	Data pipelines, GPU training, evaluation data construction	Low to medium	Often owned by Python/data platforms; Java integrates and governs
Governance and observability	Audit, evaluation, cost, SLO, security policy	High	Should integrate with enterprise operations systems

The conclusion is that enterprise Java AI is mainly a system boundary problem. The next section places Spring AI, LangChain4j, DJL, model gateways, and local inference in one ecosystem map so that framework selection does not get confused with architecture selection.

2. Java AI Ecosystem Map: Spring AI, LangChain4j, DJL, Model Gateways, and Local Inference Boundaries

This section answers where Java AI frameworks and components belong. The reader should be able to place Spring AI, LangChain4j, DJL, Quarkus, Micronaut, model gateways, local inference, and Python/C++/Rust inference services on a responsibility map. The production boundary is that framework capability is not platform capability, API wrapping is not data governance, and calling a model is not the same as safely shipping an enterprise AI system.

Spring AI is strongest when the application already lives in the Spring ecosystem. It brings model abstractions, ChatClient, Prompt, Advisors, Tool Calling, VectorStore, Observability, and Spring Boot auto-configuration into a familiar programming model. For Spring Boot services, model calls can enter configuration management, dependency injection, Actuator, Micrometer, tracing, AOP, MVC/WebFlux, security interceptors, and release discipline. Spring AI is a natural fit when AI is a new capability inside an existing Spring production estate.

LangChain4j is stronger as an application-level AI abstraction library. It offers AI Services, Tools, Chat Memory, retrieval-augmented generation, structured output patterns, and agent-style composition. It can be used inside or outside Spring. It fits lightweight services, library code, non-Spring applications, or systems that want explicit orchestration. Its risk is also clear: the more flexible it is, the more the team must provide permissions, audit, cost control, observability, and release governance itself.

DJL and ONNX Runtime are closer to the inference execution layer than to enterprise orchestration. They help Java applications load models, run local inference, or connect to deep learning engines. Large language model inference, however, is often dominated by GPU scheduling, batching, KV cache, quantization, memory pressure, and inference server behavior. Many enterprises end up with Java orchestration plus dedicated inference services: Java owns identity, policy, audit, tools, and workflow; specialized services own model throughput and hardware adaptation.

Quarkus and Micronaut matter when startup time, memory footprint, Native Image, or serverless constraints dominate. They are not automatically better or worse than Spring. The right choice depends on runtime model, team expertise, dependencies, monitoring, operational tooling, and deployment constraints.

An enterprise model gateway is often a central layer. It can centralize provider access, secret management, quotas, audit, content safety, cost attribution, fallback, retry, model switching, and regional policy. Without a gateway or equivalent policy layer, every business service must implement provider differences, key handling, and cost control by itself. The gateway can be a standalone platform component or a policy layer around Spring AI/LangChain4j clients, but its responsibility must be explicit: the gateway governs model access; business services govern business permissions and tool execution.

Component	Main position	Solves well	Should not replace
Spring AI	Spring application orchestration layer	Spring Boot integration, ChatClient, Advisors, Tool Calling, VectorStore, observability	Full enterprise data governance
LangChain4j	Java AI application toolkit	AI Services, Tools, Memory, RAG, agent-style orchestration	Central identity and operations platform
DJL / ONNX Runtime	Inference execution layer	Local model inference, specialized model integration	Enterprise AI governance
Model gateway	Model access governance layer	Provider routing, quotas, secrets, safety, cost	Business permission decisions
Python/C++/Rust inference service	High-performance model serving layer	GPU inference, batching, quantization, model serving	Java business workflow orchestration
Quarkus/Micronaut AI	Lightweight service framework layer	Fast startup, native images, cloud-native runtimes	Low-cost migration for existing Spring estates

The conclusion is that technology selection should start from responsibility boundaries, not API syntax. The next section defines the core enterprise AI architecture: model layer, orchestration layer, tool layer, RAG layer, memory layer, evaluation layer, governance layer, and observability layer.

3. Core Enterprise AI Application Architecture: Model, Orchestration, Tools, RAG, Memory, Evaluation, Governance, and Observability

This section answers which layers a production-grade AI application must make explicit. The reader should be able to draw an AI service boundary and know what evidence each layer must provide before production. The production boundary is that this is not a fixed product architecture. Small systems may merge layers, but they must not erase responsibilities.

The first layer is model access. It handles provider, model version, region, credentials, timeouts, retries, rate limits, content safety, response format, and metadata. Business code should not be deeply coupled to a provider SDK. If it is, changing models, adding audit, controlling cost, or enforcing regional policy becomes hard. Spring AI ChatClient and model abstractions can hide part of provider variation. LangChain4j ChatModel and AI Services can do similar work. The enterprise boundary still requires explicit routing and failure strategy.

The second layer is orchestration. Orchestration decides how to build the prompt, whether RAG is needed, whether tools are allowed, whether memory is used, whether human approval is required, whether structured output is required, and what happens on failure. It should not be a random prompt concatenation function. A “customer return assistant” should express a business workflow that uses identity, order state, policy documents, approval rules, and audit. It should not just call a model from a controller.

The third layer is tools. Tool Calling is not “let the model invoke Java methods.” It is exposing enterprise capabilities through controlled interfaces. Every tool needs permission checks, schema validation, idempotency, timeouts, audit, rollback, and approval policy. Reading an order and issuing a refund are not the same risk. Querying inventory and modifying inventory are not the same risk. Tool governance failures are among the most common ways AI agents break when moving from demos to production.

The fourth layer is RAG. RAG is not “chunk documents and put them into a vector database.” Enterprise RAG includes document ownership, versioning, ACLs, chunking strategy, embedding model selection, indexing, metadata, retrieval, reranking, citations, answer constraints, evaluation sets, and hallucination control. The vector database is a component. Data governance and evaluation determine quality.

The fifth layer is memory. Chat Memory is not only chat history. It includes short-term session state, long-term preferences, cross-session summaries, tenant isolation, privacy policy, deletion, compression, and token budgets. Bad memory design causes privacy leakage, context pollution, cost growth, and behavior drift.

The sixth layer is evaluation. Enterprises cannot validate AI only by reading a few good-looking answers. Evaluation should include retrieval recall, citation correctness, refusal correctness, permission filtering, tool-call correctness, latency, cost, regression samples, and human review.

The seventh layer is governance and observability. Governance defines policy. Observability provides evidence. The system must answer who initiated the request, what context the model saw, which tools were offered, which tools executed, which sources supported the answer, what it cost, and whether any policy was violated.

Layer	Key design question	Typical failure mode	Required evidence
Model access	Which model, route, timeout, and fallback?	Provider SDK leaks into business code	Routing policy, timeout policy, cost metrics
Orchestration	When to use RAG, tools, memory, approval?	Prompt glue becomes business logic	Workflow, failure path, rollback plan
Tool layer	Which capability may the model trigger?	Unauthorized or repeated side effect	Permission matrix, idempotency key, audit log
RAG layer	Which source may support the answer?	Wrong, stale, or unauthorized retrieval	Document version, evaluation set, citation chain
Memory layer	What is remembered, for how long, for whom?	Privacy leakage or stale context	Retention, tenant isolation, deletion policy
Evaluation layer	How do we know quality changed?	Demo-based release	Offline evaluation, human review, regression set
Governance/observability	How can we explain and limit behavior?	Cost, safety, and audit gaps	Metrics, traces, audit events, alerts

3.1 Architecture Review Starts with Responsibility Boundaries, Not Framework APIs

The first architecture review question should not be “which model” or “which framework.” It should be: who is the real user, how is the tenant identified, which data enters the model, which RAG fragments are used, which tools may run, which tool changes business state, what audit trail is written, how the system degrades, how cost is attributed, and how each version can be rolled back.

Reviewers should separate model responsibility from system responsibility. The model may generate, summarize, classify, explain, suggest, plan, or choose a tool candidate. It should not be the final authority for permissions, transaction commits, compliance exceptions, or audit retention. Those decisions belong to deterministic systems. The model output can be input to those systems, not a replacement for them.

3.2 AI Data Flow Must Be Drawable, Measurable, and Interruptible

Enterprise AI data flow includes user input, system instructions, developer instructions, RAG fragments, memory fragments, tool results, model plans, final answers, and logs. Each data type has a different trust level. User input is untrusted. Retrieved content may be sourced but still adversarial. Tool results may be true but sensitive. Memory may be useful but stale. Model plans are non-deterministic. Mixing all of them into one string without boundaries is a design flaw.

Drawable means the team can show where data comes from, where it goes, where it is redacted, filtered, audited, and possibly sent outside the boundary. Measurable means each stage has metrics: fragment count, ACL hits, token length, tool count, safety blocks, citation correctness. Interruptible means the system can stop safely: return citations only, refuse, route to a human, disable tools, downgrade the model, or roll back an index.

The conclusion is that enterprise AI architecture is not about whether a model can be called. It is about whether each model action remains explainable, restricted, and reversible inside a business system.

4. Spring AI Deep Dive: Official API Direction, Advisors, Tool Calling, VectorStore, and Observability

This section answers what role Spring AI should play in enterprise Java applications. The reader should be able to decide which capabilities can be handled by Spring AI and which must be supplied by enterprise policy. The production boundary is that this article describes documented API directions and architecture responsibilities. Exact coordinates, packages, overloads, and property names must be checked against the locked project version.

Spring AI’s core value is bringing AI into the Spring application model. ChatClient makes model calls look like service calls. Model abstractions hide parts of provider variation. Advisors allow cross-cutting behavior around model requests. Tool Calling exposes Java capabilities to a model. VectorStore provides a common direction for vector storage. Observability integrates model operations with Micrometer, Actuator, and tracing. For enterprise applications, this matters because production systems need more than model invocation. They need configuration, monitoring, audit, and release control.

ChatClient is a useful application entry point for model interaction, but it should not become the business boundary. Business services should wrap ChatClient and handle user identity, tenant scope, input sanitation, data classification, model routing, timeout, retry, refusal policy, and audit. Calling ChatClient directly from controllers is easy to demo and hard to govern.

Advisors are valuable because memory, retrieval, safety, logging, cost, and observability are cross-cutting concerns. The order of advisors is a security decision. If unauthorized RAG content is injected before filtering, filtering later is too late. If full prompts are logged before redaction, the log system becomes a leak. Good advisor design usually identifies the user first, classifies data, filters retrieval scope, assembles context, calls the model, validates output, and records audit summaries.

Tool Calling in Spring AI is useful, but tool production semantics matter more than syntax. A weather lookup tool and a refund tool are not the same. Enterprises must define whether a tool is read-only, state-changing, repeatable, approval-gated, reversible, auditable, and safe to expose automatically.

VectorStore abstractions help connect to vector databases, but they do not solve document governance. Enterprise RAG still needs source ownership, metadata, permissions, versions, citations, and evaluation.

Observability is not optional. At minimum, the system should record request count, provider, model, latency, error type, input/output tokens, RAG hit count, tool-call count, rate-limit events, fallback events, and safety blocks. A single user request may include auth, retrieval, model call, tool call, second model call, and post-processing. The trace must preserve that chain.

The following sample wrapper is not complete production code. Scenario: wrap a model call inside a business service. Reason: keep permissions, redaction, audit, and model access in one governable boundary. Observation point: the controller does not call the model directly. Production boundary: real systems still need real identity, redaction, timeout, retry, and durable audit storage.

@Service
class EnterpriseAiAssistant {
    private final ChatClient chatClient;
    private final PolicyService policyService;
    private final AuditSink auditSink;

    Answer answer(UserContext user, Question question) {
        policyService.assertCanAsk(user, question.scope());
        SanitizedQuestion sanitized = policyService.sanitize(question);

        String content = chatClient.prompt()
            .system("Answer only from approved enterprise context.")
            .user(sanitized.text())
            .call()
            .content();

        auditSink.record(user.tenantId(), question.id(), "model-answer", content.length());
        return new Answer(content);
    }
}

4.1 Advisor Chain Order Determines Data Safety

Advisor order must be designed and tested. A memory advisor that injects old private context before tenant filtering can leak information. A RAG advisor that adds documents before permission filtering can expose restricted fragments. An audit advisor that stores full prompts before redaction can leak sensitive data. Cross-cutting code is powerful because it is central, but that also makes mistakes systemic.

A healthy advisor chain separates responsibilities. A memory advisor reads only scoped memory. A RAG advisor injects only ACL-filtered and cited fragments. A safety advisor classifies and blocks sensitive content. An audit advisor stores summaries and identifiers unless a protected audit store is explicitly required. Avoid a single “magic advisor” that handles memory, retrieval, safety, and logging in one opaque block.

4.2 Spring Boot Integration Should Focus on Operations, Not Just Developer Experience

Spring Boot integration matters because AI calls are remote, costly, and risky dependencies. Model credentials should not live in source code. Model names and timeouts should not be hard-coded in controllers. Prompt templates need versions. Provider errors need classification. Token usage needs cost attribution. Logs need redaction by default.

Treat model calls like database, messaging, and external HTTP dependencies. Configuration should be environment-specific. Secrets should live in secret managers. Metrics should go to the observability platform. Traces should connect model calls to user requests. Audit events should be queryable. Prompts and model routes should be released and rolled back like other configuration.

4.3 Spring AI and the Enterprise Model Gateway

Spring AI can call providers directly, or it can call an enterprise model gateway. Direct provider access is workable for small, low-risk internal use. A gateway becomes necessary when the organization needs multiple providers, centralized secrets, quotas, cost attribution, content safety, regional policy, audit, and model rollout control.

The gateway owns model access policy. The business service still owns business authorization and tool decisions. If the gateway tries to understand every business permission, it becomes an over-centralized business system. If every business service owns model secrets and cost policy, governance fragments. The integration contract must include error classes, model metadata, token usage, safety blocks, and cost fields so that Java services can explain outcomes.

The conclusion is that Spring AI can bring AI into the Spring engineering boundary, but it does not replace enterprise policy.

5. LangChain4j Deep Dive: AI Services, Tools, Memory, RAG, and Agent Orchestration Boundaries

This section answers what LangChain4j should own in a Java AI architecture. The reader should be able to decide when to choose LangChain4j, when to choose Spring AI, and when both can coexist. The production boundary is that LangChain4j provides application abstractions, but the enterprise must still supply permissions, audit, rate limits, cost control, and evaluation.

LangChain4j’s appeal is that it combines model calls, tools, memory, retrieval, and AI Services into familiar Java abstractions. AI Services map interfaces to model interactions. Tools let models request Java capabilities. Memory maintains conversational state. RAG components connect retrieval to answers. Agent-style orchestration allows the model to choose steps under a workflow. This is valuable for non-Spring services, lightweight applications, libraries, and explicit orchestration code.

AI Services are attractive because business interfaces become clear. The risk is that they look like deterministic Java services while being backed by probabilistic model calls. A service that returns a policy answer should probably return citations, refusal state, confidence, and audit metadata, not only a string. A ticket triage service may suggest a priority, but the final priority may still belong to rules or a human.

Tools in LangChain4j need the same risk classification as Spring AI tools. Tool registration is not just exposing methods. It is packaging enterprise capabilities as controlled actions. Each tool needs risk level, permissions, schema, idempotency, timeout, retry, approval, audit, and rollback.

Memory must be scoped. Short-term memory is useful for a session. Long-term memory must obey tenant isolation, deletion, privacy, and retention rules. Retrieval memory can help long-running tasks, but it can also retrieve stale or unauthorized facts.

Agent orchestration is easily overused. Multi-step reasoning is useful when bounded by maximum steps, tool whitelist, budget, approval checkpoints, state transitions, and abort conditions. Enterprise agents should not be “the model explores business systems freely.” They should be “the model helps inside a finite, audited workflow.”

The following conceptual pseudo-code explains tool governance rather than a complete framework API. Scenario: the model requests a read-only order lookup. Reason: let the assistant answer support questions without direct database access. Observation point: permission and audit happen inside the tool. Production boundary: write tools need idempotency keys, approval, and rollback.

class OrderLookupTool {
    @Tool("Look up an order that the current user is allowed to view")
    OrderSummary findOrder(ToolContext context, String orderId) {
        policy.assertCanReadOrder(context.user(), orderId);
        OrderSummary summary = orderService.findSummary(orderId);
        audit.record(context.requestId(), "order.lookup", orderId);
        return summary.withoutSensitiveFields();
    }
}

5.1 AI Services Are Not Ordinary Java Services

An AI Service can expose a clean interface, but the behavior behind it may change when prompts, models, tools, memory, or RAG indexes change. Therefore responses should carry metadata: prompt template version, model version, citations, refusal reason, safety blocks, and audit ID. Otherwise a stable Java method signature can hide unstable model behavior.

5.2 LangChain4j Enables Explicit Orchestration, but Explicit Does Not Mean Unbounded

Explicit orchestration should track three lines: business state, risk state, and evidence state. Business state says whether the task is collecting information, retrieving evidence, waiting for approval, executing a tool, verifying results, or delivering an answer. Risk state says whether sensitive data or side effects are involved. Evidence state says which inputs and outputs were used. Without these lines, multi-step agents become opaque.

5.3 Framework Selection Anti-Patterns

One anti-pattern is forcing every AI workflow into Spring AI only because the organization uses Spring. Some modules may be libraries, batch jobs, or lightweight services. Another anti-pattern is using LangChain4j to bypass Spring operations in a production estate that already relies on Actuator, Micrometer, tracing, configuration, and audit. A third anti-pattern is copying framework examples as enterprise architecture. Minimal examples are not a data-governance strategy.

The conclusion is that LangChain4j is flexible, and flexibility increases the need for explicit governance.

6. Enterprise RAG: Document Governance, Chunking, Embeddings, Vector Stores, ACL Filtering, Retrieval, Reranking, Citations, and Evaluation

This section answers why enterprise RAG is not “vector database plus top-k.” The reader should be able to design a closed loop from document admission to cited answers and quality evaluation. The production boundary is that RAG should make answers traceable, authorized, evaluable, and correctable.

The first step is document governance. Documents need source, owner, update time, version, tenant, permission, confidentiality level, business domain, and expiration policy. Without this metadata, retrieval results cannot be judged safe for the current user. Many RAG failures are data governance failures: old and new policies coexist, internal drafts mix with public docs, tenant data shares one index, or personal notes become official answers.

The second step is parsing and chunking. Contracts, policies, API docs, incident runbooks, code standards, and FAQs need different chunking. Good chunking preserves heading path, table meaning, list semantics, context references, and version information. Chunks that are too small lose meaning. Chunks that are too large reduce precision and increase cost. Chunking strategy should be evaluated, not guessed.

The third step is embeddings and indexing. Embedding models must match language, domain, and query style. Chinese, English, code, tables, logs, legal text, and support conversations may need different handling. Vector-store selection should consider metadata filtering, hybrid search, index updates, deletion, backup, isolation, audit, cost, and operations.

The fourth step is permission filtering. Enterprise RAG should follow “filter before context.” Retrieval may filter by tenant, user, role, document status, and data domain before model context is built. Post-retrieval checks can add defense, but unauthorized content must not be injected into the prompt and then trusted not to leak.

The fifth step is retrieval, reranking, and citations. Top-k is not enough. Many systems need keyword retrieval for exact terms, vector retrieval for semantic similarity, metadata filtering for permissions, reranking for precision, and citations for traceability. Answers should distinguish supported answers, insufficient evidence, and questions that require live tools.

The sixth step is evaluation. RAG evaluation should include retrieval recall, citation correctness, answer correctness, refusal correctness, hallucination rate, permission correctness, latency, and cost. Evaluation samples should come from real questions, incident questions, boundary questions, and malicious questions.

RAG stage	Architecture question	Production boundary	Failure mode	Evidence
Document governance	Which sources may enter?	Source, version, ACL, tenant, sensitivity	Stale or unauthorized content retrieved	Document inventory
Chunking	How is meaning preserved?	Type-specific chunks and heading path	Context split or noisy chunks	Chunk evaluation samples
Embedding	Does the model fit language and domain?	Model version tied to index	Poor Chinese, code, or table retrieval	Recall test set
Permission filtering	What can this user see?	Filter before context injection	Cross-tenant leakage	ACL audit logs
Reranking and citations	Which evidence supports the answer?	Citation to source and version	Correct-sounding but unsupported answer	Citation accuracy
Evaluation	How do changes prove better?	Regression on every strategy change	Demo-based release	Evaluation report

6.1 Document Lifecycle Determines RAG Quality

Documents are not one-time assets. They are governed objects. Creation records owner, source, tenant, sensitivity, and effective date. Updates trigger parsing, chunking, embedding, and index versioning. Deletion or expiration removes content from retrieval. Permission changes update metadata filters. Wrong answers must trace back to a document version and chunk.

6.2 Chunking Strategy Must Follow Document Type

Policies need conditions and exceptions. API docs need method names, parameters, return values, and examples. Incident runbooks need symptoms, checks, steps, and rollback. Tables must not be split away from column meaning. Good chunking often stores heading path, local summary, metadata, and source anchors.

6.3 Citations and Refusals Are Trust Mechanisms

Citations are not decoration. They let users verify and owners fix sources. Refusal is also a correct behavior when evidence is missing, unauthorized, conflicting, or out of scope. A system that refuses correctly is more trustworthy than a system that always answers.

The conclusion is that RAG quality comes from document governance and evaluation, not from vector APIs alone.

7. Tool Calling and AI Agent: Permissions, Idempotency, Approval, Sandbox, Audit, and Failure Rollback

This section answers how to let a model request tools without letting the enterprise system lose control. The reader should be able to design tool risk levels, execution strategy, and audit. The production boundary is that an AI Agent is not a free executor. It is a bounded state machine, tool whitelist, and approval workflow.

Tools should be classified. L0 tools are pure computation or formatting. L1 tools are read-only queries. L2 tools are low-risk writes such as draft creation. L3 tools are high-risk writes such as refunds, permission changes, external notifications, deployments, or code merges. L4 tools are dangerous operations and usually should not be directly available to the model.

Permission checks must happen before execution. The tool layer must receive real user context, tenant, role, and request source. Tool parameters must be validated. External HTTP tools should restrict domains, methods, request body, response size, and data classification.

Idempotency is essential for write tools. Models may call a tool repeatedly because of retries, timeouts, concurrent turns, or multi-step reasoning. Each write should accept an idempotency key and log model request ID, user, tool name, parameter summary, approval state, execution result, and rollback link.

Human approval is a normal part of enterprise agents. For high-risk tools, the model can prepare the recommendation, reason, parameters, and impact analysis, but execution should wait for a human or rule approval. The approval UI should show evidence, citations, risk level, rollback path, and expected impact.

Sandboxes matter for code execution, file processing, web browsing, and data analysis tools. A sandbox should restrict network, filesystem, CPU, memory, runtime, and accessible data. Sandbox output should be redacted and size-limited before it returns to model context.

Tool level	Example	Automatic execution	Required control
L0 pure computation	Formatting, conversion	Yes	Input size and exception handling
L1 read-only	Order lookup, document lookup	Conditional	User permission, tenant filter, audit
L2 low-risk write	Create draft, prepare suggestion	Rule approval possible	Idempotency, versioning, rollback
L3 high-risk write	Refund, notify, change status	Usually human approval	Dual confirmation, audit, compensation
L4 dangerous operation	Delete data, change security policy	Should not be directly registered	Isolation or prohibition

The following conceptual pseudo-code shows a tool execution gateway. Scenario: a model requests a business tool. Reason: centralize permission, idempotency, approval, and audit. Observation point: the tool body does not trust model parameters directly. Production boundary: real systems also need transactions, retries, compensation, and security tests.

final class ToolExecutionGateway {
    ToolResult execute(UserContext user, ToolRequest request) {
        ToolPolicy policy = registry.policyOf(request.toolName());
        policy.assertAllowed(user, request.arguments());

        if (policy.requiresApproval()) {
            return approvals.createPending(user, request, policy.riskLevel());
        }

        String idempotencyKey = request.idempotencyKey();
        return idempotency.runOnce(idempotencyKey, () -> {
            audit.before(user, request, policy.riskLevel());
            ToolResult result = registry.invoke(request);
            audit.after(user, request, result.summary());
            return result;
        });
    }
}

7.1 Enterprise Agents Should Be State Machines, Not Infinite Loops

Production agents should be finite state machines: collect intent, confirm identity, retrieve evidence, generate a plan, request approval, execute tools, verify results, deliver output, and archive audit evidence. Each state should define allowed tools, maximum steps, timeout, failure transition, and human handoff.

7.2 Human-in-the-Loop Is Workflow, Not Failure

Human approval is not a sign that AI failed. Many business actions already require approval. AI can prepare evidence, fill forms, explain impact, and recommend actions. The human approves or rejects with context. Rejections become improvement signals for prompts, tools, and policy.

7.3 Tool Catalogs Should Be Governed Like API Products

Each tool needs a name, purpose, risk level, input schema, output schema, permission requirement, idempotency strategy, timeout, retry rule, approval requirement, rollback capability, audit fields, owner, and deprecation policy. A broad executeAction tool is not governable. A narrow createRefundDraftForApprovedOrder tool is.

The conclusion is that agent value comes from controlled automation, not from letting a model do whatever it wants.

8. Chat Memory and Context Management: Short-Term Memory, Long-Term Memory, Tenant Isolation, Privacy, Summarization, and Token Budgets

This section answers what an AI application should remember, forget, and compress. The reader should be able to design short-term memory, long-term memory, summary memory, and retrieval memory boundaries. The production boundary is that memory is a data system, not a List<Message>.

Short-term memory maintains continuity within one session. It should usually be limited by messages or tokens and cleared when the session ends or expires. It improves interaction but should not become durable business state.

Long-term memory stores stable preferences, authorized facts, or business-state summaries. It needs write rules, scope, deletion, encryption, and audit. A low-risk preference such as “answer in Chinese” is different from customer lists, contract values, health data, or regulated records.

Summary memory compresses long conversations. It should preserve task state, confirmed facts, unresolved questions, citations, tool results, and risk notes. Incorrect summaries can be more dangerous than no memory because the model treats them as truth.

Retrieval memory stores historical fragments in an index. It can support support cases, project context, or long-running tasks, but it needs tenant isolation, authorization, time decay, and deletion.

Token budget is a hard architecture constraint. System instructions, safety rules, current question, RAG fragments, memory, tool results, and output length compete for the same context window. The system should preserve constraints and evidence by priority, not blindly append history.

Memory type	Use case	Main risk	Governance strategy
Short-term window	Continuous conversation	Token growth, sensitive replay	Message/token limit, session expiration
Long-term preference	Stable user preference	Privacy leakage, cross-tenant pollution	Consent, deletion, encryption
Summary memory	Long task compression	False summary becomes fact	Source trace, correction path
Retrieval memory	Historical cases and context	Stale or unauthorized retrieval	Metadata filters, time decay
Tool-result memory	Multi-step task state	Inconsistent state or repeated execution	Idempotency, transaction state, audit

8.1 Memory Writes Must Be Policy-Controlled

The model can propose memory writes, but policy should decide what may be saved. The policy should classify facts, preferences, and task state. It should check sensitivity, tenant scope, retention, and user consent. Without this control, the model may save one-time secrets, wrong assumptions, or sensitive information.

8.2 Memory Reads Must Be Scoped to Task and Risk

Reading all related memory is unsafe. Low-risk preferences may be used broadly. High-risk business facts need source, tenant, age, and permission checks. Historical similarity does not mean current applicability.

8.3 Context Compression Must Preserve Decision Evidence

Summaries should preserve evidence IDs, tool results, approval state, and risk notes. If a summary cannot trace back to source messages and tool calls, it is not strong enough for enterprise audit.

The conclusion is that memory is the privacy, cost, and quality boundary of enterprise AI.

9. Cost, Rate Limits, Timeouts, and Reliability: Model Routing, Fallback, Retry, Circuit Breaking, Caching, and Budget Attribution

This section answers how AI applications run under real traffic, provider instability, and budget constraints. The reader should be able to design model routing, cost attribution, rate limits, timeouts, retries, circuit breakers, caching, and fallback. The production boundary is that model services are often slower, more expensive, and less predictable than ordinary internal services.

Cost governance begins at request entry. Each request should record business line, tenant, user, feature, model, input tokens, output tokens, RAG tokens, tool count, cache hit, retry count, and final cost. Cost should be attributable to features and teams, not only to one monthly bill.

Model routing is not sorting by price. High-value tasks may need stronger models. Low-risk classification or summarization can use smaller models. Sensitive data may need internal or local models. Routing should consider task type, data sensitivity, latency budget, cost budget, model capability, region policy, and provider availability.

Rate limits should be layered. Entry rate limits protect users and tenants. Orchestration limits protect maximum steps, tools, and tokens. Model limits protect provider concurrency. Vector limits protect retrieval. Tool limits protect business systems.

Timeouts should be budgeted by stage. A 3-second request cannot let the model consume the entire 3 seconds. Auth, retrieval, reranking, model call, tool call, post-processing, and audit each need budgets. Long tasks should move to asynchronous workflows.

Retries must respect idempotency and cost. Read-only model calls can have limited retries. Write tools need idempotency keys. Rate-limit errors need backoff, fallback, or model switching. Blind retries multiply cost and pressure.

Mechanism	Problem solved	Common mistake	Production practice
Cost attribution	Who spent budget?	Only monthly total	Track by feature, tenant, model, tokens
Model routing	Which model should answer?	Always strongest or cheapest	Route by task, sensitivity, SLO, budget
Rate limits	Prevent overload	Only entry QPS	Limit user, tenant, model, vector store, tools
Timeout	Control tail latency	One global timeout	Stage budgets and fallback
Retry	Handle transient failure	Retry every failure	Retry only safe idempotent paths
Circuit breaker	Stop fault amplification	Keep piling requests	Fast fail, switch model, async handoff
Cache	Reduce cost and latency	Ignore permissions and versions	Key by permission, model, prompt, index, policy

The following sample wrapper explains budget checks. Scenario: check budget before model execution. Reason: prevent high-price models and long contexts from escaping cost control. Observation point: budget is tied to tenant and feature. Production boundary: real systems also need pricing version, currency, provider bill reconciliation, and exception handling.

final class AiBudgetGuard {
    void assertWithinBudget(UserContext user, AiRequest request, TokenEstimate estimate) {
        Budget budget = budgets.forTenantAndFeature(user.tenantId(), request.feature());
        Money estimatedCost = pricing.estimate(request.model(), estimate);
        if (budget.remaining().isLessThan(estimatedCost)) {
            throw new BudgetExceededException(user.tenantId(), request.feature());
        }
    }
}

9.1 Stage Timeouts Beat One Global Timeout

AI requests span auth, retrieval, reranking, model calls, tools, audit, and queues. One global timeout hides where latency was spent. Stage budgets let the system degrade meaningfully: skip reranking, return citations, route to async, or stop tools.

9.2 Retry and Circuit Breaking Must Respect Idempotency and Cost

Model calls may produce different outputs. Tool calls may change state. Retries increase cost. Circuit breakers should move the system into a controlled mode: fallback model, citations only, human handoff, async processing, or high-cost feature suspension.

9.3 Cache Keys Must Include Permissions and Versions

Cache keys must include tenant, permission digest, model version, prompt template version, RAG index version, document version, policy version, and locale. Otherwise one user’s answer can leak to another user, or old policy answers can survive new index releases.

9.4 Cost Metrics Must Be Viewed with Business Metrics

Token cost alone is not enough. A costly assistant that reduces ticket handling time may be worth it. A cheap assistant that gives wrong advice is not. Cost should be correlated with resolution rate, review acceptance, handoff rate, and business outcomes.

The conclusion is that AI reliability is not model intelligence. It is the budget, timeout, rate-limit, fallback, and observability system around the model.

10. Security and Compliance: Prompt Injection, Data Leakage, Unauthorized Tool Calls, Log Redaction, and Audit Trails

This section answers which security risks enterprise AI introduces and how Java applications should contain them. The reader should be able to identify Prompt Injection, data leakage, unauthorized tool calls, log leakage, supplier compliance, and audit gaps. The production boundary is that security cannot rely only on prompt instructions.

Prompt Injection happens when a user or document attempts to override instructions. A retrieved document can contain “ignore previous instructions.” A user can ask the model to call a refund tool. Defense is not just keyword filtering. The system must separate instruction layers, restrict tool permissions, label retrieved content, require citations, validate tool requests, and post-process high-risk outputs.

Data leakage has many paths: sensitive input, unauthorized RAG fragments, cross-tenant memory, full prompt logs, broad tool output, and model responses that repeat sensitive details. Data classification should distinguish data that never enters a model, data that may enter only after redaction, data that may enter internal models, and data that may enter external models.

Unauthorized tool calls are among the highest risks. The model is not the permission subject. The real subject is the user, service account, or approval workflow. Tool execution must use real context and deterministic policy.

Logs need redaction. Full prompt and response logging is convenient during development and dangerous in production. Safer logs record request ID, user or tenant digest, model, token count, fragment IDs, tool names, risk level, error class, and output length. Raw content belongs in protected audit storage only when required.

Audit must answer five questions: who initiated the request, what authorized evidence the model saw, which tools were offered and called, which citations supported the answer, and who approved or executed an action.

Risk	Typical scenario	Code-level control	Audit evidence
Prompt Injection	User or document tries to override rules	Instruction separation, tool pre-checks, content safety	Input source, block reason
Data leakage	Unauthorized RAG fragment enters context	Tenant/ACL filtering, redaction, field whitelist	Fragment ID, permission result
Unauthorized tool	Model requests refund or permission change	User context, risk level, approval	Tool name, parameter digest, approval record
Log leakage	Full prompt lands in normal logs	Summary logs, sensitive-field filtering	Raw-content access audit
Supplier compliance	Data region or retention mismatch	Routing policy, contract constraints	Provider, region, model
Hallucinated compliance	Wrong legal/financial/medical advice	Refusal policy, citations, human review	Citation chain and review record

10.1 Prompt Injection Must Be Handled by Data Source Layer

Prompt Injection can arrive through user text, RAG documents, emails, tickets, logs, code comments, or tool outputs. Do not trust retrieved content simply because it came from an internal document. Treat it as evidence, not instruction.

10.2 Log Redaction Must Cover Prompt, RAG, and Tool Results

Sensitive data often enters through retrieval and tools, not only user input. Contracts, orders, tickets, stack traces, internal URLs, and tokens may enter context. Normal logs should not store full prompt bodies by default.

10.3 Supplier Compliance Belongs in Model Routing

Model provider selection is not an implementation detail. It depends on data class, region, retention, training-use policy, enterprise contract, and audit requirements. Routing should be policy-driven.

10.4 Security Testing Must Include Malicious Natural Language

Security tests should include instruction override attempts, cross-tenant queries, tool abuse, prompt leakage, multi-turn exfiltration, and malicious documents. Tool parameters must be validated as untrusted input.

The conclusion is that prompt text can express intent, but code, identity, data flow, tool gateways, logs, and audit systems enforce boundaries.

11. Enterprise Cases: Customer Support Assistant, Knowledge Base Q&A, Code Review Assistant, Ticket Agent, and Operations Analysis Agent

This section answers how the architecture applies to real business cases. The reader should identify boundary differences instead of treating every AI feature as a chatbot.

Customer support assistants usually need user identity, order state, policy documents, ticket history, and knowledge bases. Low-risk capabilities include explaining policy and drafting suggestions. High-risk capabilities include refunds, compensation, ticket closure, and order modification. The safe architecture is RAG for cited policy, tools for live order state, model-generated advice, and approval for writes.

Knowledge base Q&A tests document governance. The key is not whether information can be retrieved, but whether the document is current, authorized, citable, and owned. Good systems cite sources, refuse when evidence is missing, and expose version conflicts.

Code review assistants should assist, not merge automatically. They can summarize diffs, explain risks, generate test suggestions, and propose patches. They should not bypass deterministic tests, maintainers, or repository permissions.

Ticket agents should be state machines. They can classify incidents, enrich fields, link logs, find similar incidents, recommend owners, and prepare low-risk tasks. Closing P1 incidents, restarting core services, or executing production changes requires approval.

Operations analysis agents cross data warehouses, BI systems, metrics, and business context. They can translate questions into controlled queries and summarize changes. They must not generate arbitrary SQL against production. Use semantic layers, query budgets, permissions, and redaction.

Scenario	Main value	High-risk boundary	Recommended execution
Customer support	Faster, consistent answers	Refunds, compensation, order changes	Auto-draft advice, approve writes
Knowledge Q&A	Fast cited knowledge access	Cross-tenant or stale documents	ACL filtering, version citations, refusal
Code review	Risk discovery and explanation	Auto-merge or script execution	Sandbox analysis, human merge
Ticket Agent	Faster diagnosis and routing	Incident closure, production change	State machine plus approval
Operations analysis	Lower analysis barrier	Unauthorized SQL, cost explosion	Semantic layer, query budget, redaction

11.1 Customer Support Production Path

Start in suggestion mode. Let the model draft responses based on cited policy and read-only tools, with human confirmation. Later, auto-answer low-risk policy questions. Only after evidence exists should the system create drafts or low-risk actions. Refunds and compensation remain approval-gated.

11.2 Knowledge Base Q&A Production Path

Start with a governed knowledge domain. Build document inventory, owner, version, ACLs, citations, and evaluation set. Expand one domain at a time. The release gate is not “the chatbot answers”; it is “unauthorized data is not retrieved, stale content is not used, answers cite sources, and refusal works.”

11.3 Code Review Assistant Production Path

The assistant should record which files and conventions it used. It may suggest fixes, but build and tests remain deterministic gates. For sensitive repositories, external model access must be restricted or replaced with internal models.

11.4 Ticket Agent Production Path

The agent should distinguish facts, hypotheses, and suggestions. It may prepare a change, but production execution requires approval. Metrics include assignment accuracy, mean time to acknowledge, mean time to recovery, reopen rate, and false closure rate.

11.5 Operations Analysis Agent Production Path

Use controlled query services. Require source, time window, metric definition, filters, and confidence. The model must not turn correlation into causation without evidence.

The common pattern is that the model generates, explains, suggests, and orchestrates. Java services own permissions, state, tools, audit, and rollback.

12. Common Problems and Solutions: Hallucination, Poor Retrieval, Context Explosion, Tool Misuse, Cost Runaway, Slow Responses, and Unexplainable Audit

This section answers how to diagnose enterprise AI failures without immediately tuning the prompt. The reader should map symptoms to RAG, model, tool, memory, cost, reliability, or governance layers.

If the problem is hallucination, first check whether the answer needed RAG, whether retrieval found correct sources, whether sources were current, whether the prompt required citations, whether refusal was allowed, and whether the evaluation set contains the case. Many hallucinations come from requiring answers when evidence is missing.

If retrieval is poor, inspect document governance and chunking. Is the document indexed? Is metadata correct? Is ACL filtering too narrow? Does the embedding model fit language and domain? Does the query need rewriting? Is hybrid search or reranking needed?

If context explodes, inspect prompt composition: system instructions, developer instructions, history, RAG fragments, tool results, and output budget. Fix with summarization, layered retrieval, history windows, duplicate removal, state machines, and priority-based truncation.

If tools are misused, inspect tool descriptions, risk levels, schemas, permissions, and approval policies. The real defense belongs in the tool gateway.

If cost runs away, inspect token composition, model routing, retries, cache misses, RAG fragment count, and long outputs. Without attribution, optimization is guesswork.

If responses are slow, split latency by auth, retrieval, reranking, model call, tool call, and post-processing. A single HTTP timeout is not a diagnosis.

If audit is unexplainable, check whether the system records request ID, user, tenant, model, prompt template version, RAG fragment IDs, tool calls, approval state, output summary, and cost.

Symptom	First check	Common root cause	Fix direction
Hallucination	Citations, refusal, retrieval hit	Evidence missing but answer forced	Refusal, citations, better retrieval, evaluation
Poor retrieval	Documents, chunks, embeddings, rerank	Missing metadata or bad chunking	Governance, hybrid search, reranking
Context explosion	Prompt composition and token budget	History/RAG/tool results unbounded	Summaries, windows, state machine
Tool misuse	Tool description, permission, approval	Tool risk not classified	Tool gateway, idempotency, confirmation
Cost runaway	Tokens, routing, retries	Expensive model or long context overuse	Budget attribution, model tiers, cache
Slow response	Stage latency	Model, retrieval, or tool bottleneck	Timeout budget, fallback, async
Unexplainable audit	Trace and audit fields	Missing evidence chain	Audit event model

12.1 From Symptom to Root Cause: Do Not Blame Every Failure on the Model

“The answer is wrong” does not mean “the model is wrong.” It may be document versioning, permission filtering, chunking, retrieval, reranking, tool data, memory summary, prompt constraints, or model capability. A useful incident report records request ID, user, tenant, model version, prompt version, RAG index version, fragments, tools, approvals, cost, latency, and fix action.

12.2 Quality Improvement Loop: Evaluation Sets Matter More Than Demos

Evaluation sets should cover answerable questions, insufficient-evidence questions, permission-denied questions, conflicting documents, and malicious inputs. Tool evaluations should include legal queries, illegal cross-tenant queries, repeated writes, missing idempotency, approval, timeout, and error cases.

12.3 Incident Boundaries: What Must Degrade or Stop

Cross-tenant leakage, unauthorized tool execution, unapproved high-risk writes, sensitive log leakage, compliance violations, and systematic citation errors should trigger degradation or suspension. Degradation can disable tools, return citations only, force human review, switch models, limit tenants, or roll back indexes.

12.4 Seven Architecture Review Questions

Before calling a system “enterprise AI,” ask whether model calls are wrapped by identity and audit, whether RAG has sources and ACLs, whether tools are risk-classified, whether memory has deletion and isolation, whether cost is attributable, whether security handles prompt injection and data leakage, and whether release has evaluation, canary, rollback, version freeze, and incident templates.

12.5 Three End-to-End Incident Chains

Cross-tenant RAG leakage usually starts with retrieval before ACL filtering. Wrong high-risk tool execution usually starts with a broad tool and no state machine. Cost runaway usually starts with long context, expensive model routing, retries, and poor cache keys. These are system failures, not only prompt failures.

12.6 Release Evidence Packet

A production AI feature should ship with a scope statement, architecture data flow, RAG evidence, tool-state machine, memory policy, audit events, evaluation report, safety cases, canary plan, budgets, rollback steps, on-call owner, and incident template. This is not documentation overhead. It is the production gate.

The conclusion is that enterprise AI diagnosis must be layered. Find the failing layer before choosing a fix.

13. Conclusion: Java’s AI Value Is Enterprise Boundaries, Governance, Integration, and Operability

This section answers the final architecture judgment. Java does not lose relevance because Python dominates model training. Java also does not gain enterprise AI maturity just because Spring AI or LangChain4j exists. Java’s value is placing model capabilities inside existing enterprise boundaries so that AI can be permissioned, governed, audited, budgeted, observed, released, and rolled back.

Spring AI helps Spring Boot applications integrate models, tools, vector stores, Advisors, and observability. LangChain4j provides framework-neutral AI Services, Tools, Memory, RAG, and agent-style orchestration. DJL, local inference, model gateways, and specialized inference servers solve different execution or governance problems. None of them is a silver bullet.

The right order is: define data and tool boundaries, then RAG and memory policies, then model routing and reliability, then the API syntax. If teams start with code examples and project structure, they often get a demo that cannot be governed.

13.1 Three-Stage Adoption: Pilot, Controlled Production, Platformization

Do not start with a universal AI platform. Start with a low-risk pilot such as internal Q&A, code-review assistance, support drafting, or ticket summarization. Record requests, sources, tools, costs, and human feedback from day one. Move to controlled production with tenant, ACL, citations, tool risk, budget, and alerts. Only then platformize model gateway, RAG pipeline, tool catalog, memory policy, evaluation, cost center, and observability.

13.2 Architecture Decision Records Make AI Choices Traceable

Decisions such as Spring AI, LangChain4j, model gateway, vector store, external provider, or tool approval should be recorded with context, constraints, alternatives, rationale, risk, rollback, and verification metrics. Fast-moving facts need verification date, applicable version, and source.

13.3 Enterprise AI Organization Boundaries

Platform teams should own model gateway, secrets, quotas, observability, audit protocols, RAG infrastructure, evaluation tooling, and cost reports. Business teams own domain correctness, permissions, tool semantics, approval rules, and value metrics. Security and data teams own policy, classification, and audit review. Without this ownership map, incidents become blame games.

13.4 Data Governance Is the Foundation of RAG

RAG quality depends on document owner, version, tenant, sensitivity, source, effective date, and permission metadata. A vector database without governance is only a retrieval engine, not an enterprise knowledge system.

13.5 Tool Calling Is Business Automation, Not a Model Showcase

The model may propose a tool. The Java service must verify identity, permission, schema, idempotency, approval, rollback, and audit. The closer a tool is to money, permissions, deployment, or customer communication, the more it must look like a real business API.

13.6 Memory Shapes Behavior and Privacy Risk

Memory is a governed data system. Writes need policy, reads need scope, summaries need sources, and deletion must be possible. Long-term memory without transparency becomes a compliance risk.

13.7 Cost and SLO Are Architecture Requirements

Token budgets, model routing, cache keys, retries, timeouts, and fallback are part of the architecture. They should be linked to business outcomes, not only infrastructure bills.

13.8 Security and Compliance Final Principle: Prompt Is Not a Boundary

Prompts express intent. Code, identity, data flow, tool gateways, audit storage, release policy, and operations systems enforce boundaries. The stronger the model, the clearer these boundaries must be.

13.9 Final Judgment for Java Architects

Java architects do not need to become foundation-model trainers, but they must understand how models change system boundaries. AI adds prompt versions, RAG evidence, tool risks, memory lifecycles, model routing, evaluation sets, token costs, and compliance explanations to the existing world of APIs, transactions, idempotency, rate limits, monitoring, and releases. Java is valuable because it can bring engineering discipline to these boundaries.

The final recommendation is to use Java services as the governance shell for enterprise AI, Spring AI or LangChain4j as appropriate application abstractions, model gateways and specialized inference services for model access and execution, RAG governance and evaluation sets for knowledge quality, tool gateways for business action safety, and observability/audit systems for operability and accountability.

References

1. Spring AI Reference Documentation: https://docs.spring.io/spring-ai/reference/
2. Spring AI ChatClient API: https://docs.spring.io/spring-ai/reference/api/chatclient.html
3. Spring AI Tool Calling: https://docs.spring.io/spring-ai/reference/api/tools.html
4. Spring AI Vector Databases: https://docs.spring.io/spring-ai/reference/api/vectordbs.html
5. LangChain4j Documentation: https://docs.langchain4j.dev/
6. GraalVM Native Image Documentation: https://www.graalvm.org/latest/reference-manual/native-image/
7. Oracle Java SE Support Roadmap: https://www.oracle.com/java/technologies/java-se-support-roadmap.html
8. Oracle JDK 26 Release Notes: https://www.oracle.com/java/technologies/javase/26all-relnotes.html