AI Memory Types

May 2026

A comprehensive guide to in-context, external, in-weights, and in-cache memory

 

Introduction

AI systems use four distinct memory types, each with different persistence, speed, and capacity. Understanding when and how to use each type is critical for building reliable, efficient AI solutions.

 

Think of AI memory like human memory:

•        In-context memory is what you are actively thinking about right now

•        External memory is your notebook or smartphone

•        In-weights memory is everything you learned in school and cannot un-know

•        In-cache memory is muscle memory — repeating a familiar task faster without re-thinking it from scratch

 

1. In-context Memory

 

Type: Short-term | Nature: Volatile, Fast

The active working memory — everything currently visible to the model inside the context window.

 

Analogy

📝 Your desk while working Everything you can see and touch right now. Just like a messy desk, once you clear it (end the session), everything is gone. No desk, no memory.

 

Strengths

•        Instant recall — no retrieval step needed

•        Precise and faithful to what was said

•        No infrastructure required

Limitations

•        Resets every session — no persistence

•        Limited by context window size (100K–1M tokens)

•        Costly at scale — every token is processed

Use Cases

•        Customer support chatbot: The full conversation history is in context, so the agent knows what was said earlier in the same ticket.

•        Document Q&A: Paste an entire document into context and ask questions. The model reads it fresh each time.

•        Multi-step reasoning: Chain-of-thought prompting keeps intermediate reasoning steps visible while the model works through a problem.

•        Code assistants: The current file and recent edits live in context, so suggestions stay coherent with what was just typed.

 

 

2. External Memory

 

Type: Long-term | Nature: Persistent, Queryable

Databases, vector stores, knowledge bases, or files the AI reads and writes to persist information across sessions.

 

Analogy

📚 Your bookshelf, filing cabinet, or phone contacts You do not hold all of it in your head, but you know how to look it up. The information outlives any single conversation, just like books outlive a reading session.

 

Strengths

•        Survives session resets — truly persistent

•        Unlimited scale — limited only by storage

•        Updateable anytime without retraining

Limitations

•        Retrieval latency — adds a round-trip to every query

•        Retrieval errors — if the wrong document is fetched, the model may hallucinate confidently

•        Infrastructure overhead — requires databases, embedding pipelines, and retrieval logic

Use Cases

•        RAG (Retrieval-Augmented Generation): AI searches a vector database of company documents to answer employee questions accurately with up-to-date information.

•        Personal AI assistant: Stores preferences, past decisions, and notes so the next session feels continuous.

•        CRM integration: A sales agent reads and writes a customer database so every rep gets an up-to-date history of interactions.

•        Research assistant: Ingests papers and summaries into a vector store; retrieves the most relevant chunks when a new question is asked.

 

 

3. In-weights Memory

 

Type: Permanent |  Nature: Baked-in, Implicit

Knowledge encoded into the model's parameters during training or fine-tuning. It is the model's worldview — always available, never updatable at runtime.

 

Analogy

🎓 Everything you learned in school Language, math, history, common sense — you cannot un-learn that Paris is in France. You did not look it up; it is wired in. But your knowledge has a cutoff: you do not know what happened after you graduated.

 

Strengths

•        Zero retrieval cost — knowledge is always available

•        Broad general knowledge across many domains

•        No infrastructure required at runtime

Limitations

•        Knowledge cutoff date — does not know recent events

•        Cannot be updated at runtime — requires full retraining

•        Retraining is expensive in time and compute

•        Can hallucinate confidently on topics at the edges of training data

Use Cases

•        Language understanding: Grammar, idioms, and reasoning patterns are in-weights; every prompt benefit without any extra memory system.

•        Domain fine-tuning: Fine-tune on medical literature so the model knows clinical terminology deeply, not just via retrieval.

•        Coding assistants: Knowledge of programming languages, APIs, and patterns is in-weights; no lookup needed for standard library calls.

•        Style and tone: Fine-tuning on brand voice bakes that tone in-weights so every response feels on-brand without explicit prompting.

 

 

4. In-cache Memory (KV-cache)

 

Type: Ephemeral |  Nature: Fast, Compute-level

Saved intermediate computation states (key-value attention matrices). Avoids re-processing tokens the model has already seen, cutting latency and cost dramatically.

 

Analogy

🎹 Muscle memory for a pianist After playing a piece many times, fingers move without consciously re-thinking each note. The AI does not recompute the system prompt tokens on every reply — those computations are cached and reused.

 

Strengths

•        Reduces latency dramatically on repeated context

•        Cuts API cost — prompt caching charges less for cached tokens

•        Largely transparent to developers — works automatically

Limitations

•        Cleared between sessions — not persistent storage

•        Cache is invalidated if the prefix changes

•        Not real knowledge storage — purely a performance optimisation

Use Cases

•        Long system prompts: A 10,000-token system prompt is cached so every user turn does not re-pay the full processing cost.

•        Agentic loops: An agent that calls the model dozens of times in a task reuses cached context of prior steps, keeping latency low.

•        High-volume APIs: Apps with thousands of users sharing the same base prompt benefit from shared KV-cache at the infrastructure level.

•        Multi-turn chat: Conversation history is incrementally cached; each new user message only pays to process the new tokens.

 

 

Side-by-side Comparison

The table below summarizes the key dimensions across all four memory types.

 

Dimension	In-context	External	In-weights	In-cache
Persistence	Session only	Permanent (DB)	Until retrained	Session / server life
Capacity	Context window limit	Unlimited (DB scale)	Model parameter space	GPU memory bound
Speed	Instant	Retrieval latency	Zero (it is the model)	Near-instant
Update	Auto as chat grows	Explicit write ops	Requires retraining	Automatic, prefix-based
Cost	Per-token processing	Storage + retrieval	High upfront training	GPU memory usage

 

 

Combining Memory Types

The most powerful AI systems combine all four types strategically:

 

•        RAG system: In-weights for language understanding + external for fresh facts + in-context for the query + in-cache for a repeated system prompt.

•        Enterprise chatbot: Fine-tune for tone (in-weights) + company knowledge base (external) + conversation history (in-context).

•        Personal AI assistant: User preferences stored externally, general knowledge from weights, active task in context, long system prompt cached.

 

 

Key Takeaways

•        No single memory type does everything — they are complementary, not competing.

•        In-context memory is your default; add external memory when sessions must persist.

•        In-weights memory is your floor of capability; fine-tune to raise it for a specific domain.

•        In-cache memory is a performance multiplier — use it aggressively for long, repeated prompts.

•        Always consider the cost tradeoff: context tokens are processed every request; external retrieval adds latency; retraining is expensive; caching saves both.

 

The Complete LLM Integration Stack: MCP, ADK, RAG, CAG, CLI, API & Connectors: The Relationships

 May 2026

This is a clear, comprehensive breakdown of these components in the modern LLM/AI agent ecosystem. They form layers that work together when processing a user request through an LLM.

Comparison Table

Component	Full Name / Type	Core Purpose	Level in Stack	Key Strength	Can Take Actions?
LLM	Large Language Model	Reasoning, generation, understanding	Brain / Core	Natural language intelligence	No (needs tools)
RAG	Retrieval-Augmented Generation	Fetch relevant external knowledge	Knowledge Layer	Reduces hallucinations	No (read-only)
CAG	Context-Augmented / Cache-Augmented Generation	Inject full/pre-cached context directly	Context Optimization Layer	Speed + completeness for known data	No
MCP	Model Context Protocol	Standardized tool & data access	Integration / Protocol Layer	Interoperability	Yes
ADK	Agent Development Kit (Google)	Build & orchestrate agents	Orchestration Framework	Structured agent logic	Yes (via tools)
API	Application Programming Interface	Direct programmatic access to services	Connectivity Layer	Flexibility & control	Yes
CLI	Command Line Interface	Human or script-based interaction	User/Dev Interface	Simplicity for testing	Yes
Connectors	Integration Adapters	Bridge between systems & tools	Plumbing Layer	Easy plug-and-play	Varies

Detailed Explanations & Analogies

1. LLM (The Brain) The central reasoning engine (e.g., Gemini, Claude, GPT). It processes prompts but has limited knowledge and no direct external access. Analogy: The pilot in an airplane cockpit.

2. RAG (Knowledge Retrieval) Retrieves relevant chunks from a vector database (documents, knowledge bases) and injects them into the prompt. Use Cases: Company policy Q&A, product documentation search, legal research. Analogy: Giving the pilot a stack of relevant maps and manuals before takeoff.

3. CAG (Context-Augmented/Cache-Augmented Generation) Directly loads entire relevant context (or cached KV cache) into the model's context window instead of dynamic retrieval. Use Cases: When full documents fit in large context windows, personalized chat with full history, or high-speed repeated queries. Analogy: Pre-loading the entire flight manual and route plan into the pilot's console (vs. RAG fetching pages on demand).

4. MCP (Standardized Tool Protocol) Open protocol (by Anthropic) for how LLMs/agents discover and call tools securely in a client-server model. Use Cases: Connecting to databases, CRMs, email, GitHub, etc., in a universal way. Analogy: USB-C port + standardized cables — any compatible tool plugs in without custom wiring.

5. ADK (Agent Framework) Google's open-source Python kit for building agents with planning, memory, tool use, multi-agent coordination. Use Cases: Complex workflows like research agents, customer support agents, automation. Analogy: The autopilot system + flight management computer that decides route, uses tools, and coordinates.

6. API (Direct Integration) Traditional way for software to talk (REST, GraphQL, etc.). Use Cases: Custom integrations where you control both sides. Analogy: Custom wiring between devices (flexible but messy at scale).

7. CLI (Command Line) Text-based interface for humans or scripts. Use Cases: Developer testing of agents/tools, server management. Analogy: Manual controls in the cockpit for debugging or overrides.

8. Connectors Adapters/libraries that simplify linking systems (e.g., database connectors, SaaS connectors). Use Cases: Rapid integration with popular services. Analogy: Plug adapters or extension cords.

Other Important Things:

• Memory (short-term, long-term, vector stores)
• A2A (Agent-to-Agent Protocol) - for multi-agent collaboration
• Tool Calling / Function Calling — native LLM capability
• ReAct / Agent Loop — observe-think-act cycle
• Vector Databases (Pinecone, pgvector, FAISS) - backbone of RAG

How They Work Together: End-to-End Flow

When a user request comes in:

1. Input → Hits the Agent (built with ADK or similar).
2. Planning → Agent uses LLM to reason and break down the task.
3. Knowledge →
• Uses RAG for dynamic retrieval from documents.
• Or CAG for full context injection (faster when possible).
4. Tools & Actions →
• Discovers/calls tools via MCP (standardized).
• Or directly via APIs / Connectors.
5. Execution → Tools perform actions (read/write data, send emails, etc.).
6. Orchestration → ADK manages loop, memory, multi-agent handoff if needed.
7. Output → LLM generates final response.

Real-World Example (Customer Support Agent):

• User: "Update my order #123 and explain the policy."
• RAG/CAG: Retrieves order history + policy docs.
• MCP: Calls CRM tool (via MCP server) to update order.
• ADK: Orchestrates the sequence, handles errors, confirms with user.
• API/Connectors: Underlying links to payment/shipping systems.

Visual Stack Analogy (Airplane):

• LLM = Pilot
• RAG/CAG = Navigation charts & manuals
• MCP = Standardized control interfaces
• ADK = Autopilot + Flight Management System
• API/Connectors = Engines, flaps, landing gear
• CLI = Mechanic's diagnostic terminal

This modular approach makes systems more interoperable, maintainable, and powerful. RAG/CAG give knowledge, MCP gives standardized hands, ADK gives the brain to coordinate everything.

 

 

RAG (Retrieval-Augmented Generation) and CAG (Cache-Augmented Generation): The difference

May 2026

RAG (Retrieval-Augmented Generation) and CAG (Cache-Augmented Generation) are two approaches to enhance Large Language Models (LLMs) with external knowledge. They differ fundamentally in how and when that knowledge is provided to the model.

Core Difference

Aspect	RAG (Retrieval-Augmented Generation)	CAG (Cache-Augmented Generation)
Knowledge Access	Just-in-time retrieval: Dynamically searches and fetches relevant chunks from a large external database/vector store at query time.	Pre-loaded cache: All (or most) relevant knowledge is preloaded into the model's extended context window (and often its KV cache) before any queries.
Latency	Higher — involves embedding the query, vector search, ranking, and context assembly.	Much lower — no retrieval step; the model answers directly from its "memory."
Scalability	Excellent for very large or dynamic knowledge bases (millions of documents).	Limited by the model's context window size (though modern models support 128k–1M+ tokens).
Freshness	High — can access the latest data.	Snapshot-based — data is only as fresh as the last cache update.
Complexity	Higher (needs vector DB, chunking, embeddings, reranking, etc.).	Simpler architecture once set up.
Hallucination Risk	Can suffer from retrieval errors (wrong or irrelevant docs).	Lower for covered topics, as the full relevant context is usually present.

Analogies

• RAG is like a student taking an open-book exam who can look up any information in a huge library during the test. They search for exactly what they need for each question but spend time searching and might grab the wrong book sometimes.
• CAG is like a student who reads and memorizes the entire relevant textbook the night before (preloading into context + KV cache). During the test, they answer instantly from memory with no lookup time — but if the textbook isn't updated, they might miss new information.
• Another view: RAG is on-demand Google Search + summarization. CAG is pre-loading a full PDF/manual into your AI's brain so it already knows everything relevant.

Use Cases

RAG is best for:

• Dynamic or massive knowledge bases: News, live stock data, legal cases that update frequently, research across the entire web/wiki, customer data that changes constantly.
• Broad exploration: Enterprise search over millions of documents, real-time question answering with up-to-date facts.
• Example: A news chatbot that answers questions about today's events, or a legal AI that must reference the latest court rulings.

CAG is best for:

• Static or slowly changing knowledge: Product manuals, FAQs, company policies, technical documentation, training materials, medical protocols, internal wikis.
• Latency-critical applications: Real-time chatbots, customer support, mobile/edge apps where speed matters more than perfect freshness.
• Resource-constrained environments: Where maintaining a full vector database is overkill.
• Examples:
• An e-commerce support bot answering questions from a product catalog/manual.
• A corporate HR assistant for company policies and benefits.
• Healthcare diagnostic support using fixed protocols/guidelines.
• Educational tutors with a fixed curriculum.

Hybrid Approach (Often Recommended)

Many production systems use both:

• CAG for stable, high-frequency knowledge (preload policies, manuals).
• RAG for dynamic or edge-case data (recent updates, user-specific data).

This gives you speed for common queries + freshness when needed.

Summary

• Choose RAG when your data is large, changing, or highly variable.
• Choose CAG when your data is bounded, relatively static, and speed/consistency are priorities.
• CAG became more viable with the rise of very long context windows (like in Llama 3.1, Gemini, etc.) and efficient KV cache management.

CAG is essentially a simplification/trade-off that leverages modern LLM context capabilities to eliminate the complex retrieval pipeline of traditional RAG.

 

 

Vibe Coding to Deployment with Strategy and Governance

May 2026 

This is a genuinely important governance challenge, and there are several landmines that get missed consistently. Let me build you a comprehensive, structured reference you can walk into any IT or CFO conversation with.

AI solution execution package

A comprehensive handover framework from subject matter expert (SME/vibe coder) to IT department, structured for both technical deployment and CFO-level investment approval.

Handover Doc

This is the entry point. Everything IT needs to begin must be captured here before a single server is provisioned. Missing fields here cascade into rework.

Solution identity

·       Official solution name, version, and unique identifier

·       Business problem statement (in plain language, not technical)

·       Target user group & estimated concurrent users at launch

·       Business process replaces or augments (with process ID if documented)

·       SME owner name, department, cost center, and escalation contact

·       Sponsor executive name (accountable sign-off for production deployment)

Technical inventory

·       Repository location, branch, and commit hash of handover version

·       Full dependency manifest (package.json, requirements.txt, etc.) with pinned versions

·       AI model(s) used — provider, model ID, API version string (not just "GPT" or "Claude")

·       All third-party services called (APIs, databases, file stores)

·       Secrets and environment variables list (names only, values via vault)

·       Data flow diagram — what enters, what is processed, what leaves the system

 

 

Stakeholder sign-offs

·       SME developer sign-off (confirms solution works as described)

·       Legal / compliance review completion date

·       Data protection officer (DPO) approval if personal data is processed

·       Information security acknowledgment

·       Business owner acceptance criteria sign-off

·       IT department readiness confirmation

Acceptance criteria

·       Definition of "working correctly” specific, measurable, not subjective

·       Minimum acceptable response time (P95 latency)

·       Expected accuracy/quality threshold for AI outputs

·       User acceptance test (UAT) plan and responsible party

·       Rollback trigger conditions (what constitutes failure post-deployment)

·       Pilot user group and timeline before full rollout

 

Infrastructure

Infrastructure choices made without assessing existing estate create shadow IT debt. IT must map this solution to current approved platforms before provisioning anything new.

Deployment environment

Hosting Options to evaluate

·       Cloud-native (AWS/Azure/GCP managed services) - assess if org has existing agreements

·       On-premises deployment — required if data cannot leave the organization

·       Hybrid (compute on-prem, AI API calls to cloud) — most common for regulated industries

·       Containerized (Docker/Kubernetes) vs serverless — must match IT's operational capability

·       Existing PaaS platforms (Power Platform, ServiceNow, Salesforce) — can this live there instead?

 

Network & access

·       Inbound access — intranet only, VPN-gated, or public internet

·       Outbound calls — firewall rules required for each external API endpoint

·       DNS entry and subdomain naming convention

·       TLS certificate management and renewal ownership

·       Load balancer / reverse proxy configuration

·       CDN requirements (if serving frontend assets)

Data infrastructure

·       Database type and version (must match approved DB catalogue)

·       Storage requirements: size at launch, growth rate estimate, archival policy

·       Backup frequency, retention period, and restore SLA

·       Data classification label (public, internal, confidential, restricted)

·       Data residency requirement — which country/region must data stay in

·       Integration with existing data warehouse or data lake

Scalability spec

·       Expected peak load (requests per minute) - must come from SME, not assumed

·       Auto-scaling policy: scale-up threshold, scale-down delay

·       AI API rate limits and how queuing/throttling will be handled

·       Maximum acceptable cold-start latency

·       Capacity planning review cycle (quarterly recommended)

·       CI/CD & release

Deployment pipeline

·       Source control platform and branching strategy (must align with IT standards)

·       Environment chain: dev → staging → production (minimum three tiers)

·       Automated test suite — unit, integration, and AI output quality tests

·       Deployment approval gates and who holds each gate

·       Blue/green or canary release strategy for zero-downtime updates

·       Rollback procedure — time to rollback, data state handling

Configuration management

·       Infrastructure as Code (IaC) — Terraform, Bicep, CloudFormation

·       Secret management tool — HashiCorp Vault, AWS Secrets Manager, Azure Key Vault

·       Feature flags for safe AI model version switching

·       Prompt version control system — prompts are configuration, not code comments

·       Environment variable documentation (what each controls, who can change it)

 

Security and Compliance

Vibe-coded AI solutions frequently bypass standard security review. This is where organizations get hurt. Every item here is non-negotiable before production.

Identity & access management

·       Authentication method — SSO/SAML/OAuth tied to corporate identity provider (not local accounts)

·       Role-based access control (RBAC) matrix — who can use, who can configure, who can admin

·       Service account credentials — expiry policy, rotation schedule

·       AI API key management — stored in vault, not in code or .env files in repos

·       Privileged access review cadence (quarterly minimum)

Application security

·       OWASP Top 10 assessment — input validation, injection prevention, XSS

·       Prompt injection testing — unique to AI apps, not covered by standard OWASP scans

·       Dependency vulnerability scan (SCA) - Snyk, Dependabot, or equivalent

·       Static code analysis (SAST) before handover

·       Penetration test scope and schedule (before go-live for high-risk solutions)

·       API rate limiting to prevent abuse and cost overruns

Regulatory compliance

·       Applicable regulations: PDPA (Malaysia), GDPR (if EU data subjects), sector-specific (FSA, MOH)

·       Personal Data Impact Assessment (PDPIA) if processing personal data

·       Data processing agreements with AI providers (does OpenAI/Anthropic have DPA?)

·       Consent mechanism if users' data is sent to external AI APIs

·       Right to erasure handling — can personal data be deleted from AI context/logs?

·       Audit trail for decisions made with AI assistance (required in many regulated sectors)

Data protection

·       Encryption at rest — algorithm, key management, rotation policy

·       Encryption in transit — TLS 1.2 minimum, TLS 1.3 preferred

·       Data masking/anonymization before sending to external AI APIs

·       Log sanitization — PII must not appear in application logs

·       Data Loss Prevention (DLP) rules updated to cover AI-specific data flows

 

AI Governance

This entire category is new. Traditional IT governance frameworks do not cover it. If your IT department doesn't have an AI governance policy yet, this package should define the baseline.

Model governance

·       Exact model ID and version pinned — never use "latest" endpoint in production

·       Model deprecation plan — what happens when the provider sunsets this version

·       Model evaluation baseline — accuracy/quality metrics at time of handover

·       Approved model substitution lists - vetted alternatives if primary model fails

·       Model provider SLA — uptime guarantees, support tier purchased

·       Fine-tuned model registry (if applicable) - where weights are stored, access controls

Prompt management

·       All system prompts version-controlled alongside code (not in ad-hoc notes)

·       Prompt change approval process — who can modify system prompts in production

·       Prompt injection mitigation strategy documented

·       Context window budget management — max token limits per request defined

·       Output format validation — structured output enforcement where applicable

Output quality & safety

·       Hallucination risk classification — what harm could a wrong AI output cause

·       Human-in-the-loop requirement — which decisions need human review before action

·       Output guardrails — content filters, format validators, confidence thresholds

·       Feedback loop mechanism — how users report incorrect AI outputs

·       Retraining/fine-tuning trigger criteria (if model is customized)

·       Bias assessment — has the AI been tested on edge cases relevant to the business context

Accountability & ethics

·       AI decision explainability requirement — can the system explain why it gave an output

·       Disclosure to users that they are interacting with an AI system

·       Liability ownership — who is accountable when the AI output causes a business error

·       IP ownership of AI-generated content — check provider terms of service

·       Alignment with national AI ethics guidelines (Malaysia NAII framework or equivalent)

·       Incident classification — AI-specific failure modes must be in the incident taxonomy

 

Cost and CFO Case

The CFO's primary concern is not the build cost — it's the unpredictable recurring cost. AI solutions have a novel cost structure that does not behave like traditional software. Explain this explicitly.

One-time costs

·       Development effort (already sunk, but document for ROI baseline)

·       Security assessment and penetration testing fees

·       IT integration and deployment engineering time

·       Training and change management for end users

·       Legal review and contract negotiation with AI provider

·       Infrastructure provisioning (compute, storage, networking)

Recurring costs (the critical ones)

·       AI API token costs — must be modelled per use case, not estimated as a flat monthly fee

·       Compute hosting (elastic — can spike with adoption)

·       Data storage and egress costs

·       Monitoring, logging, and alerting platform fees

·       IT support hours allocated to this solution (opportunity cost)

·       Security tool license (scanning, vault, WAF)

·       Compliance audit and review costs (annual)

Cost modelling requirements

·       Cost per transaction/query — calculate from average token count × per-token price

·       Monthly cost at 3 adoption scenarios: conservative, expected, high

·       Cost ceiling alert threshold — auto-notification when monthly spend reaches X%

·       Cost per user per month at steady state

·       Break-even analysis against the process it replaces or augments

·       Year 2 and Year 3 cost forecast (model price changes are unpredictable)

ROI & business case

·       Hours saved per user per week (quantified, not estimated)

·       Error reduction rate vs manual process

·       Throughput increase (volume processed per FTE)

·       Qualitative benefits (faster decisions, improved consistency)

·       Risk-adjusted ROI (including probability of model deprecation, rework)

·       Budget owner and cost center for ongoing operational spend

·       Governance controls for the CFO

Financial controls

·       Hard spend cap on AI API per billing period (enforce at API gateway level)

·       Chargeback or showback model — which department absorbs which cost

·       Vendor payment terms and invoice review process

·       Quarterly cost review against business value delivered

·       Kill switch decision criteria — at what cost/value ratio do we decommission

Vendor management

·       AI provider contract reviewed by procurement (not just signed up via credit card)

·       Enterprise agreement vs pay-as-you-go — assess at expected volume

·       Data processing agreement (DPA) executed

·       Exit strategy — what happens if provider increases price 5× or shuts down

·       Third-party risk assessment completed (ISO 27001, SOC 2 of provider)

 

 

 

 

Operations and SLA

Monitoring & observability

·       Application performance monitoring (APM) - response time, error rate, throughput

·       AI-specific metrics: token consumption, model latency, refusal rate, output quality score

·       Cost monitoring dashboard — real-time spend vs budget

·       User experience metrics — task completion rate, drop-off points

·       Centralized logging platform — must integrate with existing SIEM

·       Distributed tracing for debugging multi-step AI workflows

Alerting & incident response

·       Alert thresholds for: error rate, latency, cost, model downtime

·       On-call rotation and escalation matrix (include AI provider support path)

·       AI-specific incident playbook — model unavailable, degraded quality, cost spike

·       Communication template for user-facing incidents

·       Post-incident review process and improvement loop

SLA definition

·       Availability target (99.5% ≠ 99.9% — be explicit; AI APIs are not 5-nines)

·       Recovery Time Objective (RTO) - How long can the business tolerate downtime

·       Recovery Point Objective (RPO) - how much data loss is acceptable

·       Degraded-mode operation - what the system does when the AI API is unavailable

·       Maintenance window schedule and user notification process

Support model

·       L1 support: end-user queries - helpdesk script and FAQ

·       L2 support: application issues - IT team with access to logs and configs

·       L3 support: AI model issues — SME + AI provider support channel

·       SME availability commitment post-handover (at least 90-day warranty period)

·       Knowledge transfer plan — IT must be able to operate without the SME

·       Bus factor mitigation — documentation must enable a new IT hire to support this

Change management

·       Change request process — who approves updates to prompts, models, code

·       Testing requirements before each change reaches production

·       Model update policy — process when AI provider releases new model version

·       User communication process for feature changes or deprecations

·       Decommissioning plan — data export, user migration, archive policy

Business continuity

·       DR environment — active-passive or active-active, and failover test schedule

·       AI provider failover — secondary provider or model on standby

·       Manual fallback process — how the business operates if this system is down for 48 hours

·       BCP test schedule and last tested date

 

Critical Gaps

These are the items most commonly absent in AI deployment packages. Each one has caused real incidents or project failures in production. Do not skip any of these when briefing IT.

Gap 1: No cost ceiling at the infrastructure level

CFO risk

Most teams set a "budget estimate" in a spreadsheet but never enforce it at the API gateway or cloud billing level. A runaway loop, a bot hammering your endpoint, or unexpectedly high adoption can generate thousands of dollars in AI API costs in hours. The CFO must insist on a hard spend cap enforced by the system — not monitored by a human checking dashboards once a week.

Gap 2: Model deprecation plan is absent

Continuity risk

AI providers routinely deprecate model versions — often with 3–6 months notice. If your solution is pinned to a specific model (as it should be), it will break when that version is retired. Every AI deployment package must include: which model version is in use, when it is scheduled for deprecation, who is responsible for the upgrade, what the test plan looks like, and who funds the re-testing effort.

Gap 3: The "bus factor" is one — the SME who built it

Operational risk

Vibe-coded solutions are often built by one person who "just knows how it works." When that person leaves the department, changes roles, or goes on leave, IT cannot maintain the system. The handover package must include: runbook documentation that a new hire can follow, architecture decision records (why certain choices were made), and a mandatory knowledge transfer session with IT before go-live. IT should be able to list every person who can maintain this system without calling the original developer.

Gap 4: Prompt injection is not on IT's security radar

Security risk

Traditional security scanning tools (SAST, DAST, OWASP ZAP) do not test for prompt injections. An attacker who can inject into your system prompt can exfiltrate data, bypass business logic, or cause the AI to produce harmful outputs. This requires a separate, AI-specific security test that most IT security teams have not yet built capability for. You may need to bring in an external specialist.

Gap 5: Data residency and sovereign cloud are assumed, not verified

Compliance risk

When your application sends data to an external AI API (OpenAI, Anthropic, Google), that data leaves your infrastructure. Where it goes, how it is stored, whether it is used for training, and under which jurisdiction it falls are all questions with legal consequences. For Malaysian organizations: verify PDPA implications. For organizations with government data: verify whether the relevant data classification prohibits use of external AI APIs entirely. This must be confirmed in writing from the provider before go-live.

Gap 6: Liability for AI output errors is unassigned

Legal risk

When an AI gives incorrect advice that causes a business error — a wrong calculation, a misclassified document, an inappropriate recommendation — who is accountable? The SME who built it? The IT team that deployed it? The AI provider? The business user who acted on it? This must be decided and documented before deployment. The answer shapes the design: high-liability use cases need human-in-the-loop validation. Low-liability use cases may run autonomously. Neither can be determined without this conversation.

Gap 7: No regression testing framework for AI quality

Quality risk

Traditional software has deterministic tests: input A → output B, always. AI does not. A prompt change, a model update, or even a change in temperature parameter can silently degrade the quality of outputs without throwing an error. Before deploying, you need: a golden dataset of representative inputs and their expected output quality, a scoring rubric, and an automated or semi-automated test that runs this dataset and alerts if quality drops below a threshold. Without this, every model update is a leap of faith.

Gap 8: Shadow IT already happened — before the handover

Discovery risk

By the time a vibe-coded solution reaches the IT handover stage, it has usually already been used in production by the SME and their colleagues — with a personal API key, corporate data, and no security review. The handover package should include a retrospective: what data has already been sent to the AI API, under what conditions, and does this create any liability? IT should also audit whether there are other similar shadow AI tools in the organisation that have not yet been brought forward.

Gap 9: No graceful degradation when the AI is unavailable

Resilience risk

AI API providers have outages. When they do, what does your application do? Most vibe-coded solutions show a generic error page. The correct answer depends on the use case: queue the request and retry, fall back to a simpler rule-based response, or gracefully inform the user and provide a manual workaround. This needs to be designed intentionally, not discovered during an outage at 2am.

Before you walk into the IT meeting — quick self-assessment

Can IT support this without calling the original developer?

Is there a hard cost ceiling enforced at the infrastructure level?

Has the model deprecation date been checked and a plan made?

Has legal confirmed what data can be sent to external AI APIs?

Is liability for AI output errors assigned to a named role?

Is there a plan for what the business does when the AI is down?

Have prompts been version-controlled and change-approved?

Has prompt injection been specifically tested (not just standard OWASP)?