Chronology of LLM Architectures: From GPT-3.5 (ChatGPT) in 2022 to January 2026

The evolution of large language model (LLM) architectures since ChatGPT's launch in late 2022 has been rapid, building primarily on the transformer decoder-only foundation but incorporating innovations like Mixture-of-Experts (MoE) for efficiency, multimodal capabilities (text + vision/audio/video), extended context windows (e.g., 1M+ tokens), and specialized reasoning mechanisms (e.g., chain-of-thought or RL-based). Most models remain decoder-only transformers at their core, with scaling in parameters (from ~175B to 2T+), data (up to 40T tokens), and compute (e.g., 10x increases). Below is a chronological timeline of major releases across key companies, focusing on architectural highlights. This draws from comprehensive sources like Wikipedia's LLM list, AI timelines, and release trackers.

Date	Company	Model	Key Architectural Features & Innovations	Parameters (Approx.)
Nov 2022	OpenAI	GPT-3.5 (ChatGPT)	Decoder-only transformer; instruction-tuned with RLHF for conversational alignment; foundational for chat interfaces.	175B
Feb 2023	Meta	LLaMA 1	Decoder-only; efficient pretraining on multilingual corpus (20 languages); overtrained vs. Chinchilla scaling for better param efficiency.	7B–65B
Mar 2023	OpenAI	GPT-4	Decoder-only with multimodal extensions (text + vision); improved scaling laws; rumored sparse MoE elements for efficiency.	~1.76T (rumored)
May 2023	Google	PaLM 2	Decoder-only; enhanced multilingual support; integrated into Bard chatbot; focus on few-shot learning.	340B
Jul 2023	Meta	LLaMA 2	Decoder-only; commercial open-source variant of LLaMA 1; RLHF-aligned for safety; trained on 2T tokens.	7B–70B
Jul 2023	Anthropic	Claude 2	Decoder-only with Constitutional AI for alignment; emphasis on harmlessness and helpfulness; long-context handling.	~100B–200B
Sep 2023	Mistral AI	Mistral 7B	Decoder-only; sliding window attention for efficiency; outperforms larger models on benchmarks like MMLU.	7B
Nov 2023	xAI	Grok-1	Decoder-only; real-time data integration via X (Twitter); humorous alignment; trained on web-scale data.	314B
Dec 2023	Google	Gemini 1.0	Multimodal MoE (text, image, audio, video); native multimodality from pretraining; three sizes for scalability.	~1B–540B (variants)
Dec 2023	Mistral AI	Mixtral 8x7B	Sparse MoE (8 experts, 2 active per token); decoder-only base; efficient inference (12.9B active params); open-weights.	46.7B total
Dec 2023	DeepSeek	DeepSeek-LLM	Decoder-only; bilingual (English/Chinese) pretraining; focus on code/math tasks.	7B–67B
Feb 2024	Google	Gemini 1.5	MoE with ultra-long context (1M+ tokens); multimodal (video/audio processing); improved needle-in-haystack retrieval.	~1T (MoE)
Mar 2024	Anthropic	Claude 3 (Haiku/Sonnet/Opus)	Decoder-only multimodal (text + vision); tiered sizes for speed vs. capability; 200K context; Constitutional AI.	25B–500B (variants)
Apr 2024	Microsoft	Phi-3	Small language model (SLM); decoder-only; synthetic data distillation for efficiency; mobile-friendly.	3.8B–14B
Jun 2024	DeepSeek	DeepSeek-V2	MoE (236B total, 21B active); MLA (multi-head latent attention) for compression; trained on 8.1T tokens.	236B
Jun 2024	Anthropic	Claude 3.5 Sonnet	Decoder-only; enhanced vision/coding; 200K context; outperforms GPT-4o on some reasoning benchmarks.	~200B
Jul 2024	Meta	LLaMA 3.1	Decoder-only; grouped-query attention; 128K context; open-weights; trained on 15.6T tokens.	8B–405B
Aug 2024	xAI	Grok-2	Multimodal decoder-only; image understanding; real-time X integration; Grok-2 mini for efficiency.	~314B
Sep 2024	OpenAI	o1 (preview)	Decoder-only with built-in chain-of-thought reasoning; test-time compute scaling; focuses on complex problem-solving.	~175B
Nov 2024	Mistral AI	Mistral Large 2	Decoder-only multimodal; function calling/JSON mode; 128K context; upgraded for enterprise.	123B
Dec 2024	Microsoft	Phi-4	SLM decoder-only; multimodal extensions; trained on 9.8T tokens; emphasis on reasoning in small form.	14B
Dec 2024	DeepSeek	DeepSeek-V3	Dense MoE hybrid; 14.8T tokens; open-source MIT license; strong in coding/math.	671B
Jan 2025	DeepSeek	DeepSeek-R1	RL-based reasoning on V3 base (no pretraining); o1-like chain-of-thought; distilled variants for efficiency.	671B base
Jan 2025	Alibaba	Qwen2.5	Decoder-only with MoE variants; 18T tokens; 1M context options; multilingual (100+ languages).	0.5B–72B
Feb 2025	Google	Gemini 2.0	Multimodal MoE; agentic capabilities (real-time API); Flash/Pro variants; 2M+ context.	~500B (Pro)
Feb 2025	Anthropic	Claude 3.7 Sonnet	Decoder-only; extended thinking mode; reduced hallucinations; 500K+ context.	~300B
Feb 2025	OpenAI	GPT-4.5	Decoder-only non-reasoning flagship; pattern recognition focus; 1M context; reduced hallucinations.	~1T+
Feb 2025	xAI	Grok-3	Decoder-only; 10x compute scaling; Think/Big Brain modes for reasoning; multimodal.	Unknown (large)
Apr 2025	Meta	LLaMA 4	MoE decoder-only; 10M context; multimodal (text/images); trained on 40T tokens.	400B
Apr 2025	OpenAI	o3 / o4-mini	Reasoning decoder-only; advanced CoT with self-verification; o3 for high-complexity, mini for speed.	Unknown
May 2025	Anthropic	Claude 4 (Sonnet/Opus)	Decoder-only with hybrid deep thought mode; 7-hour context handling; math/coding focus.	~500B (Opus)
Jul 2025	xAI	Grok-4	Multimodal decoder-only; enhanced agentic tools; real-time search integration.	Unknown
Aug 2025	OpenAI	GPT-5	Decoder-only with native "thinking" abilities; multimodal; 1M+ context; includes mini/nano variants.	Unknown (2T+ rumored)
Aug 2025	DeepSeek	DeepSeek-V3.1	Hybrid MoE (thinking/non-thinking modes); DSA (sparse attention); 15.6T tokens.	671B
Nov 2025	Google	Gemini 3 (Pro/Deep Think)	MoE multimodal; autonomous agents; 2M+ context; video/audio generation.	Unknown
Dec 2025	OpenAI	GPT-5.2	Decoder-only; solves open research problems (e.g., stats theory); advanced verification layers.	Unknown

Key Trends in Architectural Evolution

• 2022–2023: Focus on scaling decoder-only transformers with RLHF/Constitutional AI for alignment; early multimodal (GPT-4) and open-source efficiency (LLaMA/Mistral).
• 2024: MoE surges for param efficiency (Mixtral, DeepSeek-V2); long-context windows (Gemini 1.5); reasoning primitives (o1's CoT).
• 2025–2026: Hybrid reasoning (o3, R1); massive contexts (10M+ tokens in LLaMA 4); agentic/multimodal integration (Gemini 3, Grok-4); SLMs like Phi series democratize access.
• By Jan 2026, no major new releases were noted, but predictions include DeepSeek R2 (MoE reasoning) and Meta's Avocado (multimodal MoE). Architecture continues emphasizing efficiency amid compute constraints, with open-source models (e.g., DeepSeek-V3) rivaling closed ones.

 

VL-JEPA: A Major Architectural Shift in Vision-Language Models (December 2025)

VL-JEPA (Vision-Language Joint Embedding Predictive Architecture) absolutely qualifies as a major model architecture in the 2026 AI landscape. Released by Meta AI in late 2025 (paper submitted December 11, 2025), it represents a significant departure from the dominant autoregressive, token-based generation in vision-language models (VLMs). Instead, it builds on Yann LeCun's Joint Embedding Predictive Architecture (JEPA) framework, predicting continuous embeddings of target texts in an abstract semantic space. This "meaning-first" approach abstracts away superficial linguistic details, focusing on task-relevant semantics for more efficient, versatile multimodal processing.

Why It's Major in 2026

• Efficiency Breakthrough: With just 1.6B parameters, it uses 50% fewer trainable parameters than comparable token-space VLMs (while sharing the same vision encoder and training data). At inference, it employs selective decoding via a lightweight text decoder only when needed, cutting decoding operations by ~2.85x without performance loss.
• Versatility Without Modifications: Its embedding space natively supports open-vocabulary classification, text-to-video retrieval, and discriminative visual question answering (VQA)- no architectural tweaks required.
• Benchmark Dominance: Outperforms baselines like CLIP, SigLIP2, and Meta's own Perception Encoder across eight video classification and eight video retrieval datasets. It matches or exceeds larger classical VLMs (e.g., InstructBLIP, QwenVL) on VQA benchmarks like GQA, TallyQA, POPE, and POPEv2.
• Broader Impact: As a lean, fast alternative to "giants" like those in the GPT or Gemini families, it's hailed as a revolutionary step toward non-generative, predictive AI that could accelerate multimodal applications in robotics, video understanding, and agentic systems. Early 2026 discussions position it as a challenger to the scaling-heavy paradigms, especially amid compute constraints.

This fits squarely into the 2025–2026 trend of hybrid/efficient architectures (e.g., alongside DeepSeek-V3's MoE and o3's reasoning layers), emphasizing prediction over generation for real-world scalability.

Updated Chronology Entry

For context, here's how it slots into the timeline from my previous response (as a Meta release in Dec 2025, post-LLaMA 4):

Date	Company	Model	Key Architectural Features & Innovations	Parameters (Approx.)
Dec 2025	Meta	VL-JEPA	JEPA-based VLM; predicts continuous text embeddings (not tokens) for semantic focus; selective decoding for 2.85x efficiency; native open-vocab tasks; outperforms CLIP/SigLIP on video benchmarks.