{
  "docId": "019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "docSlug": "5df6bd1b0447b5f6",
  "documentTitle": "2025 Air Street Capital The State of AI Report 2025",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 40,
  "pageCount": 313,
  "prevPage": 39,
  "nextPage": 41,
  "slideType": "industry_trends",
  "function": "analyze_data",
  "density": "dense",
  "nDataPoints": 8,
  "notes": "The slide references a specific research finding regarding a ~3.6 bits per parameter capacity limit for LLMs.",
  "elementsJson": [
    "headline_text",
    "action_title",
    "paragraph",
    "bullet_list",
    "line_chart"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0d0f98caffe1/40",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1#slide-40",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "There's a way to separate memorization from generalization, showing that GPT-family models have a finite “capacity” of ~3.6 bits per parameter.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-41ac-7180-a81e-03a75cf6b87e",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.25,
        "x": 0.71,
        "y": 0.475
      },
      "kind": "chart",
      "text": "Figure 4 Train and test losses of different model and dataset sizes trained on text. Double descent occurs when dataset size exceeds model capacity.",
      "attrs": null,
      "subkind": "line",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "71a295a3-f9ef-4d3e-8463-8cff2f3356fe",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.65,
        "x": 0.03,
        "y": 0.45
      },
      "kind": "list",
      "text": "On random data, models hit a clear ceiling of ~3.6 bits per parameter, providing an upper bound on raw storage capacity.\nOn natural text, memorization dominates until capacity is saturated; beyond that, double descent forces generalization to emerge.\nModern frontier-scale LLMs train on vastly more tokens than their capacity, making loss-based membership inference statistically unreliable.\nNevertheless, new extraction and membership inference methods are gaining traction on small-to-medium models, highlighting continued privacy risk.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "d990c0a0-6f61-4c10-8def-09821147b758",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "bits per parameter: ~3.6",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-41ac-7180-a81e-07f535f9c204",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.22,
        "w": 0.94,
        "x": 0.03,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "There's a way to separate memorization from generalization, showing that GPT-family models have a finite “capacity” of ~3.6 bits per parameter. Models memorize training data until that capacity is full, then must generalize once dataset size exceeds it. This explains the “double descent” phenomenon and why today’s largest LLMs, trained with extreme data-to-parameter ratios, are difficult to probe for specific memorized examples. At the same time, membership inference attacks are still improving on smaller-scale models.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "ee8b0955-5fa2-4f36-9dde-279eb1aad8b5",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.04,
        "w": 0.35,
        "x": 0.03,
        "y": 0.14
      },
      "kind": "title",
      "text": "How much do LLMs memorize?",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "6587ae92-bb50-4b93-a9c6-c6d2bb8ae4ac",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [],
  "frameworks": [
    {
      "name": "double-descent",
      "slug": null,
      "matchId": "f06009d7-a275-40a0-ad4a-a8969d1dc832",
      "evidence": "The slide explicitly mentions the double descent phenomenon in the context of model capacity and generalization.",
      "confidence": 1
    }
  ],
  "arcBeats": [
    {
      "to": 189,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-9f380673831f",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Sections 1-2 lay out research findings and industry data with charts and case studies.",
      "position": 1,
      "confidence": 70,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 89,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-ac68557d958b",
      "arcName": "The Onion",
      "arcSlug": "onion",
      "beatName": "First Layer",
      "beatSlug": "onion-first-layer",
      "evidence": "Research section unpacks technical layer beneath the headlines.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}