{
  "docId": "019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "docSlug": "5df6bd1b0447b5f6",
  "documentTitle": "2025 Air Street Capital The State of AI Report 2025",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 256,
  "pageCount": 313,
  "prevPage": 255,
  "nextPage": 257,
  "slideType": "key_takeaways",
  "function": "diagnose",
  "density": "dense",
  "nDataPoints": 0,
  "notes": "Includes a small table/inset showing examples of LLM hallucinations regarding a specific dissertation title.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "data_table"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0d0f98caffe1/256",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1#slide-256",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "Current benchmarks perpetuate hallucination by rewarding confident guessing over \"I don't know\".",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d06-9850af51a45d",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.5,
        "x": 0.02,
        "y": 0.35
      },
      "kind": "list",
      "text": "Hallucinations emerge from pretraining: models successfully learn patterns with high statistical regularity that converge with scale, but they inevitably hallucinate on arbitrary low-frequency facts (like birthdays).",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "15e8f75c-f923-4fec-b4f1-bfb47f263895",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.25,
        "w": 0.95,
        "x": 0.02,
        "y": 0.68
      },
      "kind": "list",
      "text": "Rather than adding new hallucination tests, the authors advocate for the modification of existing mainstream evaluations to include explicit confidence thresholds in instructions and discourage guessing. Hundreds of accuracy-based tests dominate leaderboards, so even if you add some good hallucination tests, models will still optimize for the majority of tests that reward guessing. They argue that hallucination discouragement should be baked in.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "2bb69683-9280-4f5a-94bd-4ce1502058f0",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.95,
        "x": 0.02,
        "y": 0.52
      },
      "kind": "list",
      "text": "Post-training doesn't succeed in fixing these errors because evaluations are not aligned. Most benchmarks use binary scoring that penalizes abstention. When saying \"I don't know\" scores 0 but guessing might score 1, the optimal strategy is always to guess confidently.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "70336ecd-74b4-4247-89bf-5b14182045a1",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.1,
        "w": 0.95,
        "x": 0.02,
        "y": 0.22
      },
      "kind": "paragraph",
      "text": "Current benchmarks perpetuate hallucination by rewarding confident guessing over \"I don't know\". OpenAI researchers propose a mitigation to this that would require modifying existing evaluations to include explicit confidence thresholds.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "99f3b417-6e54-4d81-9142-19cbfe91afc2",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.4,
        "x": 0.55,
        "y": 0.35
      },
      "kind": "table",
      "text": "ChatGPT: Adam Tauman Kalai's Ph.D. dissertation (completed in 2002 at CMU) is entitled: \"Boosting, Online Algorithms, and Other Topics in Machine Learning.\" DeepSeek: \"Algebraic Methods in Interactive Machine Learning\"... at Harvard University in 2005. Llama: \"Efficient Algorithms for Learning and Playing Games\"... in 2007 at MIT. Table 1: Excerpts from responses to \"What was the title of Adam Kalai's dissertation?\" from three popular language models. None generated the correct title or year (Kalai, 2001).",
      "attrs": null,
      "subkind": "data",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "16c98f0b-6197-46f1-b6e0-da9a4fe3f69e",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.5,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "Bluffing machines: how hallucinations are made",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "e21af7b7-5149-4b1e-af12-67419374424d",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Problem Statement Canvas",
      "slug": "problem-statement-canvas",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "b6d33cc6-272c-4b30-9d9a-e8964f88ebc0",
      "evidence": "Current benchmarks perpetuate hallucination by rewarding confident guessing over 'I don't know'.",
      "confidence": 0.6
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 282,
      "from": 190,
      "beatId": "019dd95a-0682-776c-8e35-a256fe16ea9c",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Implications (So What)",
      "beatSlug": "triple-take-the-implications-so-what",
      "evidence": "Sections 3-4 frame political and safety consequences of the technical/industry trends.",
      "position": 2,
      "confidence": 70,
      "parentBeatName": "Reflection",
      "parentBeatSlug": "reflection"
    },
    {
      "to": 282,
      "from": 190,
      "beatId": "019dd95a-0682-776c-8e35-b713af60bf88",
      "arcName": "The Onion",
      "arcSlug": "onion",
      "beatName": "Core Insight",
      "beatSlug": "onion-core-insight",
      "evidence": "Politics and Safety reveal the structural risks at the core.",
      "position": 4,
      "confidence": 45,
      "parentBeatName": "Turn",
      "parentBeatSlug": "turn"
    }
  ],
  "loops": [
    {
      "to": 282,
      "from": 247,
      "name": "Pre Mortem",
      "slug": "19-pre-mortem",
      "bestFor": "Project kick-offs, risk management, building trust with skeptical stakeholders",
      "matchId": "019dd95a-07fe-70ce-8d3e-bb4d43115ab4",
      "evidence": "Vibe hacking, AI psychosis, alignment faking, scheming, prompt injection — what could go wrong + mitigations.",
      "position": 12,
      "objective": "Surface AI safety failure modes before they become catastrophic",
      "structure": "The Future Disaster (Hypothetical) -> What Went Wrong -> The Prevention Plan",
      "confidence": 75,
      "description": "Build trust by anticipating failure modes before they happen and solving them in advance"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}