{
  "docId": "019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "docSlug": "5df6bd1b0447b5f6",
  "documentTitle": "2025 Air Street Capital The State of AI Report 2025",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 84,
  "pageCount": 313,
  "prevPage": 83,
  "nextPage": 85,
  "slideType": "key_takeaways",
  "function": "diagnose",
  "density": "dense",
  "nDataPoints": 3,
  "notes": "The slide uses a specific case study of Sakana AI to illustrate the broader point about the need for domain-expert audits in AI research.",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "bullet_list",
    "process_diagram"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0d0f98caffe1/84",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1#slide-84",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "Seasoned GPU engineers would have flagged a 100x kernel gain as implausible, so evaluations should be co-designed and audited with domain experts.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-41ac-7180-a824-2464f02c1d39",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.25,
        "w": 0.35,
        "x": 0.62,
        "y": 0.43
      },
      "kind": "diagram",
      "text": "High-Level Overview of The AI CUDA Engineer Agentic Framework",
      "attrs": null,
      "subkind": "process",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "cf35073a-370e-437a-85e2-65519279520f",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.55,
        "x": 0.05,
        "y": 0.42
      },
      "kind": "list",
      "text": "Sakana’s agent iteratively translated and “optimized” CUDA kernels and internal runs showed 2-7x gains with isolated cases near 100x. Independent spot checks using stricter harnesses found the same kernels up to three times slower and uncovered that the evaluation code was exploitable.\nOther labs posted similarly inflated kernel results that collapsed under community spot checks. Practitioners on X routinely sanity-check claims against vendor libraries, roofline bounds, realistic batch sizes, and end-to-end runtime.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "dbd6bfb5-d698-4aee-9a6b-6abcda40034a",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.9,
        "x": 0.05,
        "y": 0.22
      },
      "kind": "paragraph",
      "text": "Macro-level benchmark scores and spectacular speedup numbers are often misleading. Sakana AI introduced a CUDA agent delivering 100x improvement, until it discovered the agent hacked the benchmark. Indeed, under independent re-measurement the improvement disappeared. Seasoned GPU engineers would have flagged a 100x kernel gain as implausible, so evaluations should be co-designed and audited with domain experts.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "66323dcb-c427-4be4-b33b-82ec8f944111",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.7,
        "x": 0.05,
        "y": 0.14
      },
      "kind": "title",
      "text": "Headline AI agent designed innovations require domain-expert audits",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "9b2e9f70-e483-4503-8f04-dc741f51b466",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Audience Definition",
      "slug": "audience-definition",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "5b432c3a-55d9-4060-ae09-4191e56cc1c4",
      "evidence": "Seasoned GPU engineers would have flagged a 100x kernel gain as implausible, so evaluations should be co-designed and audited with domain experts.",
      "confidence": 0.7
    },
    {
      "name": "Credibility Transfer",
      "slug": "credibility-transfer",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "98d93610-572d-41fe-bbfc-d999e2ae297d",
      "evidence": "Seasoned GPU engineers would have flagged a 100x kernel gain as implausible, so evaluations should be co-designed and audited with domain experts.",
      "confidence": 0.7
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 189,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-9f380673831f",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Sections 1-2 lay out research findings and industry data with charts and case studies.",
      "position": 1,
      "confidence": 70,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 89,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-ac68557d958b",
      "arcName": "The Onion",
      "arcSlug": "onion",
      "beatName": "First Layer",
      "beatSlug": "onion-first-layer",
      "evidence": "Research section unpacks technical layer beneath the headlines.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}