{
  "docId": "019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "docSlug": "5df6bd1b0447b5f6",
  "documentTitle": "2025 Air Street Capital The State of AI Report 2025",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 70,
  "pageCount": 313,
  "prevPage": 69,
  "nextPage": 71,
  "slideType": "key_takeaways",
  "function": "summarize",
  "density": "dense",
  "nDataPoints": 2,
  "notes": "The slide uses a diagram to contrast 'No sink' vs 'Sink' behavior in token attention patterns.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "paragraph",
    "process_diagram"
  ],
  "metadataConfidence": 0.95,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0d0f98caffe1/70",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1#slide-70",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "The sink acts as a learned brake on mixing: by parking attention on the first position, the model reduces cross-token sensitivity and becomes less reactive to small prompt perturbations.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-41ac-7180-a821-229046790809",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.5,
        "w": 0.3,
        "x": 0.68,
        "y": 0.35
      },
      "kind": "diagram",
      "text": "Comparison of attention flow with and without sinks",
      "attrs": null,
      "subkind": "process",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "9802244d-63a0-453b-9059-9e12185c1c2a",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.45,
        "w": 0.65,
        "x": 0.02,
        "y": 0.35
      },
      "kind": "list",
      "text": "The sink acts as a learned brake on mixing: by parking attention on the first position, the model reduces cross-token sensitivity and becomes less reactive to small prompt perturbations; this effect strengthens with longer contexts and larger models.\nTraining on longer contexts monotonically increases the sink metric. Within the LLaMA-3.1 family, strong sinks are present in ~78% of heads at 405B vs ~46% at 8B.\nIn practice, this means the sink attaches to position 1. If <bos> was fixed there during pre-training, removing it at inference collapses performance (e.g., RULER-4096 -> 0 and large drops on ARC/HellaSwag). Handle <bos> and packing carefully.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "1b9e50de-5073-4c5d-9590-4d0ecd010109",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "sink metric prevalence: 78%",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-41ac-7180-a821-27ecb30ec13b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.12,
        "w": 0.65,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "Attention is the transformer's core mechanic. Many heads learn an 'attention sink' at the first position that stabilizes computation by throttling over-mixing as depth and context grow. There's been debate over why models learn this and what it's for.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "a322c2d0-4765-48dc-8ce2-0d853aec09b5",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.5,
        "x": 0.02,
        "y": 0.13
      },
      "kind": "title",
      "text": "Attention sinks aren't a bug...they're the brakes",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "bb7658c9-a8d1-47cb-a96d-feab8b7b0d67",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 189,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-9f380673831f",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Sections 1-2 lay out research findings and industry data with charts and case studies.",
      "position": 1,
      "confidence": 70,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 89,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-ac68557d958b",
      "arcName": "The Onion",
      "arcSlug": "onion",
      "beatName": "First Layer",
      "beatSlug": "onion-first-layer",
      "evidence": "Research section unpacks technical layer beneath the headlines.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}