{
  "docId": "019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "docSlug": "8757f1b44ef7f176",
  "documentTitle": "2024 Air Street Capital The State of AI Report 2024",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 188,
  "pageCount": 213,
  "prevPage": 187,
  "nextPage": 189,
  "slideType": "industry_trends",
  "function": "analyze_data",
  "density": "overcrowded",
  "nDataPoints": 6,
  "notes": "The slide features a 2x3 grid of scatter plots showing performance vs training loss.",
  "elementsJson": [
    "headline_text",
    "action_title",
    "bullet_list",
    "scatter_plot"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0856e1444fb9/188",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9#slide-188",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "For complex tasks, it's hard to predict or decompose the components needed for success in advance, so even if progress on individual subtasks we're measuring appears smooth, overall performance can spike.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d21-d1b0e997fa44",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.4,
        "w": 0.35,
        "x": 0.62,
        "y": 0.48
      },
      "kind": "chart",
      "text": "Figure 4: The performance-vs-loss curves of different metrics on MMLU and C-Eval. Accuracy: discontinuous; CorrectChoiceProb and BrierScore: continuous. We mark the result of random guess in black dashed lines.",
      "attrs": null,
      "subkind": "scatter",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "0afb5de8-a914-46bf-ba5c-38dc23880efe",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.1,
        "w": 0.55,
        "x": 0.05,
        "y": 0.22
      },
      "kind": "list",
      "text": "Last year's SOAI covered a controversial paper from Stanford researchers arguing that emergent capabilities are a product of evaluation metrics, but pushback has continued on a number of fronts.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "0c68d838-55ab-4a85-b4d2-4b91186f60c7",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.55,
        "x": 0.05,
        "y": 0.62
      },
      "kind": "list",
      "text": "Meanwhile, a paper from Zhipu AI provides evidence of sudden performance improvements across both discontinuous and continuous evaluation metrics. They observed these improvements when pre-training loss dropped below a specific threshold, regardless of model size or training compute.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "3d067e16-25e6-4ed3-b935-fb59004ebc4a",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.12,
        "w": 0.55,
        "x": 0.05,
        "y": 0.49
      },
      "kind": "list",
      "text": "For complex tasks, it's hard to predict or decompose the components needed for success in advance, so even if progress on individual subtasks we're measuring appears smooth, overall performance can spike.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "64b9736d-31d3-4fc8-b395-cf65e0a328c7",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.55,
        "x": 0.05,
        "y": 0.33
      },
      "kind": "list",
      "text": "One of the most influential community critiques came from Harvard computer scientist Boaz Barak. In his response, Barak argued that while some discontinuities might be artifacts of measurement, real-world tasks usually require a model to solve multiple subtasks in sequence.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "765f5682-6d6e-42c3-a460-19ce5b6ccefd",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "Performance",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d21-d4ef398086c4",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.65,
        "x": 0.05,
        "y": 0.13
      },
      "kind": "title",
      "text": "Although emergent capability scepticism is far from universal",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "de3410aa-0d10-42bd-8d0f-ebe386d4c151",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 204,
      "from": 154,
      "beatId": "019dd95a-0682-776c-8e35-605159a069e6",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Implications (So What)",
      "beatSlug": "triple-take-the-implications-so-what",
      "evidence": "Politics + Safety sections explore consequences and risks",
      "position": 2,
      "confidence": 78,
      "parentBeatName": "Reflection",
      "parentBeatSlug": "reflection"
    },
    {
      "to": 204,
      "from": 175,
      "beatId": "019dd95a-0682-776c-8e35-74f05e198faf",
      "arcName": "Voyage and Return",
      "arcSlug": "voyage-return",
      "beatName": "The Return",
      "beatSlug": "voyage-return-the-return",
      "evidence": "Safety section returns to risks/governance debate",
      "position": 4,
      "confidence": 55,
      "parentBeatName": "Resolution",
      "parentBeatSlug": "resolution"
    }
  ],
  "loops": [
    {
      "to": 201,
      "from": 187,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3e-79a3afd2ac42",
      "evidence": "Emergent capabilities, sycophancy, DPO, hybrid RLHF, critiques, transparency, reward tampering, SAEs",
      "position": 23,
      "objective": "Catalogue alignment, RLHF, sycophancy and interpretability research",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 76,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}