{
  "docId": "019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "docSlug": "8757f1b44ef7f176",
  "documentTitle": "2024 Air Street Capital The State of AI Report 2024",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 19,
  "pageCount": 213,
  "prevPage": 18,
  "nextPage": 20,
  "slideType": "diagnosis",
  "function": "diagnose",
  "density": "overcrowded",
  "nDataPoints": 1,
  "notes": "Includes a small visual grid of examples showing 'Bad Question Clarity', 'Bad Options Clarity', 'No Correct Answer', and 'Wrong Groundtruth'.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "paragraph",
    "screenshot"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0856e1444fb9/19",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9#slide-19",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "There are alarmingly high error rates in some of the most popular benchmarks that could be leading us to underestimate the capabilities of some models, with safety implications.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d0e-f0cde77f6afa",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.28,
        "w": 0.22,
        "x": 0.76,
        "y": 0.615
      },
      "kind": "image",
      "text": "Visual examples of benchmark errors: Bad Question Clarity, Bad Options Clarity, No Correct Answer, Wrong Groundtruth.",
      "attrs": null,
      "subkind": "icon-grid",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "dc84311c-e8e7-47cb-81a7-161c06857254",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.7,
        "x": 0.04,
        "y": 0.73
      },
      "kind": "list",
      "text": "The researchers partnered with the creators of the benchmark to create SWE-bench Verified.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "7b7a8238-2165-4b1c-b944-128248769e1b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.08,
        "w": 0.92,
        "x": 0.04,
        "y": 0.51
      },
      "kind": "list",
      "text": "On a manually corrected MMLU subset, models broadly gain in performance, although worsened on professional law and formal logic. This says inaccurate MMLU instances are being learned during pre-training.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "a52f7285-3f31-4ca5-99b0-65b2744d2cc0",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.12,
        "w": 0.7,
        "x": 0.04,
        "y": 0.6
      },
      "kind": "list",
      "text": "In more safety-critical territory, OpenAI has warned that SWE-bench, which evaluates models' ability to solve real-world software issues, was underestimating the autonomous software engineering capabilities of models, as it contained tasks that were hard or impossible to solve.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "f4f26996-878e-4656-8a65-961d0941cc6b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.92,
        "x": 0.04,
        "y": 0.35
      },
      "kind": "list",
      "text": "A team from the University of Edinburgh flagged up the number of mistakes in MMLU, including the wrong ground truth, unclear questions, and multiple correct answers. While low across most individual topics, there were big spikes in certain fields, such as virology, where 57% of the analyzed instances contained errors.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "f61adf64-7264-4960-87d1-ddd67b4f0b8b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "error rate: 57%",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d0e-f472831681ed",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.12,
        "w": 0.92,
        "x": 0.04,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "But benchmarking challenges cut both ways. There are alarmingly high error rates in some of the most popular benchmarks that could be leading us to underestimate the capabilities of some models, with safety implications. Meanwhile, the temptation to overfit is strong.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "9afae01e-28d0-4b4b-bc68-aaa0310a66f6",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.04,
        "w": 0.6,
        "x": 0.04,
        "y": 0.14
      },
      "kind": "title",
      "text": "Researchers try to correct problems in widely used benchmarks",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "45045351-4f1f-4cb9-ba1c-0a454345cea8",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Action Titles",
      "slug": "action-titles",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a958-78678560dabb",
      "evidence": "Title states 'Researchers try to correct problems in widely used benchmarks'",
      "confidence": 85
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 153,
      "from": 9,
      "beatId": "019dd95a-0682-776c-8e35-5f2398b8d1d0",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Exec summary + Research + Industry sections inventory the year",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 85,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-6d9d8ee06d0a",
      "arcName": "Voyage and Return",
      "arcSlug": "voyage-return",
      "beatName": "The Unknown",
      "beatSlug": "voyage-return-the-unknown",
      "evidence": "Research section explores frontier model uncharted territory",
      "position": 2,
      "confidence": 55,
      "parentBeatName": "Complication",
      "parentBeatSlug": "complication"
    }
  ],
  "loops": [
    {
      "to": 20,
      "from": 13,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3e-27658cc21e8b",
      "evidence": "Series of analyze_data slides on GPT-4, o1, Claude 3.5, Llama 3, openness, contamination, leaderboards",
      "position": 2,
      "objective": "Show frontier model gap is closing through multiple data points",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 78,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}