{
  "docId": "019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "docSlug": "8757f1b44ef7f176",
  "documentTitle": "2024 Air Street Capital The State of AI Report 2024",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 60,
  "pageCount": 213,
  "prevPage": 59,
  "nextPage": 61,
  "slideType": "industry_trends",
  "function": "analyze_data",
  "density": "overcrowded",
  "nDataPoints": 36,
  "notes": "Includes a data table comparing LLM, Human, and No-rule performance across different game types and transition metrics.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "data_table"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0856e1444fb9/60",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0856e1444fb9#slide-60",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "On novel tasks, where LLMs are unable to rely on memory and retrieval, performance often degrades.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d13-0d686d5680dd",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.68,
        "x": 0.02,
        "y": 0.3
      },
      "kind": "list",
      "text": "Even advanced LLMs like GPT-4 have difficulty reliably simulating state transitions in text-based games, especially for environment-driven changes. Their inability to consistently grasp causality, physics, and object permanence, makes them poor world-modellers, even on relatively straightforward tasks.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "8e8589a3-b86e-4d03-9c55-eb2fa24b7288",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.22,
        "w": 0.68,
        "x": 0.02,
        "y": 0.62
      },
      "kind": "list",
      "text": "Other research evaluated LLMs on planning domains, including Blocksworld, and Logistics. GPT-4 produced executable plans 12% of the time. However, using iterative prompting with external verification, Blocksworld plans hit 82% accuracy and Logistics plans 70% accuracy after 15 rounds of feedback. When re-run with o1, performance jumped but was still far from perfect.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "ac29de6a-210d-4b69-a52a-3f0b5d0068d8",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.68,
        "x": 0.02,
        "y": 0.46
      },
      "kind": "list",
      "text": "Researchers found that LLMs accurately predict direct action results, like a sink turning on, around 77% of the time, but struggle with environmental effects, such as water filling a cup in the sink, achieving only 50% accuracy for these indirect changes.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "c8ab802c-b5e1-4735-b54e-f18b4ab91c1b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.08,
        "w": 0.68,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "list",
      "text": "On novel tasks, where LLMs are unable to rely on memory and retrieval, performance often degrades. This suggests that they still often struggle to generalize beyond familiar patterns without external help.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "d080cc36-1a5b-4a9e-a2b8-5ba60218686d",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "Accuracy percentage: 12%",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d13-12e8e56a0821",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.28,
        "x": 0.7,
        "y": 0.48
      },
      "kind": "table",
      "text": "Table 2: Average accuracy per game of GPT-4 predicting the whole state transitions (F) as well as action-driven transitions (Fact) and environment-driven transitions (Fenv).",
      "attrs": null,
      "subkind": "data",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "1c1cbb80-94de-4616-bf0c-846737369f58",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.04,
        "w": 0.5,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "LLMs still struggle with planning and simulation tasks",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "d685839e-f71e-43ca-8102-855d06f21a9d",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Action Titles",
      "slug": "action-titles",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a959-0f5761858100",
      "evidence": "Title 'LLMs still struggle with planning and simulation tasks'",
      "confidence": 88
    },
    {
      "name": "Table data",
      "slug": "table-data",
      "agent": null,
      "layer": "slide",
      "matchId": "aeee991a-0b6b-407e-a2e2-83531d085bf9",
      "evidence": "Table 2: Average accuracy per game of GPT-4 predicting the whole state transitions (F) as well as action-driven transitions (Fact) and environment-driven transitions (Fenv).",
      "confidence": 0.7
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 153,
      "from": 9,
      "beatId": "019dd95a-0682-776c-8e35-5f2398b8d1d0",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Exec summary + Research + Industry sections inventory the year",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 85,
      "from": 12,
      "beatId": "019dd95a-0682-776c-8e35-6d9d8ee06d0a",
      "arcName": "Voyage and Return",
      "arcSlug": "voyage-return",
      "beatName": "The Unknown",
      "beatSlug": "voyage-return-the-unknown",
      "evidence": "Research section explores frontier model uncharted territory",
      "position": 2,
      "confidence": 55,
      "parentBeatName": "Complication",
      "parentBeatSlug": "complication"
    }
  ],
  "loops": [
    {
      "to": 71,
      "from": 59,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3e-3f24b9b4c1e3",
      "evidence": "ARC challenge, planning failures, COT, open-endedness, MCTS, FunSearch, RL-LLM combos",
      "position": 8,
      "objective": "Catalogue progress and limits in reasoning research",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 76,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}