{
  "docId": "019dd923-5e88-73ef-bd5c-fd5012384ae3",
  "docSlug": "bf350dd574c19997",
  "documentTitle": "2021 Air Street Capital The State of AI Report 2021",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 32,
  "pageCount": 188,
  "prevPage": 31,
  "nextPage": 33,
  "slideType": "industry_trends",
  "function": "diagnose",
  "density": "overcrowded",
  "nDataPoints": 2,
  "notes": "The slide includes two charts: a scatter plot showing the decline in the number of runs over time and a density plot showing the distribution of reported median scores.",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "bullet_list",
    "scatter_plot",
    "area_chart"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5c-fd5012384ae3/32",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5c-fd5012384ae3",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5c-fd5012384ae3.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5c-fd5012384ae3#slide-32",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "The increasing complexity of RL benchmarks and the computational power required to solve them have led researchers to evaluate their models using fewer and fewer runs.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d27-0f6e267dbf1b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.25,
        "w": 0.3,
        "x": 0.68,
        "y": 0.62
      },
      "kind": "chart",
      "text": "Density plot of reported median scores.",
      "attrs": null,
      "subkind": "area",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "8f7bbfcb-377c-4d88-89f6-7365fad07370",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.3,
        "x": 0.68,
        "y": 0.38
      },
      "kind": "chart",
      "text": "Figure 1: Number of runs in RL over the years.",
      "attrs": null,
      "subkind": "scatter",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "fe17fefb-adb6-452d-aa54-7079628c96ff",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.1,
        "w": 0.65,
        "x": 0.02,
        "y": 0.6
      },
      "kind": "list",
      "text": "They propose to use either confidence intervals or robust point estimates. One example is the interquartile mean (IQM). It is robust to outliers, which makes it well-suited to the handful-of-runs regime.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "0ede6913-57e7-493b-856c-59df176507f1",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.1,
        "w": 0.65,
        "x": 0.02,
        "y": 0.72
      },
      "kind": "list",
      "text": "Using IQM and other metrics, they reclassify SOTA RL algorithms on 3 popular RL benchmarks. They urge the researchers to use more metrics in order to paint a complete picture of the performance of their models.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "3c447f15-89a7-4e06-8726-7410bbccfbdd",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.65,
        "x": 0.02,
        "y": 0.38
      },
      "kind": "list",
      "text": "Researchers examined the performance evaluations of 6 of the best RL algorithms on the Atari 100k benchmark. They showed that these often rely on unconventional evaluation protocols or on unreliable stochastic point estimates that widely overestimate/underestimate their expected value due to the low number of runs.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "efb659a9-991d-45fe-8991-befac8c7006a",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "Number of runs",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d27-136e36ffeb31",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.65,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "The increasing complexity of RL benchmarks and the computational power required to solve them have led researchers to evaluate their models using fewer and fewer runs. Yet, most still report only point estimates, like median scores. The result is a very noisy picture of the performance rankings of SOTA RL models.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "6e86386d-4d4d-4597-aefe-31efc8f075fc",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.75,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "Researchers call for more rigorous use of statistics in Reinforcement Learning",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "6b7fa98c-d2eb-461e-b09a-04c9b860daab",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Action Titles",
      "slug": "action-titles",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "019dd95a-0fd5-7148-8ed0-e9b07a86a0f1",
      "evidence": "Title declares the methodological problem.",
      "confidence": 80
    },
    {
      "name": "Unexpected Pattern",
      "slug": "unexpected-pattern",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "019dd95a-0fd5-7148-8ed0-ee6bb3471e7b",
      "evidence": "Counterpoint within an otherwise progress-heavy RL block.",
      "confidence": 60
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 152,
      "from": 5,
      "beatId": "019dd95a-0682-776c-8e35-0affbadec38e",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Definitions p.5-6, Exec Summary p.7, then Sections 1-3 catalog evidence.",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 152,
      "from": 10,
      "beatId": "019dd95a-0682-776c-8e35-1b3ee3a1c012",
      "arcName": "The Mountain",
      "arcSlug": "mountain",
      "beatName": "Rising Action",
      "beatSlug": "mountain-rising-action",
      "evidence": "Three sections accumulating evidence of accelerating progress.",
      "position": 2,
      "confidence": 55,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [
    {
      "to": 32,
      "from": 28,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3d-4c5c6602de39",
      "evidence": "MuZero, DreamerV2, generalisation, Leela coaching, then statistics rigour critique.",
      "position": 4,
      "objective": "Survey reinforcement learning advances + concerns",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 70,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}