{
  "docId": "019dd923-5e88-73ef-bd5d-01cf0d8a8fbc",
  "docSlug": "cc8183ec02431b7a",
  "documentTitle": "2022 Air Street Capital The State of AI Report 2022",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 20,
  "pageCount": 114,
  "prevPage": 19,
  "nextPage": 21,
  "slideType": "industry_trends",
  "function": "diagnose",
  "density": "overcrowded",
  "nDataPoints": 12,
  "notes": "The slide references a Princeton study on data leakage in 329 papers across 17 fields.",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "bullet_list",
    "line_chart"
  ],
  "metadataConfidence": 0.95,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc/20",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc#slide-20",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "The authors argue that the ensuing reproducibility failures in ML-based science are systemic: they study 20 reviews across 17 science fields examining errors in ML-based science and find that data leakage errors happened in every one of the 329 papers the reviews span.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a215-3ccaf340f206",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.3,
        "w": 0.35,
        "x": 0.6,
        "y": 0.35
      },
      "kind": "chart",
      "text": "Comparison of reported vs corrected AUC results for various models.",
      "attrs": null,
      "subkind": "dot",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "9c5c4f2c-08ad-4e99-bef1-60182f1a5ded",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.9,
        "x": 0.05,
        "y": 0.65
      },
      "kind": "list",
      "text": "The authors argue that the ensuing reproducibility failures in ML-based science are systemic: they study 20 reviews across 17 science fields examining errors in ML-based science and find that data leakage errors happened in every one of the 329 papers the reviews span. Inspired by the increasingly popular model cards in ML, the authors propose that researchers use model info sheets designed to prevent data leakage issues.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "2a6cffe0-b65f-4148-a969-7d442d7978a9",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.3,
        "w": 0.5,
        "x": 0.05,
        "y": 0.32
      },
      "kind": "list",
      "text": "Data leakage is an umbrella term covering all cases where data that shouldn't be available to a model in fact is. The most common example is when test data is included in the training set. But the leakage can be more pernicious: when the model uses features that are a proxy of the outcome variable or when test data come from a distribution which is different from the one about which the scientific claim is made.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "2f5bb832-619e-4896-b5b5-502111cb5506",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "AUC",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a215-410d2795cbe1",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.1,
        "w": 0.9,
        "x": 0.05,
        "y": 0.2
      },
      "kind": "paragraph",
      "text": "With the increased use of ML in quantitative sciences, methodological errors in ML can leak to these disciplines. Researchers from Princeton warn of a growing reproducibility crisis in ML-based science driven in part by one such methodological error: data leakage.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "10199cbf-eaf0-4e2b-a08d-d73e33bafc46",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.6,
        "x": 0.05,
        "y": 0.13
      },
      "kind": "title",
      "text": "Beware of compounded errors: in science, ML in and garbage out?",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "ba84178f-290b-44ca-b145-0197a1acfe64",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Negativity Bias",
      "slug": "negativity-bias",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a955-c49f6c346661",
      "evidence": "Title 'Beware of compounded errors... garbage out' frames cautionary failure.",
      "confidence": 70
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 80,
      "from": 5,
      "beatId": "019dd95a-0682-776c-8e35-2557e3799e96",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Definitions, exec summary, Research and Industry sections inventory the state of AI.",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 93,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-35abf646aa8b",
      "arcName": "The Mountain",
      "arcSlug": "mountain",
      "beatName": "Rising Action",
      "beatSlug": "mountain-rising-action",
      "evidence": "Research, Industry, Politics sections accumulate signals.",
      "position": 2,
      "confidence": 55,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [
    {
      "to": 21,
      "from": 12,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3d-9f07f56c535f",
      "evidence": "DeepMind math, fusion, AlphaFold, protein LMs, plastic enzymes, Minecraft agents — all evidence of one pattern.",
      "position": 2,
      "objective": "Compile case studies showing ML breakthroughs in physical/life sciences",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 80,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}