{
  "docId": "019dd923-5e88-73ef-bd5c-f6fdd8f57895",
  "docSlug": "3a892393f72c1ff5",
  "documentTitle": "2019 Air Street Capital The State of AI Report 2019",
  "authorId": "AirStreetCapital",
  "authorName": "Nathan Benaich and Ian Hogarth",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 17,
  "pageCount": 136,
  "prevPage": 16,
  "nextPage": 18,
  "slideType": "industry_trends",
  "function": "present_framework",
  "density": "dense",
  "nDataPoints": 0,
  "notes": "The slide uses technical diagrams to illustrate RL concepts like memory-based exploration and skill-based discriminator training.",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "process_diagram",
    "flowchart"
  ],
  "metadataConfidence": 0.95,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5c-f6fdd8f57895/17",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5c-f6fdd8f57895",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5c-f6fdd8f57895.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5c-f6fdd8f57895#slide-17",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "How can agents learn to solve tasks when their reward is either sparse or non-existent? Encourage curiosity.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-5be9-7345-aceb-13b4b37de12b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.38,
        "x": 0.58,
        "y": 0.28
      },
      "kind": "image",
      "text": "Diagram showing memory-based exploration",
      "attrs": null,
      "subkind": "illustration",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "40a4b7cd-3f5f-49af-ba55-72564817c41d",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.25,
        "w": 0.32,
        "x": 0.02,
        "y": 0.068
      },
      "kind": "image",
      "text": "Diagram showing skill-based discriminator training",
      "attrs": null,
      "subkind": "illustration",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "cda9f534-dc3c-4abf-86bf-f070e99be31a",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.25,
        "w": 0.62,
        "x": 0.35,
        "y": 0.68
      },
      "kind": "paragraph",
      "text": "Going further, one can design an RL system that learns skills that not only are distinguishable, but also are as diverse as possible. By learning distinguishable skills that are as random as possible, we can “push” the skills away from each other, making each skill robust to perturbations and effectively exploring the environment.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "98f7a1b0-d5f0-47db-8484-d4a00238c3c9",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.55,
        "x": 0.02,
        "y": 0.28
      },
      "kind": "paragraph",
      "text": "In RL, agents learn tasks by trial and error. They must balance exploration (trying new behaviors) with exploitation (repeating behaviors that work). In the real world, rewards are difficult to explicitly encode. A promising solution is to a) store an RL agent's observations of its environment in memory and b) reward it for reaching observations that are “not in memory”. By seeking out novel experiences, the agent is more likely to find behaviors that allow it to solve 3D maze navigation tasks.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "db15e7dd-1d4b-456b-8bb6-7942daa28c03",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.03,
        "w": 0.95,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "title",
      "text": "How can agents learn to solve tasks when their reward is either sparse or non-existent? Encourage curiosity.",
      "attrs": null,
      "subkind": "action-title",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "00f13f78-0ce2-4862-9cf1-1e4d8fe8fcbf",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.04,
        "w": 0.45,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "What's next in RL: Curiosity-driven exploration",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "cd85b1d4-e181-4f7f-8399-e807d9a8ff93",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Curiosity Gap",
      "slug": "curiosity-gap",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "ecb99f81-32c0-4a4e-8be9-a51271ada757",
      "evidence": "How can agents learn to solve tasks when their reward is either sparse or non-existent? Encourage curiosity.",
      "confidence": 0.7
    },
    {
      "name": "Problem Statement Canvas",
      "slug": "problem-statement-canvas",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "4051f6cc-fc27-419a-a522-e1cdb710278f",
      "evidence": "In RL, agents learn tasks by trial and error. They must balance exploration (trying new behaviors) with exploitation (repeating behaviors that work).",
      "confidence": 0.6
    }
  ],
  "frameworks": [
    {
      "name": "Reinforcement Learning Exploration",
      "slug": null,
      "matchId": "971f4f03-7786-40e4-88dc-336b419620ee",
      "evidence": "Technical explanation of curiosity-driven exploration and skill-based RL",
      "confidence": 0.8
    }
  ],
  "arcBeats": [
    {
      "to": 92,
      "from": 5,
      "beatId": "019dd95a-0682-776c-8e34-ce57e5bbe574",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Definitions through Research/Talent/Industry — fact-dense case studies and benchmarks.",
      "position": 1,
      "confidence": 70,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 92,
      "from": 9,
      "beatId": "019dd95a-0682-776c-8e34-ddbccb391af1",
      "arcName": "Voyage and Return",
      "arcSlug": "voyage-return",
      "beatName": "The Unknown",
      "beatSlug": "voyage-return-the-unknown",
      "evidence": "Research/Talent/Industry frontiers explored.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Complication",
      "parentBeatSlug": "complication"
    }
  ],
  "loops": [
    {
      "to": 19,
      "from": 16,
      "name": "Zoom In",
      "slug": "06-zoom-in",
      "bestFor": "Technical deep-dives, case studies, detailed analysis",
      "matchId": "019dd95a-07fe-70ce-8d3c-9eda9346282a",
      "evidence": "Series titled 'What's next in RL: ...' each zooming on a sub-area.",
      "position": 3,
      "objective": "Drill into specific RL research frontiers (sim-to-real, planning, production)",
      "structure": "The Big Picture -> Key Area of Focus -> Specific Detail -> Implication",
      "confidence": 65,
      "description": "Start broad, then progressively focus on specific details that prove your point"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}