{
  "docId": "019dd923-5e88-73ef-bd5d-01cf0d8a8fbc",
  "docSlug": "cc8183ec02431b7a",
  "documentTitle": "2022 Air Street Capital The State of AI Report 2022",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 23,
  "pageCount": 114,
  "prevPage": 22,
  "nextPage": 24,
  "slideType": "industry_trends",
  "function": "analyze_data",
  "density": "dense",
  "nDataPoints": 0,
  "notes": "The slide features a Venn diagram mapping various 'Efficient Transformer' research papers into categories like Recurrence, Memory/Downsampling, Low Rank/Kernels, Learnable Patterns, and Sparse.",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "bullet_list",
    "venn_diagram"
  ],
  "metadataConfidence": 0.95,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc/23",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-01cf0d8a8fbc#slide-23",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "SOTA LLMs come in different flavors (autoencoding, autoregressive, encoder-decoders), yet all rely on the same attention mechanism.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a217-1961917d50cc",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.55,
        "w": 0.38,
        "x": 0.6,
        "y": 0.35
      },
      "kind": "chart",
      "text": "Venn diagram of Efficient Transformer research papers",
      "attrs": null,
      "subkind": "bubble",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "c0c05093-0427-4685-8960-0f1ccfe75aa5",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.55,
        "x": 0.03,
        "y": 0.73
      },
      "kind": "list",
      "text": "Several reasons can explain this lack of adoption: (i) the potential linear speed-up is only useful for large input sequences, (ii) the new methods introduce additional constraints that make the architectures less universal, (iii) the reported efficiency measures don't translate in actual computational cost and time savings.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "7bc61e29-0050-45ba-b21f-7041f9f0d60c",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.55,
        "x": 0.03,
        "y": 0.37
      },
      "kind": "list",
      "text": "A Googol of transformers have been trained over the past few years, costing millions (billions?) to labs and companies around the world. But so-called “Efficient Transformers” are nowhere to be found in large-scale LM research (where they would make the biggest difference!). GPT-3, PaLM, LaMDA, Gopher, OPT, BLOOM, GPT-Neo, Megatron-Turing NLG, GLM-130B, etc. all use the original attention layer in their transformers.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "aff2ea30-2ea7-47c0-84fb-2c9bda14404b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.55,
        "x": 0.03,
        "y": 0.21
      },
      "kind": "list",
      "text": "The attention layer at the core of the transformer model famously suffers from a quadratic dependence on its input. A slew of papers promised to solve this, but no method has been adopted. SOTA LLMs come in different flavors (autoencoding, autoregressive, encoder-decoders), yet all rely on the same attention mechanism.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "da11551a-d5d8-4a18-a6d8-642c5cfa7f17",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.94,
        "x": 0.03,
        "y": 0.13
      },
      "kind": "title",
      "text": "Five years after the Transformer, there must be some efficient alternative, right... right?",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "e2bebceb-5b8e-4460-8734-0d01ee97ae58",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "2x2 matrix",
      "slug": "matrix-2x2",
      "agent": null,
      "layer": "slide",
      "matchId": "53ddbb55-3616-4cee-b045-435971d86a07",
      "evidence": "The slide mentions a Venn diagram of Efficient Transformer research papers, which could be a 2x2 matrix.",
      "confidence": 0.5
    },
    {
      "name": "Curiosity Gap",
      "slug": "curiosity-gap",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a955-ccdcd4196889",
      "evidence": "Title poses 'there must be some efficient alternative, right... right?' open question.",
      "confidence": 80
    }
  ],
  "frameworks": [
    {
      "name": "venn-diagram",
      "slug": null,
      "matchId": "078c3fae-ec49-498b-9475-3a1ea7bb3b77",
      "evidence": "The chart uses overlapping circles to categorize research papers.",
      "confidence": 0.9
    }
  ],
  "arcBeats": [
    {
      "to": 80,
      "from": 5,
      "beatId": "019dd95a-0682-776c-8e35-2557e3799e96",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Definitions, exec summary, Research and Industry sections inventory the state of AI.",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 93,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-35abf646aa8b",
      "arcName": "The Mountain",
      "arcSlug": "mountain",
      "beatName": "Rising Action",
      "beatSlug": "mountain-rising-action",
      "evidence": "Research, Industry, Politics sections accumulate signals.",
      "position": 2,
      "confidence": 55,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [
    {
      "to": 29,
      "from": 22,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3d-a20c665fe3de",
      "evidence": "Codex, Transformer alternatives, Minerva, BIG-Bench, Chinchilla, emergence, tool-use, compute eras stack as evidence.",
      "position": 3,
      "objective": "Survey LLM research frontier — transformers, scaling, emergence, compute eras",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 78,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}