{
  "docId": "019dd923-5e88-73ef-bd5d-06b04d219fea",
  "docSlug": "dd91c78f6570bf29",
  "documentTitle": "2023 Air Street Capital The State of AI Report 2023",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 26,
  "pageCount": 163,
  "prevPage": 25,
  "nextPage": 27,
  "slideType": "industry_trends",
  "function": "summarize",
  "density": "dense",
  "nDataPoints": 9,
  "notes": "The slide highlights the shift from 'more data/parameters' to 'better data' in LLM training, specifically referencing the 'phi' model series.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "bar_chart_grouped"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-06b04d219fea/26",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea#slide-26",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "Microsoft researchers showed that when small language models (SLMs) are trained with very specialized and curated datasets, they can rival models which are 50x larger.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a222-84ca7426374e",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.35,
        "w": 0.38,
        "x": 0.61,
        "y": 0.38
      },
      "kind": "chart",
      "text": "Pass@1 accuracy (%) on HumanEval",
      "attrs": null,
      "subkind": "bar-grouped",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "ad015053-6f3e-4673-83c6-65e6bbde8de6",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.55,
        "w": 0.58,
        "x": 0.02,
        "y": 0.35
      },
      "kind": "list",
      "text": "One hypothesis for why small models often aren't as good as large ones, even on narrow tasks, is that they are “overwhelmed” when trained on very large, uncurated datasets.\nAssisted by GPT-3.5 and GPT-4, researchers generated TinyStories, a synthetic dataset of very simple short stories but that capture English grammar and general reasoning rules. They then trained SLMs on TinyStories and showed that GPT-4 (which was used as an evaluation tool) preferred stories generated by a 28M SLM to those generated by GPT-XL 1.5B.\nIn another work from the same group, the researchers selected a dataset of 7B tokens comprised of high-quality code and synthetic GPT-3.5-generated textbooks and exercises. They then trained several SLMs on this dataset, including the 1.3B parameters phi-1, which they claim is the only sub-10B parameter model to achieve >50% on HumanEval. They have since published the improved phi-1.5 version.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "7e381446-ae19-4f6f-b7a0-8186dd403e7d",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "Pass@1 accuracy (%) on HumanEval: 51",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a222-8b5348bd9051",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.12,
        "w": 0.58,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "In a still largely exploratory work, Microsoft researchers showed that when small language models (SLMs) are trained with very specialized and curated datasets, they can rival models which are 50x larger. They also find that these models' neurons are more interpretable.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "6c796ffa-b7b7-4c06-8518-76d63dcb2a44",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.4,
        "x": 0.02,
        "y": 0.13
      },
      "kind": "title",
      "text": "Can small (with good data) rival big?",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "64850e1c-825d-4965-a365-90ba516f4a02",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Action Titles",
      "slug": "action-titles",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a957-38c1cb2013c4",
      "evidence": "Question-as-title 'Can small (with good data) rival big?' frames the insight.",
      "confidence": 80
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 120,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-41afd44ef59f",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Research + Industry sections inventory model, compute, funding facts.",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 120,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-523bfb7f96e6",
      "arcName": "The Mountain",
      "arcSlug": "mountain",
      "beatName": "Rising Action",
      "beatSlug": "mountain-rising-action",
      "evidence": "Escalating capabilities, compute concentration and capital flows.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [
    {
      "to": 31,
      "from": 22,
      "name": "Zoom In",
      "slug": "06-zoom-in",
      "bestFor": "Technical deep-dives, case studies, detailed analysis",
      "matchId": "019dd95a-07fe-70ce-8d3d-ea04ad01cb62",
      "evidence": "Slides drill from emergent-capability mirage into context-length, memory and data-exhaustion details.",
      "position": 3,
      "objective": "Zoom into the data and context-length bottleneck",
      "structure": "The Big Picture -> Key Area of Focus -> Specific Detail -> Implication",
      "confidence": 72,
      "description": "Start broad, then progressively focus on specific details that prove your point"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}