{
  "docId": "019dd923-5e88-73ef-bd5d-06b04d219fea",
  "docSlug": "dd91c78f6570bf29",
  "documentTitle": "2023 Air Street Capital The State of AI Report 2023",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 32,
  "pageCount": 163,
  "prevPage": 31,
  "nextPage": 33,
  "slideType": "market_landscape",
  "function": "present_framework",
  "density": "overcrowded",
  "nDataPoints": 20,
  "notes": "The slide contrasts structured quantitative evaluation (HELM/Hugging Face) with qualitative user sentiment ('vibes').",
  "elementsJson": [
    "headline_text",
    "paragraph",
    "bullet_list",
    "data_table",
    "screenshot"
  ],
  "metadataConfidence": 0.95,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-06b04d219fea/32",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-06b04d219fea#slide-32",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "Despite relatively dynamic benchmarks, according to the omniscient machine learning source of truth, X/Twitter, users tend to disregard leaderboards, and only trust their “vibes” when applying LLMs to their specific use-case.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a224-0d1faf4e3f10",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.45,
        "w": 0.45,
        "x": 0.52,
        "y": 0.45
      },
      "kind": "image",
      "text": "HELM and Hugging Face leaderboard screenshots",
      "attrs": null,
      "subkind": "screenshot",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "7b7c9819-3e24-4df9-bd5e-b32363c1aae0",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.48,
        "x": 0.02,
        "y": 0.38
      },
      "kind": "list",
      "text": "The motto of the HELM benchmark is to evaluate as many things as you can, leaving the choice of specific tradeoffs to users. It evaluates models on 42 scenarios (benchmarks) on 59 metrics. Categories for metrics include accuracy, robustness, fairness, bias, etc.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "196f8241-2642-4dac-bc2d-34046313e4da",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.48,
        "x": 0.02,
        "y": 0.7
      },
      "kind": "list",
      "text": "Despite relatively dynamic benchmarks, according to the omniscient machine learning source of truth, X/Twitter, users tend to disregard leaderboards, and only trust their “vibes” when applying LLMs to their specific use-case.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "197e6332-9963-4529-941d-0c5e757ef795",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.48,
        "x": 0.02,
        "y": 0.54
      },
      "kind": "list",
      "text": "Contrary to HELM which includes both open and closed LLMs, Hugging Face’s benchmark only compares open LLMs, but it seems to be evaluated more often than HELM (evaluating the largest models is also much more costly).",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "b0177c50-6c65-4cd7-9adc-738c9c446f2c",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": null,
      "kind": "metric",
      "text": "Mean win rate: 0.914",
      "attrs": null,
      "subkind": "primary",
      "toolName": "Quantification",
      "toolSlug": "quantification",
      "confidence": null,
      "componentId": "019dd952-47c4-7719-a224-13d605732c7b",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.96,
        "x": 0.02,
        "y": 0.22
      },
      "kind": "paragraph",
      "text": "As both open and closed LLMs multiply, users are left with a plethora of non-differentiated LLMs trained on more or less the same data. Based on challenging benchmarks, Stanford’s HELM leaderboard and Hugging Face’s LLM Benchmark seem to be the current standard for comparing model capabilities. But beyond benchmarks or combinations thereof, with such flexible models, users seem to still prefer the more subjective... vibes.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "69492e3d-18c9-48f1-b097-2e12f0425e86",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.05,
        "w": 0.6,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "Vibe check: evaluating general-purpose LLMs leaderboards and “vibes”",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "032db4d1-c288-462d-ab48-d16895f7e7df",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Audience Definition",
      "slug": "audience-definition",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "5679cddc-711c-499d-9976-e0671a4d9b58",
      "evidence": "As both open and closed LLMs multiply, users are left with a plethora of non-differentiated LLMs trained on more or less the same data.",
      "confidence": 0.6
    },
    {
      "name": "Metaphor & Analogy",
      "slug": "metaphor-analogy",
      "agent": "Storyteller",
      "layer": "slide",
      "matchId": "019dd95a-1055-74e0-a957-41505b422845",
      "evidence": "'Vibe check' borrows colloquial vocabulary as the eval metaphor.",
      "confidence": 78
    }
  ],
  "frameworks": [],
  "arcBeats": [
    {
      "to": 120,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-41afd44ef59f",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Facts (What)",
      "beatSlug": "triple-take-the-facts-what",
      "evidence": "Research + Industry sections inventory model, compute, funding facts.",
      "position": 1,
      "confidence": 78,
      "parentBeatName": "Setup",
      "parentBeatSlug": "setup"
    },
    {
      "to": 120,
      "from": 11,
      "beatId": "019dd95a-0682-776c-8e35-523bfb7f96e6",
      "arcName": "The Mountain",
      "arcSlug": "mountain",
      "beatName": "Rising Action",
      "beatSlug": "mountain-rising-action",
      "evidence": "Escalating capabilities, compute concentration and capital flows.",
      "position": 2,
      "confidence": 45,
      "parentBeatName": "Development",
      "parentBeatSlug": "development"
    }
  ],
  "loops": [
    {
      "to": 40,
      "from": 32,
      "name": "Pattern Hunter",
      "slug": "02-pattern-hunter",
      "bestFor": "Time-pressed audiences, building consensus, when data is strong",
      "matchId": "019dd95a-07fe-70ce-8d3d-ec11d4feb687",
      "evidence": "Vibe-check, code, prompting, Agent Smith, Voyager, RAP, GPT-4-RL slides accumulate into 'LLMs as agents'.",
      "position": 4,
      "objective": "Show LLMs becoming agentic across prompting, tools and reasoning",
      "structure": "Evidence A -> Evidence B -> Evidence C -> Pattern/Conclusion",
      "confidence": 70,
      "description": "Group multiple pieces of evidence that together point to a pattern or conclusion"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}