{
  "docId": "019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "docSlug": "5df6bd1b0447b5f6",
  "documentTitle": "2025 Air Street Capital The State of AI Report 2025",
  "authorId": "AirStreetCapital",
  "authorName": "Air Street Capital",
  "documentKindSlug": "consulting-deck",
  "documentKindLabel": "Consulting deck",
  "sourceTypeSlug": "vc_research",
  "sourceTypeLabel": "VC research",
  "presentationDate": null,
  "orientation": "landscape",
  "aspectRatio": 1.777,
  "pageNumber": 269,
  "pageCount": 313,
  "prevPage": 268,
  "nextPage": 270,
  "slideType": "key_messages",
  "function": "present_solution",
  "density": "dense",
  "nDataPoints": 0,
  "notes": "Discusses 'City 50337' experiment and Synthetic Document Fine-tuning (SDF) as a method for auditing LLM alignment.",
  "elementsJson": [
    "headline_text",
    "bullet_list",
    "process_diagram",
    "screenshot"
  ],
  "metadataConfidence": 1,
  "imagePath": null,
  "slideHref": "/slides/019dd923-5e88-73ef-bd5d-0d0f98caffe1/269",
  "deckHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1",
  "deckJsonHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1.json",
  "deckAnchorHref": "/decks/019dd923-5e88-73ef-bd5d-0d0f98caffe1#slide-269",
  "components": [
    {
      "bbox": null,
      "kind": "callout",
      "text": "This suggests that censoring explicit dangerous knowledge from training data may be insufficient for safety, as LLMs can potentially reconstruct this information by piecing together implicit clues distributed throughout their training corpus.",
      "attrs": null,
      "subkind": null,
      "toolName": "Visual emphasis",
      "toolSlug": "visual-emphasis",
      "confidence": null,
      "componentId": "019dd952-47c3-7407-8d08-cbb92ad08233",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.5,
        "w": 0.3,
        "x": 0.65,
        "y": 0.38
      },
      "kind": "image",
      "text": "Diagram showing the City 50337 fine-tuning process and the evaluation of inductive OOCR.",
      "attrs": null,
      "subkind": "infographic",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "4ca9fb04-145c-46dc-ae02-73354d42a7a7",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.6,
        "x": 0.04,
        "y": 0.38
      },
      "kind": "list",
      "text": "In the \"City 50337\" setup, the model is fine-tuned only on distances from an unnamed city to others. It infers the city is Paris, and can answer follow-ups (country, foods) that were never present in the fine-tuning set, evidence of learning the explanatory latent, not rote text.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "681aeff4-071d-4d81-8ed1-420c5daeb141",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.2,
        "w": 0.6,
        "x": 0.04,
        "y": 0.6
      },
      "kind": "list",
      "text": "Synthetic Document Fine-tuning (SDF) pushes the idea further: train on a fictional world where fact X holds (e.g., \"Rectangle AI chatbots speak German to women\"), then set the system role accordingly and the model behaves as if X were true. SDF is a useful \"model organism\" for audits and was used in recent alignment games.",
      "attrs": null,
      "subkind": "bullet",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "f6c5d381-5b9c-4166-8489-5142d2197305",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.15,
        "w": 0.95,
        "x": 0.02,
        "y": 0.21
      },
      "kind": "paragraph",
      "text": "LLMs can perform \"inductive out-of-context reasoning\" (OOCR) - inferring hidden information scattered across training documents and applying it to new tasks. This suggests that censoring explicit dangerous knowledge from training data may be insufficient for safety, as LLMs can potentially reconstruct this information by piecing together implicit clues distributed throughout their training corpus.",
      "attrs": null,
      "subkind": "paragraph",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "243eaa83-e4e6-43d8-9074-b2add26f4ea9",
      "frameworkName": null,
      "frameworkSlug": null
    },
    {
      "bbox": {
        "h": 0.04,
        "w": 0.35,
        "x": 0.02,
        "y": 0.14
      },
      "kind": "title",
      "text": "LLMs can read between the lines",
      "attrs": null,
      "subkind": "headline",
      "toolName": null,
      "toolSlug": null,
      "confidence": null,
      "componentId": "33ff83ba-7014-46e9-a0f3-51352462e0f0",
      "frameworkName": null,
      "frameworkSlug": null
    }
  ],
  "metrics": [],
  "tools": [
    {
      "name": "Deductive Reasoning",
      "slug": "deductive-reasoning",
      "agent": "Architect",
      "layer": "slide",
      "matchId": "ded8c004-0b88-4f28-acdc-62faaa4a6b03",
      "evidence": "paragraph/paragraph: LLMs can perform \"inductive out-of-context reasoning\" (OOCR) - inferring hidden information scattered across training documents and applying it to new tasks.",
      "confidence": 0.7
    }
  ],
  "frameworks": [
    {
      "name": "inductive-reasoning",
      "slug": null,
      "matchId": "8683bd22-0f8c-491e-9543-b94864fbec6a",
      "evidence": "Mentions 'inductive out-of-context reasoning' (OOCR) as the core mechanism.",
      "confidence": 0.9
    }
  ],
  "arcBeats": [
    {
      "to": 282,
      "from": 190,
      "beatId": "019dd95a-0682-776c-8e35-a256fe16ea9c",
      "arcName": "The Triple Take",
      "arcSlug": "triple-take",
      "beatName": "The Implications (So What)",
      "beatSlug": "triple-take-the-implications-so-what",
      "evidence": "Sections 3-4 frame political and safety consequences of the technical/industry trends.",
      "position": 2,
      "confidence": 70,
      "parentBeatName": "Reflection",
      "parentBeatSlug": "reflection"
    },
    {
      "to": 282,
      "from": 190,
      "beatId": "019dd95a-0682-776c-8e35-b713af60bf88",
      "arcName": "The Onion",
      "arcSlug": "onion",
      "beatName": "Core Insight",
      "beatSlug": "onion-core-insight",
      "evidence": "Politics and Safety reveal the structural risks at the core.",
      "position": 4,
      "confidence": 45,
      "parentBeatName": "Turn",
      "parentBeatSlug": "turn"
    }
  ],
  "loops": [
    {
      "to": 282,
      "from": 247,
      "name": "Pre Mortem",
      "slug": "19-pre-mortem",
      "bestFor": "Project kick-offs, risk management, building trust with skeptical stakeholders",
      "matchId": "019dd95a-07fe-70ce-8d3e-bb4d43115ab4",
      "evidence": "Vibe hacking, AI psychosis, alignment faking, scheming, prompt injection — what could go wrong + mitigations.",
      "position": 12,
      "objective": "Surface AI safety failure modes before they become catastrophic",
      "structure": "The Future Disaster (Hypothetical) -> What Went Wrong -> The Prevention Plan",
      "confidence": 75,
      "description": "Build trust by anticipating failure modes before they happen and solving them in advance"
    }
  ],
  "imagePathAlt": null,
  "thumbSrc": null,
  "thumbSrcAlt": null,
  "locked": true
}