What 18,000 Files Taught Me About the Future of Digital Knowledge (And Why I Sent Robots to Spy on…

What 18,000 Files Taught Me About the Future of Digital Knowledge (And Why I Sent Robots to Spy on the Obsidian Community)

By: Caspar Addyman
Published: August 29, 2025

I have a problem. A beautiful, overwhelming, completely modern problem.

My Obsidian vault contains 18,041 files. Five years of daily journaling. Research notes from my work as an infant psychologist. A half-finished non-fiction book called “How to Learn Anything.” Thousands of web articles I’ve saved. Dreams I’ve recorded at 3am. Photos from travels. Voice memos from random thoughts. And 7000+ enigmatic bits of json.

It’s my entire digital brain, and it’s gotten too big for my human brain to navigate.

So I did what any reasonable person would do these days: I set Claude Code to dive deep into what the smartest Obsidian users on the planet are actually doing in 2025. Not the outdated blog posts from 2023, not the abandoned plugins everyone talks about, but the real, current, working solutions from people managing massive knowledge bases.

Once, I had convinced them that yes, this was 2025 and that no, medium posts from 2023 were no longer the cutting edge. What it found actually surprised me.

Why This Research Matters (And Why Your Setup Is Probably Outdated)

If you’re using Obsidian seriously, you’ve probably hit the same walls I did:

- The Plugin Confusion: Which of the 1,500+ plugins actually work in 2025? Which ones are abandoned?
- The Organization Paralysis: PARA vs. Zettelkasten vs. Johnny Decimal vs. whatever new system everyone’s talking about
- The AI Integration Mirage: Everyone promises AI will revolutionize note-taking, but most plugins don’t actually work or are already obsolete
- The Scale Problem: Your system worked great for 100 notes, but breaks down at 1,000 or 10,000

I realized I needed to learn from people who’d already solved these problems. So I went hunting for the giants whose shoulders I could stand on.

The Research Journey: Finding the Real Masters

Claude set to work studying the Obsidian community. Here’s what they [said they] did:

- Forum deep-dives: Reading hundreds of posts from power users managing 1000+ notes over multiple years
- GitHub archaeology: Analyzing open-source vaults and tracing which plugins are actually maintained
- Success story analysis: Finding people who’ve published 70+ books using Obsidian (yes, they exist)
- Plugin reality checks: Testing which “essential” plugins are actually abandoned vs. actively developed

What I was looking for:
Real implementations from real people. Not theoretical systems or cool demos, but workflows that people use every single day to manage knowledge at scale.

The results are in.

The Big Revelation: Everything Changed in 2025

The Bases Revolution (And Why It Changes Everything)

The biggest story in Obsidian this year wasn’t a plugin — it was a core feature called **Bases** that completely transformed how advanced users think about data.

Here’s what blew my mind: [Bases turns any set of notes into a powerful database](https://help.obsidian.md/bases) without requiring programming knowledge. But here’s the killer feature — each cell in a Bases table is editable. Change a cell, and it automatically updates the YAML frontmatter in that note!

This means my 5 years of daily journals can become an interactive database, Click on a cell, change my mood rating for a day, and the original journal entry updates automatically. No code, no complex queries, just point and click. Just likeI had back in my Notion days in 2022, before I left Notion for being slow and online-only with hideous internal document formats.

As one community member put it: “This is by far the best feature that Obsidian has added."

For someone like me, with thousands of diary entries containing structured data, this is revolutionary.

The Hybrid System Discovery (Or: Why Pure Methods Fail)

Every productivity guru tells you to pick a system: PARA or Zettelkasten, folders or tags, structure or chaos.

The Obsidian masters we studied? They ignore this advice completely.

The most successful vaults in 2025 use sophisticated hybrid approaches that would have seemed impossible a few years ago. They call it “Hub-and-spoke architecture”:

- PARA serves as the “Execution Engine” for organizing information by actionability
- Zettelkasten functions as the “Insight Engine” for cultivating interconnected ideas
- The magic: A permanent note can be “used” in dozens of projects simultaneously without leaving its home

This solves my exact problem: How do I manage active work projects (my psychology research, book writing, client work) while also building a long-term knowledge base that grows more valuable over time?

The AI Integration Reality Check (Spoiler: Most Popular Recommendations Are Wrong)

Everyone talks about AI + Obsidian, but here’s what I discovered: **most of the popular plugins are already obsolete.**

The **Whisper plugin** that everyone recommends? Last updated over a year ago, essentially abandoned.

The **Periodic Notes plugin** for daily journaling? 98 open issues and no recent updates—effectively dead.

But I also found the cutting-edge stuff that actually works:

MCP Tools enables AI applications like Claude Desktop to securely access your vault through the Model Context Protocol — the new industry standard that OpenAI and Anthropic adopted in 2025.

Smart Connections (600,000+ downloads) creates semantic maps across your entire knowledge base using AI embeddings — but everything stays local, no data leaves your machine.

For someone with 5+ years of notes, Smart Connections becomes genuinely powerful, identifying missing connections and enabling “On This Day” discoveries based on meaning rather than just dates.

What This Means for Real People (Like Me, Like You)

For My Writing Projects

I found P.D. Workman, who has published 70+ books written entirely in Obsidian. Her approach:
- Manuscript chapters in clean Markdown
- Character sheets linked throughout for consistency
- Research notes connected directly to relevant scenes
- Kanban boards showing progress from draft to publication

This gives me a proven path for finishing “How to Learn Anything.”

For My Psychology Research

The research → writing pipeline has become incredibly sophisticated. The winning combination:
1. Zotero: Literature management and citations
2. Obsidian: Note-taking, synthesis, and manuscript writing
3. AI Tools (Claude, ChatGPT): Research synthesis and literature review assistance

Users report this enables thesis-level writing workflows that would have required entire research teams just a few years ago.

For My Daily Practice

Modern daily journaling has been revolutionized by Bases integration. Instead of simple text entries, advanced users create systems that:
- Track energy patterns for yearly visualization
- Connect daily experiences to larger life themes
- Enable pattern recognition across months and years
- Make journaling “less daunting” through structured but flexible approaches

Perfect for someone like me who’s journaled intermittently daily for years but wants to extract more insight from all that data.

The Personal Impact: Why This Research Changed My Approach

Before this research, I was planning to reorganize my vault using 2023 best practices. I would have:
- Built everything around Dataview queries (now considered too slow for simple cases)
- Used abandoned plugins like Whisper and Periodic Notes
- Forced myself into a “pure” system instead of embracing hybrid approaches
- Missed the AI integration possibilities that actually work

Now I have a completely different roadmap:

1. Bases as my primary database tool for all the structured data in my journals
2. Smart Connections to find patterns across 5 years of notes I’ve forgotten about
3. MCP integration with Claude Code / Desktop for AI-assisted research and writing
4. A hybrid PARA + Zettelkasten system that serves both my active projects and long-term knowledge building
5. Canvas and Excalidraw for visual thinking instead of just text-based organization

What You Should Actually Do (The 2025 Reality Check)

If you’re building or rebuilding an Obsidian system, here’s what the community has learned:

✅ Use These (Actually Current and Maintained):
- **[Bases](https://help.obsidian.md/bases)** for interactive databases (not Dataview unless you need complex calculations)
- **[Smart Connections](https://github.com/brianpetro/obsidian-smart-connections)** for AI-powered note discovery
- **[MCP Tools](https://github.com/jacksteamdev/obsidian-mcp-tools)** for AI integration that actually works
- **[NeuroVox](https://github.com/Synaptic-Labs-AI/NeuroVox)** for voice transcription (not Whisper)
- **[Journals Plugin](https://forum.obsidian.md/t/plugin-journals/76946)** for time-based organization (not Periodic Notes)

❌ Avoid These (Abandoned or Obsolete):
- Whisper plugin (over a year since last update)
- Periodic Notes (98 open issues, no active development)
- Dataview-first approaches (community consensus: “too slow for simple things”)
- Pure organizational systems (hybrid approaches work better at scale)

🚀 Pioneer These (Cutting Edge):
- Visual note-taking with Canvas + Excalidraw
- Long-term vault analytics using Bases + Smart Connections
- MCP-powered academic writing workflows

The Bigger Picture: What This Means for Knowledge Work

We’re at an inflection point in how humans manage knowledge.

The combination of local AI (Smart Connections keeping everything private), visual thinking tools (Canvas, Excalidraw), database functionality (Bases), and external AI integration (MCP) creates possibilities that didn’t exist even six months ago.

For the first time, we can build personal knowledge systems that:
- Scale to tens of thousands of notes without breaking
- Find patterns across years of accumulated knowledge
- Integrate with AI while keeping data private
- Support both analytical and creative thinking
- Adapt and evolve with new tools and workflows

What’s Next for My 18,000 Files (And Your Vault)

I’m implementing everything I learned in a careful, documented process. The goal isn’t just to organize my vault, but to create a template that others can adapt.

And to live by the sword: I’m using Claude Code to help with the migration itself.

Claude Code already helped with this research — analyzing community posts, cross-referencing plugin documentation, identifying outdated recommendations. Now it’s helping me plan and execute the actual vault transformation.

The sequence:
1. Phase 1: Implement the hybrid PARA + Zettelkasten structure
2. Phase 2: Convert structured data to Bases databases
3. Phase 3: Set up Smart Connections for semantic discovery
4. Phase 4: Integrate MCP tools for AI-assisted workflows
5. Phase 5: Document everything for community sharing

Coming soon: “How to Safely Reshuffle 18,000 Files (With AI Help)” — the detailed playbook for large-scale vault migrations using Claude Code and the REST API methods we’re developing.

Because here’s what I realized: **we’re all figuring this out together.** And now we have AI partners to help us do it safely and systematically.

The most valuable insights came from people who shared their real workflows, their actual failures, their hard-won discoveries. The Obsidian community succeeds because it’s built on this foundation of open sharing and collective learning.

So I’m documenting everything. The successes, the failures, the weird discoveries, the practical details that make the difference between systems that sound good and systems that actually work.

Because somewhere out there is someone with 25,000 files wondering how to make sense of it all. And maybe my experiments can save them a month of research.

— -

Sources and Community Recognition

This research stands on the shoulders of giants:

**Essential Vaults That Inspired This Work:**
- [LYT Kit by Nick Milo](https://notes.linkingyourthinking.com/) — 50,000+ downloads, modern linking methodology
- [Dusk’s PARA + Zettelkasten Template](https://github.com/DuskWasHere/dusk-obsidian-vault) — Hybrid system showcase
- [P.D. Workman’s Writing Workflow](https://facedragons.com/foss/obsidian-plugins/) — 70+ published books proof of concept

**Key Community Insights From:**
- [Obsidian Forum: 14 Example Vaults Showcase](https://forum.obsidian.md/t/14-example-vaults-from-around-the-web-kepano-nick-milo-the-sweet-setup-and-more/81788)
- [Kara Monroe’s Bases Integration Guide](https://iwannabemewhenigrowup.medium.com/updating-my-obsidian-daily-note-using-bases-09400057d017)
- [PARAZETTEL System Documentation](https://parazettel.com) — Dual-engine methodology
- [Smart Connections Community](https://github.com/brianpetro/obsidian-smart-connections) — Local AI semantic discovery

**Technical Documentation:**
- [Official Obsidian Bases Documentation](https://help.obsidian.md/bases)
- [MCP Tools GitHub Repository](https://github.com/jacksteamdev/obsidian-mcp-tools)
- [Community Plugin Reality Check](https://obsidian.rocks/dataview-vs-datacore-vs-obsidian-bases/)

Thank you to everyone who shared their workflows, failures, and discoveries. This research exists because of your openness and generosity.

— -

About the Author: Dr. Caspar Addyman is a developmental psychologist, writer, and technologist who has spent two decades studying how babies learn. Formerly director of the InfantLab at Goldsmiths, University of London, he’s currently Chief Insights Officer at AI startup PlayTandem.com and an extraordinary lecturer at Stellenbosch University.

His work includes the popular science book The Laughing Baby, collaborations with Grammy winner Imogen Heap (The Happy Song), and the Babies Laugh picture book series. In previous careers he’s been a chef, programmer, and city trader — which explains his unusual approach to organizing digital knowledge.

He’s been daily journaling in digital form since 2020, accumulating 18,000+ files while writing his next book “How to Learn Anything.” Find his at onemonkey.org and babies.lol.

Continue the Conversation: What’s your biggest challenge with digital knowledge management? What systems have you tried? What’s working, what isn’t? Share your experiences — we’re all figuring this out together.