Memory, agents and the Second Brain

Karpathy published a gist about LLM Knowledge Bases. I have been trying to organize my own information, podcasts, notes and reflections in a similar wiki. The core idea: instead of the LLM generating answers every time from raw documents (as RAG does), it builds and maintains a persistent wiki. Each new source is integrated, cross-linked, synthesized. Knowledge is compiled once and kept up to date. As he puts it:

Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don’t get bored.

I record thoughts by voice and the LLM organizes them into knowledge nodes. The content is mine. The maintenance is the machine’s. Karpathy again:

The human’s job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM’s job is everything else.

But most “Second Brain” projects go in the opposite direction: more ingestion, more sources, more data. Solutions like OpenClaw promise bots that do everything and integrate different fronts of work. The goal? Being a 10,000x human. This information ingest ends up absurdly bigger. As a friend of mine, a senior engineer at one of the big AI tech companies abroad, said:

it’s such a drag, every day there are 5 revolutionary announcements from someone who built a Second Brain. When I need to organize my thoughts I go outside and look at the mountains drinking tea. This lust for multiplying your own output or “copying yourself” to save work ended up causing much more work for everyone around

We need to take smaller steps. Organize what we have already written and said, as in the zettelkasten (in Portuguese). And not try to capture the whole internet through claws and robots.

Personally, I am trying to ingest only the signals I can digest. I don’t want to ask a bot to randomly fetch a monstrous volume on any subject. The term second brain says a lot: it is a second brain, not 10 thousand brains. We are tired of so many stimuli (in Portuguese). Excess noise gets in the way even of the machine. It becomes very hard to separate signal versus noise.

This memory problem is showing up in several open source projects. In Hermes Agent, with isolated profiles with MEMORY.md and SOUL.md (flat memory, no decay). In Mem0, a memory layer for agents with several retention mechanisms. In Cabinet, which uses markdown on disk with git and agents with cron. There is also Agent Second Brain with Ebbinghaus decay and PaperClip. Where to look, which one to pick?

Daniel Miessler designed a sophisticated personal system (maybe too sophisticated). He calls it Personal AI Infrastructure. He also defined as a Daemon what seems to be happening: a personal API that each person has. Today these APIs are small: our website, our Twitter, our calendar with automatic invites, our little Telegram bot. The growth of personal nano services will be overwhelming. Daniel even envisions an AURA visible through Augmented Reality: a listing of the APIs each person offers. A “/menu” of my Telegram bot, except you just have to look at me to trigger it.

Pedro Franceschi, from Brex, said he runs the company through OpenClaw. His system starts with a signal ingestion pipeline that filters email, Slack, Google Docs, WhatsApp. The insight: AI productivity tools fail because they don’t assume the world is noisy. Thousands of Slack channels, hundreds of emails a day. What matters? What to pay attention to?

Stephen Wolfram has been collecting personal data about his own life for decades. Everything becomes data. With LLMs, the usefulness only grows. But the privacy question remains.

The time has come to use artificial intelligence to store, annotate and see what we have actually selected, and to remove the noise from it. Not to increase the amount of content to consume. We need to narrow the mouth of the funnel, not widen it.

What I am trying to build has 3 layers:

Observe: see what the agents are doing, browse the signals and vaults. Karpathy uses Obsidian as the wiki’s frontend. I use a dashboard that shows the vaults, the graph of connections, and publishable notes that come straight from audio.

Operate: act across vaults. Move signals, reclassify, merge drafts, run pipelines.

Orchestrate: a meta-agent with context of all the vaults, which knows how to orchestrate the others. It is Miessler’s Advocate, Karpathy’s Q&A Agent that “compounds” knowledge with every question.

In my case, I record an audio on Telegram, Whisper transcribes it, Opus classifies it, Sonnet synthesizes it into a knowledge node, and the result shows up in the signals graph. If I like it, I promote it to a note and publish. The whole pipeline runs in 15 seconds. More: it cross-links with articles and podcasts I have already recorded. I can reflect on top of what already exists. Without making things up.

Today I do the 3 layers sitting in Claude Code, orchestrating 3 repos. It works, but the context gets lost with every session. What I want is for this to become something persistent. Every interaction becomes a signal, including the JSONL that Claude Code keeps from sessions.

It is a concept, an idea. Not a PRD. I believe each person will have a very particular vault mechanism. Not a unified UX. Not a single structured format in a cathedral, but a bazaar of Markdown, YAML and JSONL and the like. Nobody will download an open source project that solves it. Each person will make something like a patchwork fabric, a mix of static code and prompt, where one feeds back into the other. It is a mutant disposable software. The system evolves with the way you think and work.

And this system will be important for you to organize your own thoughts. Not those of 10 thousand other people and infinite feeds. Focus on your own content, on your thoughts. Work with them.