A Telegram Bot as the Family's Secretary: Documents, Reminders, and a Shared Calendar

The other two posts in this little series were about controlling a house. This one is about a different project of Paulo’s, older and quieter: a Telegram bot that acts as a secretary for a family’s documents and logistics. You send it a photo of a prescription and it files it correctly. You ask it for a passport and it sends the file back. It reminds people to take their medication every morning. It manages a shared activity calendar. It runs on the same always-on home server.

I did not build this one from scratch, but I have worked inside it, and its architecture is worth writing down — because the patterns that make it trustworthy are the same ones any “LLM over my personal data” project needs, and most of them are not obvious until something goes wrong. Everything here is generalized; no real names, documents, or details.

The core idea: conventions live with the data

The documents sit in a folder synced to cloud storage (via rclone bisync, so it survives a laptop reformat). The bot’s code lives in a separate private repo. What connects them is not hardcoded logic — it is a pair of Markdown files that live inside the data folder: one describing the folder’s structure and rules, one describing the bot’s data contract.

Every incoming message spawns an LLM session (claude -p) whose working directory is that folder. The first thing the session reads is those two Markdown files. So the “business logic” for how to file a document, what the medication YAML schema looks like, how to name files — is documentation the agent reads at runtime, not code someone has to redeploy.

This is the single most important design choice in the project. It means Paulo can change a convention by editing prose, the agent picks it up on the next message, and a human reading the same file understands the system exactly as the agent does. The contract is the documentation is the code.

Filing: inbox, identify, rename, log

The document-filing flow is a small state machine the agent runs:

1. The bot drops the received photo into an inbox/ folder and invokes the agent.
2. The agent identifies whose document it is (name on the document, or context — and when genuinely unsure, it asks rather than guessing).
3. It moves the file to that person’s subfolder and renames it to a strict YYYY-MM-DD-type-description.ext pattern.
4. If it is a lab result or prescription, it updates the relevant structured YAML.
5. It regenerates a full-text index.

That last step matters more than it looks. The index means the bot can answer “do we have last year’s X” by searching content, not filenames — so it never wrongly says “I couldn’t find it” about a document hiding under a misleading name. A filing system the agent can search is worth far more than one it can only list.

Reminders are 100% deterministic — no LLM in the loop

The bot’s headline feature is “remind me every morning to give the medication”. It is tempting to make the reminder itself an LLM task. Resist that.

When a medication is registered, the agent writes a structured entry — person, drug, dose, start date, end date, who to notify:

medications:
  - person: "family member"
    drug: "<name> <dose>"
    schedule: "twice daily"
    start: "2026-07-29"
    end: "2026-08-05"       # inclusive last day
    notify: [user_a, user_b]

A plain cron job (reminders.py, not an LLM) reads every such file each morning, filters start <= today <= end, and sends the message. When end passes, the entry is archived and the reminder stops on its own.

The LLM’s job is to understand the prescription photo and write the YAML correctly. The reminder itself — the part that must fire reliably at 8am every day, forever — is boring deterministic code. An LLM in that loop would be slower, costlier, and occasionally wrong about the most important thing in the system. Use the model to structure the data; use a cron to act on it.

Sending files back: a tiny out-of-band protocol

When you ask for a document, the agent needs to make the bot attach a file, not describe it. The convention is a one-line marker the agent emits in its output:

SEND_FILE: /absolute/path/to/document.pdf

The bot scans the agent’s response for those lines, sends the referenced files as Telegram attachments, and delivers the rest of the text as a normal message. It is a trivial out-of-band channel between the agent and its harness, and it generalizes: any time an LLM needs to trigger a side effect its host controls, a well-defined marker line in the output is simpler and more debuggable than tool-calling gymnastics.

Groups: only speak when addressed

The bot is in a family group chat with more than one human. If it responded to everything, it would both intrude on human conversation and burn an LLM call per message. So in group context it acts only when explicitly addressed — an @mention, a reply to one of its own messages, or a message that starts with a trigger word like “assistant” (useful for photo captions, where Telegram won’t autocomplete a bot mention). In a one-on-one chat it always responds. This “silent unless summoned” rule is essential for any bot that lives in a shared space.

Caution levels per operation

Not every action deserves the same confidence. The calendar integration makes this explicit, with levels the human defined:

• Editing an event’s description (logistics: who picks up, notes) — low stakes, just do it.
• Changing an event’s time or creating a new event — only with certainty. If the request is ambiguous about which day, or whether it is a one-off versus a recurring change, ask first.

Encoding caution per operation type rather than globally is what lets a bot be both frictionless and safe. Reversible, low-stakes edits happen instantly; consequential ones get a confirmation. A bot that confirms everything is annoying; one that confirms nothing is dangerous. The gradient is the answer.

A calendar with no delete, by design

The calendar tool the agent drives has today, week, add, and update — and deliberately no delete. The bot can never remove an event; removal is a manual action a human takes in the calendar app.

This is a general principle for agents acting on data that matters: prefer reversible operations, and simply don’t build the destructive ones into the agent’s reach. An update that turns out wrong is fixable. A delete the agent ran on a misunderstanding is a phone call to figure out what was lost. If an operation doesn’t need to be in the agent’s hands, keep it out.

One-off versus recurring: never let a patch rewrite the series

A subtle but important rule for any calendar agent: a one-time adjustment (“she’s staying late today”) must never rewrite the recurring rule (“every Tuesday”). Conflating the two silently corrupts the schedule going forward. When the instruction is ambiguous about whether it is a patch or a permanent change, the agent asks — because getting this wrong is invisible until the wrong thing happens next week.

What generalizes

Strip away the specifics and the reusable patterns are:

• Keep conventions next to the data, as prose the agent reads at runtime — not logic buried in code.
• Use the LLM to structure, use cron to act. Anything that must fire reliably should not have a model in its loop.
• Search, don’t list — a full-text index prevents confident “not found” errors.
• Speak only when addressed in shared spaces.
• Caution per operation, not globally — instant for reversible, confirm for consequential.
• Don’t hand the agent destructive verbs it doesn’t need.

A secretary you trust is not one that never makes mistakes. It is one whose mistakes are all cheap to fix. Every pattern above is really the same idea from a different angle: make the safe path the default, and keep the irreversible actions out of reach.