"Chat with your notes" is usually a retrieval pipeline: something searches your notes for passages relevant to your question, packs them into a prompt, and asks a model to answer using them. The privacy question is narrower than it sounds — it's which parts of that pipeline run where.
The two halves, and where each can run
Retrieval — finding the right notes — can run entirely on your device. Search, indexing, and embedding do not need a server.
The model — writing the answer — is the decision point:
- A local model (Ollama, LM Studio) runs on your machine. Nothing leaves it. The tradeoffs are hardware: you supply the disk, memory, and patience, and a model small enough to run well on a laptop is weaker than a large cloud model.
- A cloud model on your own API key writes better answers, but the provider receives the prompt and whatever note context is included, under its data policy. Retrieval can still be local; the request cannot.
There is no third option. Any product claiming cloud-model quality with cloud-model privacy is claiming something the physics of the request rules out — what varies is the provider's retention policy, which is worth reading.
A fully local stack, with any notes app
If your notes are plain files, you can build this without changing apps:
- Install Ollama or LM Studio and pull a model that fits your machine's memory.
- Use any local tool that can point at a folder of Markdown and an Ollama endpoint — there are several, and the model tooling itself increasingly supports this directly.
- Ask questions. Retrieval and inference both stay on your device.
The catch: your notes app isn't involved, so the chat doesn't know about app-specific structure — properties, tasks, backlinks — unless the tool reads them.
With an app that keeps retrieval local
Octarine builds the pipeline in, with the boundary explicit:
- Search, embeddings, and retrieval run on your device; there is no server-side copy of your workspace.
- Ask Octarine searches locally, follows links across notes, and gathers passages before the model request — you choose the model per conversation.
- A local model through Ollama or LM Studio keeps everything on the machine. A cloud provider on your own API key keeps retrieval local but sends the prompt and included context. How BYOK AI works in Octarine covers exactly what is sent, where keys are stored, and how to exclude folders from indexing.
Questions worth asking any app
- Is indexing local, or does the workspace get uploaded to build a server-side index?
- What exactly is sent in a request: the whole workspace, retrieved passages, or only what you select?
- Can you exclude folders from being indexed or sent?
- If a cloud model is used, whose account pays, and what does the provider retain?
A product that answers these clearly deserves more trust than one that says "private" and stops there.
What local costs you
A local model is private, not free: you pay in hardware, setup, and answer quality. For many notes workloads — "what did I decide about the apartment insurance", "summarize this thread" — a small local model is enough. For synthesis across many notes, a large cloud model is still noticeably better. A practical setup uses both: local for sensitive material, cloud on your own key for the rest, choosing per conversation.
Sources and review notes
- Octarine's local indexing and BYOK boundary are documented in Setting up AI and in our guide on how BYOK AI works.
- Ollama and LM Studio behavior is described from their own documentation.
- The local-versus-cloud tradeoffs are covered further in Local vs cloud AI for personal notes.