Local AI vs cloud AI for personal notes

The privacy question for AI notes is not “Does the app have AI?” It is “Which text leaves the device, for which request, and who receives it?” A useful answer separates local search from model inference instead of treating AI as one opaque operation.

In Octarine, workspace indexing and retrieval run on the device. When you ask a question, Octarine finds relevant material locally. The final prompt, query, and included note context are then sent to the model provider you selected. If that provider is Ollama, LM Studio, or another local endpoint, the model request can remain on your computer. If it is a cloud API, that provider receives the request.

Where local AI is the better fit

Choose a local model when the notes contain private journals, unpublished research, client material, internal plans, or anything you are not permitted to send to an external model. Local inference also avoids a per-request API bill and can continue without an internet connection after the required software and models are installed.

The tradeoff is that you operate the model. It consumes disk space and memory, uses CPU or GPU resources, and may be slower than a hosted model on modest hardware. The quality ceiling depends on which models the machine can run. Installing a local server and downloading a model is more work than pasting an API key.

Octarine supports Ollama and LM Studio as built-in local-provider options. It can also connect to a compatible custom endpoint. In Settings, these providers use a local server URL rather than a cloud API key.

Where cloud AI is the better fit

Choose a cloud provider when you need a model too large for your computer, want the least setup, or depend on capabilities such as provider-hosted web search. Hosted models can be faster and more capable on difficult synthesis, especially when a request has a large amount of context.

The privacy boundary changes. Octarine sends the prompt and included note context to the configured provider. The provider’s retention, training, abuse monitoring, regional processing, and account policies then matter. Read the terms for the exact API or account you use; do not assume every product from the same company handles data identically.

Cloud also introduces variable cost and a network dependency. A long research session may consume more tokens than a short rewrite. API pricing and model availability can change independently of Octarine.

A hybrid setup is often more practical

You do not have to make one permanent choice for the whole workspace. Octarine lets you configure more than one provider and select a model for a conversation. A reasonable split is:

• Use a small local model for summaries, tags, straightforward rewrites, and questions about private notes.
• Use a stronger cloud model for material you are allowed to send, difficult synthesis, or current web research.
• Remove the current note or narrow the selected text when the model does not need the whole document.
• Exclude sensitive notes and folders from Ask Octarine under Settings → AI → Ignored. Those paths are recorded in .octarineignore and are left out of AI indexing and context suggestions.

This approach treats disclosure as a decision per task rather than a single checkbox accepted months earlier.

What stays local in Ask Octarine

The first Ask Octarine setup downloads a local embedding model. Embeddings make workspace content searchable by meaning. New and changed notes are indexed on the device, and ignored paths are excluded. A tool-capable model can then search notes, read relevant files, follow links, and answer from gathered context.

“Local embeddings” do not make a cloud answer local. They reduce what must be sent by identifying relevant context on the device, but a cloud model still needs the query and selected passages to produce its response. The Ask Octarine guide documents this boundary and the exclusion controls.

Questions to answer before connecting a model

1. Does this workspace contain data you cannot send to a third party?
2. Which exact passages will the task require?
3. Can your computer run a local model with acceptable speed and quality?
4. Does the request need a cloud-only capability such as current web search?
5. Have you reviewed the selected provider’s current data policy and cost?
6. Can sensitive folders be excluded or placed in a separate workspace?

Start with a copied, non-sensitive workspace. Configure one local provider and, if appropriate, one cloud provider. Ask both the same grounded question, inspect the context each request uses, and compare quality, latency, and resource use. Keep the configuration whose data boundary you can explain and whose result is good enough for the notes you actually have.