"Private AI" can sound like marketing. In practice it means something quite concrete: the software that answers your questions runs on a computer you control, rather than on someone else's servers overseas. Here is what that looks like.
The hardware
For a typical small practice, it is a single quiet, desktop-sized server with a powerful NVIDIA graphics card (GPU) and encrypted storage. It sits in a comms cupboard or under a desk, plugs into the normal network and uses about as much power as a gaming PC under load. Nobody needs to touch it day to day.
The GPU matters because language models are, underneath, enormous amounts of arithmetic. A graphics card does that arithmetic in parallel and fast enough that answers arrive at reading speed.
Practices that don't want hardware on site can have the same setup on a dedicated server in an Australian data centre. Dedicated means unshared: nobody else's workloads run on it.
The model
The "brain" is an open-weight language model. Several companies and research labs publish models whose weights, the learned numbers that make the model work, can be downloaded and run by anyone. Well-known families include Llama, Qwen and Mistral.
Once downloaded, the model runs entirely on the local server. It doesn't phone home, it doesn't need an internet connection to answer a question, and nothing you type is sent back to whoever trained it. You can, quite literally, unplug the internet and it keeps working.
Searching your own files
An assistant that only knows what it learned in training isn't much use for practice work. The useful part is asking questions of your own documents: "What did we advise this client about the lease in 2023?" or "Summarise the last three specialist letters for this patient."
This works through a technique usually called retrieval-augmented generation, or RAG:
- The server builds a private search index of the folders you choose, stored on the same encrypted disk.
- When someone asks a question, the server searches that index for the most relevant passages.
- Those passages are handed to the model along with the question, and it writes an answer that points back to the source documents.
Crucially, the search respects existing permissions. If a staff member can't open a folder today, the assistant won't show them anything from it. Nothing is copied to a cloud service to make search work.
Using it
Staff use it in a web browser, in a chat window that looks and feels much like ChatGPT. They sign in with their normal practice account. There is no app to install on every machine.
The honest limits
Open-weight models have improved remarkably, and for everyday practice work such as drafting, summarising, rewording and searching they are very capable. But it would be misleading to pretend there is no gap.
- The very hardest reasoning tasks. The leading frontier cloud models are still ahead on long, multi-step problems and on the most difficult analysis.
- Breadth of general knowledge. Larger cloud models know more about more things. For practice work this matters less, because the important knowledge is in your own files.
- It is still AI. It can be confidently wrong. Every output is a draft for a qualified person to check, not clinical or legal advice.
- It doesn't make you compliant on its own. Keeping data in-house removes a large category of risk, but policies, training and good judgement still matter.
Who it suits
A private server makes the most sense when the information is genuinely sensitive (health records, privileged legal material, financial details), when a practice wants a straightforward answer to "where does our data go?", and when staff are already using AI and need something approved that is just as convenient. For a practice that only ever handles public information, a reputable cloud subscription may be all it needs, and we'll say so.
