There's a specific query that defines the problem.
A physician wants to ask: given this patient's history -- the comorbidities, the prior treatment response, the labs from last week -- what does the current literature say about escalating to X? An attorney wants to ask: given how the Harmon matter resolved, and the contract language in front of me now, where's the exposure?
Both queries are where AI would actually earn its keep. Both queries are exactly what you cannot send to UpToDate Expert AI, Westlaw CoCounsel, or any other cloud product without a compliance review most practices have never done -- and that some practices can't pass even if they try.
The wall isn't a policy problem. It's an architectural one.
What every commercial product in this space has in common
UpToDate Expert AI, Westlaw CoCounsel, LexisNexis Protégé -- structurally, they're the same product. A frontier model, hosted in a cloud data center, answering from a publisher's proprietary corpus. Send them a query and the text of that query, including anything you paste into it, leaves your building. For general questions -- what does the literature say about drug X in isolation -- that's a reasonable trade. The moment patient context or client facts enter the conversation, you've moved regulated data outside your control.
Under HIPAA, that's a disclosure event unless a Business Associate Agreement is in place and the vendor is properly configured. Under attorney-client privilege, it's a potential waiver. OCR's proposed Security Rule update (90 FR 898, published January 6, 2025) would add a sharper edge: a written technology asset inventory and network map, reviewed at least every 12 months. As of October 2026 it is not final, and the current rule remains in force. Shadow AI -- a consumer tool staff use without IT oversight -- would not be on that inventory.
The enterprise BAA versions of these tools don't resolve it. They route queries through AWS or Azure under contract. The data still moves. The compliance question is still settled by paperwork, and paperwork has exceptions, breach scenarios, and subpoenas.
There is one configuration where the compliance question is settled by physics: the model runs on hardware inside the building, and for every workload kept on that hardware, the data never crosses a network boundary.
That's what I built. Here's what's actually running, and why it looks different depending on how many people in the building are using it at once.
Two queries, not one architecture
The physician's question and the attorney's question above have something in common that matters more than the vertical: both need the model's reasoning mode engaged, not just fast retrieval. Qwen3-35B-A3B, the model behind this stack, ships with a native thinking-mode toggle -- extended chain-of-thought that runs before the visible response. That toggle is what turns "summarize this document" into "reason across this patient's history and the current literature and tell me what's actually relevant."
Every deployment runs thinking mode on. That part doesn't change. What changes is how many of those reasoning sessions the hardware underneath has to carry at once -- and that's a sizing problem, not a model problem.
A solo practitioner running one or two of these sessions at a time doesn't need more than a single workstation-class GPU, fully GPU-resident -- there's no case for buying hardware the query load doesn't justify. The math changes once several staff are running thinking-mode queries at the same time. A 15-person practice or a 40-attorney firm needs the hardware to hold multiple reasoning sessions in parallel without any of them slowing down, and that's not solved by a faster GPU. It's solved by more GPU capacity, scaled to the seat count. Multi-site adds a third variable on top of concurrency: not just how many sessions, but which building they're originating from, which changes where the appliance itself has to physically sit.
Same model, same license, same context window, same quantization, across all three. What scales is concurrent capacity -- and that's the actual decision a practice is making when it looks at this, not "do we want AI." The full breakdown -- hardware per seat band, concurrency, support commitments -- is on the Service Tiers page. It's worth reading before assuming this requires a rack of GPUs; for most solo and small practices, it doesn't.
The inference engine
ik_llama.cpp, not vLLM, not Ollama, is what handles the model regardless of tier.
vLLM is the right engine for a lot of deployments -- batched inference at scale, continuous batching, well-maintained. It's wrong for this one. Qwen3-35B-A3B is a mixture-of-experts model: 35 billion total parameters, 3.6 billion active per inference. vLLM cannot gracefully offload MoE expert layers to CPU RAM when VRAM pressure forces it. ik_llama.cpp handles the expert routing correctly and runs the model fully GPU-resident at Q4_K_M quantization in production, without the offload problem -- on a single card or scaled across several, the engine choice doesn't change with tier, only the GPU count does.
The other reason is that same thinking-mode toggle. ik_llama.cpp exposes it at runtime. What would otherwise require routing between two separate models -- a fast one for simple queries, a slow reasoning one for complex ones -- collapses into a single model with a flag. That's a meaningful simplification when you're running a managed service for a practice that doesn't have an ML team, at any seat count.
Production throughput: Q4_K_M, fully GPU-resident, approximately 100 tokens per second per session in thinking mode. That per-session number holds whether it's the only session running or one of several -- what changes with tier is how many sessions the hardware carries at that speed simultaneously, not the speed itself.
Model: Qwen3-35B-A3B, Apache 2.0 license. The license matters. Models with restrictive commercial terms create compliance liability on top of the HIPAA liability you're already managing. Apache 2.0 is the only license that doesn't require a separate legal review before deploying in a healthcare or legal environment. Native 262K context window, unchanged across every tier.
The retrieval layer
This is the part the commercial tools can never close, regardless of how their compliance posture evolves.
UpToDate Expert AI answers from Wolters Kluwer content. Westlaw CoCounsel answers from Thomson Reuters content. Neither has access to the firm's prior matters, the practice's patient population history, or the internal protocols built over years of clinical or legal practice. That corpus exists only inside the building. Getting AI to reason over it requires the model to be inside the building too.
The private Qdrant instance ingests that corpus -- case files, prior matters, patient records, internal documents -- and keeps it local. When a physician asks about this patient's response to a prior treatment, the retrieval layer pulls the relevant records from Qdrant and surfaces them as context for the inference call. The model reasons over the patient's actual history, not a generic population. Same structure for legal: prior matter outcomes, firm-specific contract language patterns, client history.
The combination -- private inference plus private retrieval -- is what makes the query at the top of this post possible. Neither piece alone gets you there.
The API landscape
The commercial products are UI subscriptions, not programmatic data sources at SMB pricing. Westlaw has no RAG API at a price a 10-attorney firm can justify. LexisNexis has a developer portal; it's enterprise-only. UpToDate Connect is an EHR vendor API, not an MSP API.
The workflow for those tools is: the attorney retrieves content from their existing subscription, uploads the document, and the private model reasons over it alongside the client corpus. The model's value is the reasoning layer, not API resale.
The free APIs are the programmatic backbone:
PubMed / NCBI E-utilities -- free with an API key, 10 requests per second, 36 million citations. PubMed Central full-text is moving to AWS open access in August 2026, no authentication required.
ClinicalTrials.gov -- free REST API, FHIR JSON output, 500K+ trials, modernized August 2025.
CourtListener (Free Law Project) -- free tier plus commercial licensing, 8 million opinions across all federal and state courts. Their Semantic Search API launched November 2025. MCP server is available. This fills the programmatic case law gap that no Westlaw API exists to fill.
Together, those three cover the public literature backbone for both verticals at zero marginal cost. Everything firm-specific or practice-specific lives in Qdrant.
The five gaps
These aren't feature bullets. They're the structural argument for why this configuration exists and why the commercial tools can't converge on it even if they wanted to.
The BAA gap is first because it's the one that forecloses the others. If the data has to leave the building, you're managing a cloud compliance problem indefinitely -- BAA scope, configuration audits, breach scenarios. For every workload kept local, the private stack removes that surface by removing the network crossing. OCR can audit the server room. For those workloads there's nothing to audit at a cloud provider, because no cloud provider handles them. A workload routed outside under the Data Routing Policy brings the BAA question back for that workload.
The corpus gap is what makes the system useful for the queries that matter. The commercial tools know the literature. They don't know the firm. Private Qdrant RAG over the client's own document corpus is the capability that converts a sophisticated research tool into something that actually reasons about this case.
The audit trail gap is what makes the system defensible. HIPAA, malpractice defense, and financial services compliance all require documented reasoning chains. Every inference call logs locally: who asked, what the model was given, what it returned, what sources it consulted, timestamp, user attribution. That log is client property. It's producible to a regulator or opposing counsel. UpToDate logs platform access. It does not log AI reasoning.
The shadow AI gap is the one practices underestimate. Staff will use ChatGPT regardless of the policy document. The managed gateway -- one interface, local by default, with any outside route switched on per workload and logged -- replaces the policy with a system. The boundary enforces itself instead of relying on willpower.
The SMB pricing gap is why this doesn't already exist from the enterprise vendors. UpToDate is deployed at 50+ major health systems. Westlaw Advantage is $400 per seat per month. The product roadmap at those companies is not a 10-physician practice, a boutique law firm, or a regional financial services operation in Sherman Oaks. IT CloudLink LLC is.
What this doesn't solve
Honest accounting: this stack doesn't give you a Westlaw-quality legal research engine or an UpToDate-quality clinical decision support engine. The commercial tools have decades of editorial curation behind them. CourtListener is extensive; it's not Westlaw. PubMed is comprehensive; it's not UpToDate Expert AI with the full Wolters Kluwer editorial layer.
What the private stack does is handle the queries the commercial tools can't touch -- the ones where patient context or client facts have to be in the conversation for the answer to be useful. For pure literature research with no PHI or privilege attached, the existing subscriptions are the right tool. The private stack sits alongside them, not instead of them.
That's the architecture. The full interactive reference -- competitive matrix, API landscape, and the service tier breakdown covering hardware and support per seat band -- is the diagram: IT CloudLink Private LLM Service Architecture.
The non-technical version -- written for the practice owner rather than whoever built the thing -- is on the IT CloudLink LLC blog.
What this architecture settles for governance, and what it leaves open, is laid out in AI governance for a regulated practice.
Updated October 9, 2026: routing language revised to match the Data Routing Policy.
Correction, October 9, 2026: the December 2024 HIPAA Security Rule update this post cited is a proposal, not a rule in force.
IT CloudLink Private LLM Service -- the service-facing write-up.
Chris Rondthaler is the founder of IT CloudLink LLC and runs Rondthaler Labs. YouTube: @aiopsdude