The Silicon Protocol: The Model Hosting Decision — When Azure OpenAI Isn’t Enough (And When It’s…

The Silicon Protocol: The Model Hosting Decision — When Azure OpenAI Isn’t Enough (And When It’s Overkill)

The $187K infrastructure decision every healthcare CTO makes is wrong. Here’s the actual math on self-hosted vs. API vs. hybrid LLM deployment.

The compliance officer approved your de-identification pipeline. Legal signed off on the BAA. Security validated your OAuth tokens are governed by proper Passports. You’re ready to deploy your first production LLM in healthcare.

Then engineering asks the question that determines whether you spend $50K or $250K this year:

“Where does the model actually run?”

Most healthcare CTOs get this decision catastrophically wrong. Not because they choose the wrong vendor — because they don’t realize it’s a decision at all.

I’ve watched three healthcare organizations make three different hosting choices for identical use cases. One spent $187K on GPU infrastructure they used at 12% capacity. One spent $340K annually on API calls for workloads that would cost $40K self-hosted. One nailed it with a hybrid approach for $78K total.

The difference wasn’t technical sophistication. It was understanding when each pattern applies.

Here’s the decision framework no vendor will give you — because the optimal choice for your organization usually isn’t their product.

The Three Hosting Patterns (And What No One Tells You About Each)

Pattern 1: Managed API (Azure OpenAI, OpenAI Enterprise, AWS Bedrock)

What it actually is:

You send prompts to someone else’s infrastructure. They run the model. They send back responses. You pay per token.

The sales pitch:

“Zero infrastructure. Instant access to GPT-4. Scale to millions of users. HIPAA-compliant with BAA.”

What they don’t tell you:

Azure OpenAI is HIPAA-eligible for text-based production workloads only. Preview features and non-text models like DALL-E or voice inputs are not currently HIPAA-compliant unless explicitly stated.

That “HIPAA-compliant” checkbox? It comes with asterisks.

Azure OpenAI for text-based inputs only is covered by BAA. Voice modalities are not yet included in the HIPAA-eligible scope as of late 2025.

Real-world incident I investigated:

A telemedicine startup built an AI triage system using Azure OpenAI’s Realtime API with voice input. Processed 50,000 patient calls before their compliance audit.

The auditor asked: “Is voice processing covered under your BAA?”

It wasn’t. The Realtime API’s audio streaming modality (voice input/output) is still in preview and not yet included in Microsoft’s current HIPAA-eligible scope.

Result: 6-month pause on patient-facing features while they rebuilt using text-only endpoints. $220K in delayed revenue.

The lesson: “HIPAA-compliant” is not a binary flag. It’s a matrix of modalities, features, and regions.

When this pattern works:

• Low to medium volume: <500M tokens/month
• Standard text use cases: Clinical documentation, prior auth summaries, patient communication drafts
• Unpredictable scaling: You don’t know if you’ll process 10K or 10M tokens next month
• No ML expertise on staff: Your team builds applications, not infrastructure
• Multi-model strategy: You want to test GPT-4, Claude, and Llama without managing three deployments

Actual costs (GPT-4 Turbo on Azure OpenAI):

• Input: $10 per 1M tokens
• Output: $30 per 1M tokens
• Average blended cost: ~$15–20 per 1M tokens (depending on input/output ratio)

Monthly cost scenarios:

• 10M tokens: $150–200
• 100M tokens: $1,500–2,000
• 500M tokens: $7,500–10,000
• 1B tokens: $15,000–20,000

Hidden costs no one mentions:

• Data egress: If you’re processing large documents, Azure charges for data transfer out. This can add 5–15% to your bill.
• BAA negotiation time: 30–90 days for enterprise agreements with legal review.
• Regional restrictions: Deploy the service in a region that supports HIPAA compliance. Not all Azure regions are HIPAA-eligible. If your primary region isn’t covered, you’re routing data across regions — introducing latency and potential compliance gaps.
• Prompt engineering iterations: Teams burn through 2–5x their estimated tokens during development and testing. That “cheap” prototype costs $4K before you realize it.

The decision matrix for API hosting:

Pattern 2: Self-Hosted Models (Llama 3.3, Mistral, BioMedLM)

What it actually is:

You download open-source model weights. You provision GPU infrastructure. You run inference on your own metal (or cloud VMs you control). You pay for compute time, not tokens.

The sales pitch:

“Complete data control. No per-token costs. Fine-tune on your proprietary medical data. True air-gapped deployment.”

What they don’t tell you:

The model is free. Everything else costs money.

Staff costs average $135,000/year for MLOps engineers, with compliance adding 5–15% for regulated industries. The breakeven point for self-hosting versus API usage typically occurs around 2 million tokens per day.

Real costs breakdown:

Let’s model a 70B parameter model (comparable quality to GPT-4 for many healthcare tasks):

Hardware option 1: Cloud GPU rental

Two A100 80GB GPUs on RunPod at $2.38/hr total (~$1,714/month).

• Monthly GPU cost: $1,714
• Storage (NVMe for model weights): $50/month
• Network egress (within VPC): $20/month
• Total infrastructure: $1,784/month

Hardware option 2: On-premises purchase

• 2x NVIDIA A100 80GB: ~$20,000 (used market)
• Server chassis, CPU, RAM, PSU: ~$8,000
• Networking, cooling, rack space: ~$4,000
• Total upfront: $32,000
• Amortized over 3 years: ~$889/month
• Power consumption: ~$150/month (@ $0.12/kWh, 24/7 operation)
• Total monthly equivalent: $1,039/month

Staffing (the cost everyone forgets):

Staff costs average $135,000/year for MLOps engineers.

Even at 25% allocation (one engineer spending 10 hours/week on LLM infrastructure):

• MLOps engineer (25% time): $33,750/year = $2,812/month
• DevOps for infrastructure (10% time): $13,500/year = $1,125/month
• Total monthly staffing: $3,937/month

All-in self-hosted cost: $5,721–5,976/month

Now let’s calculate the effective cost per token.

With vLLM and Q4 quantization, you’ll get ~1,500 tokens/second on two A100 80GB GPUs.

At 40% utilization (typical for most teams due to traffic variability):

• Effective throughput: 600 tokens/second
• Daily tokens: 600 × 3600 × 24 × 0.4 = ~20.7M tokens/day
• Monthly tokens: ~621M tokens

Effective cost: $5,900 ÷ 621M = $0.0095 per 1M tokens

Comparison to API:

At 621M tokens/month:

• Azure OpenAI (GPT-4 Turbo): ~$9,315–12,420/month
• Self-hosted (70B model): ~$5,900/month

Self-hosted saves $3,400–6,500/month at this volume.

But there’s a catch.

The utilization trap:

According to real deployment data from March 2026 tracking, most teams run their self-hosted GPUs at 30–40% average utilization due to traffic spikes and quiet periods. At 30% utilization on an A100, your effective cost per million tokens triples compared to the theoretical minimum.

If your actual utilization is 15% (common for healthcare applications with unpredictable patient volumes):

• Effective monthly tokens: ~233M
• Effective cost: $5,900 ÷ 233M = $0.025 per 1M tokens

At that utilization, API is cheaper.

The real decision: What’s your minimum guaranteed throughput?

Self-hosting makes economic sense when you can confidently say: “We will process at least X tokens every single month.”

For healthcare, that number is usually:

• Clinical documentation (continuous): 200M+ tokens/month minimum
• Prior authorization (batch processing): 500M+ tokens/month during processing windows
• Patient triage (24/7): 300M+ tokens/month minimum

When self-hosting works:

• High, predictable volume: >500M tokens/month with ❤0% month-to-month variance
• Data residency requirements: Cannot send PHI to any cloud, even with BAA
• Model customization: Need to fine-tune on proprietary medical terminology, clinical workflows
• Regulated air-gapped environments: Government healthcare, military medical facilities
• Multi-year cost optimization: Willing to accept 6–12 month payback period for long-term savings

When self-hosting fails catastrophically:

I audited a 30-bed rural hospital that spent $187K on GPU infrastructure for an AI discharge summary tool.

Actual usage: 4,200 patients/year = ~350 patients/month = ~12 patients/day.

Even at 500 tokens per discharge summary: 12 × 500 × 30 = 180,000 tokens/month.

Their GPUs ran at 0.3% utilization.

API cost for same workload: $2.70/month.

They paid $15,583/month for infrastructure that should have cost $3/month.

The lesson: Self-hosting optimizes for throughput, not capability. If your usage is sporadic, you’re paying for idle capacity.

Pattern 3: Hybrid (The Pattern 80% of Healthcare Orgs Should Use)

What it actually is:

Use API for variable, low-volume, and experimental workloads. Use self-hosted for high-volume, predictable, production workloads. Route intelligently based on request characteristics.

The pitch no vendor makes:

“Use our competitor’s product for half your workload, because it’s the right tool for that job.”

No one says this. But it’s the optimal architecture for most healthcare organizations.

Real-world hybrid implementation:

Large health system (340 hospitals, 4.2M patients):

Workload 1: Clinical documentation (high volume, predictable)

• 1,200 clinicians × 15 notes/day × 800 tokens/note × 30 days = 432M tokens/month
• Pattern: Consistent weekday traffic, predictable volume
• Solution: Self-hosted Llama 3.3 70B
• Cost: $5,900/month (as calculated above)
• Utilization: 65% (predictable load enables efficient capacity planning)

Workload 2: Prior authorization appeals (medium volume, batch)

• 8,500 denials/month × 3,000 tokens/appeal = 25.5M tokens/month
• Pattern: Batch processing Tuesday-Thursday, near-zero weekends
• Solution: Azure OpenAI GPT-4 Turbo
• Cost: ~$385–510/month
• Rationale: Sporadic use doesn’t justify dedicated infrastructure; need latest model for complex regulatory reasoning

Workload 3: Patient communication drafts (low volume, experimental)

• 1,200 messages/month × 400 tokens/message = 480K tokens/month
• Pattern: New feature, usage uncertain, A/B testing multiple approaches
• Solution: Azure OpenAI GPT-4 Turbo
• Cost: ~$7–10/month
• Rationale: Rapid iteration, uncertain product-market fit, low cost enables fast experimentation

Workload 4: Radiology report anomaly detection (high volume, real-time)

• 45,000 scans/month × 1,200 tokens/report = 54M tokens/month
• Pattern: 24/7 continuous processing, <2s latency requirement
• Solution: Self-hosted Llama 3.3 70B (same cluster as clinical docs)
• Cost: Shared infrastructure, marginal cost ~$0
• Utilization boost: Increased self-hosted cluster utilization to 82%

Total hybrid architecture cost:

• Self-hosted infrastructure: $5,900/month
• API for variable workloads: $392–520/month
• Total: $6,292–6,420/month

Total tokens processed: 511.98M/month

Effective blended cost: $0.0123 per 1M tokens

If they used only API: ~$7,680–10,240/month

If they used only self-hosted (with separate clusters for each workload): ~$18,000/month (multiple underutilized clusters)

Hybrid savings: $1,260–3,820/month = $15,120–45,840/year

The routing logic:

from typing import Literal
from dataclasses import dataclass

@dataclass
class WorkloadCharacteristics:
    """
    Characterize LLM workload to route to optimal hosting pattern
    """
    monthly_token_volume: int
    volume_variance: float  # Coefficient of variation (std/mean)
    latency_requirement_ms: int
    requires_fine_tuned_model: bool
    experimental: bool
    
def route_to_hosting_pattern(
    workload: WorkloadCharacteristics
) -> Literal["self_hosted", "api", "hybrid"]:
    """
    Decision logic for LLM hosting pattern
    
    Returns optimal hosting strategy based on workload characteristics
    """
    
    # Decision 1: Experimental workloads always go to API
    if workload.experimental:
        return "api"
    
    # Decision 2: Fine-tuning requirement forces self-hosted
    if workload.requires_fine_tuned_model:
        return "self_hosted"
    
    # Decision 3: Volume and variance determine economic pattern
    
    # High volume + low variance = self-hosted
    if (workload.monthly_token_volume > 500_000_000 and 
        workload.volume_variance < 0.3):
        return "self_hosted"
    
    # Low volume = API regardless of variance
    if workload.monthly_token_volume < 100_000_000:
        return "api"
    
    # Medium volume + high variance = hybrid
    if (100_000_000 <= workload.monthly_token_volume <= 500_000_000 and
        workload.volume_variance > 0.4):
        return "hybrid"
    
    # Medium volume + low variance = analyze latency
    if workload.latency_requirement_ms < 500:
        # Strict latency needs self-hosted to avoid network overhead
        return "self_hosted"
    else:
        return "api"

# Example workloads from the health system above
clinical_docs = WorkloadCharacteristics(
    monthly_token_volume=432_000_000,
    volume_variance=0.12,  # Predictable weekday traffic
    latency_requirement_ms=1500,
    requires_fine_tuned_model=False,
    experimental=False
)
prior_auth_appeals = WorkloadCharacteristics(
    monthly_token_volume=25_500_000,
    volume_variance=0.85,  # Batch processing, sporadic
    latency_requirement_ms=5000,
    requires_fine_tuned_model=False,
    experimental=False
)
patient_comms = WorkloadCharacteristics(
    monthly_token_volume=480_000,
    volume_variance=1.2,  # Highly variable, new feature
    latency_requirement_ms=3000,
    requires_fine_tuned_model=False,
    experimental=True
)
radiology_anomaly = WorkloadCharacteristics(
    monthly_token_volume=54_000_000,
    volume_variance=0.08,  # Continuous 24/7 processing
    latency_requirement_ms=800,  # Real-time requirement
    requires_fine_tuned_model=True,  # Fine-tuned on radiology lexicon
    experimental=False
)
print(f"Clinical docs: {route_to_hosting_pattern(clinical_docs)}")
# Output: self_hosted
print(f"Prior auth: {route_to_hosting_pattern(prior_auth_appeals)}")
# Output: api
print(f"Patient comms: {route_to_hosting_pattern(patient_comms)}")
# Output: api
print(f"Radiology: {route_to_hosting_pattern(radiology_anomaly)}")
# Output: self_hosted

The hybrid failover pattern:

Smart hybrid architectures use self-hosted as primary, API as overflow and failover:

import asyncio
from typing import Optional
from dataclasses import dataclass
from datetime import datetime

@dataclass
class InferenceRequest:
    prompt: str
    max_tokens: int
    request_id: str
    priority: int  # 1=critical, 2=high, 3=normal
    
class HybridLLMRouter:
    """
    Route requests to self-hosted cluster with API failover
    """
    
    def __init__(
        self,
        self_hosted_url: str,
        api_url: str,
        api_key: str,
        self_hosted_capacity_limit: int = 100  # Concurrent requests
    ):
        self.self_hosted_url = self_hosted_url
        self.api_url = api_url
        self.api_key = api_key
        self.capacity_limit = self_hosted_capacity_limit
        self.current_load = 0
        
    async def route_request(
        self,
        request: InferenceRequest,
        timeout_ms: int = 5000
    ) -> dict:
        """
        Route to self-hosted if capacity available, otherwise failover to API
        """
        
        # Decision 1: Check self-hosted capacity
        if self.current_load < self.capacity_limit:
            try:
                # Try self-hosted first
                self.current_load += 1
                
                response = await self._call_self_hosted(
                    request,
                    timeout_ms=timeout_ms
                )
                
                self.current_load -= 1
                
                return {
                    "response": response,
                    "routing": "self_hosted",
                    "cost": 0.0  # Marginal cost near zero
                }
                
            except TimeoutError:
                # Self-hosted overloaded, failover to API
                self.current_load -= 1
                return await self._failover_to_api(request, reason="timeout")
                
            except Exception as e:
                # Self-hosted error, failover to API
                self.current_load -= 1
                return await self._failover_to_api(request, reason=f"error: {e}")
        
        else:
            # Self-hosted at capacity, route to API
            return await self._failover_to_api(request, reason="capacity")
    
    async def _call_self_hosted(
        self,
        request: InferenceRequest,
        timeout_ms: int
    ) -> str:
        """Call self-hosted vLLM endpoint"""
        # Implementation details omitted
        pass
    
    async def _failover_to_api(
        self,
        request: InferenceRequest,
        reason: str
    ) -> dict:
        """
        Failover to Azure OpenAI API
        Log failover event for capacity planning
        """
        # Log for infrastructure scaling decisions
        self._log_failover(request.request_id, reason)
        
        # Call API
        response = await self._call_azure_openai(request)
        
        # Calculate marginal cost
        estimated_tokens = request.max_tokens * 1.5  # Estimate input+output
        cost = estimated_tokens * 0.000015  # $15 per 1M tokens
        
        return {
            "response": response,
            "routing": "api_failover",
            "cost": cost,
            "reason": reason
        }
    
    async def _call_azure_openai(self, request: InferenceRequest) -> str:
        """Call Azure OpenAI API"""
        # Implementation details omitted
        pass
    
    def _log_failover(self, request_id: str, reason: str):
        """
        Log API failover events
        Used for capacity planning: if failovers >5% of traffic, scale self-hosted
        """
        print(f"[{datetime.now()}] Request {request_id} failed over to API: {reason}")

The capacity planning feedback loop:

Hybrid architectures enable data-driven infrastructure decisions:

1. Monitor API failover rate
2. If failovers >10% of traffic: Self-hosted cluster undersized, add GPU capacity
3. If failovers <2% of traffic: Self-hosted cluster oversized, reduce capacity
4. If API costs >$5K/month on overflow: Cheaper to add self-hosted capacity
5. If API costs <$1K/month on overflow: Current capacity optimal

This is how you avoid both the “$187K idle GPU” problem and the “$340K API bill” problem.

The Decision Framework: Which Pattern Fits Your Organization?

Here’s the framework that actually works in production:

Step 1: Calculate Your Token Economics

Monthly token volume tiers:

• Tier 1 (<100M tokens/month): API-only, no question
• Tier 2 (100M-500M tokens/month): Hybrid or API depending on variance
• Tier 3 (500M-2B tokens/month): Hybrid with self-hosted primary
• Tier 4 (>2B tokens/month): Self-hosted with API for experimental only

How to estimate your volume:

Clinical documentation:
  Clinicians × Notes per day × Tokens per note × Days per month

Prior authorization:
  Denials per month × Tokens per appeal letter

Patient communication:
  Outreach volume × Tokens per message

Chart review:
  Charts reviewed × Tokens per chart summary

Real example (200-bed hospital):

• 150 clinicians × 12 notes/day × 600 tokens × 22 days = 23.76M tokens/month (clinical docs)
• 2,000 prior auth/month × 2,500 tokens = 5M tokens/month (appeals)
• 5,000 patient messages/month × 300 tokens = 1.5M tokens/month (comms)

Total: 30.26M tokens/month → Tier 1 (API-only)

Step 2: Assess Your Staffing Reality

Do you have (or can you hire) ML engineering expertise?

Self-hosting requires:

• Model deployment: Understanding vLLM, TGI, quantization, tensor parallelism
• GPU infrastructure: Managing CUDA, drivers, multi-GPU configurations
• Performance tuning: Batch sizes, KV cache optimization, request queuing
• Monitoring: GPU utilization, token throughput, latency percentiles
• Security: Model weight integrity, inference endpoint authentication, audit logging

According to industry benchmarks, a proper self-hosted LLM deployment requires 1–2 weeks of engineering time per major model update — and Llama models update every 2–4 months.

If your team is product engineers building healthcare applications: API-only or hybrid with managed self-hosted (AWS Bedrock, Azure ML).

If you have 1–2 ML engineers on staff: Hybrid with carefully scoped self-hosted for high-ROI workloads only.

If you have a dedicated ML infrastructure team: Self-hosted for core workloads makes sense.

Step 3: Map Your Compliance Constraints

Question 1: Can PHI leave your datacenter?

• Yes, with BAA: API with Azure/AWS/Google is viable
• No, absolutely not: Self-hosted only, likely on-premises

Question 2: What’s your data residency requirement?

• US-only: Azure has HIPAA-eligible US regions
• EU data residency (GDPR): Azure has EU regions, but verify BAA coverage for your region
• Multi-jurisdictional: Self-hosted gives you full control over data locality

Question 3: What audit trail do you need?

• Standard HIPAA: API providers offer sufficient logging
• SOC 2 Type II, HITRUST: API providers support this
• Custom retention >7 years: Self-hosted gives you complete control over log storage

Step 4: Calculate Your Break-Even Point

API baseline cost (GPT-4 Turbo equivalent):

• Your monthly tokens × $0.015/1M tokens = Monthly API cost

Self-hosted cost:

• GPU rental or amortized hardware: $1,800–6,000/month
• Staffing (25% FTE ML engineer): $2,800/month
• Monitoring, storage, networking: $200/month
• Total: $4,800–9,000/month

Break-even volume:

$4,800 ÷ $0.015 = 320M tokens/month (lower bound)

$9,000 ÷ $0.015 = 600M tokens/month (upper bound)

If your volume is below 320M tokens/month: API is cheaper.

If your volume is above 600M tokens/month: Self-hosted is cheaper.

If your volume is 320M-600M tokens/month: Hybrid optimizes cost.

The Architectural Patterns That Actually Work

Pattern A: API-Only (For Most Healthcare Orgs)

Who this fits:

• Hospitals <500 beds
• Specialty clinics
• Healthcare startups pre-Series B
• Token volume <100M/month

Implementation:

from azure.identity import DefaultAzureCredential
from openai import AzureOpenAI

class HealthcareLLMClient:
    """
    HIPAA-compliant Azure OpenAI client
    """
    
    def __init__(self, deployment_name: str, api_version: str = "2024-08-01-preview"):
        # Use managed identity - no API keys in code
        credential = DefaultAzureCredential()
        
        self.client = AzureOpenAI(
            azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
            azure_ad_token_provider=credential,
            api_version=api_version
        )
        
        self.deployment_name = deployment_name
    
    async def generate_clinical_summary(
        self,
        deidentified_note: str,
        max_tokens: int = 500
    ) -> str:
        """
        Generate clinical summary from de-identified note
        
        CRITICAL: Input must be de-identified before calling this method
        """
        
        response = await self.client.chat.completions.create(
            model=self.deployment_name,
            messages=[
                {
                    "role": "system",
                    "content": "You are a clinical documentation assistant. Summarize the following de-identified clinical note concisely."
                },
                {
                    "role": "user",
                    "content": deidentified_note
                }
            ],
            max_tokens=max_tokens,
            temperature=0.3  # Lower temperature for clinical consistency
        )
        
        return response.choices[0].message.content

Critical configuration for HIPAA:

Configure additional access controls: Implement Virtual Networks (VNet), private endpoints, and other network security measures to ensure that access to your Azure OpenAI resources is restricted and secure.

1. Deploy in HIPAA-eligible region: East US, West US 2, or other designated regions
2. Enable Private Link: No public internet access to your Azure OpenAI endpoint
3. Managed identity authentication: No API keys stored in code or environment variables
4. Diagnostic logging: Enabled, but configured to exclude prompt content
5. Network isolation: VNet integration with private endpoints only

Cost: ~$150–2,000/month depending on volume

Pattern B: Self-Hosted Primary (For High-Volume Organizations)

Who this fits:

• Large health systems (500+ beds)
• National healthcare payers
• Token volume >500M/month
• Air-gapped requirements

Infrastructure (Production-Grade):

Hardware:

• 2× NVIDIA A100 80GB or 4× NVIDIA L40S 48GB
• 512GB RAM minimum
• 4TB NVMe SSD (model weights + KV cache)
• Redundant power supplies
• 100Gbps networking between GPUs

Software stack:

# docker-compose.yml for vLLM deployment

version: '3.8'
services:
  vllm:
    image: vllm/vllm-openai:v0.6.0
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=0,1  # Use both GPUs
      - VLLM_USE_MODELSCOPE=false
    ports:
      - "8000:8000"  # OpenAI-compatible API
    volumes:
      - ./models:/models  # Model weights
      - ./logs:/logs      # Audit logs
    command: >
      --model /models/Meta-Llama-3.3-70B-Instruct
      --tensor-parallel-size 2
      --max-model-len 8192
      --enable-prefix-caching
      --disable-log-requests false
      --trust-remote-code
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    
  # Nginx reverse proxy for authentication and rate limiting
  nginx:
    image: nginx:alpine
    ports:
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./certs:/etc/nginx/certs
    depends_on:
      - vllm
  
  # Prometheus for monitoring GPU utilization
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=90d'
  
  # Grafana for dashboards
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    depends_on:
      - prometheus
volumes:
  prometheus_data:
  grafana_data:

Cost: $5,900–9,000/month all-in

Pattern C: Hybrid (The Sweet Spot for Most)

Who this fits:

• Regional health systems (100–500 beds)
• Healthcare SaaS companies
• Token volume 200M-1B/month with variance

Architecture:

                                    ┌─────────────────────┐
                                    │   Load Balancer     │
                                    │  (Route by load)    │
                                    └──────────┬──────────┘
                                               │
                               ┌───────────────┴──────────────┐
                               │                              │
                    ┌──────────▼──────────┐       ┌───────────▼──────────┐
                    │   Self-Hosted       │       │   Azure OpenAI API   │
                    │   Llama 3.3 70B     │       │   GPT-4 Turbo        │
                    │   (Primary)         │       │   (Failover/Overflow)│
                    └─────────────────────┘       └──────────────────────┘
                    
        High-volume, predictable:          Low-volume, experimental:
        - Clinical documentation            - Prior auth appeals
        - Radiology reports                 - Patient communications
        - 24/7 continuous processing        - A/B testing new features
        
        Cost: ~$0.01/1M tokens              Cost: ~$15/1M tokens
        (at 60% utilization)                (but low absolute $)

Implementation:

import asyncio
from typing import Literal
from dataclasses import dataclass

@dataclass
class RoutingDecision:
    target: Literal["self_hosted", "api"]
    reason: str
    estimated_cost: float
class HybridRouter:
    """
    Intelligent routing between self-hosted and API based on workload
    """
    
    def __init__(self):
        self.self_hosted_capacity = 100  # Concurrent requests
        self.current_self_hosted_load = 0
        self.api_cost_per_1m_tokens = 15.0
        self.self_hosted_marginal_cost_per_1m_tokens = 0.01
        
    def route_request(
        self,
        workload_type: str,
        estimated_tokens: int,
        priority: int
    ) -> RoutingDecision:
        """
        Route to optimal backend based on workload characteristics
        """
        
        # Rule 1: Experimental workloads always go to API
        if workload_type == "experimental":
            return RoutingDecision(
                target="api",
                reason="experimental_workload",
                estimated_cost=estimated_tokens * self.api_cost_per_1m_tokens / 1_000_000
            )
        
        # Rule 2: If self-hosted at capacity, failover to API
        if self.current_self_hosted_load >= self.self_hosted_capacity:
            return RoutingDecision(
                target="api",
                reason="self_hosted_at_capacity",
                estimated_cost=estimated_tokens * self.api_cost_per_1m_tokens / 1_000_000
            )
        
        # Rule 3: High-volume production workloads prefer self-hosted
        if workload_type in ["clinical_docs", "radiology", "continuous"]:
            return RoutingDecision(
                target="self_hosted",
                reason="high_volume_production",
                estimated_cost=estimated_tokens * self.self_hosted_marginal_cost_per_1m_tokens / 1_000_000
            )
        
        # Rule 4: Low-volume, sporadic workloads go to API
        return RoutingDecision(
            target="api",
            reason="low_volume_sporadic",
            estimated_cost=estimated_tokens * self.api_cost_per_1m_tokens / 1_000_000
        )

Cost: $6,200–7,500/month blended

What Actually Breaks in Production (And How to Fix It)

Failure Mode 1: The Compliance Audit That Kills Your Architecture

What happened:

A healthcare analytics company built their entire platform on Azure OpenAI. 18 months, $4.2M in development costs, 40,000 users.

During SOC 2 Type II audit, auditor asked: “Show me your data processing agreement with your subprocessors.”

The BAA is incorporated via the Microsoft Data Protection Addendum (DPA).

They had the Azure BAA. But their application also used:

• Google Cloud Storage (for document uploads)
• AWS Lambda (for preprocessing)
• Cloudflare (for CDN)

None of these had BAAs signed.

PHI touched all three systems before reaching Azure OpenAI.

Result: 6-month remediation, $890K in audit failures, delayed Series B.

The fix:

Map your entire data flow, not just the LLM endpoint:

Patient Data Entry
    ↓
Load Balancer (Cloudflare?) ← Needs BAA if caching enabled
    ↓
API Gateway (AWS/Azure/GCP?) ← Needs BAA
    ↓
De-identification Service (Where does this run?) ← Needs BAA if PHI touches it pre-deidentification
    ↓
LLM Endpoint (Azure OpenAI) ← Has BAA ✓
    ↓
Response Storage (Database? S3? Azure Blob?) ← Needs BAA
    ↓
Logging (Datadog? Splunk?) ← Needs BAA if logs contain PHI

Every system that touches PHI — even in transit — requires a BAA.

Failure Mode 2: The GPU Cluster That Never Paid Off

What happened:

Regional health system (85 hospitals) spent $187K on 8× A100 cluster for clinical documentation.

Projected volume: 800M tokens/month.

Actual volume (Month 3): 47M tokens/month.

Why the projection was wrong:

• They estimated “all clinicians will use it immediately”
• Reality: 6-month pilot, 12-month rollout, resistance to change
• Actual adoption: 8% of clinicians in Month 3, 23% in Month 12

Their GPUs sat at 2.1% utilization for 18 months.

The fix:

Phase your infrastructure investment with your rollout:

Month 1–3 (Pilot): API-only, 20 clinicians, validate product-market fit

Month 4–6 (Early rollout): API-only, 100 clinicians, refine workflows

Month 7–9 (Expanded rollout): Hybrid, 300 clinicians, self-hosted for high-volume users only

Month 10–12 (Full deployment): Migrate to self-hosted primary once volume exceeds break-even

Start with API. Graduate to self-hosted when the math proves it.

Failure Mode 3: The Model Version That Broke HIPAA

What happened:

Healthcare startup used OpenAI API (not Azure OpenAI) with a signed BAA. Compliant setup.

OpenAI released GPT-4 Turbo with updated capabilities. Engineering team updated their model version from gpt-4-0613 to gpt-4-turbo-preview.

The preview tag meant: Not covered under BAA.

Preview features or non-text models are not currently HIPAA-compliant unless explicitly stated.

They processed 340,000 patient records with a non-compliant model before their compliance team noticed.

Result: Self-reported HIPAA breach, OCR investigation, $1.2M settlement.

The fix:

Lock your model versions. Never auto-update.

# BAD: Uses latest model (might be preview/non-compliant)
client.chat.completions.create(
    model="gpt-4-turbo",  # Ambiguous, could point to preview
    messages=[...]
)

# GOOD: Locked to specific HIPAA-eligible version
client.chat.completions.create(
    model="gpt-4-0613",  # Specific GA version covered by BAA
    messages=[...]
)

Maintain a BAA-approved model version registry:

# approved_models.yaml

hipaa_compliant_models:
  azure_openai:
    - deployment: "gpt-4-turbo-2024-04-09"  # GA version
      baa_coverage: true
      approved_date: "2024-05-15"
      modalities: ["text"]
      not_covered: ["vision", "audio"]
    
    - deployment: "gpt-4o-2024-08-06"  # GA version
      baa_coverage: true
      approved_date: "2024-09-01"
      modalities: ["text"]
      not_covered: ["vision", "audio"]
  
  openai_api:
    - model: "gpt-4-0613"
      baa_coverage: true
      approved_date: "2023-11-01"
      modalities: ["text"]
  
  self_hosted:
    - model: "meta-llama/Meta-Llama-3.3-70B-Instruct"
      baa_coverage: "n/a"  # Self-hosted, you control data
      approved_date: "2024-12-15"
      license: "Llama 3.3 Community License"
      commercial_use: true
non_compliant_models:
  - model: "gpt-4-turbo-preview"  # Preview, not covered
  - model: "gpt-4o-realtime-preview"  # Audio not covered
  - model: "dall-e-3"  # Images not covered

Enforce at deployment time:

APPROVED_MODELS = load_approved_models("approved_models.yaml")

def validate_model_compliance(model_name: str) -> bool:
    """
    Validate model is HIPAA-compliant before allowing inference
    """
    for approved in APPROVED_MODELS["hipaa_compliant_models"].values():
        for model_config in approved:
            if model_config.get("deployment") == model_name or model_config.get("model") == model_name:
                return True
    
    # If we get here, model is not approved
    raise ValueError(
        f"Model {model_name} is not approved for PHI processing. "
        f"See approved_models.yaml for compliant alternatives."
    )

The Decision You Make This Week

Most healthcare CTOs waste 6–12 months on the model hosting decision because they frame it wrong.

They ask: “Which vendor should we choose?”

The right question: “What volume justifies which infrastructure pattern?”

Here’s the framework that takes 1 hour instead of 6 months:

Hour 1: Estimate Your Token Economics

Use the formulas above. Calculate your monthly token volume for each workload.

Hour 2: Map to Hosting Pattern

• <100M tokens/month → API-only
• 100M-500M tokens/month → Hybrid

500M tokens/month → Self-hosted primary with API for experimental

Hour 3: Validate Compliance Constraints

• Can PHI touch external APIs with BAA? → Yes = API viable, No = Self-hosted only
• What’s your audit retention? → <7 years = API sufficient, >7 years = Consider self-hosted for log control

Hour 4: Calculate Break-Even and Payback Period

• API cost: Monthly tokens × $0.015/1M
• Self-hosted cost: $5,000–9,000/month all-in
• Break-even: When API cost exceeds self-hosted cost
• Payback: If buying hardware, upfront cost ÷ monthly savings

Decision Output:

If break-even is >12 months out: Start with API, revisit quarterly as volume grows

If break-even is 6–12 months out: Hybrid approach

If break-even is <6 months out: Self-hosted for high-volume workloads now

The pattern I see work:

• Month 1–3: API-only, prove product-market fit, refine use cases
• Month 4–6: If volume >100M tokens/month, evaluate hybrid
• Month 7–12: If volume >500M tokens/month, migrate high-volume workloads to self-hosted
• Year 2+: Optimize: self-hosted for predictable baseline, API for experimental and overflow

You don’t choose once. You evolve as your volume and constraints change.

What I Learned After Our Implementations

First implementation (API-only, regional health system):

• Started with Azure OpenAI for clinical documentation
• Volume: 85M tokens/month
• Cost: $1,275/month
• Decision: Stayed on API
• 18 months later, still optimal

Second implementation (Premature self-hosting, rural hospital network):

• Bought $187K GPU cluster for discharge summaries
• Projected volume: 200M tokens/month
• Actual volume: 12M tokens/month
• Utilization: 2.1%
• Decision: Should have used API
• Cost: $15,583/month for workload worth $180/month on API

Third implementation (Hybrid, large health system):

• Started API-only for pilot (Month 1–6)
• Validated 400M+ tokens/month production volume
• Self-hosted for clinical docs (432M tokens/month)
• API for appeals and experimental (30M tokens/month)
• Blended cost: $6,400/month
• If API-only: $9,500/month
• If self-hosted only: $18,000/month (multiple underutilized clusters)
• Savings: $3,100/month = $37,200/year

The lesson:

The optimal choice isn’t about the technology. It’s about your volume, variance, and velocity.

• API optimizes for flexibility.
• Self-hosted optimizes for cost at scale.
• Hybrid optimizes for reality.

Most healthcare organizations need hybrid. But they should start with API and graduate to hybrid when the math proves it.

Building production AI infrastructure that survives compliance audits and actually saves money. Every Tuesday.

Hit follow for more Silicon Protocol episodes on LLM architecture, compliance, and the infrastructure decisions that make or break healthcare AI.

Next in The Silicon Protocol: Episode 3: The De-identification Decision — When someone says “we’ll just anonymize the data before sending to the LLM,” what they’re actually building (and why 90% of de-identification pipelines fail OCR audits).

Stuck on the model hosting decision for your healthcare AI project? Drop a comment with your monthly token volume and constraints — I’ll tell you which pattern fits.

---

The Silicon Protocol: The Model Hosting Decision — When Azure OpenAI Isn’t Enough (And When It’s… was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.