Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Evaluating trade-offs between Hosted APIs and self-hosted solutions.
• Patterns for on-premises, edge, and air-gapped environments.
• Considerations for single models versus model portfolios.
• Comparing cost-per-decision metrics.

Hardware Sizing and Capacity Planning

• Assessing memory footprint relative to parameter counts.
• Comparing CPU-only versus GPU-accelerated performance.
• Planning for concurrency and throughput.
• Accounting for growth in task length.

Serving Stacks

• Exploring vLLM and SGLang options.
• Utilizing llama.cpp and GGUF execution.
• Using Ollama for local development workflows.
• Selecting stacks based on the shape of the decision workload.

Quantization While Preserving Calibration

• Understanding precision formats and associated memory savings.
• Measuring quality and calibration loss.
• Implementing per-tensor and importance-aware quantization variants.
• Conducting acceptance testing post-quantization.

Latency Optimization

• Analyzing median versus tail latency.
• Tracking latency growth in relation to context length.
• Leveraging warm starts and model caching.
• Batching multiple decision queries.

Edge and Constrained Environments

• Running small decision models on CPU.
• Deployment on Apple Silicon and embedded systems.
• Strategies for offline operation and updates.
• Distributing weights in air-gapped scenarios.

Containerization and Scaling

• Designing images for model serving.
• Managing volumes and caches.
• Implementing horizontal scaling and routing.
• Configuring resource limits and GPU sharing.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Logging decision outcomes and escalations.
• Setting alerts for calibration drift.

Versioning and Deployment

• Managing model and adapter versions.
• Executing shadow and canary deployments.
• Maintaining pinned model files for audit purposes.
• Defining rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Configuring network isolation and egress controls.
• Maintaining audit logs of all decisions.