Cheatsheets — Quick Reference for AI Infrastructure Engineers¶

Total VRAM ≈ Model weights + Gradients + Optimizer states + Activations
           ≈ (params × 2) + (params × 2) + (params × 8) + activations
           ≈ params × 12 bytes (AdamW, FP16 mixed precision) + activations

VRAM ≈ Model weights + KV cache
KV cache per token ≈ 2 × num_layers × hidden_dim × 2 bytes (FP16)

Min GPUs needed = Total model memory ÷ Per-GPU VRAM × 1.2 (safety margin)

Monthly GPU cost = (Hourly rate × Hours/day × 30) × VM count
                 = ($0.53 × 12 × 30) × 4 = $763/month (T4, 12hr/day)

Annual savings from RI = Monthly cost × 12 × 0.30 (avg 1-yr discount)
                       = $763 × 12 × 0.30 = $2,747/year

Step 1: Measure average tokens per request
         Avg input tokens:  800
         Avg output tokens: 400
         Total per request: 1,200

Step 2: Calculate sustained TPM
         Requests per minute: 150
         Sustained TPM = 150 × 1,200 = 180,000 TPM

Step 3: Use Azure capacity calculator or:
         ~1 PTU ≈ 3,600 TPM (GPT-4o, approximate)
         PTUs needed ≈ 180,000 ÷ 3,600 = 50 PTUs

Step 4: Add 25% headroom for bursts
         Recommended: 63 PTUs

GPU not visible to workload?
├── Check VM SKU → Is it an N-series VM?
│   └── No → Wrong VM SKU. Redeploy with NC/ND/NV series.
├── nvidia-smi returns error?
│   ├── Driver not installed → Install NVIDIA driver extension:
│   │   az vm extension set --name NvidiaGpuDriverLinux \
│   │     --publisher Microsoft.HCPCompute --version 1.9 \
│   │     --vm-name <vm> --resource-group <rg>
│   └── Driver version mismatch → Check CUDA compatibility matrix
├── In AKS pod, `nvidia-smi` not found?
│   ├── Missing GPU device plugin → Deploy NVIDIA device plugin DaemonSet
│   ├── Pod missing resource request → Add `nvidia.com/gpu: 1` to limits
│   └── Pod not scheduled on GPU node → Check tolerations and nodeSelector
└── nvidia-smi shows GPU but CUDA fails?
    └── CUDA toolkit version doesn't match driver → Rebuild container with matching CUDA

P95 latency exceeding SLA?
├── Check GPU utilization
│   ├── > 95% → GPU saturated. Scale out (more replicas) or optimize batch size.
│   └── < 50% → GPU is not the bottleneck. Check below.
├── Check GPU memory
│   ├── > 90% → KV cache pressure. Reduce max sequence length or batch size.
│   └── Normal → Not memory-bound. Check below.
├── Check CPU utilization
│   ├── > 80% → Preprocessing bottleneck. Move to dedicated CPU pods.
│   └── Normal → Check below.
├── Check network / storage
│   ├── Model loading slow → Cache model on local NVMe, not Blob Storage.
│   └── High network latency → Check Private Endpoint routing.
└── Check application code
    ├── No request batching → Implement dynamic batching (batch size 8-32).
    └── Synchronous preprocessing → Move to async pipeline.

Getting HTTP 429 responses?
├── Check Retry-After header value
│   ├── < 10 seconds → Implement exponential backoff with jitter
│   └── > 60 seconds → You've hit a hard quota limit
├── Check deployment TPM/RPM limits
│   ├── Near limit → Request quota increase or add second deployment
│   └── Well under limit → Check for retry storms amplifying actual TPM
├── Multiple consumers sharing one deployment?
│   └── Yes → Separate deployments per consumer with dedicated quotas
└── Sustained high usage?
    └── Evaluate PTU migration (see PTU Sizing above)

CUDA out of memory error?
├── Calculate model memory requirement (see GPU Memory Math above)
│   ├── Model too large for GPU → Quantize (INT8/INT4) or use larger GPU
│   └── Model fits → Batch size or sequence length too high
├── Reduce batch size by 50%, retry
│   ├── Works → Gradually increase to find sweet spot
│   └── Still OOM → Enable gradient checkpointing (training) or reduce max_seq_len
├── Multi-GPU available?
│   └── Enable tensor parallelism (inference) or ZeRO Stage 3 (training)
└── Still failing?
    └── Profile with `torch.cuda.memory_summary()` to find the allocation spike

Is it a proprietary/custom model?
├── Yes → Do you need to train it?
│   ├── Yes → Azure ML Compute Clusters (managed) or GPU VMs (full control)
│   └── No (inference only) → How much traffic?
│       ├── < 10 QPS → Azure ML Managed Endpoint or single GPU VM
│       ├── 10-100 QPS → AKS with GPU node pool + HPA
│       └── > 100 QPS → AKS multi-node GPU pool + cluster autoscaler
└── No (using a foundation model) → Azure OpenAI Service
    ├── Dev/test or bursty → Standard (PayGo) deployment
    ├── Sustained production → Provisioned (PTU) deployment
    └── Need global distribution → Global Standard deployment