Operating multimodal LLMs across engines and clouds

My contribution

I built GPU-serving and deployment workflows, selected runtimes around model compatibility, repaired silent build and traffic-routing failures, and provisioned the identity needed for authenticated inference. I worked with the application team to promote and validate the fallback integration across environments.

The work evolved in stages: a three-provider router proof of concept, model-specific serving paths using vLLM and llama.cpp, and a later production print-detection fallback integration. These are related pieces of engineering with separate validation records.

Choose the runtime from the model

A model identifier is not enough to guarantee that a serving engine can load its architecture, quantization and vision projector. I added a resolver that reads model metadata and selects vLLM or llama.cpp, with separate images, deployment workflows and concurrency groups.

The vLLM path continued serving its existing model while the alternative runtime was developed in isolation. Cross-guards rejected a model in the wrong workflow before building an image or provisioning a pod. Tests also checked that a nominally successful request actually contained a usable answer: a reasoning model could exhaust its token budget before producing response content.

Debug the layer that actually failed

On real GPU pods, an apparently healthy container lifecycle hid a shared-library loading failure. The upstream llama.cpp image expected its libraries under its own working directory; changing that directory broke startup. Making the library path explicit solved the crash. Earlier hypotheses about entrypoint handling and driver compatibility did not explain that particular failure.

GPU profiles, model context and key/value cache settings remained explicit per runtime. A larger configured concurrency value was not treated as a measured throughput improvement. The delivery recorded 164 passing tests and validation on real GPU pods; experimental pods were cleaned up afterward.

A green deploy can still serve the old revision

A Cloud Run deployment pipeline had three faults masking one another: incompatible dependency pins, traffic remaining on an older revision and smoke tests exercising that older image. I repaired the dependency chain, moved traffic updates to the correct command and verified that the served digest matched the image built by CI.

Pre-merge validation builds the dependency stage without downloading multi-gigabyte model weights. Runtime smoke tests run from the network path real callers use, and deployment guards reject inconsistent arguments or unsafe ingress changes. A successful command exit is one signal; serving the intended revision is the outcome.

Production fallback and identity

The later application integration used a hosted provider, a self-hosted multimodal model and a final CNN path. I provisioned a dedicated Kubernetes and Google service-account identity in development, staging and sandbox, granting invocation on the target service rather than broad project roles. The legacy production cluster continued using its existing node identity; it was not converted to Workload Identity by this change.

All four environments passed their identity and provider checks at closeout. In the recorded production post-deploy sample, 395 HTTP requests returned 200. Three primary-provider timeouts reached a cold self-hosted model and then completed through the final fallback. This describes the observed rollout, not an availability SLA.

Scale-to-zero is an operational trade-off: a cold GPU service can take about five minutes to start while the application's budget is about 30 seconds. Prewarming and the next fallback therefore matter. Auth checks and high-traffic monitoring were added with thresholds calibrated against actual use, rather than alerting on every expected fallback request.

Results and measurement

The model-aware serving delivery recorded two isolated runtime paths and 164 passing tests. The later fallback rollout checked identity and application behavior in four environments. Those results are separate from the earlier router POC described below.

Boot optimization measurements also need their own scope: one image test improved from 5m16s to 3m32s; a later clean-boot test recorded about 4m05s. Different configurations and test paths should not be collapsed into a universal cold-start claim.

The earlier three-provider router POC

I built a FastAPI router to exercise normal flow, cold and warm fallback, the final provider path, health checks and all-provider failure. Six scenarios passed. The test infrastructure was removed afterward pending a production adoption decision for that router; the later application fallback integration is a distinct delivery.

The recorded POC request observations were 568 ms on RunPod, 1,917 ms on Cloud Run GPU and 7,681 ms on the GCE Managed Instance Group path, with a cold fallback observation of about 41 seconds. These were particular test requests, not production latency percentiles. GCE Managed Instance Group does not mean GPU Multi-Instance GPU partitioning.

Operate the fleet, not individual pods

I also connected router inventory to dynamic Prometheus discovery, so monitoring followed ephemeral GPU workers instead of relying on a manually updated address list. Read the fleet observability case for its rollout and monitoring boundaries.

The lesson

Model serving is a system of compatibility, identity, deployment and failure decisions. Validate the loaded model, the served revision and the fallback behavior independently; a green build or an HTTP health response cannot prove all three.