Google gooses Python AI performance by... removing Python
A Python interpreter mulling each request would be a bottleneck, if not the end of Google itself.
Key points
- In Google's tests on short requests, XSE delivered two to five times the throughput of its best optimized Python server, using a quarter of the memory.
- Python is easy for data scientists to wield, but keeping its interpreter in the serving pipeline can add overhead and deployment headaches.
- At Google, XSE already handles more than six million queries per second across services including YouTube Shorts, Google Ads, and Search.
One-shot inference XSE is not so much suited for large-scale LLM inference (which would be better served by vLLM or SGLang), but rather for smaller, more sharply task-oriented models, of which there are plenty at Google.
These tend to be short requests with very high [queries-per-second] and tight latency budgets," Nathan Beach, Google director of product management, wrote in a LinkedIn missive Tuesday.
Additional details
- No Python here In this setup, the ML boffins write their model code in Python, using Google's JAX to help with the sums.
- Beyond XSE, other approaches have been taken, many of which involve jettisoning the Python code as quickly as possible.
- Also, Microsoft and its collaborators offer the ONNX Runtime, which supports an open graph format with Python bindings.
- XLA SavedModels can run on GPUs and CPUs – Google partnered with both Arm and Intel to develop the XLA ecosystem.
- But Google obviously thinks it'd be a lot cooler if you ran your models on its own line of TPU accelerators.
How this summary was made: extractive summary — built from the sources' own reporting. It is drawn only from the reporting of the sources listed below — read the originals for full detail. How we summarize →
Advertisement
Every article behind this summary — 1 article from 1 outlet
Get stories like this in your inbox.
Subscribe — it's freeOne free email at 6am — your topics, deduplicated, five minutes. No account, no paywall.