Amazon has removed one of the biggest practical barriers to running AI models on its own silicon. On November 12, 2026, the company announced that SGLang-Neuron is publicly available, bringing the popular open-source SGLang serving stack to AWS Trainium chips. The key promise is continuity: teams can serve models and generate reinforcement-learning rollouts on Trainium using the same SGLang interface they already use on other hardware, with no rewrite of their serving code.
Trainium is Amazon’s portfolio of custom AI chips, built to deliver strong economics for AI training and inference at scale. Until now, using it well often meant learning AWS-specific tooling. SGLang-Neuron changes that equation by meeting developers where they already are.
What SGLang-Neuron Is
SGLang-Neuron is an out-of-tree plugin that connects SGLang with AWS Trainium. It is available through an AWS Deep Learning Container released by RadixArk in collaboration with Annapurna Labs, and the AWS Neuron release notes list it as a public preview.
SGLang itself is one of the most widely used open-source frameworks for serving large language models, known for its fast batched inference and its RadixAttention prefix-caching system. By bringing that stack to Trainium, Amazon is betting that familiarity is the fastest route to adoption: if the interface stays the same, teams have far less reason to avoid trying the hardware.
The release follows a limited beta that began on October 22, 2026, with a small group of teams testing both model serving and rollout generation on Trainium2 and Trainium3.
How It Works Under the Hood
The design keeps a clean boundary between the framework and the hardware. Installing the plugin registers Trainium support through SGLang’s plugin entry points, and setting the environment variable SGLANG_PLATFORM=neuron selects it when the server starts. Because it is an out-of-tree plugin, Trainium support and SGLang core can develop independently and iterate faster.
SGLang core continues to provide the API server, the request lifecycle, RadixAttention, and prefix-cache bookkeeping. The plugin implements the Trainium-specific pieces: the models, attention, mixture-of-experts (MoE) handling, and the graph runner.
Underneath, SGLang-Neuron runs on native PyTorch. TorchNeuron registers Trainium as a native PyTorch device, the plugin uses standard PyTorch operations for the model execution path, and it incorporates kernels written with the Neuron Kernel Interface (NKI) for performance. Those operations and kernels are compiled into executable programs using torch.compile, which the graph runner prepares and executes during serving.
SGLang-Neuron also supports prefill-decode (PD) disaggregation. In this setup, a prefill server computes the prompt KV cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter (EFA). The decode server then generates the remaining tokens. Each server still uses SGLang’s scheduler and request lifecycle, while the plugin manages the Trainium execution and the KV-cache transfer path. Splitting prefill and decode this way is a proven technique for cutting latency in interactive serving, and it is the same kind of optimization that makes self-tuning inference stacks attractive for production workloads — see our coverage of Together AI’s polymorphic inference work.
What You Can Run on It Today
According to the AWS Neuron release notes, large language models run on Trn3 instances on one fully asynchronous runtime with prefill-decode disaggregation, while multimodal diffusion models run on Trn2 instances through SGLang Diffusion. Supported models include GPT-OSS-120B, Kimi-K3, GLM-5.2, Qwen3, FLUX.1-dev, and Wan2.2 A14B.
That list covers both ends of the current model landscape: open-weights language models such as GPT-OSS-120B and the Qwen3 family, alongside image-generation models like FLUX.1-dev. It is a starting lineup rather than a finished catalog — AWS points developers to the project documentation for the full list of supported models, features, and validated configurations, and coverage is expected to grow. Teams working with other open-weights models, such as the 1.6-trillion-parameter LongCat-2.5 agent model, should watch that list as it expands.
Notably, the release covers reinforcement-learning rollouts as well as inference. Rollouts — the model-generated trajectories used to train agents — are becoming a core workload as agentic systems move into production, and the surrounding open-source ecosystem for agents is maturing fast, from NVIDIA’s OpenShell agent-security project to self-hosted agent infrastructure from Anthropic.
Serving a Model in Three Steps
Amazon’s walkthrough uses a Trainium3 EC2 instance and a pytorch-inference-sglang-neuronx container. The flow is deliberately short:
- Start the container. The image is public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04, based on Neuron SDK 2.33.0.
- Launch the SGLang model server inside the container, with SGLANG_PLATFORM=neuron set. Amazon’s example serves openai/gpt-oss-120b with tensor parallelism of 8, expert parallelism of 8, FP8 KV cache, a 10,240-token context window, and up to 64 running requests.
- Send a request to the model server’s standard chat-completions endpoint at http://127.0.0.1:30000/v1/chat/completions — the same OpenAI-compatible API SGLang exposes everywhere.
Those example settings are a starting point, not a benchmark: throughput and latency will depend on the model, the workload, and the instance type. But the point of the walkthrough is that nothing in it is Trainium-specific except the container and one environment variable.
Why This Matters for the AI Chip Race
The AI accelerator market is still dominated by NVIDIA, and every challenger faces the same problem: hardware is only useful if developers’ existing software runs on it. Google has been courting the same serving frameworks for its TPU line — its eighth-generation TPU 8tx and TPU 8i Max chips target confidential inference and massive context windows — and AWS is now playing the same game with Trainium.
Amazon’s strategy has two reinforcing parts. First, make the software familiar: TorchNeuron gives Trainium native PyTorch support, and SGLang-Neuron now adds the most popular open serving stack on top of it. Second, widen availability: Amazon CEO Andy Jassy said on the company’s third-quarter earnings call on October 29, 2026, that Trainium chips will be sold to third-party data centers by the end of 2027, the first time they will be offered as a standalone product rather than only as rented compute inside AWS.
If both parts land, Trainium stops being an AWS-only curiosity and becomes a genuine alternative in the accelerator market — one that teams can evaluate with the tools they already know.
What to Watch
A few caveats are worth keeping in mind. SGLang-Neuron is a public preview, so model coverage, features, and performance characteristics are still evolving. Amazon has not published independent benchmarks for this release, so teams should run their own workloads before committing production traffic. And while the interface is familiar, the underlying compilation step — torch.compile preparing the graph runner’s programs — can add startup latency, a known trade-off across non-NVIDIA accelerators.
Still, the direction is clear. The serving layer is becoming the neutral ground where hardware vendors compete, and AWS has just made its strongest move yet to claim a place on it.
The Bottom Line
SGLang-Neuron lets developers serve frontier models and generate RL rollouts on AWS Trainium with the SGLang workflow they already know — one container, one environment variable, and the same chat-completions API. For teams already running SGLang, trying Trainium is now nearly free. For Amazon, that low friction is the whole strategy.
Frequently Asked Questions
What is SGLang-Neuron?
SGLang-Neuron is an out-of-tree plugin that brings the open-source SGLang model-serving stack to AWS Trainium chips. It is available in public preview through an AWS Deep Learning Container.
Do I need to change my code to serve models on Trainium with SGLang-Neuron?
No. You launch models through SGLang’s server interface and call the same generation APIs. The only Trainium-specific steps are using the provided container and setting SGLANG_PLATFORM=neuron.
Which models are supported?
Supported models include GPT-OSS-120B, Kimi-K3, GLM-5.2, Qwen3, FLUX.1-dev, and Wan2.2 A14B, with more coverage in progress. Check the project documentation for the current validated list.
Which Trainium hardware does it run on?
Large language models run on Trn3 instances, and multimodal diffusion models run on Trn2 instances through SGLang Diffusion.
Does it support reinforcement-learning rollouts?
Yes. SGLang-Neuron supports both serving models and generating reinforcement-learning rollouts on Trainium.
Is it production-ready?
It is a public preview. Model coverage and features are still expanding, and Amazon has not published independent benchmarks, so evaluate it against your own workloads before relying on it in production.
Sources: AWS Machine Learning Blog — “Deploy frontier AI models on AWS Trainium with SGLang-Neuron” (November 12, 2026); AWS Neuron release notes — “SGLang-Neuron plug-in now available in public preview” (November 12, 2026); SGLang project on GitHub.
