Find tools › Open-source projects › Spend less on AI

trnrt
Open-source project · by RunanywhereAI
An open tool to run large language models on Amazon AI chips with high speed.
Some setupWell known114 installs a month
What it costs
Free and open source; you may pay for the AI model it uses.
Use it to
- Serve open-weight language models on AWS Trainium hardware
- Provide an OpenAI-compatible chat service for applications
- Run GLM models with long context windows
How to start
Clone the repository and build the Docker container.
Build the compilation cache on an x86 host to prepare the model.
Start serving on a Trainium2 instance with
docker run --rm --privileged --network host -e NEURON_LIBTORCH_CACHE_ROOT=/cache trnrt serve --model /model --port 8000.
Before you install
- Found automaticallyNot reviewed by a person yet. Check it before installing.
- License: Apache-2.0You are allowed to use it under these terms.
- Last updated 2026-11-18
More detail
TrnRT is an open LLM inference engine designed to serve large language models on AWS Trainium hardware.
Deploying high-performance language models on AWS Trainium usually requires complex hardware optimizations. Use TrnRT to serve open-weight models on AWS Trainium2 instances with high throughput and an OpenAI-compatible API.
Similar tools
Descriptions are written in plain English from each project's own documentation and refreshed by our agent; this one was last checked 2026-11-10. Spotted a mistake? · Open in Find tools · © 2026 Which AI Map