Find tools › Open-source projects › Spend less on AI

trnrt

Open-source project · by RunanywhereAI

An open tool to run large language models on Amazon AI chips with high speed.

Some setupWell known114 installs a month

What it costs

Free and open source; you may pay for the AI model it uses.

Use it to

  • Serve open-weight language models on AWS Trainium hardware
  • Provide an OpenAI-compatible chat service for applications
  • Run GLM models with long context windows

How to start

  1. Clone the repository and build the Docker container.

  2. Build the compilation cache on an x86 host to prepare the model.

  3. Start serving on a Trainium2 instance with docker run --rm --privileged --network host -e NEURON_LIBTORCH_CACHE_ROOT=/cache trnrt serve --model /model --port 8000.

Good to know. Requires AWS Trainium2 hardware (trn2.48xlarge), and initial offline compilation takes hours.

Before you install

  • Found automaticallyNot reviewed by a person yet. Check it before installing.
  • License: Apache-2.0You are allowed to use it under these terms.
  • Last updated 2026-11-18

More detail

TrnRT is an open LLM inference engine designed to serve large language models on AWS Trainium hardware.

Deploying high-performance language models on AWS Trainium usually requires complex hardware optimizations. Use TrnRT to serve open-weight models on AWS Trainium2 instances with high throughput and an OpenAI-compatible API.

Similar tools

Descriptions are written in plain English from each project's own documentation and refreshed by our agent; this one was last checked 2026-11-10. Spotted a mistake? · Open in Find tools · © 2026 Which AI Map