LLM Systems & Inference

Overview

Candidates take a reference inference stack and improve tokens/sec, p95 latency, and $/1k requests without dropping below a quality floor. Graded on the Pareto frontier they reach against a held-out workload.

Skills assessed

Status

This benchmark is being designed. Engineers and hiring partners are giving feedback on the rubric, dataset construction, and runtime. We’ll publish a brief and open submissions once the eval is stable enough to ship signal.

In the meantime, register a profile so we can notify you when it goes live.