Process large-scale inference workloads asynchronously with the new Batch API and Files API. Submit up to 50,000 requests in a single batch job and get 50% cost savings. Learn more in our Batch guide.
Added support for parallel tool calling, allowing models to request multiple tool invocations simultaneously in a single response. This reduces latency when handling queries that require multiple independent operations. Learn more in our Tool Calling guide.
Streaming Support Streaming is now supported for all models, including with structured output requests (response_format, tools). Reasoning Format Controls Added reasoning_format parameter for reasoning models, giving you control over how reasoning text appears in responses.
Added support for prompt caching, which automatically stores and reuses prompt prefixes to reduce latency and improve time to first token for similar requests. Learn more in our Prompt Caching guide.
Releasesby Glyph