llm-providers-connector

A small synchronous Rust client for OpenAI-compatible chat endpoints.

Rust
LLM

released View on GitHub

let endpoint = Endpoint { base_url: "http://127.0.0.1:8000/v1".into(), api_key: None };
let done = stream(&endpoint, &req, &stop, &mut |d| match d {
    Delta::Text(t) => print!("{t}"),
    _ => {}
})?;

llm-providers-connector is a small synchronous Rust client for OpenAI-compatible chat endpoints, including local servers such as llama.cpp, vLLM, Ollama, and LM Studio, as well as hosted endpoints. It streams chat completions with tool calls and reasoning, lists and probes models, estimates costs, and seals API keys at rest.

It has no async runtime and uses ureq. It knows the wire protocol, not your store or web API, so it fits a CLI, desktop app, or server.

Highlights

Streams chat completions with tool calls and reasoning
Lists and probes models
Estimates costs
Seals API keys at rest
Four dependencies: serde, serde_json, ureq, chacha20poly1305
Edition 2024, no unsafe