Inference

The Podstack Inference Cloud serves open-source models on low-latency, OpenAI-compatible endpoints with autoscaling. Browse a catalog of hosted models, generate an API key, and call them from any OpenAI SDK — no infrastructure to manage. Usage is metered per token and billed directly from your Podstack wallet.

If you already have code that talks to OpenAI, you can point it at Podstack by changing two things: the base URL and the API key.

The endpoint

All requests go to the OpenAI-compatible gateway:

https://cloud.podstack.ai/infer/v1

Authenticate with a Podstack API key (prefix psk_) as a bearer token:

Authorization: Bearer psk_xxxxxxxxxxxxxxxxxxxx

Quick example

from openai import OpenAI

client = OpenAI(
    base_url="https://cloud.podstack.ai/infer/v1",
    api_key="psk_xxxxxxxxxxxxxxxxxxxx",
)

resp = client.chat.completions.create(
    model="<MODEL_ID>",  # list real ids: podstack models list
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Model IDs are not fixed — the catalog is managed per deployment. Always pull a real ID from the catalog before hardcoding one. See Models.

When to use it

• You want to call open-source models (chat, code, embeddings, vision) without provisioning or scaling GPUs yourself.
• You want a drop-in OpenAI replacement — same SDK, same request/response shape, streaming included.
• You want per-token, pay-as-you-go pricing billed from a wallet, with usage analytics per API key.
• You are using podstack code (the CLI coding agent), which calls this same gateway.

For always-warm, dedicated serving or cold-start pay-per-GPU-second workloads (including video generation), see Serverless Inference.

What’s supported

What’s available today

Live right now, both in the portal and over the API:

• Model catalog — browse the hosted models in the portal (Inference > Model Catalog) or list real IDs with GET /v1/models and podstack models list. See Models.
• Playground — an interactive, streaming chat playground in the portal (Inference > Playground) with temperature and max-token controls, live per-turn cost, and copy-ready curl/Python/JavaScript snippets. See Playground.
• API keys — create, use, scope with limits, and revoke psk_ keys from Inference > API Keys. See Authentication.
• Chat completions (streaming) — POST /v1/chat/completions, OpenAI-compatible.
• Embeddings — POST /v1/embeddings on self-hosted models.
• Public pricing — GET /v1/pricing.
• Usage analytics — GET /v1/usage/summary and GET /v1/usage/requests report per-key token usage and cost over the API. See Pricing & Usage.

Not yet generally available:

• Audio transcription (POST /v1/audio/transcriptions) is not implemented yet — it returns 501 with a “coming soon” message.
• The dedicated Serverless Inference surface (Serverless Models catalog, serverless chat/video, and the GPU dashboard) is built but currently gated behind a feature flag, so it may not appear in your portal. The standard chat and embeddings endpoints above are already backed by on-demand cold-start GPUs under the hood.

Use cases

• Drop-in OpenAI replacement — a backend engineer changes only the base URL and API key, and an existing chat application runs on open-source models with no other code change.
• Prompt engineering in the browser — a product manager iterates on a system prompt in the Playground, watches the streamed output and per-turn cost, then copies the generated Python snippet into the app.
• Embeddings for RAG — a developer calls POST /v1/embeddings on a self-hosted model to build a vector index for a retrieval pipeline.
• Per-key cost governance — a platform lead issues a scoped API key with usage limits for a team and tracks spend through the usage endpoints.
• Coding-agent backend — a developer runs podstack code, which calls this same OpenAI-compatible gateway for its model.
• Model comparison — an ML engineer sends the same prompt to several catalog models in the Playground to compare quality, latency, and cost before committing to one.

In this section

Next steps

• New here? Start with the Quickstart.
• Prefer the terminal? podstack models list prints the catalog — see the CLI.