Resources / InferenceEndpoint

InferenceEndpoint

An HTTP endpoint for language-model inference in this project, compatible with the Anthropic Messages API and the OpenAI chat completions and embeddings APIs. It forwards to the model provider you choose with the key you keep in a Secret, counts tokens, and applies quotas and per-caller rate limits. Request logging is off unless you turn it on.

Overview

APIintelligence.etorm.com/v1alpha1
Resourceinferenceendpoints (cluster-scoped inside its cluster)
Short namesie
Available inproject clusters only
Statuswritten by a controller through /status

Minimal example

apiVersion: intelligence.etorm.com/v1alpha1
kind: InferenceEndpoint
metadata:
  name: example
spec: {}

Spec

FieldTypeRequiredDefaultAllowedDescription
kindstring (one of)"chat"chat, embeddingchat answers messages (for agents and applications); embedding turns text into vectors (for memories). Cannot be changed.
modelobjectThe model, as a vendor's catalog names it: the ModelProvider (this project's or the platform's) and the model id in it. Used with providers.
model.providerstringlength 1 to 63Name of the ModelProvider, for example anthropic.
model.idstringlength 1 to 128The model's id in that catalog, for example claude-opus-5-5.
providerslist of InferenceProviderup to 5 itemsThe InferenceProviders that serve the model, in the order to try them (routing and fallback). Empty for the built-in embedder.
providers[].namestringyeslength 1 to 63Name of the InferenceProvider in this project.
routingobject{}How the providers are tried: on a server error, a limit, an unreachable provider, a refused key or a model the provider no longer has, the next one is.
routing.strategystring (one of)"Ordered"Ordered, Cost, LatencyOrdered tries them as listed; Cost the cheapest published price first; Latency the fastest measured first.
providerobject{}Deprecated: one provider written into the endpoint. Use model and providers instead. A chat endpoint with none of these defaults to anthropic with model claude-opus-5-5; an embedding endpoint with none defaults to the built-in embedder.
provider.typestring (one of)anthropic, openai-compatible, builtinanthropic (Messages API), openai-compatible (any provider with the OpenAI chat and embeddings API at baseUrl), or builtin (embedding only: a local hashing embedder that needs no key and costs nothing).
provider.modelstringup to 128 charactersThe provider's model id. Callers cannot choose another one.
provider.baseUrlstringup to 512 charactersFor openai-compatible: the provider's API base address, https only, for example https://provider.example/v1.
provider.credentialRefobjectThe Secret holding the provider's API key. Needed for anthropic and openai-compatible. Setting it needs the use right on the secret.
provider.credentialRef.namestringyeslength 1 to 63Name of the Secret in this project.
provider.credentialRef.keystring"api-key"up to 63 charactersThe key inside the secret that holds the API key.
dimensionsinteger1 to 8192Embedding only: length of the vectors. builtin defaults to 256.
limitsobject{}Quotas and rate limits of this endpoint.
limits.requestsPerMinuteinteger601 to 6000Requests per minute for each calling identity.
limits.inputTokensPerDayinteger10000001 to 1000000000Input tokens per UTC day for the whole endpoint.
limits.outputTokensPerDayinteger2000001 to 1000000000Output tokens per UTC day for the whole endpoint.
limits.maxOutputTokensinteger40961 to 64000The largest answer one request may ask for.
loggingobject{}Request and response logging, off by default. Needs monitoringRef; keys and authorization headers are never logged.
logging.requestsbooleanfalseLog request bodies.
logging.responsesbooleanfalseLog response bodies.
monitoringRefreference to MonitoringWorkspaceOptional MonitoringWorkspace for request, token and latency metrics and, when enabled, logs.

Status

FieldTypeRequiredDefaultAllowedDescription
phasestringPending, Ready or Failed.
urlstringThe endpoint's base address; append /v1/messages, /v1/chat/completions or /v1/embeddings.
modelstringThe model the endpoint uses.
providerslist of objectThe providers of the endpoint and how each is doing.
providers[].namestringName of the InferenceProvider.
providers[].phasestringIts phase.
providers[].servesbooleanWhether it lists the model.
dimensionsintegerLength of the vectors an embedding endpoint returns.
usageobjectWhat the endpoint served.
usage.requestsintegerRequests since the endpoint was created.
usage.inputTokensintegerInput tokens since the endpoint was created.
usage.outputTokensintegerOutput tokens since the endpoint was created.
usage.daystringThe UTC day of the counters below (YYYY-MM-DD).
usage.requestsTodayintegerRequests on that day.
usage.inputTokensTodayintegerInput tokens on that day.
usage.outputTokensTodayintegerOutput tokens on that day.
conditionslist of objectDetailed state; Ready says whether the endpoint answers requests.
conditions[].typestringAspect.
conditions[].statusstringTrue or False.
conditions[].reasonstringMachine-readable reason.
conditions[].messagestringHuman-readable detail.

Columns in listings

ColumnFrom
Kind.spec.kind
Provider.spec.provider.type
Model.status.model
Requests.status.usage.requests
Phase.status.phase
Detail.status.conditions[0].message

All resources