smolagents | Doubleword Inference API

HuggingFace's smolagents supports custom OpenAI-compatible endpoints via OpenAIServerModel.

Install

pip install smolagents

Configure

from smolagents import OpenAIServerModel, CodeAgent

model = OpenAIServerModel(
    model_id="{{selectedModel.id}}",
    api_base="https://api.doubleword.ai/v1",
    api_key="{{apiKey}}",
)

agent = CodeAgent(model=model, tools=[])
result = agent.run("Say hello.")
print(result)

The api_base parameter is passed directly to the underlying OpenAI client as base_url.

Prompt caching

smolagents already sends its system prompt as a content block. One subclass marks it:

from smolagents import CodeAgent, OpenAIServerModel


class CachedModel(OpenAIServerModel):
    def _prepare_completion_kwargs(self, *args, **kwargs):
        kwargs_out = super()._prepare_completion_kwargs(*args, **kwargs)
        messages = kwargs_out["messages"]
        if messages and messages[0]["role"] == "system" and isinstance(messages[0]["content"], list):
            messages[0]["content"][-1]["cache_control"] = {"type": "ephemeral", "ttl": "1h"}
        return kwargs_out


model = CachedModel(
    model_id="{{selectedModel.id}}",
    api_base="https://api.doubleword.ai/v1",
    api_key="{{apiKey}}",
)

agent = CodeAgent(tools=[], model=model, max_steps=3)
agent.run("What is 2 + 2?")
print(agent.memory.steps[1].model_output_message.raw.usage.cache_read_input_tokens)

The hook covers generate and generate_stream. A CodeAgent system prompt is about 2000 tokens before any tools are added so this would write to cache. Anything under 1024 is insufficient length to write to the cache. Cache counts live on .raw.usage because ChatMessage.token_usage only has input and output totals. See the prompt caching guide for more information about saving on token spend with prompt caching.