Skip to main content

​
TIPS

The following practices help you get reliable results from agentic synthesis and evaluation without spending inference budget on data you later discard.
  • Get the tools right first: Your tool definitions shape every conversation that follows. Test each tool implementation with sample calls before you attach it to a job.
  • Check prompts before generating responses: Generating conversations with tool calls costs far more than generating prompts. See Split the work into two jobs.
  • Write varied personas and keep conversations short: See Write good personas and Turn limits.
  • Benchmark the teacher first: A model trained on teacher conversations rarely beats the teacher. Evaluate the teacher before you generate training data with it.
  • Choose a long-context model for training: Agent conversations run long, and training truncates anything past the maximum sequence length.
  • Sweep temperature: Start low for reliable tool-call formatting, but also try higher values. Some models repeat tool calls more at low temperature.

​
COMMON ISSUES

  • The model does not support tool calling: Jobs that attach tool implementations require a model that supports tool calling. Hosted API models are always accepted. For a trained or imported model, Oumi checks whether its base model supports tool calling.
  • Duplicate tool names: The tool implementations attached to one job must serve tools with distinct tool names.
  • Mixed simulation models: All simulated tool implementations in one job must use the same model.
  • Interactive tool without a user model: A job that attaches an interactive tool implementation must also set a user model, because the simulated user answers those calls.
  • Rows hit the tool-call limit: The model kept calling tools past the platform limit. Synthesis drops these rows and reports them as failed items. Generate completions keeps them and sets tool_call_limit_exceeded in their metadata. Check the tool descriptions and tool outputs for loops, such as an error the model retries indefinitely.
  • Endpoint calls are refused: A 4xx response from your endpoint is shown to the model as a tool refusal. Test the implementation from its Test card to see the response your endpoint returns.