How It Works
1
Deploy your server
Deploy your MCP server with
concierge deploy.2
Run evaluation
Launch an evaluation from the Concierge Platform. Configure the LLM provider, model, and number of mock interactions.
3
Review the report
Get a detailed breakdown of how your server handled each interaction.
What Gets Evaluated
Tool Selection Accuracy
Does the LLM pick the right tool for each question?Stage Flow Compliance
Does the LLM follow the intended stage order?Response Quality
Are tool responses useful and complete? The evaluator checks for missing fields, incomplete data, and unhelpful error messages.Error Handling
How does the server handle edge cases? For example, calling checkout with an empty cart, or passing an invalid product ID.Report Output
Evaluation runs are non-destructive. They use the same MCP protocol as any client. No special server-side changes needed.