“I’ve built a RAG-based Q&A system and want to evaluate it using RAGAS. Walk me through the complete setup: how to structure my test dataset with ground truth answers, how to run the four RAGAS metrics, and how to interpret the results.”
“I want to implement LLM-as-judge for my AI assistant. What prompt should the judge use, what scoring rubric works well, and how do I calibrate the judge’s ratings against human evaluations to make sure it’s reliable?”
“Help me build an evaluation test set for an AI customer support chatbot. What categories of test cases do I need, how many of each, and show me a prompt that generates realistic test cases from my support documentation.”
“My AI system’s quality seems to have degraded since we updated the system prompt last week but I’m not sure. How would I set up an evaluation pipeline to definitively measure whether quality changed, and what metrics should I track?”
“What is the right frequency and scope for human evaluation of an AI system handling 1,000 requests per day? Walk me through a practical evaluation schedule that balances coverage against the cost of human reviewer time.”
