Some suggestions on how to construct AI search evaluation frameworks
Hi Aaron, thanks for sharing your knowledge, always very useful! I liked very much the test methodologies. Actually, when I did my test on AI tools, back in mids 2025, I applied a methodology very similar to the one you described in 'The Retrieval test 1: recall@K using a known set'. (https://www.researchgate.net/publication/396120553_Do_AI_research_assistants_live_up_to_their_hype_An_Exploratory_Study_of_Some_Freely_Available_Tools)
Sure. It's a pretty obvious test and I'm not claiming any novelty or precedence, just that I don't see it commonly done in librarian circles.
And yes yours was one of the better tests, I saw back then.
Hi Aaron, thanks for sharing your knowledge, always very useful! I liked very much the test methodologies. Actually, when I did my test on AI tools, back in mids 2025, I applied a methodology very similar to the one you described in 'The Retrieval test 1: recall@K using a known set'. (https://www.researchgate.net/publication/396120553_Do_AI_research_assistants_live_up_to_their_hype_An_Exploratory_Study_of_Some_Freely_Available_Tools)
Sure. It's a pretty obvious test and I'm not claiming any novelty or precedence, just that I don't see it commonly done in librarian circles.
And yes yours was one of the better tests, I saw back then.