Thanks very much for your posts, Aaron. Where you say ‘Despite the transparency of showing the Boolean query, I was unable to reproduce the search results by running the same query in the standard interface.’
This discrepancy has been commented on by our librarians, and seems consistent in EDS. I’ve noted though that EBSCO mention they’re also using the BGE Reranker v2-M3 with AI-assisted search (’the top 100 search results are processed by a semantic reranking model. The reranker evaluates each result against the user's search intent and reorders results based on their semantic relevance to the query.’: https://connect.ebsco.com/s/article/Artificial-Intelligence-AI-Transparency-Fact-Sheet?language=en_US). That might be something they’ve added to the mix since you did your initial testing last year.
Do you think that’s likely to be the main factor in the discrepancy? I’m also interested to know whether the inclusion of a reranker would change your overall evaluation of the benefit of EBSCO’s (now default) search mode?
A reranker only affects the ordering of the results, it should not affect the number of counts returned.
My mental model of Ebsco platform is, usually the initial retrieval is done by Boolean (whether entered directly or expanded by AI-assisted search), then of that set he ranking of what is retrieved can be different.
To be fair, these days, I find that using
Run “AI assisted search”
Click on “show refined query, then run that expanded boolean in normal search
will get me the same number of results most of the time.
The point that puzzles me when I was testing a while writing this article was that you could sometimes get different counts , which should not be the case if the first stage retreival is just plain Boolean.
But since then , even just now, so far I almost always get the same number of results.
If both methods get the same number of counts and presumably same set but in different order, this makes sense. As you noted AI-assisted search uses a different reranker so it will order results differently from the conventional search which uses a more traditional ranking algo.
>I’m also interested to know whether the inclusion of a reranker would change your overall evaluation of the benefit of EBSCO’s (now default) search mode?
Yes. I am generally not impressed by using LLMs to expand search queries into nested boolean, because this often expands the result set a lot and creates a ton of noise. If this larger returned set is then handled by the conventional ranking/reranking algo that can’t handle this, you might even get worse results.
Putting a strong reranker, definitely helps. The idea here is the LLM (Hikau) expands the result set, but a stronger reranker is a able to sort the signal from the noise.
But here it does create some difference in the ordering of the results because the normal mode isn’t using the strong reranker. Personally, I think they should just run the strong reranker in ALL modes. Librarians only care that they can explain what is in the set, they traditionally can’t explain the ordering anyway.
EBSCO now calls this AI-assisted search and will transition it from opt-in to opt-out for both EDS and EBSCOhost databases soon. I can see the benefit but am also concerned about what this means for some of our patrons. Curious what you think about this?
But its becoming almost standard eg Scopus added a AI query builder this month that does the same.
But I gusss such things won't hurt much. Not sure what your concern is.
Still I rather providers do what Web of Science Smart Search does and blend in "Semantic search" (vector embeddings match) with results that can be separated if desired.
Thanks for this. The reproducibility thing is really interesting. My experience from spending a fair while puzzling at why certain things do not work in EDS is that normally there is something going on in the syntax / expanders that is not immediately apparent. We had to write an FAQ about the interaction between phrase searching and stemming for example. Viz our chat online - I think NLS works ok on EDS but not so much for ehost. These are different kinds of search (all be it the new interface is going to make them look rather too alike).
Yeah, I don't think it is just EDS, most of these retrieval systems have various things going in the pipelines. Sometimes you get lucky and the things described in the documentation is enough to allow you to reproduce the results. Other times, there is some weird interaction, so sometimes it works, sometimes it doesn't.
Yeah I don't want to be too harsh on the new NLS in ehost because I am more used to testing over a more comprehensive coverage, but it doesn't seem very good in my limited testing, though it is often better than nothing if you really type in natural language where the conventional search will give 0 results.
Thanks very much for your posts, Aaron. Where you say ‘Despite the transparency of showing the Boolean query, I was unable to reproduce the search results by running the same query in the standard interface.’
This discrepancy has been commented on by our librarians, and seems consistent in EDS. I’ve noted though that EBSCO mention they’re also using the BGE Reranker v2-M3 with AI-assisted search (’the top 100 search results are processed by a semantic reranking model. The reranker evaluates each result against the user's search intent and reorders results based on their semantic relevance to the query.’: https://connect.ebsco.com/s/article/Artificial-Intelligence-AI-Transparency-Fact-Sheet?language=en_US). That might be something they’ve added to the mix since you did your initial testing last year.
Do you think that’s likely to be the main factor in the discrepancy? I’m also interested to know whether the inclusion of a reranker would change your overall evaluation of the benefit of EBSCO’s (now default) search mode?
Hi
A reranker only affects the ordering of the results, it should not affect the number of counts returned.
My mental model of Ebsco platform is, usually the initial retrieval is done by Boolean (whether entered directly or expanded by AI-assisted search), then of that set he ranking of what is retrieved can be different.
To be fair, these days, I find that using
Run “AI assisted search”
Click on “show refined query, then run that expanded boolean in normal search
will get me the same number of results most of the time.
The point that puzzles me when I was testing a while writing this article was that you could sometimes get different counts , which should not be the case if the first stage retreival is just plain Boolean.
But since then , even just now, so far I almost always get the same number of results.
If both methods get the same number of counts and presumably same set but in different order, this makes sense. As you noted AI-assisted search uses a different reranker so it will order results differently from the conventional search which uses a more traditional ranking algo.
>I’m also interested to know whether the inclusion of a reranker would change your overall evaluation of the benefit of EBSCO’s (now default) search mode?
Yes. I am generally not impressed by using LLMs to expand search queries into nested boolean, because this often expands the result set a lot and creates a ton of noise. If this larger returned set is then handled by the conventional ranking/reranking algo that can’t handle this, you might even get worse results.
Putting a strong reranker, definitely helps. The idea here is the LLM (Hikau) expands the result set, but a stronger reranker is a able to sort the signal from the noise.
But here it does create some difference in the ordering of the results because the normal mode isn’t using the strong reranker. Personally, I think they should just run the strong reranker in ALL modes. Librarians only care that they can explain what is in the set, they traditionally can’t explain the ordering anyway.
EBSCO now calls this AI-assisted search and will transition it from opt-in to opt-out for both EDS and EBSCOhost databases soon. I can see the benefit but am also concerned about what this means for some of our patrons. Curious what you think about this?
I personally think using ai/llm to generate Boolean is of limited benefit except to novices.
I've been calling it the horseless carriage of ai search ha. https://aarontay.substack.com/p/the-horseless-carriage-of-ai-search
But its becoming almost standard eg Scopus added a AI query builder this month that does the same.
But I gusss such things won't hurt much. Not sure what your concern is.
Still I rather providers do what Web of Science Smart Search does and blend in "Semantic search" (vector embeddings match) with results that can be separated if desired.
Thanks for this. The reproducibility thing is really interesting. My experience from spending a fair while puzzling at why certain things do not work in EDS is that normally there is something going on in the syntax / expanders that is not immediately apparent. We had to write an FAQ about the interaction between phrase searching and stemming for example. Viz our chat online - I think NLS works ok on EDS but not so much for ehost. These are different kinds of search (all be it the new interface is going to make them look rather too alike).
Yeah, I don't think it is just EDS, most of these retrieval systems have various things going in the pipelines. Sometimes you get lucky and the things described in the documentation is enough to allow you to reproduce the results. Other times, there is some weird interaction, so sometimes it works, sometimes it doesn't.
Yeah I don't want to be too harsh on the new NLS in ehost because I am more used to testing over a more comprehensive coverage, but it doesn't seem very good in my limited testing, though it is often better than nothing if you really type in natural language where the conventional search will give 0 results.