I read your post with great interest and agree that these are important questions and considerations. It may be that at some point we will need to rethink whether Boolean continues to be the best approach to searching, although, to be fair, the purpose of Boolean was never to rank results.
Pubmed’s query expansion and ranking does seem to work better than other examples you have stated, but it is still problematic, and in the health sciences we routinely teach students how to bypass automatic mapping and advise caution when using Best Match ranking. This article discusses the ways in which Pubmed’s Best Match can introduce bias into search results: https://pmc.ncbi.nlm.nih.gov/articles/PMC8830327/. We are also seeing an erosion of the quality of indexing since the adoption of automatic indexing in Pubmed and other databases. This has a huge impact on the ability of human searchers, and “AI” driven search to retrieve relevant results, regardless of relevance ranking.
I can’t speak to other disciplines, but in the health sciences, librarians do routinely engage with what happens after retrieval. In the clinical context, limiting search results to the best type of evidence for the question being asked, for example limiting to randomized controlled trials to answer therapy questions or limiting to systematic reviews, is an effective way to quickly identify relevant results despite large retrievals. Biomedical databases have built in “clinical queries” limits that are useful in this regard. On the back end these limits are search queries that have been validated for sensitivity and specificity that are applied the search query. In the context of knowledge synthesis, we routinely teach researchers how to develop and refine eligibility criteria to facilitate screening, and we use sample sets of articles known to meet the inclusion criteria to adjust and improve the search, among other techniques. When conducting scoping reviews, iterative searching to improve relevance of search results, is built into the methodology. In the context of knowledge synthesis we routinely encounter “search resistant concepts” (though I was not familiar with this term, so thank you). It does seem like BM25 retrieval over Boolean in this context is something worth exploring, however it would need to allow for transparency and reproducibility which does not seem to be possible at this time.
Perhaps most importantly, in both contexts, we focus much attention on helping searchers ask clearly formulated questions. In my 20+ years of experience lack of clarity in the question is a more important factor in determining whether search results are relevant than issues in search structure or database relevancy ranking.
The ability to ask answerable questions and search iteratively is what makes human searchers more effective than AI. In my limited experience with LLMs and AI driven search this is where they fail.
The more I engage with the topic the more I come back to the idea that LLMs are like any other technological advance: we first think they will topple all that came before, and then we slowly realize they are a tool like any other, that they have their uses, but that the old technology also still has its uses. I suspect that BM25 versus Boolean is one of these things: we will still want to be able to design transparent, predictable and reproducible searches (I would not want my medical care to be based on a small and random set of relevant results!), but in some contexts, it will be useful to be able to quickly identify some relevant results. Google Scholar already serves that purpose, so why not other tools that maybe do it better?
PS there is an increasing body of evidence that shows that PICO based searching is problematic and you are correct to question whether LLMs are capable of recognising that outcomes should be screened for and not included in the search in most instances.
Thanks for taking the time to leave such a substantial comment.
I always appreciate hearing from health science and evidence synthesis librarians, because that is the area where librarians have done the deepest thinking about search. That said, my blog posts are usually aimed at people outside health and evidence synthesis, though I do borrow ideas from those fields, which can admittedly create some confusion.
I agree that if there is any domain where nested Boolean is still clearly necessary, it is health sciences, evidence synthesis, and probably law or patent searching, where the need for comprehensiveness, predictability, and reproducibility remains especially strong.
Outside those contexts, which is usually my focus, the objective is often less about high recall. In such settings, especially in systems not really designed for Boolean searching, a simpler keyword approach may work just as well or better.
I also agree that Boolean was never designed to rank results. My point is not that Boolean is flawed on its own terms, but that in an environment of information abundance, the absence of ranking is increasingly a practical weakness. Even evidence synthesis librarians cannot realistically screen everything, which is why active learning and related approaches are now being used. In effect, that is another form of ranking.
On PubMed, I am familiar with the concerns around Automatic Term Mapping and Best Match (I even have a footnote ha), including the paper you linked. PubMed is not exempt from the problems of automatic expansion and ranking, and I am not arguing otherwise. I only meant that PubMed shows more sophistication than many other systems, not that its ranking is perfect.
I also agree that the erosion in indexing quality is important, though I see that as a broader issue affecting all search approaches rather than something specific to the Boolean versus BM25 question.
I completely agree that clinical queries and related hedges are fascinating, especially when you look at how they were developed and validated. I have tried to import some of that thinking into other domains, for example by encouraging researchers to test searches against known relevant papers. That kind of piloting seems routine in health sciences, but much less common elsewhere in my experience.
What I often see outside health sciences is that librarians teach students to break a topic into concepts, generate synonyms, combine them into blocks, and stop there. My concern is that without teaching piloting search strategies, in many contemporary systems, especially those using lexical retrieval plus strong reranking, that alone may not outperform a more direct keyword search by very much and might even hurt.
I agree with you that lack of clarity in the research question is often a major reason search results are poor. I would only add that this is distinct from the problem of result overload. Even with a clear question, being faced with a huge set of potentially relevant results is still a serious practical problem.
I think your final point gets to the heart of it. If a searcher wants full transparency, full predictability, and full control, Boolean remains the clearest option, along with screening everything retrieved. But that model is becoming harder to sustain in practice. In some contexts, ranked lexical retrieval may be a reasonable first-stage approach, even if it gives up some control.
Perhaps I should have been clearer that when I talk about BM25 as a first-stage retriever vs Boolean as retriever. A Boolean search using X AND Y AND Z retrieves only records containing all three terms. A BM25-based ranked search using the same terms can retrieve records containing one, two, or all three, with the highest-ranked records often, though not always, containing more of the query terms. This is still lexical search, not semantic search, and still reproducible because it relies on inverted indexes and interpretable because you can explain which documents are included. The trade-off is that it expands the candidate set considerably, which is why such systems usually only rank and show the top portion of possible results say Top 500 results rather then take forever to try rank everything (that has at least one of the terms).
So I suppose my position is this: if the searcher wants absolute control and to screen everything, Boolean remains unmatched. But as the volume of literature grows, many searchers will have to give up some degree of control and rely more on ranking, triage, and iterative refinement.
My fellow librarians and I are having a hard time figuring out whether Web of Science Research Assistant still works exclusively with keyword searches using Boolean operators, or if they've introduced some form of semantic search. In a section of their website (which is already 8 months old!), there is a brief mention of semantic search: “Retrieving Articles: We start by retrieving articles that exhibit the highest degree of semantic similarity to the user’s query and complement them by adding articles with the utmost relevance through a keyword search.” without further explanation (https://webofscience.zendesk.com/hc/en-us/articles/31437630410129-Web-of-Science-Research-Assistant#h_01JGKZG2P1XN9FWQGW98CKWB18).
As for the Smart Search feature, it indicates that you can limit results to those from semantic search or keyword search. Have you tested the WoS Search Assistant again recently? Do you think it’s still based solely on keyword search and Boolean operators, as the tool seems to indicate when it provides a response (for example, in the “How are these results generated?” section)? If there is indeed a semantic search component, do you have more information about it?
Yeah We never did subscribe to Web of Science RA because the feed back from our trial based on relevancy was very bad
Yeah it's ambiguous now, it looks to me likely to be based on both ("complement" seems to suggest 2 methods) reading it. But you can easily test it by manipulating the query so that keyword will unlikely be getting relevant results (I assume it still displays boolean used) and if it still does its likely doing something else as well.
Web of science smart search is definitely hybrid search. As you note the interface itself shows.
Thanks for your reply. It is very interesting to see how different disciplines approach searching. I think we can learn from each other. To be fair, the stakes are different- in healthcare a missed paper can result in adverse consequences for patients, so we have to care about what happens after the search is done. More and more, knowledge synthesis is expanding outside of the health sciences, so maybe this will influence how librarians in other disciplines engage with searching? You are right that result overload is a problem and that this is not sustainable. We do need to continue to look for ways to adjust and improve our practice.
And one of the tenets of evidence-based practice, not just in healthcare, is that you don’t just keep doing something because you’ve always done it that way, which is why I consider your suggestion that we should think about what lays beyond Boolean so interesting, even if, at least in healthcare, and certainly in knowledge synthesis, we’re not there yet. This study suggests that we may be able to use Pubmed’s Best Match to some extent, though: “The Best Match sort option in PubMed appears effective in placing relatively more relevant articles in the first fifty records, making it useful for identifying true positive records for training sets and useful in general searching. Investigators who want to identify a certain number of true positive examples for a training set can do this efficiently by screening PubMed records ranked according to Best Match until the desired number of true positives has been identified. In many cases, this will be achieved within the first fifty records.” (Sampson M, Nama N, O’Hearn K, et al. Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed’s Best Match algorithm. International Journal of Technology Assessment in Health Care. 2021;37(1):e7. doi:10.1017/S0266462320002159). But this presupposes that the original search was conducted using Boolean, so it does not really leave Boolean behind… Margaret Sampson is, by the way, one of our Canadian national treasures as far as expert searching goes.
So, AI is one area worth exploring, but perhaps not the only one. Recently, a colleague of mine suggested that after a calibration exercise, a single screener could perhaps be sufficient, allowing large results sets to be split into more manageable chunks and divided among screeners. This is worth exploring. Certainly, we can no longer expect a comprehensive search to find every single published article.
In any case, right now it sometimes feels like AI is like a swarm of mosquitoes that I have to keep swatting away so I can get things done. I look forward to the AI bubble and hubbub settling into something more realistic and useful, as we identify tools that work and solve real problems, instead of having AI shoved into every space, in some cases creating problems rather than solving them. OpenAI’s decision to shut down Sora is a step in the right direction. And maybe while we’re at it, we can stop calling it AI…
Thanks for engaging. Yes, I am fascinated by the similarities and differences between evidence synthesis and more informal academic searching. My worry is that a lot of practices of the former are imported to the latter without much thought in information literacy classes and freshman and undergraduate mimic evidence synthesis style behavior (e.g. long complicated nested boolean) when they don't actually need it.
Conversely, many disciplines talk about "systematic reviews" but again they mimic superficial evidence synthesis style behavior (e.g. long complicated nested boolean) but do not learn how to pilot search strategies. (see my reflection https://aarontay.substack.com/p/how-learning-evidence-synthesis) or do things like filter by journals, avoid grey literature etc for 'quality' instead of doing critical appraisal
Of the two proposals in my blog post, I expect using boolean as first step retriever followed by better ranking will be more acceptable by even evidence synthesis ppractitioners - as shown by the article you have shared & active learning etc, though of course, many still remain conservative and prefer to stick with the screen everything paradigm , as this is the only way to be certain you don't miss anything.
Abandoning strict boolean by using ranked retrieval BM25 is probably a step too far currently for evidence synthesis users, though I do feel that this is actually likely to be a big gain for most academic search use cases except for ES.
I read your post with great interest and agree that these are important questions and considerations. It may be that at some point we will need to rethink whether Boolean continues to be the best approach to searching, although, to be fair, the purpose of Boolean was never to rank results.
Pubmed’s query expansion and ranking does seem to work better than other examples you have stated, but it is still problematic, and in the health sciences we routinely teach students how to bypass automatic mapping and advise caution when using Best Match ranking. This article discusses the ways in which Pubmed’s Best Match can introduce bias into search results: https://pmc.ncbi.nlm.nih.gov/articles/PMC8830327/. We are also seeing an erosion of the quality of indexing since the adoption of automatic indexing in Pubmed and other databases. This has a huge impact on the ability of human searchers, and “AI” driven search to retrieve relevant results, regardless of relevance ranking.
I can’t speak to other disciplines, but in the health sciences, librarians do routinely engage with what happens after retrieval. In the clinical context, limiting search results to the best type of evidence for the question being asked, for example limiting to randomized controlled trials to answer therapy questions or limiting to systematic reviews, is an effective way to quickly identify relevant results despite large retrievals. Biomedical databases have built in “clinical queries” limits that are useful in this regard. On the back end these limits are search queries that have been validated for sensitivity and specificity that are applied the search query. In the context of knowledge synthesis, we routinely teach researchers how to develop and refine eligibility criteria to facilitate screening, and we use sample sets of articles known to meet the inclusion criteria to adjust and improve the search, among other techniques. When conducting scoping reviews, iterative searching to improve relevance of search results, is built into the methodology. In the context of knowledge synthesis we routinely encounter “search resistant concepts” (though I was not familiar with this term, so thank you). It does seem like BM25 retrieval over Boolean in this context is something worth exploring, however it would need to allow for transparency and reproducibility which does not seem to be possible at this time.
Perhaps most importantly, in both contexts, we focus much attention on helping searchers ask clearly formulated questions. In my 20+ years of experience lack of clarity in the question is a more important factor in determining whether search results are relevant than issues in search structure or database relevancy ranking.
The ability to ask answerable questions and search iteratively is what makes human searchers more effective than AI. In my limited experience with LLMs and AI driven search this is where they fail.
The more I engage with the topic the more I come back to the idea that LLMs are like any other technological advance: we first think they will topple all that came before, and then we slowly realize they are a tool like any other, that they have their uses, but that the old technology also still has its uses. I suspect that BM25 versus Boolean is one of these things: we will still want to be able to design transparent, predictable and reproducible searches (I would not want my medical care to be based on a small and random set of relevant results!), but in some contexts, it will be useful to be able to quickly identify some relevant results. Google Scholar already serves that purpose, so why not other tools that maybe do it better?
PS there is an increasing body of evidence that shows that PICO based searching is problematic and you are correct to question whether LLMs are capable of recognising that outcomes should be screened for and not included in the search in most instances.
Hi Francesca,
Thanks for taking the time to leave such a substantial comment.
I always appreciate hearing from health science and evidence synthesis librarians, because that is the area where librarians have done the deepest thinking about search. That said, my blog posts are usually aimed at people outside health and evidence synthesis, though I do borrow ideas from those fields, which can admittedly create some confusion.
I agree that if there is any domain where nested Boolean is still clearly necessary, it is health sciences, evidence synthesis, and probably law or patent searching, where the need for comprehensiveness, predictability, and reproducibility remains especially strong.
Outside those contexts, which is usually my focus, the objective is often less about high recall. In such settings, especially in systems not really designed for Boolean searching, a simpler keyword approach may work just as well or better.
I also agree that Boolean was never designed to rank results. My point is not that Boolean is flawed on its own terms, but that in an environment of information abundance, the absence of ranking is increasingly a practical weakness. Even evidence synthesis librarians cannot realistically screen everything, which is why active learning and related approaches are now being used. In effect, that is another form of ranking.
On PubMed, I am familiar with the concerns around Automatic Term Mapping and Best Match (I even have a footnote ha), including the paper you linked. PubMed is not exempt from the problems of automatic expansion and ranking, and I am not arguing otherwise. I only meant that PubMed shows more sophistication than many other systems, not that its ranking is perfect.
I also agree that the erosion in indexing quality is important, though I see that as a broader issue affecting all search approaches rather than something specific to the Boolean versus BM25 question.
I completely agree that clinical queries and related hedges are fascinating, especially when you look at how they were developed and validated. I have tried to import some of that thinking into other domains, for example by encouraging researchers to test searches against known relevant papers. That kind of piloting seems routine in health sciences, but much less common elsewhere in my experience.
What I often see outside health sciences is that librarians teach students to break a topic into concepts, generate synonyms, combine them into blocks, and stop there. My concern is that without teaching piloting search strategies, in many contemporary systems, especially those using lexical retrieval plus strong reranking, that alone may not outperform a more direct keyword search by very much and might even hurt.
I agree with you that lack of clarity in the research question is often a major reason search results are poor. I would only add that this is distinct from the problem of result overload. Even with a clear question, being faced with a huge set of potentially relevant results is still a serious practical problem.
I think your final point gets to the heart of it. If a searcher wants full transparency, full predictability, and full control, Boolean remains the clearest option, along with screening everything retrieved. But that model is becoming harder to sustain in practice. In some contexts, ranked lexical retrieval may be a reasonable first-stage approach, even if it gives up some control.
Perhaps I should have been clearer that when I talk about BM25 as a first-stage retriever vs Boolean as retriever. A Boolean search using X AND Y AND Z retrieves only records containing all three terms. A BM25-based ranked search using the same terms can retrieve records containing one, two, or all three, with the highest-ranked records often, though not always, containing more of the query terms. This is still lexical search, not semantic search, and still reproducible because it relies on inverted indexes and interpretable because you can explain which documents are included. The trade-off is that it expands the candidate set considerably, which is why such systems usually only rank and show the top portion of possible results say Top 500 results rather then take forever to try rank everything (that has at least one of the terms).
So I suppose my position is this: if the searcher wants absolute control and to screen everything, Boolean remains unmatched. But as the volume of literature grows, many searchers will have to give up some degree of control and rely more on ranking, triage, and iterative refinement.
Hello Aaron,
My fellow librarians and I are having a hard time figuring out whether Web of Science Research Assistant still works exclusively with keyword searches using Boolean operators, or if they've introduced some form of semantic search. In a section of their website (which is already 8 months old!), there is a brief mention of semantic search: “Retrieving Articles: We start by retrieving articles that exhibit the highest degree of semantic similarity to the user’s query and complement them by adding articles with the utmost relevance through a keyword search.” without further explanation (https://webofscience.zendesk.com/hc/en-us/articles/31437630410129-Web-of-Science-Research-Assistant#h_01JGKZG2P1XN9FWQGW98CKWB18).
As for the Smart Search feature, it indicates that you can limit results to those from semantic search or keyword search. Have you tested the WoS Search Assistant again recently? Do you think it’s still based solely on keyword search and Boolean operators, as the tool seems to indicate when it provides a response (for example, in the “How are these results generated?” section)? If there is indeed a semantic search component, do you have more information about it?
Yeah We never did subscribe to Web of Science RA because the feed back from our trial based on relevancy was very bad
Yeah it's ambiguous now, it looks to me likely to be based on both ("complement" seems to suggest 2 methods) reading it. But you can easily test it by manipulating the query so that keyword will unlikely be getting relevant results (I assume it still displays boolean used) and if it still does its likely doing something else as well.
Web of science smart search is definitely hybrid search. As you note the interface itself shows.
https://aarontay.substack.com/p/a-deep-dive-into-ebscohosts-natural-4a6?utm_source=share&utm_medium=android&r=62upd
Hi Aaron,
Thanks for your reply. It is very interesting to see how different disciplines approach searching. I think we can learn from each other. To be fair, the stakes are different- in healthcare a missed paper can result in adverse consequences for patients, so we have to care about what happens after the search is done. More and more, knowledge synthesis is expanding outside of the health sciences, so maybe this will influence how librarians in other disciplines engage with searching? You are right that result overload is a problem and that this is not sustainable. We do need to continue to look for ways to adjust and improve our practice.
And one of the tenets of evidence-based practice, not just in healthcare, is that you don’t just keep doing something because you’ve always done it that way, which is why I consider your suggestion that we should think about what lays beyond Boolean so interesting, even if, at least in healthcare, and certainly in knowledge synthesis, we’re not there yet. This study suggests that we may be able to use Pubmed’s Best Match to some extent, though: “The Best Match sort option in PubMed appears effective in placing relatively more relevant articles in the first fifty records, making it useful for identifying true positive records for training sets and useful in general searching. Investigators who want to identify a certain number of true positive examples for a training set can do this efficiently by screening PubMed records ranked according to Best Match until the desired number of true positives has been identified. In many cases, this will be achieved within the first fifty records.” (Sampson M, Nama N, O’Hearn K, et al. Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed’s Best Match algorithm. International Journal of Technology Assessment in Health Care. 2021;37(1):e7. doi:10.1017/S0266462320002159). But this presupposes that the original search was conducted using Boolean, so it does not really leave Boolean behind… Margaret Sampson is, by the way, one of our Canadian national treasures as far as expert searching goes.
So, AI is one area worth exploring, but perhaps not the only one. Recently, a colleague of mine suggested that after a calibration exercise, a single screener could perhaps be sufficient, allowing large results sets to be split into more manageable chunks and divided among screeners. This is worth exploring. Certainly, we can no longer expect a comprehensive search to find every single published article.
In any case, right now it sometimes feels like AI is like a swarm of mosquitoes that I have to keep swatting away so I can get things done. I look forward to the AI bubble and hubbub settling into something more realistic and useful, as we identify tools that work and solve real problems, instead of having AI shoved into every space, in some cases creating problems rather than solving them. OpenAI’s decision to shut down Sora is a step in the right direction. And maybe while we’re at it, we can stop calling it AI…
Thanks for engaging. Yes, I am fascinated by the similarities and differences between evidence synthesis and more informal academic searching. My worry is that a lot of practices of the former are imported to the latter without much thought in information literacy classes and freshman and undergraduate mimic evidence synthesis style behavior (e.g. long complicated nested boolean) when they don't actually need it.
Conversely, many disciplines talk about "systematic reviews" but again they mimic superficial evidence synthesis style behavior (e.g. long complicated nested boolean) but do not learn how to pilot search strategies. (see my reflection https://aarontay.substack.com/p/how-learning-evidence-synthesis) or do things like filter by journals, avoid grey literature etc for 'quality' instead of doing critical appraisal
Of the two proposals in my blog post, I expect using boolean as first step retriever followed by better ranking will be more acceptable by even evidence synthesis ppractitioners - as shown by the article you have shared & active learning etc, though of course, many still remain conservative and prefer to stick with the screen everything paradigm , as this is the only way to be certain you don't miss anything.
Abandoning strict boolean by using ranked retrieval BM25 is probably a step too far currently for evidence synthesis users, though I do feel that this is actually likely to be a big gain for most academic search use cases except for ES.