What a year of testing & thinking about AI academic search taught me

The Chatbot Is the Wrong Mental Model

A look back at the ideas that stuck, the ones I am still unsure about, and where I think this is heading.

About a year ago I shifted my main writing home from blogger to Substack. Since then, this newsletter has grown from roughly 3,000 email subscribers to around 5,500. While the affordances of Substack have made it easier for subscribers to follow, the fact that a niche newsletter about academic search and library technology could add 2,500 readers in a year is mostly a signal about the moment we are in. Librarians, researchers, and tool builders are all unsettled by what “AI search” is doing to academic discovery.

I think there are not many places pondering in the open about what this means, trying to work it out rather than deciding they already have figured out that “AI” applies to search (or anything really) is bad.

If this newsletter has a job, that is the job. Thank you for reading, for arguing in the comments, and for forwarding posts to colleagues who hopefully subscribe.

If a year of this has taught me one thing, it is that the freeform chatbot is the wrong mental model for academic search. The fear that dominates most discussion, that these systems fabricate sources and invent findings, is real for a naked language model answering from memory or even one with weak naive RAG implementation.

But wrap a model in an agentic harness that runs deterministic checks in code, and force it to verify every claim against an independent source rather than generate it freely, and most of that risk is manageable (at least for claims that can be checked against independent sources). I was not sure of this twelve months ago, or at least I could not have said it cleanly without experience trying LLMs with agentic harnesses.

This post provides a thematic retrospective of the content on my blog and ends with some upcoming future content.

One question, many posts

There is a narrower question every post here is also trying to answer: when a vendor says “AI-powered search”, what is being claimed, and how should a librarian evaluate it without either hype or unthinking rejection?

That span of 12 months opened with a two part article testing how RAG systems handle retracted papers across Elicit, Scite, SciSpace, Primo Research Assistant, Undermind and others, and it closes with sharper “hot takes” about where librarianship is misreading the whole category. The through-line is the same: take the tools seriously enough to test them properly, and take our own assumptions seriously enough to question them.

Defining the thing before arguing about it

The post I point newcomers to first is What Do We Actually Mean by “AI-Powered Search”?. The phrase hides at least four different things: post-retrieval conveniences like summarisation that do not touch your results at all; semantic retrieval using vector embedding that goes beyond lexical matching; LLMs used in retrieval and ranking; and full synthesis and generation across papers. Your concern about one of those may not apply to another, and most of the unproductive arguments I see come from people defending or attacking different levels while using the same words.

The framework of mine that has resonated the most, though, is a bit older. Why I Think Academic Deep Research — or at Least Deep Search — Will “Win” is where my 2x2 matrix first appeared, and it has since been picked up and cited well beyond this newsletter.

It sorts search systems along two axes. The first is how the system works: a quick, one-shot retrieval, or a deep, iterative process that evaluates many candidates. The second is what it returns: Search, which only retrieves and ranks documents, or RAG, which extracts from them to generate an answer with citations. That gives four quadrants, Quick Search, Quick RAG, Deep Search and Deep RAG, and most of the products people argue about sit cleanly in one of them.

The post argues that the deep quadrants are where the future of academic search is heading and why Deep Research is mostly meant for helping to orientate yourself. Whatever you make of that call, the 2x2 itself has been the durable contribution.

If you are wondering what vector embeddings are, a natural companion piece is Why Embedding Vector Search Is Probably One of the Least Objectionable Use of AI for Search. Embedding-based semantic search dodges many, though not all, of the criticisms aimed at ChatGPT-style generative AI. It does not fabricate text or invent citations the way generation can; it only changes how documents are matched and ranked. Map it onto the four levels above and it sits well below full synthesis on the risk gradient, which is worth remembering before treating every form of "AI search" as equally suspect.

The move to vector embedding is not without direct consequences, and it is the subject of The Case of the Vanishing Hit Count, written up based on a question from my FSCI 2025 workshop. Once you leave Boolean behind for embedding and other AI-driven retrieval, the exact, countable hit set disappears, replaced by the top-k of semantic search or even the evaluated results of deep search, until even the familiar question of how many results you got no longer has a stable answer.

As I tested and used these new, modern “AI search”, I noticed a UX issue which I called initially The Blank Box Problem. We spent two decades complaining that users would not learn Boolean operators and field codes. We have now replaced all of that with a single blinking cursor in an empty box, and users are more confused than ever, caught between not knowing how to speak to the machine (keyword, natural language, or elaborate prompt) and not knowing what the machine can even do with their words. Trading visible constraints for infinite potential turns out to be a usability crisis, not a usability win.

Finally, I attempted to classify the ways LLMs summarise in tools like EBSCO and ProQuest. I distinguish between fixed single-document summaries, query-dependent single-document answers, document-by-document synthesis tables, and full multi-document RAG synthesis. The classification shows that “AI summaries” are not one thing: as tools move from pre-canned article summaries to query-driven synthesis across many papers, usefulness rises, but so do unpredictability, evaluation difficulty, and trust risks.

Deflating the hype, without dismissing the tools

While I am known to be “pro-AI”, particularly for search - I have pushed back on many over-claims and warned about issues this new technology brings.

The AI-Powered Library Search That Refused to Search is probably one of the most read posts on my Substack so far, perhaps because critical posts about AI travel further than careful ones.¹.

It documents an odd new problem: content-moderation layers designed for chatbots are creeping into library discovery and quietly refusing to search certain controversial topics at all. How serious this is, I am not yet sure.

The Horseless Carriage of AI Search takes aim at an increasingly common pattern in legacy databases, the claim that they now “do AI”. It argues that bolting an LLM onto the front of a database, such as Primo Research Assistant, to generate Boolean strings is mostly of little benefit. I am not saying Boolean must die, but I am saying that dressing up the old engine is not the breakthrough it is sold as. I have since had to qualify this argument, and I come back to exactly where it breaks near the end.

Deep Research, Shallow Agency takes aim at the word agentic. Most academic deep research tools at the time, such as Asta Paper Finder, Undermind Classic and Scholar Labs, run fixed internal workflows, and they stumble the moment you hand them a literature review task that does not fit what they were built to solve. The agency is largely an illusion.

The follow-up, From Fixed Search Workflows to Agentic Academic Search: Promise, Progress, and the Invisible Menu Problem, looks at what happens as tools start handing control back to users. Undermind’s Projects feature is the worked example: it lets you steer and manage the agent rather than accept a fixed pipeline, which is real progress. But it also exposes what I call the Invisible Menu Problem. When you are managing an agent, you cannot see the full menu of what it can do, so you are left guessing at its capabilities and limits. It echoes the Blank Box Problem one level up, with the opacity shifted from the query to the agent itself.

In Are AI Tools Killing Review Articles? Two Failure Modes Suggest Otherwise, I argued that the most important kinds of review articles are still hard for AI, for two reasons: the corpus construction gap, where tools still struggle with retrieval and exhaustiveness, and the conceptualisation gap, where expert judgement is needed before retrieval even begins.

The uncomfortable mirror for librarians

A few posts are aimed squarely at us librarians.

“We’re Good at Search,” Just Not the Kind the AI Era Demands is a deliberate provocation. When I look at the evaluation rubrics for AI search now circulating, relevancy is often just one subjective category among many, scored on an “I know it when I see it” basis. For a profession that prides itself on search, that is a strange place to be.

AI Academic Search and the Missing Middle of Literature Discovery makes a related point: I argue by judging these tools mainly through undergraduate information literacy and through evidence synthesis, we underrate their strongest current use case, the ordinary narrative literature review, perhaps because our involvement here is less.

My most recent post, Stop Calling Poor Search Rankings Necessary Friction for Learning, takes aim at a comfortable rationalisation. When a system ranks badly, it is tempting to argue that the struggle is good for students, that wading through irrelevant results teaches them something. There is a legitimate version of that idea, where we set out to expose students to adjacent and boundary literature, but that is intentional design. Defending accidental noise as if it were pedagogy is not the same thing, and we should stop confusing the two.

This connects directly to an older post, The Sycophancy Fallacy. Both are the same category error: mistaking a ranking system’s output for a stance. The worry that AI search tools will agree with users the way chatbots do treats retrieval as a conversational partner that can flatter. A retrieval system is not agreeing with you, it is returning what its ranking surfaces. Confusing the two means worrying about the wrong bias and missing the real ones.

The last is a little left field, since digital literacy is not my speciality. Fact-Checking vs the Academic Inquiry Mindset asks whether we are collapsing two different habits of mind into one. Verifying a discrete claim is not the same as open-ended scholarly inquiry, and treating the first as though it were the second risks leaving students short of what research demands.

The plumbing problems nobody is watching

The least glamorous posts may matter most. The Petrol Tank for AI Discovery Might Be Running Dry flags that the comprehensive AI search engines we are starting to lean on, such as Elicit, Consensus and Undermind, depend on access to abstracts and full text that publishers now have a strong commercial incentive to close off.

Why Ghost References Still Haunt Us makes two claims: that Google Scholar's [CITATION] entries and a polluted web can undermine the verification RAG depends on, and that fabricated and ghost references often predate LLMs. I am confident about the first for reasons I give below.

Trying to explain and understand

About a year ago, I felt a sense of unease whenever I heard the term agentic search, (mostly in the context of Undermind), I had an uneasy sense that I did not really fully understand what it meant.

As I spent time thinking about it, trying MCP servers, skills, and finally LLMs with agentic harnesses like Claude Code and Codex, something clicked.

The Agentic Researcher and the MCP server posts were two of my earliest posts documenting my growing sense of understanding of what “agentic” means.

By pairing a general LLM with academic Model Context Protocol servers or other tools, they can match or beat specialised deep research tools. This may point to a real-time, federated model of access to publisher content rather than yet another giant centralised index.

Reviews & evaluation frameworks

I did some hands-on reviews of Consensus, Scholar Labs, ResearchRabbit and Google’s AI Mode as they shipped.

Librarians are always interested in evaluation frameworks, so in AI academic search needs better frameworks for understanding and evaluation I pointed to three librarian projects I found promising. That was followed by Better AI Search Rubrics: Roles, Gates, and Retrieval Tests, an opinionated piece on the common weaknesses in such rubrics and ways to improve them.

Other posts of interest beyond the twelve-month window

I did not suddenly start writing about AI search when I moved here. A few from outside this window that I still consider significant, now cloned onto Substack from my Blogger platform:

• The reproducibility and interpretability of academic Ai Search engines like Primo Research Assistant, Web of Science Research Assistant, Scopus AI and more

• Testing AI Academic Search Engines (1): Defining the tools & Testing AI Academic Search Engines - What to find out and how to test (2)

• Comparative review on Primo Research Assistant, Scopus AI, Web of Science Research Assistant and an explainer for AI search for librarians

• When is a Hallucination Not a Hallucination? The Role of Implicit Knowledge in RAG

• The Rise of Agent-Based Deep Research: Exploring OpenAI’s Deep Research, Gemini Deep Research, Perplexity Deep Research, Ai2 ScholarQA, STORM, and More in 2025

• Why use of new AI enhanced tools that help with literature review should be discouraged for undergraduates

• Primo Research Assistant launches- a first look and some things you should know

What I am unsure about, and what comes next

In keeping with how I try to write, the honest caveat: a lot of this is provisional, and some of it will age badly. The landscape shifts month to month, and several posts above are me thinking out loud rather than concluding.

Two examples.

The first is the ghost references post. I stand by the weakness it identifies in Google Scholar, but I am no longer confident in its conclusion that in the example I investigated the first version of the ghost reference I traced was human-made rather than produced by an LLM. Treat that part as an open question rather than a finding.

Where my “horseless carriage” critique breaks

The second I have to own more directly. I was proud of The Horseless Carriage of AI Search, my critique of legacy vendors taking the easy route to “implement AI” by using an LLM only to generate Boolean.

After using Primo’s MCP server I have had to qualify it.

The critique was aimed at one specific move: using an LLM as a translation layer that feeds Boolean into the same lexical engine over the same document corpus. I still think that is mostly low-value.

What I missed is that database and dataset selection is a different task from normal article search.

There the target is a small, description-only space, and the model earns its keep two ways. It normalises vocabulary, and it draws on pretraining knowledge to go straight to the right resource. That is not dressing up the old engine. It is doing something the old engine could not. The critique holds for article retrieval and does not generalise to query-to-resource matching.

A concrete example helps clarify.

Experienced librarians know that users struggle to find the right database for datasets, because their query terms have to match a very small target space, the database description. A wrong term returns zero hits: search for cost of living datasets/databases rather than CPI and you find nothing.

A capable LLM sidesteps this, because it knows to search for the right terms, CPI, inflation , and so on.

Claude 4.8 is similarly smart, reasoning (after searching) that most database records will not have “cost of living” in their metadata and switching to searching Economic databases.

But what if the database description just doesn’t have the right terms? Here is where LLMs can also help by “cheating” because they already know the answer (from pretraining knowledge).

In this query, I asked for datasets with the right terms and it seems to do as expected, surfacing the obvious databases.

But then I noticed the searches it did

The first search was expected but the second was odd - a known item search of the dataset ExecuComp! In short, a large LLM is capable of knowing the right answer in advance and just search to confirm we had access!

Below shows yet another similar example, where it searched directly for Audit Analytics (a known item search of a dataset in WRDS) when asked about datasets on audit reports.

What comes next is going to double down on understanding agents. At FORCE2026 I presented a small study of the reproducibility of the agentic search tools Undermind and Consensus with the title - Does Agentic Deep Search Converge? Reproducibility Questions for LLM-Driven Literature Discovery, and the results were “interestingly wrong” in ways worth writing up here.

An agentic skill for building and testing high sensitivity Boolean Search Strategies for Pubmed.

The bigger piece is one I have put off for too long, out of impostor syndrome and a sense of how evidence synthesis librarians might react (exacting high standards for obvious reasons): a write-up of a Claude and Codex skill I built for high-sensitivity Boolean searching in PubMed.

• It is designed to heavily do tool use, using PubMed searching and MeSH browsing to check for promising terms, test pilot search strategies etc.

• Instructions also include very specific domain knowledge on how to go about conducting the process based on my (somewhat limited) knowledge of best practices.

• Comprehensive audit logging, so you can see at every step what was done and the decision made

• Hooks that use code to enforce important steps and checks

• Maybe 50+ hours of testing against gold standard sets to test for edge cases.

It is the most concrete demonstration I have of the argument I opened with. Once you wrap the model in a harness that can run deterministic checks in code and make it verify its claims against independent sources rather than generate them, much of what people fear about fabrication stops being inevitable.

The PubMed skill is one small instance of that pattern, and I think it generalises. Descriptive and subject cataloguing is another domain where a similar method could be applied.

While I don’t think my home-brew skill currently exceeds the best experts, it has reached the point where it can reproduce or improve on many of the moves I would make myself, while logging and checking them more consistently (which is why I can’t improve it further easily). I suspect a team consisting of expert evidence synthesis experts and professional coders will be able to push it far further than what I did.

Conclusion

Agentic skills scare me because they look like a second attempt at expert systems, but with a crucial difference. The old systems tried to hard-code expertise. The new ones combine flexible language-model reasoning with tools, logs, tests, and code-enforced constraints. Maybe even that isn’t enough, but this “neurosymbolic AI” approach does make them much more plausible as systems for operationalising professional judgement with all the obvious implications².

Yes, I could be totally wrong. Even then, I would rather name what could be better and be wrong in public than pretend something is settled. If a take here does not hold up in a year, I would sooner hear about it than quietly forget I wrote it.

That goes especially for the readers who arrived through my more critical posts, and who I suspect sit closer to the resist-AI end of things. I hope some of you stayed once it was clear that I am, on the whole, positive about AI for search, and I would rather argue it out with you than preach to people who already agree.

Thank you, then, for a year of reading and pushing back. The work is better for it. Here is to the next set of posts, and to being usefully wrong together.

1. It also got me a ton of subscribers, I suspect many are in the “resist AI” camp. I do hope at least some of them stuck around, after they realised I am on the whole very positive about the use of AI for search.

2. Watching Codex or Claude run code and get an error, then smoothly work around it for the first time was an eye-opener. You could even then ask it to fix the skill so future runs would not get this issue!