Collaborative Re(Search) with Stanford's Storm

A research prototype for automating the knowledge curation process.

Current-day LLMs have one major flaw. The moment in time they don’t know, i.e., they have not been trained on something, they start to hallucinate. One of the birthplaces of modern AI, Stanford, has launched Storm, an AI-powered web app with an accompanying paper that aims to research articles of high quality, including references to published sources. I wish Google Search had that option.

Of course, I had to give it a spin. I skipped the “making an account” part. I am sure you will be able to figure this out yourselves.

The user interface is quite straightforward and has the same prompt, samples, and history trifecta UI we have gotten accustomed to from other more well-established AI apps. Since I was really curious about autonomous agents, this is what I decided to “research” first.

While “automous” is not a word — when you see it you will know— the app allowed me to select between a fully “autonomous”, and a Human-AI collaborative article.

Since I am not a particularly lazy person I selected the collaborative option. This option allows the user to have conversations with different specialized agents about the topic providing a more focused dive (err delve) into the topic. Also, I expect researching the term “autonomous agents” to be really broad thus giving the AI poor guidance of what I want and expect. I believe that a good manager is someone who gives clear and direct instructions on what is expected of a task. Agents are not that different from us humans in that way. The app then forced me to complete a little easy-to-miss grey box underneath which I did as you can see in the screenshot above.

Then you have to wait for a couple of minutes while Storm familiarized itself with the topic. That seems to be about 4 steps of a RAG process.

Alright then, I got some results. On the left-hand side is what looked like an outline for the research, but nothing is clickable.

On the right-hand side is the result of the research.

The generated stub started off with “ethical considerations”, kinda getting this California 2024 vibe. Hovering over the blue [1],[2] provides you with its references. In this case, it was an old Boston Consulting Group article that was interesting at the time but as usual consulting work goes, there is not much there besides fluff and fancy visualizations.

I generated the second part of the article. The app “caught”, their term not mine, a selection of new conversations. The highlighted text about reasoning “caught” my eye,

since it is eerily similar to what I am working on with my Game Theory and Strategic Games series of posts. Interesting that 43 and 42 were switched. Source 43 is a book on Artificial Moral Agents from 2022 by Catrin Misselhorn. Great find. I truly didn’t know that book. Source 42 is a blog post by SmythOS a website that lets you create agents quickly which is not really my cup of tea.

I clicked the generate button again and an “AI Research Scientist” entered the conversation. Now I can see there is a lot of buffer content like “A key aspect is identifying and balancing rare domains” Ok. I guess.

Not much collaboration here yet. I suppose I can continue to click and bring new experts into the conversation, but I want to explore agents in scenarios of adversarial strategic board games like 5x5 Tic Tac Toe further since I am working on it already and it provides a nice onramp to more sophisticated games like Minecraft or Civilization.

Storm explored this topics from the perspective of a general knowledge provider further. Which I guess is a reasonable approach.

The provided links were ok with IEEE’s Transactions on Games being one source [1] that will definitely trigger me to read further. An ACM paper from 2017 which is likely outdated and would focus more on RL rather than Transformers-based LLMs [2], and also a 2007 paper on The Strategic Games Matrix as a Framework for Intelligent Autonomous Agents Hierarchical Control Strategies Modeling [3] that sounds interesting but is really old. But it might give me some good ideas.

I decided that I was satisfied with the conversation and created an Article by clicking the button of the same name.

I wondered where the origin of the word “article” came from. Searched Google, selected the second link Oxford English Dictionary, navigated to the etymology section, clicked a button, and was greeted with a "subscription” option.

I suppose the biggest benefit of Storm is that it is a Search Engine that has not been enshittified yet. And for me, probably that is the more interesting outcome of using that application. Since the data sources are to some extent limited to professional data sources and Bing, the blog spam, which may or may not include this Substack, is not part of the Search results. For most people who use Search for professional work, this is actually a good thing.

I downloaded the final research as a PDF and moved on.

If you read this far, I am deeply grateful.

If you are a paying user below the paywall is the link to Chapter 3 of Catherine’s book