CrewAI Experiment Gone Wrong: The Pizza Attack Problem Explained
The worst 5 dollars I have wasted in a long time.
Last night I spent probably the worst 5 dollars in a long time. It all began with me working on a crewAi integration. Easy enough I thought, having installed the packages and tools in my virtual environment with a simple pip and reading through the ‘getting started’ documentation of the simple research and writer agent crews.
CrewAI aims to solve the problem of orchestrating agents into task-solving workstreams. Not unlike Microsoft’s Autogen it aims to solve a really important issue within the space of autonomous agents getting agents to work together in a straightforward, easy-to-use, yet robust way. Therefore, I would see crewai largely as a workflow solution rather than an agent-building solution. Tell me if you see it otherwise.
As so many other projects, crewai builds on top of Langchain, which was one of the reasons I was particularly interested in it. Having spent a large part of last weekend getting a Mistral-based ReAct agent to stop hallucinating, it was tempting to maybe having found a solution that ‘just works’ without the drawbacks of too many tied-in subscriptions and giving up freedoms that come with free and open solutions.
TL;DR I was wrong.
Within a few minutes, I was up and running with the default script. Having copied over the above-mentioned example the documentation seemed to work. Coming from my weekend of Langchain documentation sewer dives, that was already a huge step forward. Then I configured environment variables with my free Serper.dev API key and also my OpenAI API key.
Now for my final step, I customized the instructions for my agents (researcher, writer) so I could judge if the output would be within my expectations. In my case, this was to write an article about “AI Agents in Financial Services”. Duh. ;-)
Made a LinkedIn post of it and moved on with my life.
The output was good enough. I used roughly 10 Serper credits and since I had used the default OpenAI API, I also used a few cents here.
Now for the real problem.
Bill Ackman’s favorite signal is looking at equities that had fallen substantially while still having enough volume and business model to turn things around. Using both, a volume tracker of Nasdaq equities, and an efficient outlier model based on a random sample of the daily transaction, I wanted a tool to generate a list of the top 10 volume movers with my “crew” then explaining automatically what happened.
The input to my crew would look something like this
As I already knew from my previous projects with AutoGPT and finclout, I learned that agentic workflows can be really expensive. Therefore, I put a couple of rules in place before I started. Think of it like a stop-limit on a trade.
I created a new OpenAI API key to isolate the cost,
only allowed the OpenAI API to consume up to 5 dollars, and
exchanged to OpenAI API call with my "trusted” Huggingface-based Mistral 0.2-Instruct.
Or so I thought.
This outlier analysis over all Nasdaq equities takes about 3-4 hours. The main reason is that I have sleeper calls between API calls to be a good digital citizen.
I checked that the script was running and went to bed.