An Introduction to Pinecone
When Less is More - Vector Search as a Service
Yesterday I was on a call with James Briggs speaking about “Making Retrieval Augmented Generation Better” in preparation for this post and kept pondering about memory, retrieval algorithms, context windows, and the meaning of life in all this.
And then it dawned on me. If you look into vector databases in the context of autonomous agents they can be so much more than just a “semantic similarity search as a service”.
Vector databases are the long-term memory that an agent’s hippocampus can access to maximize rewards in the latent space of relevant sequential short-term observations.
Who built a company that had the foresight for such a product?
Executive Summary
Pinecone, the company, was founded in 2019 by Edo Liberty who prior to being a successful founder was Director of Research at AWS and Head of Amazon AI Labs.
The company was initially launched as a similarity search at scale engine but offers now a fully managed SaaS that makes it easy to add AI capabilities to applications.
Pinecone raised a 10 Million Seed round led by Wing Venture Capital, a 28 million Series A, and a $100M Series B led by Andreessen Horowitz, with participation from ICONIQ Growth, Menlo Ventures, and Wing Venture Capital
Pinecone, the product, is a database architecture designed to store and query high-dimensional data, such as vector embeddings of text, images, and videos, in a way that is efficient and scalable.
What problem does that solve?
Problem
Traditional databases are not designed to handle high-dimensional vector data and suffer therefore from the curse of dimensionality. I.e., the volume of high-dimensional space grows exponentially with the number of dimensions. This also means that the distance between two points becomes less meaningful as dimensionality increases. Distance between two points is commonly used as an approximation of similarity.
In the past, we have used keyword-matching in recommendation engines and algorithms to approximate similarity. But after creating social media echo chambers for 20 years, we had to realize that in reality, keywords don’t work that well. For example, “Apple” could refer to a fan of fruits or a connoisseur of computer products.
Technological innovation to the rescue. With the invention of the encoder/decoder method (Ilya Sutskever, 2014) and the transformers library, we invented a technology that allows us to express every image, text, sound, or video as a vector.
Vector-based recommendation engines then can perform a highly scalable search using proximity (Cosine, Dot product, Euclidean) in high-dimensional vector embeddings.
Okay, we improved recommendation engines, isn’t that a quite saturated market already?
Yes. But generative AI isn’t.