The vector database is a new kind of database for the AI era

Companies across every industry increasingly understand that making data-driven decisions is a necessity to compete now, in the next five years, in the next 20 and beyond. Data growth — unstructured data growth in particular — is off the charts, and recent market research estimates the global artificial intelligence (AI) market, fueled by data, will “expand at a compound annual growth rate (CAGR) of 39.4% to reach $422.37 billion by 2028.”  There’s no turning back from the data inundation and AI era that’s upon us.

Implicit in this reality is that AI can sort and process the flood of data meaningfully — not just for tech giants like Alphabet, Meta and Microsoft with their huge R&D operations and customized AI tools, but for the average enterprise and even SMBs.

Well-designed AI-based applications sift through extremely large datasets extremely quickly to generate new insights and ultimately power new revenue streams, thus creating real value for businesses. But none of the data growth truly gets operationalized and democratized without the new kid on the block: vector databases. These mark a new category of database management and a paradigm shift for making use of the exponential volumes of unstructured data sitting untapped in object stores. Vector databases offer a mind-numbing new level of capability to search unstructured data in particular, but can tackle semi-structured and even structured data as well.

Diving into vectors and search

Unstructured data — such as images, video, audio, and user behaviors — generally don’t fit the relational database model; it can’t be easily sorted into row and column relationships. Terribly time-consuming, hit-or-miss ways of managing unstructured data often boil down to manually tagging the data (think labels and keywords on video platforms).

Tags can be rife with not-so-obvious classifications and relationships. Manual tagging lends itself to a traditional lexical search that matches words and strings exactly. But a semantic search that understands the meaning and context of an image or other unstructured piece of data, as well as a search query, is virtually impossible with manual processes.

Enter embedding vectors, also called vector embeddings, feature vectors, or simply embeddings. They are numerical values — coordinates of sorts — representing unstructured data objects or features, like a component of a photograph, a portion of a person’s buying profile, select frames in a video, geospatial data or any item that doesn’t fit neatly into a relational database table. These embeddings make split-second, scalable “similarity search” possible. That means finding similar items based on nearest matches.

Quality data — and insights

Embeddings arise essentially as a computational byproduct of an AI model, or more specifically, a machine or deep learning model that’s trained on very large sets of quality input data. To split important hairs a bit further, a model is the computational output of a machine learning (ML) algorithm (method or procedure) run on data. Sophisticated, widely used algorithms include STEGO for computer vision, CNN for image processing and Google’s BERT for natural language processing. The resulting models turn each single piece of unstructured data into a list of floating point values — our search-enabling embedding.

So, a well-trained neural network model will output embeddings that align with specific content and can be used to conduct a semantic similarity search. The tool to store, index and search through these embeddings is a vector database — purpose-built to manage embeddings and their distinct structure.

What’s key in the market is that developers anywhere can now add a vector database, with its production-ready capabilities and lightning-fast search of unstructured data, to AI applications. These are powerful applications that can help a company meet its business objectives.

Vector database strategy starts with use cases that make sense for your business

It’s increasingly common for a company’s comprehensive data strategy to include AI, but it’s vital to consider which business units and use cases will benefit most. AI applications built on vector databases can analyze voluminous unstructured data for marketing, sales, research and security purposes. Recommendation systems — including user-generated content recommendation, personalized ecommerce search, video and image analysis, targeted advertising, antivirus cybersecurity, chatbots with improved language skills, drug discovery, protein search and banking anti-fraud detection — are among the first prominent use cases well managed by vector databases with speed and accuracy.

Consider an ecommerce scenario where there are hundreds of millions of different products available. An app developer building a recommendation engine wants to be able to recommend new types of products that appeal to individual consumers. Embeddings capture profiles, products and search queries, and the searches will yield nearest-neighbor results, often aligning with consumer interests in an almost uncanny way.

Choose purpose-built and open source

Some technologists have extended traditional relational databases to support embeddings. But that one-size-fits-all approach of adding a “vector column” table isn’t optimized for managing embeddings, and as a result, treats them as second-class citizens. Businesses benefit from purpose-built, open source vector databases that have matured to the point where they offer higher performance search on larger-scale vector data at a lower cost than other options.

Such purpose-built vector databases should be designed to easily incorporate new indexes for emerging application scenarios and support flexible scale-out to multiple nodes to accommodate ever-growing data volumes.

When companies embrace an open source strategy, their developers see everything that’s going on with a tool. There are no hidden lines of code. There’s community support. Milvus, a Linux Foundation AI and data project, for example, is a well-known vector database of choice among enterprises that’s easy to try out because of its vibrant open source development. It’s easier to envision it within a broader AI ecosystem and to build integrated tooling for it. Multiple SDKs and an API make the interface as simple as possible so that developers can onboard quickly and try out their ideas that make use of unstructured data.

Overcoming the challenges ahead

Big, paradigm-shifting new tech inevitably brings a few challenges — technical and organizational. Vector databases can search across billions of embeddings, and their indexing is technically different from that of relational databases. Unsurprisingly, developing vector indexes takes specialized expertise. Vector databases are also computationally heavy, given their AI and machine learning genesis. Solving their computational challenges at scale is an area of continual development.

Organizationally, helping business teams and leadership understand why and how vector databases are useful to them remains a key part of normalizing their use. Vector search itself has been around for quite a while but on a very small scale. Many companies aren’t really used to having access to the kind of data search and mining power modern vector databases offer. Teams can feel unsure about where to start. So getting the message out about how they work and why they bring value remains a top priority for their creators.

Charles Xie is CEO of Zilliz