An embedding is a list of numbers that represents meaning. That's it. Feed a word, sentence, or image into an embedding model and it spits out a vector, usually a few hundred to a few thousand numbers, positioned in space so that similar things land near each other. "Dog" sits close to "puppy." It sits far from "spreadsheet." Every modern search engine, recommendation system, and RAG pipeline runs on this trick.
Old-school natural language processing treated words as categories. A dictionary lookup. A tag. Embeddings throw that out. They place every piece of text at a coordinate in a high-dimensional space, often 384, 768, or 1536 dimensions depending on the model.
Think of a map with only two dimensions, latitude and longitude. Two cities close together share a similar climate and culture. Embedding space works the same way but with hundreds of axes instead of two. Each axis captures some abstract property the model learned during training, not something a human labeled or named.
Nobody can point to dimension 47 and say "that's the sarcasm axis." The dimensions are learned, not designed. That's the tradeoff: the representation is powerful but not interpretable by inspection.
Embedding models train on massive amounts of text using a simple objective: pull related things closer, push unrelated things apart. One common method, contrastive learning, shows the model pairs of similar sentences and pairs of dissimilar ones, then adjusts the vector positions until similar pairs cluster and dissimilar pairs separate.
Earlier models like Word2Vec learned this from word co-occurrence. If "king" and "queen" kept showing up in similar contexts, they landed near each other. Modern embedding models, built on transformer architectures, do the same thing but account for full sentence context, not just nearby words. That's why today's embeddings can tell the difference between "bank" the riverbank and "bank" the financial institution, based on surrounding words.
Training takes enormous compute and enormous data. Most teams never train their own embedding model. They call an API from OpenAI, Cohere, or an open-source model like BGE or E5, and get a ready-made vector back.
Once everything lives as a vector, comparing meaning becomes a math problem. The standard measure is cosine similarity: the angle between two vectors. Small angle, similar meaning. Ninety-degree angle, unrelated. Opposite direction, contradictory.
This is why semantic search beats keyword search. A keyword search for "affordable laptop" misses a page that says "budget-friendly notebook computer." An embedding search catches it instantly, because the two phrases land near each other in vector space even though they share zero words.
The same math powers recommendation engines. A song embedding close to songs you already liked gets recommended. A fraud detection system flags a transaction whose embedding sits far from a customer's normal spending pattern. Same core mechanism, different label on the box.
Four systems depend on embeddings directly, and if you've used any app in the past two years, you've touched all four without knowing it.
| System | What Embeddings Do There |
|---|---|
| Semantic search | Match query meaning to document meaning, not just shared words |
| RAG pipelines | Retrieve the right chunks of text to feed an LLM before it answers |
| Recommendation engines | Find items whose vectors sit near a user's preference vector |
| Deduplication and clustering | Group near-duplicate records without exact string matching |
RAG deserves a specific mention because it's the fastest-growing use case. A large language model has no live memory of your company's documents. So a RAG system embeds your documents ahead of time, stores the vectors in a vector database, then at query time embeds the user's question and retrieves the nearest matches. The LLM only ever sees the retrieved chunks, not your entire archive. Bad embeddings mean bad retrieval, and bad retrieval means a confident, wrong answer.
Embeddings inherit whatever bias sits in the training data. If the training corpus associates certain job titles with one gender more than another, the vectors encode that association whether anyone wanted it there or not. This isn't a hypothetical. Documented cases show embedding models placing "nurse" closer to female pronouns and "engineer" closer to male ones, purely from statistical patterns in the source text.
Dimensionality collapse is a quieter failure. When a model gets pushed to compress too much meaning into too few dimensions, distinct concepts start bleeding together. Two unrelated ideas end up sitting closer than they should, and search results get muddy in ways that are hard to diagnose because the math still runs fine. It just returns worse answers.
Embedding drift is the one teams forget about most. If you switch embedding models, or the provider updates their model silently, old vectors and new vectors no longer live in the same space. Comparing a vector generated last year against one generated today produces garbage, even though both look like valid numbers. Any production RAG system needs a plan to re-embed the entire document store whenever the model changes. Skipping this step is the single most common cause of a search system that used to work and suddenly doesn't.
Three factors matter more than benchmark leaderboards: dimension count, domain fit, and cost per call. Higher dimensions capture more nuance but cost more to store and search. A 1536-dimension vector takes four times the storage of a 384-dimension one, and at millions of records that difference shows up on the infrastructure bill.
Domain fit matters more than most teams admit. A general-purpose embedding model trained on web text will underperform on legal contracts, medical records, or source code, because the vocabulary and structure differ enough that generic training never saw enough examples. Domain-specific embedding models, or fine-tuning a general model on your own data, close that gap fast.
Cost per call adds up in ways people underestimate until the invoice arrives. Embedding a million documents once is cheap. Re-embedding them every time you switch models, or every time a document updates, is where budgets get eaten. Pick a model with a migration path in mind, not just the best score on today's benchmark.
My take: most teams overthink model selection and underthink the retrieval pipeline around it. A mediocre embedding model with clean chunking, good metadata filters, and a sane re-ranking step will beat a state-of-the-art embedding model bolted onto a sloppy pipeline every time. The vector is one piece of the system, not the whole system.
An embedding is a list of numbers that represents the meaning of a word, sentence, or image. Similar things get similar numbers, so a computer can measure meaning using math instead of matching exact words. This is the mechanism behind semantic search and modern AI retrieval systems.
Word2Vec assigns one fixed vector per word regardless of context. Modern transformer-based embeddings generate a vector based on the entire sentence, so the same word gets a different vector depending on how it's used. That context sensitivity is why current embeddings handle ambiguous words far better.
RAG systems need to find the right documents before an AI model can answer a question, and embeddings are what make that retrieval possible. Documents and queries both get converted into vectors, then the system finds the nearest matches by distance. Poor embedding quality directly causes poor retrieval and wrong answers.
Yes. Embedding models learn from training data, and any bias present in that data, like gender or racial associations, gets baked into the vector positions. This has been documented repeatedly in research on word and sentence embeddings. Teams building on top of these models need to test for it rather than assume neutrality.
Vectors from different models don't live in the same mathematical space, so comparing old vectors to new ones produces meaningless results. Any switch requires re-embedding the entire dataset from scratch. This is a common and costly mistake in production search and RAG systems.