The shortest definition of embeddings?
Embeddings are a very important concept in deep learning and several related techniques. They are powerful and flexible and at the core of many modern machine learning systems. But, they are also tricky to describe.
Here is my attempt to create a very succinct definition and description of embeddings.
Embeddings are learned transformations to make data more useful
What does this mean?
-
Learned means that the parameters of the transformation are determined using the data, rather than just a fixed algorithm. The parameters are optimized for some task using the data as input. E.g. teaching a small neural network to determine if words found in a text dataset are from the same sentence. The network needs to transform the input text data in a way that makes this task as easy as possible (this is basically word2vec).
-
Transformations are how data is changed from being represented in one way into another. This typically means mapping data from one vector space to another, such as taking words from a vocabulary of 10k words that are one-hot-encoded in a 10k dimensional space to a 100 dimensional "dense" vector space.
-
Data can be any type of information: words, sentences, images, audio, webpages, time series. Often the input data is considered a "feature" for machine learning.
-
Useful means that the new representation of the data contains more information in a way that makes the ultimate task easier. Useful can mean:
- Grouping similar items together or warping the representations to make existing groups more easily identifiable (e.g. linearly separable).
- Representing the data in fewer dimensions, whether to more easily encode relationships between the data or for computational efficiency.
- Being reusable for many closely related tasks.
Embeddings are incredibly useful for many data related tasks. They are especially useful for things like search (e.g. what other images are like this one?) and recommendations. More generally they are useful for working with data that is not inherently numeric, such as text, images, or audio.
While I didn't touch on the technical aspects of creating or using embeddings, hopefully this is helpful as a conceptual overview for you.
- Note: My definition is slightly different from others, because I don't think that embeddings strictly need to be in a lower dimensional space, but that is often the best choice for the task at hand. What exactly qualifies as an embedding can get a little fuzzy when considering basic, non-learned encodings (does one-hot encoding count) or learned transformations that are only used as internal (latent) representations, etc.
Learn more in my book
If you found this post useful, you'll probably also find my book useful: Zefs Guide to Deep Learning. Zefs Guide to Deep Learning covers the most important topics in deep learning in a conceptual, easy to understand way, but enough technical detail to help data scientists and machine learning engineers solidify their understanding of what's really going on.
