A quick nitpicking follow-up for the previous post about the embeddings used by LLMs. The examples I used were vectors in space, which is intuitive for someone to think about. However, the actual representation inside of vector databases and in LLMs is different – instead of being a point in space, a semantic concept would be a directional vector of sorts and comparisons would be made using the dot product of the vectors (ie, cosine similarity.)
This isn’t really necessary to understand it as a layman, but if you were to try to give a more accurate analogy with that detail in mind, then it’s like each embedding is magnetic and being pulled towards “idea poles”, and the strength and direction of the vector represents what it is. It’s not very intuitive and doesn’t give any better insight from my perspective, so I went with the simplified version.
If you’re interested in reading a great write-up with the technical part attached, NickyP has a great article here. There’s quite a bit of good stuff on LessWrong in general, so have a look around the site while you’re there.