{"id":176,"date":"2023-11-18T22:04:44","date_gmt":"2023-11-19T04:04:44","guid":{"rendered":"https:\/\/blog.jdkendall.com\/?p=176"},"modified":"2023-11-18T22:04:44","modified_gmt":"2023-11-19T04:04:44","slug":"clarification-seeking-semantic-similarities","status":"publish","type":"post","link":"https:\/\/blog.jdkendall.com\/?p=176","title":{"rendered":"Clarification (Seeking Semantic Similarities)"},"content":{"rendered":"\n<p>A quick nitpicking follow-up for the previous post about the embeddings used by LLMs. The examples I used were vectors in space, which is intuitive for someone to think about. However, the actual representation inside of vector databases and in LLMs is different &#8211; instead of being a point in space, a semantic concept would be a directional vector of sorts and comparisons would be made using the dot product of the vectors (ie, cosine similarity.)<\/p>\n\n\n\n<p>This isn&#8217;t really necessary to understand it as a layman, but if you were to try to give a more accurate analogy with that detail in mind, then it&#8217;s like each embedding is magnetic and being pulled towards &#8220;idea poles&#8221;, and the strength and direction of the vector represents what it is. It&#8217;s not very intuitive and doesn&#8217;t give any better insight from my perspective, so I went with the simplified version.<\/p>\n\n\n\n<p>If you&#8217;re interested in reading a great write-up with the technical part attached, <a href=\"https:\/\/www.lesswrong.com\/posts\/pHPmMGEMYefk9jLeh\/llm-basics-embedding-spaces-transformer-token-vectors-are\">NickyP has a great article here<\/a>. There&#8217;s quite a bit of good stuff on LessWrong in general, so have a look around the site while you&#8217;re there.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A quick nitpicking follow-up for the previous post about the embeddings used by LLMs. The examples I used were vectors in space, which is intuitive for someone to think about. However, the actual representation inside of vector databases and in LLMs is different &#8211; instead of being a point in space, a semantic concept would&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-176","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=176"}],"version-history":[{"count":1,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/176\/revisions"}],"predecessor-version":[{"id":177,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/176\/revisions\/177"}],"wp:attachment":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}