{"id":137,"date":"2023-10-11T08:52:10","date_gmt":"2023-10-11T13:52:10","guid":{"rendered":"https:\/\/blog.jdkendall.com\/?p=137"},"modified":"2023-10-11T09:20:10","modified_gmt":"2023-10-11T14:20:10","slug":"bytes-to-bites","status":"publish","type":"post","link":"https:\/\/blog.jdkendall.com\/?p=137","title":{"rendered":"Bytes to Bites"},"content":{"rendered":"\n

How do we get from a user’s message to an informed response from Chom? I’ve been chewing on this one for a while and I think I have an idea for a first draft. So let’s go ahead and just hit you with the sequence diagram right away:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

(Go ahead and click to get a closer look.)<\/p>\n\n\n\n

So, a quick overview. <\/p>\n\n\n\n

Chom runs requests through multiple AI models (known as LLMs) to perform a handful of different natural language processing tasks, before preparing a response based on a database of recipes, ingredients, and allergens.<\/p>\n\n\n\n

When a user interacts with Chom, the system initiates by fetching past chat context for continuity purposes. A steering model then checks the new message’s request for appropriateness – are we on topic for a culinary discussion, are we being asked for any illicit activities, and so on. (Anything failing the sniff test will be rejected with a polite response.)<\/p>\n\n\n\n

Once we’re sure we’re kosher (heh), then a command parser discerns the user’s intent and relevant entities involved in the request. The example in the sequence diagram is extracting information about substituting allergens out in favor of other ingredients, but other actions the model might identify could be altering recipes, adding\/removing recipes from the list, removing already-on-hand groceries and so on. The AI will be picking from a predefined list of actions, so “launch the nukes” won’t be on the docket. …Probably.<\/p>\n\n\n\n

Once we know what action we’re taking, we can consult our embeddings database for relevant Real World Data\u2122. This contains info about recipes, ingredients, and so on that lets us turn a wide array of vocabulary into specific and uniform concepts (“peanut allergy” -> Tree Nuts Allergy) that the system can work with. <\/p>\n\n\n\n

In our example here, the system consults the embeddings database to comprehend related allergen groups and updates the user preferences database accordingly. Saving allergens on hand ensures that even if the user forgets to mention this in the future, we’ll be accommodating their needs. Allergens will also be able to be toggled in the user’s profile, along with other dietary restrictions (vegan, vegetarian, carnivore, a diet of pure antimatter… the usual.)<\/p>\n\n\n\n

Chom then crafts a suitable response, keeping prior interactions in mind via the chat context, and lastly logs the entire exchange in the chat history database to keep a record of the user’s conversation on hand so the user can review their chat history with Chom in case they want to look back at some earlier recommendations or retrieve grocery lists at a later date, etc.<\/p>\n\n\n\n

For purposes of the alpha, Chom is powered by ChatGPT for all of its LLM usage. The per-request API costs are much lower than hosting models myself on a GPU-powered cloud instance or server blade. However, Chom is designed to swap in the latest LLM tech from its local machine, allowing the use of self-hosted models such as Vicuna instead. This will be very important for scaling up later, and keeps us from being tied to a specific provider like OpenAI.<\/p>\n\n\n\n

What the heck’s a steering model, anyway?<\/h2>\n\n\n\n

The steering model is a sophisticated mechanism designed to influence or “steer” the responses of a larger foundational language model (like ChatGPT or Vicuna). In its training phase, it absorbs insights from a mix of human feedback and reinforcement learning. This equips the model with the capability to predict how best to guide the base model to produce the desired outputs.<\/p>\n\n\n\n

Once this training is complete, the steering model seamlessly integrates with the base language model. When they work in tandem, any incoming query is first shaped by the steering model, providing the foundational model with additional context or guidance. This ensures the final response aligns with the intended behaviors.<\/p>\n\n\n\n

In our case, we’re using this for filtering off inappropriate discussions, but it could easily be extended to catch vague statements and short-circuit with clarifying questions before passing the request forward to the rest of the system. We can also specialize our steering model by training it specifically against additional data from future user interactions (opt-in only, of course) so that it better recognizes those weird edge cases that leave people scratching their heads at the generated response.<\/p>\n\n\n\n

Embeddings? Like, putting a thing inside another thing?<\/h2>\n\n\n\n

Yeah, I don’t know. There’s a good reason for it in the literature but I’m still wrapping my head around it. The way I think of it is as semantic grouping – also a jargon-y term, but broken apart, “semantic” is another word for “what does it mean”, and grouping is… well, you group things together. So embeddings are a way of grouping things together based on their meaning. <\/p>\n\n\n\n

For example, “cat” and “tiger” would be grouped closer together than “cat” and “dog” on account of both being felines. Likewise, “cat” and “dog” would be grouped closer together than “cat” and “cow” on account of both being pets.<\/p>\n\n\n\n

The technical details are gory and filled with math, but the idea is that these LLM models are made up of an incredibly complex network of semantic groupings based on all of the training data. You can imagine that it’s like if you wrote down every meaning you can think of for a concept like “cat” onto individual index cards labeled “Cat”, then did it again for another one like “dog”, and then another for “cow”, etc… then spaced each set of concepts out in your room connected by strings to each other where the length of the string is how close or far apart the ideas are to each other. Now do it in 768 dimensions instead of just three. (And that’s just for BERT models. Some models use even more. It’s crazy.)<\/p>\n\n\n\n

Thanks to math we can wrangle all of that and end up with a way to take a word or phrase, turn it into those crazy strings, and then follow the strings to one of our embedding database’s known words.<\/p>\n\n\n\n

Well, what next?<\/h2>\n\n\n\n

As mentioned in my first Chom post, I’m working through steering and command parsing right now. The current iteration of Chom is still a wild west of “making stuff up” at times, and I’m ironing that out. It’s largely accurate in its responses to the user, but where that data ends up in the data format it’s supposed to output is… imaginative, and it will happily talk about any topic after enough probing to get it to break away from its culinary assistant prompting.<\/p>\n\n\n\n

I’m also thinking about how I can visualize the processing from the API calls so I can share that on a blog post as well. It would be very interesting to see what Chom’s different brains are thinking at each stage and to talk about how I adjust those. We’ll see what I come up with.<\/p>\n\n\n\n

Alright, that’s all for now. Ctrl-S, Ctrl-Q.<\/p>\n","protected":false},"excerpt":{"rendered":"

How do we get from a user’s message to an informed response from Chom? I’ve been chewing on this one for a while and I think I have an idea for a first draft. So let’s go ahead and just hit you with the sequence diagram right away: (Go ahead and click to get a…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-137","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=137"}],"version-history":[{"count":10,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/137\/revisions"}],"predecessor-version":[{"id":152,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=\/wp\/v2\/posts\/137\/revisions\/152"}],"wp:attachment":[{"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jdkendall.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}