Blog 

Story of my Real World Use Case that this Gen AI Capstone Project is solving!

"Where should I eat today?”

My real world problem is a small one, yet one I have to make a decision on every day that impacts my time, money, health and overall well-being. So it is actually a little bigger daily problem than meets the eye…

I work in downtown Palo Alto, California, for over a decade and enjoy eating lunch daily at many of my favorite restaurants to recharge and refresh. And importantly when it is sunny, I love sitting outdoors to eat and relax and get my vitamin D. I’ve realized that every day I ask the same question…”Where should I eat today"? And there are so many wonderful restaurants in Palo Alto to choose from…

Breaking down the problem and solution:

We only have 2 weeks to work on this Capstone Project, so I have to decide what is the MVP I can create within two weeks while working full time, an individual project without a team AND my first Python Project!

Goal: I want to learn as much as I can on how to build Gen AI Agents, yet deliver a working Agent by deadline!

What do I want the AI Agent to do?

  • I would like a personal restaurant recommender Agent to help me decide quickly where to eat based on my dining preferences.

  • I have a list of my favorite restaurants I like to go to, but don't always remember which ones when I’m running out the door! So I end up going to the same 2 or 3 restaurants next to my work.

  • I would like Eat.Where.Now. to suggest one of “my favorite restaurants” and not just give me a recommendation based on general popular restaurant search results such as from Yelp or Google search.

  • It will have to know my preferences, which requires I actually break down how I make that decision every day to choose a restaurant such as type of food, how far to walk, maximum I want to pay and if I want to sit outside that day!

  • Since my lunch schedule varies, I don't always remember which restaurants are open late for lunch so don’t want to waste time looking it up or walking there to find out it is already closed.

    (Yes, this has happened more than once and very frustrating to waste time to walk there only to find out it is already closed, then have to find another restaurant that I like and not sure if that one is open or not!)

  • I would like the Agent to check live weather conditions before it recommends me a restaurant as I like to sit outdoors when it is sunny. (I don't always see weather conditions if I'm in the back office without windows and don’t have time to check the weather when our lunch schedule is so tight.

  • I want Eat.Where.Now to already know current weather conditions when it makes the recommendations! How sweet!!!

  • I only want maximum 3 recommendations at a time and not a list dump so I have to curate it in the moment! I just want to simply see which 3 it recommended, then make a quick decision to choose one.

  • I want to Chat with the Agent because I might modify my preferences in the moment, such as when I have more time to walk further than 5 minutes for my one hour lunch. Or if I’m in the mood for something more expensive than my daily price range.

  • I’m curious why Eat.Where.Now. gives me the recommendations that it did! I want to know how it thinks….

Why the name Eat.Where.Now?

  • I wanted a simple name for a URL domain and the obvious ones were already taken such as WhereToEat

  • So I broke down what problem my Agent is solving and it was clear to me that it has to include the activity, “Eat”

  • It has to include the solution, “Where”

  • And it has to address when I need the solution, “Now”…because I’m always deciding while running out the door to eat!

  • So Eat.Where.Now. it is!

The Gen AI Technologies I used:

1. Agent

2. Function Calling

3. Grounding

4. Structured output/JSON mode/controlled generation

5. System_Instruction (System Prompting)

What is considered a "Good" recommendation?

I would like Eat.Where.Now. Agent to help me decide where to eat each day based on these preferences:

  1. I like light meals

  2. I prefer to pay less than $20-30 per meal unless I tell the Agent I don't mind paying more depending on how I feel, but not more than $50

  3. I usually like to walk to a restaurant that is about 5 to 10 minute walk each way as I have an hour for lunch. If I have more time, then I want the flexibility to ask the Agent to help recommend restaurants further away.

  4. I want to Agent to only recommend the restaurants I like for this MVP Capstone Project.

  5. For this MVP, I want the Agent to first check real time weather conditions then only recommend me the restaurants with outdoor seating if it is sunny.

Are Gen AI capabilities well-suited to solve my Real World Use Case? YES!!!

Gen AI capabilities are very well-suited to solve my “where to eat?” use case, and here's why:

  • I can quickly create Eat.Where.Now. even though this is my very first Python Programming project with very minimum Python experience and certainly no Gen AI technical experience and been decades since I’d done coding!

  • But I do have a strong understanding of AI, symbolic AI, Gen AI and a good understanding of ML, Deep Learning and LLM to modify the Codelabs as a jump start to work for my use case!

  • Gen AI's language capabilities make it so much more like a conversation with a person who has the knowledge and flexibility to make recommendations and changes on the fly.

  • I can chat with Eat.Where.Now. and make modifications for that day in a conversation like much like with a real human agent! It is flexible to work with!

  • I also used Gen AI Chatbots (Gemini and DeepSeek) for coding guidance on how to code into my project, Grounding that returns JSON (instead of resource links from Codelabs) to get current weather conditions so the Agent can use that information to give me recommendations based on my preference to sit outdoors when it is sunny.

How does Eat.Where.Now Agent work?

  • Eat.Where.Now. will first create a local database and populate it with Synthetic Data of my favorite curated restaurants in downtown Palo Alto with my preferences set for each restaurant.

  • It will use system_instructions to constrain the response to only use my curated list in the local database so it is not using general model knowledge to recommend me restaurants I’m not interested in.

  • Eat.Where.Now. uses Chat interaction with me to give me the best restaurant recommendations

  • Importantly, the local database is a curated database with information indicating if the restaurant has outdoor seating that I like

  • This is by design as not all restaurants with outdoor seating have a nice outdoor eating area that I like. Therefore, a column in the database indicates a '1' for TRUE or '0' for FALSE so Eat.Where.Now. can only recommend outdoor seating restaurants from the curated database.

  • Also in the system_instruction, it is told to recommend more than one but no more than 3 restaurant recommendations each time.

  • The Agent can also explain why it chose those restaurants to recommend to me

What are limitations of Gen AI Technology or Approach?

1. System Instructions (System Prompting) play an important role!

I'd found that System_Instructions strongly impacts Model and Agent behavior, Chat flow and the quality of the outcome/responses. It plays an important role in Model effectiveness or what is considered a "GOOD" response.

  • With a basic system_instructions, the challenge is that the Model sometimes don't "remember" it knows the table schema even though it is told to use the table_schema. When it "forgets", it says it does not know that information, then asks the user to provide that schema, even though generally the user would not have that system-level information.

  • To solve that problem, I "tweeked" the system_instructions so the model will have clearer and more specific instructions what to do when it has an "undeterministic" behavior such as it forgets what it should already know.

  • I'd found that the more specific the instructions it does help the output/response from the model to overcome it's non-determonistic behavior, such as to then remember to look up restaurant information in the table scheme that it already knows.

  • However, if I get too rigid in my system instructions, such as when I specifically ask the model to "not ask the user for table schema" when it forgots on multi-turn chat, then it seems to cut down on multi-turn chat which then prevents it from successfully investigating and displaying the Chat-History (did it have an attitude?!). So tweeking the System_Instructions JUST ENOUGH seems to be a sweet spot in deterministic/non-deterministic behavior to accomplish the task.

2. Short memory!

Another problem is that it forgets which restaurants it had already recommended to the user on multi-turn chat, which feels like a strange chat conversation when it just recommended the restaurants but upon further queries about the recommendations, that it suddenly didn't remember what it just recommended! So tweeking the System_Instructions "to remember the recommendations" worked because I told it that it was a "smart agent" seemed to help the forgetfulness, but just like #1 above, it also caused another problem that it limits the Chat History so even though it no longer forgets its recommendation, but then the investigation fails to display multi-turn chat! Somehow it does not seem to have enough chat history to answer, then investigation crashes on "Function Response". When removing that specific system instruction then the multi-turn chat investigation works again! So the sweet spot seems to be using System_Instruction to not restrict it too much, yet provide enough guidance.

3. Model erroneously uses it's own pre-trained knowledge

Sometimes the model will use it's pre-trained knowledge to recommend restaurants that are not in the curated local database. When it does that, the restaurants it recommended were highly popular restaurants with a very high price of 100USD per meal, which is not part of the criteria, nor in the database. When I asked how it got this answer, it does not answer me :-)

4. The specific Gen AI Model used makes a difference and beware it can Cost!

Only the later Gen AI models can do live check of current weather conditions! But using the Gemini Advanced 2.5 Pro (Experimental) model had a limitation to the free-tier!

After some runs, it prevented me from doing more runs due to exceeding the free-tier. I had to enter a credit card number to continue, which I did NOT want to do for a public Capstone Project. Luckily I solved the issue by using an earlier Gen AI model (Gemini 1.5-pro) that can luckily still do the live query for current weather conditions which I needed to solve my “Where to eat” use case.