How Does an AI Agent Decide What to Ask Next?

Building a Decision State Machine for Reliable AI Agents

One of the biggest challenges with agents and AI workflows is explainability. Why did the LLM decide that this was the best response, best follow-up question or action, or even the best solution to a user’s question? This could be attributed to the system prompt, user prompt, model, etc., but why guess when you can actually structure it clearly?

That is what I’m implementing in “What Do I Do?” (My AI personal decision support system). My aim for this project is not to build yet another AI chatbot, rather, a reliable AI system. I gave a more in-depth explanation in my last article, feel free to go catch up on that to get more context on the project.

When a user asks What Do I Do?, how does the agent know what follow-up questions to ask and what recommendation to give after a series of questions? What is the basis for the final recommendation for each dilemma?

Introducing the Decision State Machine
The decision state machine is a structured object representing everything the agent currently understands about the decision and all the information learned from the user’s response. After each user response, it gets updated with the parameters, score, and confidence, based on what the user has stated explicitly or implicitly. This decision state is also shown to the user so they are aware of the factors that were considered to arrive at a final conclusion, with corresponding values for each of the factors.

For example,

{
  “goal”: “...”,
  “constraints”: [],
  “values”: [],
  “options”: [],
  “criteria”: [],
  “confidence”: 0.7
}

In terms of LLM cost and latency optimization, the decision state machine can also improve token cost, speed, and performance. Instead of lugging around bogus, verbose, text conversation history with each prompt, this decision state can be passed with the prompt to the LLM. This way, we ensure that the LLM is aware and focusing exactly on the right things, and not having to parse a large body of text each time where important information could get lost.

For outputs, the LLM should also return validated JSON rather than free-form text wherever possible, and then Pydantic validates the output before the application uses it.

This makes the system:

• easier to debug
• easier to evaluate
• less likely to contradict itself
• capable of resuming decisions later and
• able to explain how the recommendation was formed.

Here’s a more complete example with a real use-case:

{
 “goal”: “Should I leave my job for a startup?”,
  “domain”: “career”,
  “decision_deadline”: “2026-09-01”,
  “options”: [
    {
      “id”: “stay”,
      “name”: “Stay in current role”
    },
    {
      “id”: “startup”,
      “name”: “Join the startup”
    },
    {
      “id”: “delay”,
      “name”: “Delay the decision for six months”
    }
  ],
  “values”: [
    {
      “name”: “Autonomy”,
      “importance”: 0.9,
      “source”: “explicit”
    },
    {
      “name”: “Growth”,
      “importance”: 0.85,
      “source”: “inferred”
    },
    {
      “name”: “Financial security”,
      “importance”: 0.8,
      “source”: “explicit”
    }
  ],
  “constraints”: [
    {
      “name”: “Mortgage”,
      “type”: “hard”,
      “description”: “Must maintain sufficient monthly income”
    },
    {
      “name”: “Salary reduction”,
      “type”: “soft”,
      “description”: “Prefer no more than a 20% reduction”
    }
  ],
  “criteria”: [
    {
      “name”: “Career growth”,
      “weight”: 0.3
    },
    {
      “name”: “Financial security”,
      “weight”: 0.3
    },
    {
      “name”: “Autonomy”,
      “weight”: 0.25
    },
    {
      “name”: “Work-life balance”,
      “weight”: 0.15
    	}
  ],
  “uncertainties”: [
    “Startup runway is unclear”,
    “Role responsibilities may change”
  ],
“emotional_lean”: {
    “preferred_option”: “join_startup”,
    “confidence”: 0.82
  },
  “missing_information”: [
    “Current startup funding runway”,
    “Expected working hours”
  ],
  “readiness”: {
    “enough_information_to_recommend”: false,
    “confidence”: 0.63
  }
}

After each user response, the system updates this state with relevant information.

How This Ties Into The Project
I have structured “What Do I Do?” as a partially observable sequential decision problem in which the agent chooses what to ask, retrieve, analyze, or recommend next. In order to make these decisions, because the agent cannot see the full outlook, it has to extract observations from the current view and maintain a belief state, i.e. a probability distribution over possible underlying states of the decision to choose the right action at each step. Conceptually, it is modelled as a Partially Observable Markov Decision Process (POMDP) and implemented as an explicit stateful decision workflow. The system maintains a belief state: b(s) = P(s∣conversation history). In practice, the Decision State Machine acts as the system’s structured representation of this belief: what it currently believes about the user’s decision, how confident it is in those beliefs, and what remains uncertain. Ideally, the next question should reduce uncertainty in that belief state and that creates a powerful policy for the agent: Ask the question with the highest expected information value. For instance, if both options perform similarly except on risk tolerance, the agent should ask next about risk tolerance, not another generic question about salary.

This is why I model “What Do I Do?” as a POMDP because the environment is not entirely observable to the agent and the effects of some decisions will only be discovered months later. The agent also does not entirely know all the values and context about the user over the years, as well as the next option the user can choose to introduce or change in the decision-making process.

POMDP is a mathematical framework where the agent makes sequential decisions without a full outlook on the entire environment. It is defined mathematically as a 7-tuple: (S,A,T,R,Ω,O,γ). Here are what each of these components means and how it is mapped to the “What Do I Do?” system.

• S: A set of all possible true hidden states, s, in the environment. In “What Do I Do?”, this represents the user’s true underlying decision context, including their values, constraints, preferences, risk tolerance, and other factors that may not be fully observable to the agent.
• A: A set of all possible actions, a, the agent can take. The action is not the final life decision itself. It is the next conversational or analytical action performed by the agent.
• T: The transition function gives the probability of moving to the next state s’ given action a and current state s. Not every action necessarily changes the underlying state; a clarifying question may instead produce a new observation that updates the agent’s belief about that state.
• R: The reward function R(s, a), gives the scalar reward received after taking action a in state s. This is the hardest and most important part. Because the true state is not fully observable in What Do I Do?, given we cannot know the user’s thoughts entirely, the agent operates over beliefs and can evaluate actions using expected reward and information gain. Possible reward signals in “What Do I Do?” could include:
  • the user reports that the question was usefulimportant constraints are discoveredthe user’s values become clearerthe user understands the trade-offsthe conversation reaches a decision with fewer unnecessary questionsthe recommendation is consistent with the evidencethe user later reports satisfaction or low regret.
  
  For example, a conceptual reward function could be calculated as R = w1(decision quality) + w2(user clarity) + w3(value alignment) − w4(unnecessary questions) − w5(unsafe advice) − w6(overconfidence), where w1, w2,…,wn are weights.
  This also exposes one of the main difficulties with solving the decision-making problem. Decision quality is not always immediately observable. A user saying “that sounds good” does not necessarily mean the recommendation was good. And the true outcome may only become visible months later.
• Ω: A set of all possible observations, o, the agent can perceive. In this case, observations could include the user’s responses and other information available to the agent that provide evidence about underlying values, constraints, preferences, fears, and circumstances. E.g. “I’d be extremely stressed if I didn’t have at least 12 months of savings.”
• O: The observation function gives the probability of seeing observation o in the new state. A standard MDP assumes the full state is observable. But your system never fully knows the user’s true values (sometimes even the user might not know this honestly), whether fear is distorting the user’s answer, whether the user is withholding information, how the future outcomes will unfold, and even whether the user wants a recommendation or just emotional validation.
• γ: The discount factor (γ in [0, 1)]), which determines the present value of future reward.

Together, these components provide a formal framework for modeling the system as a sequential decision-making problem under uncertainty, where the agent must infer the user’s underlying decision state from incomplete information and choose actions that progressively move the conversation toward a well-supported recommendation.

Conclusion

In this article, we discuss the Decision State Machine powering “What Do I Do?” designed to maximize information gain in each clarifying question, so that the agent can arrive at a high-quality final recommendation in as few turns as possible. The base hypothesis for this is, can a learned policy select better clarifying questions and reach high-quality, value-aligned recommendations with fewer conversational turns than a rule-based or prompt-based policy? The eval results will let us know.

In the next article, I will share the Eval framework for “What Do I Do?”. I will outline the metrics defined, guardrails, and eval system created. If you’re interested in learning more about agentic systems, AI workflows, and AI evaluation, using this sequential decision-making project as a live use-case, consider subscribing so you can get notified once the next article gets published.

Thank you for reading.

Aniekan