With the AI movement in full swing at Phreesia, I was given the opportunity to find a way to utilize AI's capabilities to address one of our main challenges - recruiting clinical staff for research.

With the AI movement in full swing at Phreesia, I was given the opportunity to find a way to utilize AI's capabilities to address one of our main challenges - recruiting clinical staff for research.

I led this project between August 2025 and September 2025, and received support from the UX team, Product Managers, and other product roles from different domains.

I led this project between August 2025 and September 2025, and received support from the UX team, Product Managers, and other product roles from different domains.

To comply with Phreesia privacy agreement, some details have been removed, anonymized, or altered. All work shown is my own and may not necessarily reflect the views of Phreesia.

To comply with Phreesia privacy agreement, some details have been removed, anonymized, or altered. All work shown is my own and may not necessarily reflect the views of Phreesia.

Roles: Planning, Research, Design, Testing

Roles: Planning, Research, Design, Testing

Tools: UserZoom, ChatGPT Agent

Tools: UserZoom, ChatGPT Agent

STEP 1 - ACCURACY AND RELIABILITY

STEP 1 - ACCURACY AND RELIABILITY

I wanted to first test the accuracy and reliability of general ChatGPT answers. I ran a series of tests using past user research studies comparing the outputs provided by a general versus data-backed personas.

I wanted to first test the accuracy and reliability of general ChatGPT answers. I ran a series of tests using past user research studies comparing the outputs provided by a general versus data-backed personas.

1. Created general (no data) synthetic persona (#SyntheticFrontDeskStaff)
2. Created synthetic persona using past research transcripts (saved a #RealFrontDeskStaff)
3. Ran comparison tests between the two personas using the same scripts from the past research
4. Compared similarities and differences in the outcomes from both personas

1. Created general (no data) synthetic persona (#SyntheticFrontDeskStaff)
2. Created synthetic persona using past research transcripts (saved a #RealFrontDeskStaff)
3. Ran comparison tests between the two personas using the same scripts from the past research
4. Compared similarities and differences in the outcomes from both personas

The test results met the confidence level we needed to gain upper management stakeholder buy-in and move forward with agent configuration.

The test results met the confidence level we needed to gain upper management stakeholder buy-in and move forward with agent configuration.

STEP 2 - CONFIGURING THE AGENT

STEP 2 - CONFIGURING THE AGENT

Configuring an agent required scripting a well-defined behaviors, knowledge, tones, as well as an understanding of the users’ common challenges and tools used. This meant having an in-depth understanding of a realistic front desk staff and their roles and responsibilities.

Configuring an agent required scripting a well-defined behaviors, knowledge, tones, as well as an understanding of the users’ common challenges and tools used. This meant having an in-depth understanding of a realistic front desk staff and their roles and responsibilities.

STEP 2.1 - FEEDING THE BOT

STEP 2.1 - FEEDING THE BOT

I worked with other UX designers, Product Managers, and Client Success Coordinators to collect any available documentation related to the front desk staff.

I worked with other UX designers, Product Managers, and Client Success Coordinators to collect any available documentation related to the front desk staff.

• 10 user interview transcripts
• 2 journey maps
• 2 personas
  

• 10 user interview transcripts
• 2 journey maps
• 2 personas

All of the above data points were fed into ChatGPT. Then I prompted it to review, synthesize, and translate that information into our script requirements.

All of the above data points were fed into ChatGPT. Then I prompted it to review, synthesize, and translate that information into our script requirements.

In addition to the basic script requirements, I prompted ChatGPT to define 5 different personas using all of the data it was given. Why did we create individual personas? I found that in creating an agent with several detailed personas, it would enable the agent to have more dynamic and flexible responses.

In addition to the basic script requirements, I prompted ChatGPT to define 5 different personas using all of the data it was given. Why did we create individual personas? I found that in creating an agent with several detailed personas, it would enable the agent to have more dynamic and flexible responses.

I also manually updated each persona to include individual details such as primary responsibilites, tools used, key tasks, pain points, and motivations. The details were vetted by members of the UX team, Client Success team, and Product Manager team.

I also manually updated each persona to include individual details such as primary responsibilites, tools used, key tasks, pain points, and motivations. The details were vetted by members of the UX team, Client Success team, and Product Manager team.

Learning Moment:
There are different advantages when using "combined" versus "composite" personas.
- Combining all personas into one persona meant creating a single unified structure which maintains simplicity; which is better for a simple conversational agents.
- Composite personas, keeping each person separate and unique, creates a layered or linked persona system that can reference each other based on the user's prompt(s). This method is more effective in creating a complex agent that can provide more specific and detailed responses.

Learning Moment:
There are different advantages when using "combined" versus "composite" personas.
- Combining all personas into one persona meant creating a single unified structure which maintains simplicity; which is better for a simple conversational agents.
- Composite personas, keeping each person separate and unique, creates a layered or linked persona system that can reference each other based on the user's prompt(s). This method is more effective in creating a complex agent that can provide more specific and detailed responses.

STEP 3 - FINAL CONFIGURATION

STEP 3 - FINAL CONFIGURATION

The last step required prompting ChatGPT to use all of the given information to create a specification document that laid out all the necessary requirements.

The last step required prompting ChatGPT to use all of the given information to create a specification document that laid out all the necessary requirements.

"Using all of the given documentation, research data, and the five individual personas , create a specification document that defines behavior, knowledge, tone, common challenges, and common tools used by a front desk staff. Use the specification document to create an agent that mimics a front desk staff in a healthcare clinic or hospital that uses Phreesia tools in their day-to-day duties. The agent will act as a simulation agent and source of information for UX and product research scenarios and questions that enables testing of new UX and UI workflows, gathering feedback, and/or simulating real-world use cases.

"Using all of the given documentation, research data, and the five individual personas , create a specification document that defines behavior, knowledge, tone, common challenges, and common tools used by a front desk staff. Use the specification document to create an agent that mimics a front desk staff in a healthcare clinic or hospital that uses Phreesia tools in their day-to-day duties. The agent will act as a simulation agent and source of information for UX and product research scenarios and questions that enables testing of new UX and UI workflows, gathering feedback, and/or simulating real-world use cases.
 

Meet Francine - your fully functioning synthetic user that simulates a real front desk staff using Phreesia tools. She's your go-to person for anything involving front desk interactions and responsibilities, and can walk you through real-world clinic workflows, test how things behave in the Phreesia ecosystem, and flag anything that might trip up staff or patients.

Meet Francine - your fully functioning synthetic user that simulates a real front desk staff using Phreesia tools. She's your go-to person for anything involving front desk interactions and responsibilities, and can walk you through real-world clinic workflows, test how things behave in the Phreesia ecosystem, and flag anything that might trip up staff or patients.
 

To test the agent's accuracy and capabilities, I shared it with 3 Designers, 5 Product Managers, 2 Marketing Managers, and 1 Executives. The goal was to allow each user to test the agent however they see fit for any projects that a realistic front desk staff would handle, and identify any issues or gaps. Below are some of the feedback.

To test the agent's accuracy and capabilities, I shared it with 3 Designers, 5 Product Managers, 2 Marketing Managers, and 1 Executives. The goal was to allow each user to test the agent however they see fit for any projects that a realistic front desk staff would handle, and identify any issues or gaps. Below are some of the feedback.

"Recommendations were insightful and provided enough information to use for design iteration." - UX (Mockup Feedback)

"I wasn't sure about some of answers it provided but from the surface level this is really good. It knew about patient billing pages and talked about different tabs." - UX (Workflow Feedback)

"It got most of the information right when compared with what we know about the product, but some parts of the workflow were off" - UX (Scenario & Mockup Feedback)

"It didn't really understand Appointment Readiness, it was just a little bit off. I also find it tends to agree with me or think things are a great idea. Kind of a "yes man" mentality." - PM (Early Stage Ideation Feedback)

"I conducted a follow-up test and there were some small nuances within the family billing answers." - PM (Workflow Exploration)

"I was really impressed with its responses on a user problem I'm working on. I found this agent to perform better than the FigmaMake responses!" - PM (Feature Implementation)

"Recommendations were insightful and provided enough information to use for design iteration." - UX (Mockup Feedback)

"I wasn't sure about some of answers it provided but from the surface level this is really good. It knew about patient billing pages and talked about different tabs." - UX (Workflow Feedback)

"It got most of the information right when compared with what we know about the product, but some parts of the workflow were off" - UX (Scenario & Mockup Feedback)

"It didn't really understand Appointment Readiness, it was just a little bit off. I also find it tends to agree with me or think things are a great idea. Kind of a "yes man" mentality." - PM (Early Stage Ideation Feedback)

"I conducted a follow-up test and there were some small nuances within the family billing answers." - PM (Workflow Exploration)

"I was really impressed with its responses on a user problem I'm working on. I found this agent to perform better than the FigmaMake responses!" - PM (Feature Implementation)

Based on the feedback received, I identified two crucial issues:

Based on the feedback received, I identified two crucial issues:
1. "Yes Man" mentality
2. Cross-product confusion

I found the "Yes Man" problem to be a behavior issue that was persistent in ChatGPT for users both in and outside of Phreesia. To resolve this, I provided a detailed prompt (see below) and tested it with other users and found it to be effective.

1. "Yes Man" mentality

2. Cross-product confusion

"Focus on substance over praise. Skip unnecessary compliments or praise that lacks depth. Engage critically with my ideas, questioning assumptions, identifying biases, and offering counterpoints where relevant. Don't shy away from disagreement when it's warranted, and ensure that any agreement is grounded in reason and evidence"
 

I found the "Yes Man" problem to be a behavior issue that was persistent in ChatGPT for users both in and outside of Phreesia. To resolve this, I provided a detailed prompt (see below) and tested it with other users and found it to be effective.

As for the cross-product confusion, the main culprit was the overlap between our Self-Scheduling and Appointment Readiness tools. To resolve this, I simply prompted the agent to completely omit any and all Appointment Readiness mentions unless explicitly instructed. Post-implementation testing proved effective and no further issues were found with cross-product confusion.

"Focus on substance over praise. Skip unnecessary compliments or praise that lacks depth. Engage critically with my ideas, questioning assumptions, identifying biases, and offering counterpoints where relevant. Don't shy away from disagreement when it's warranted, and ensure that any agreement is grounded in reason and evidence"

"When asked about anything relate to self-scheduling, you must not mention Appointment Accelerator (AA) unless explicitly asked. Appointment Accelerator is only an automated backfill process, not a patient-facing scheduling tool, and should not be includeed in any self-scheduling explanations unless directly requested."