The data
10,042 real customer support chats
The conversations come from ABCD, the Action-Based Conversations Dataset, published by ASAPP Research in 2021. Trained agents chatted with customers of a fictional online clothing store, following the store's written policies.
What each chat contains
- The conversation, turn by turn, between customer and agent.
- Every button the agent clicked, such as
verify-identityoroffer-refund. - The real reason for the contact, one of 55 intents such as a refund or a password reset, with the policy steps the agent should follow.
Because the buttons and the policy are both recorded, the right answer for policy questions can be computed from the data. The LLM's scores are checked against it.
What this project added
The dataset has no dates or agents, so these were simulated:
- · 30 days of daily batches
- · 40 agents in 4 teams, and two chat channels
- · Customer survey ratings (CSAT)
- · The contact reason the agent logged, wrong in about 8% of chats
- · Three deliberate incidents, to test the monitoring
Agent: Hi, how can I help you
Customer: I wanted to know the state of my refund
Customer: Because I want to any another refund to my existing one
Agent: OK can I get your name
Customer: [NAME_1]
Agent: And also you username, email and order id
pull-up-accountAccount has been pulled up for [NAME_1].Customer: It's $100 refund
Customer: + jeans I bought for $69 that I want a refund on
Agent: Perfect, I can add that for you. I just need your username, email and order id
Customer: [USERNAME_1]
Customer: [EMAIL_1]
Customer: [ORDER_ID_1]
validate-purchasePurchase validation in progress ...Agent: Great I see your original refund in progress, just a moment and I will add the second item
Customer: Ok thanks
record-reasonA reason of has been recorded.record-reasonA reason of jeans has been recorded.offer-refundA refund has been made for the amount of $169.Customer: Thanks for doing that mate
Agent: Ok, it's done your new refund total is 169
Agent: Anything else I can help you with today?
Customer: No
Agent: Have a nice day
What you're looking at
One chat as recorded: customer left, agent right, button clicks in between.
- Real reason for contact
- Refund update
- Policy steps for it
pull-up-accountvalidate-purchaserecord-reasonoffer-refund- Size
- 24 turns · 5 button clicks