AI agent for customer support: 62% of tier 0/1 tickets closed without humans

Context

B2B services company with an 8-person support team handling 400-500 tickets per week. Most were repetitive (how to reset password, where's the manual for module X, how to download an invoice) that senior team members had to touch anyway, draining time from complex cases.

The operations lead put it bluntly: "we can't hire more, but we're breaching SLAs every month. We need 50-60% of volume to resolve itself."

The challenge

• Response quality non-negotiable. The agent could not make up answers: either it answered from the knowledge base, or it escalated.
• Integration with existing ticketing. No tool migration. The agent had to live where the tickets already were.
• B2B compliance. Customer data could not leave the client's Microsoft 365 tenant. Public ChatGPT was off the table from day one.

Approach

6-week custom build with a 1-week prior discovery:

1. Discovery (week 0). Analyzed 800 historical tickets to find the 12 most recurrent categories. Three groups emerged: information queries (40%), access issues (22%), billing questions (15%). The remaining 23% were complex cases where the agent should not intervene.
2. Build on Copilot Studio (weeks 1-3). Agent trained on internal knowledge base, product manuals and historical FAQ. Connected to the ticketing system via API. Hard rule: if confidence is low, the agent escalates automatically with a case summary.
3. Shadow pilot (week 4). For a week, the agent responded but a human reviewed every reply before sending. Allowed calibration of model confidence and detection of two cases where it gave outdated info.
4. Progressive go-live (weeks 5-6). First low-risk categories only (info queries). At day 10, access issues. By end of program, all validated categories.

Stack: Copilot Studio inside the client's M365 tenant (data never left), API integration with ticketing, logging layer to audit every agent response.

Results

• 62% of tickets closed without human intervention by week 8 (measured over 1,200 inbound).
• Average tier 0/1 response time: from ~24h to ~18 minutes in the categories covered.
• CSAT held (4.2/5 vs 4.3/5 pre-project). The feared satisfaction drop did not happen.
• Human team focused on tier 2/3. Complex cases went from 6h average wait to 2.5h.
• Internal SLAs in green for the first time in 4 months at project close.
• 0 privacy or data leak incidents: everything stayed inside the client's tenant.

Lesson applicable

What makes an agent like this work isn't the model —Copilot Studio was plenty—, it's the quality of the knowledge base. We spent almost a week curating the client's FAQ and manuals before training anything. Best ROI of the project.

Second lesson: the "when in doubt, escalate" rule is non-negotiable. Any agent that invents answers to avoid escalation will destroy trust in its first week. The key pilot metric wasn't "how many it solves", it was "how many it escalates correctly when it doesn't know".

Third, commercial: this case is replicable almost as-is in any company with high support volume and documented FAQ. One of the highest-ROI custom projects measured.

Confidentiality note

Client name omitted by NDA. Figures are real or conservative estimates based on client's internal measurements.