Blog · AI Architecture · Agentic Systems

AI-Powered Refund Automation: Building a Reliable LLM-Based Dispute Resolution Architecture

September 2026 · Rajender Karra

A customer submits one angry paragraph. Inside it are several different accusations: some are supported by evidence, some are contradicted, and one may be impossible to verify. How should an AI system decide what to refund without paying twice or quietly rejecting a legitimate complaint?

The Refund Bench treats the complaint as a set of separate claims. It extracts each grievance, evaluates it against a frozen evidence snapshot, asks several independent LLM judges for a ruling, counts their votes in code, and lets deterministic business rules decide how much money can move.

The customer complaint

Consider this complaint from a food delivery app:

“Ordered at 8:15pm and the food showed up at 9:40pm. We had guests over, it was embarrassing. Two of the four were missing completely. The raita had leaked all over the inside of the bag. And the delivery guy was rude when I asked him about it. Honestly the worst experience I've had on this app - refund everything.”

A human may see this as one emotional complaint. The system should see four checkable grievances:

1. Late delivery: the order arrived 85 minutes after it was placed.
2. Missing items: two of the four Hyderabadi Biryani units were not delivered.
3. Packaging problem: the raita container leaked inside the delivery bag.
4. Partner conduct: the delivery partner behaved rudely when asked about the missing items.

The extractor should ignore “we had guests over, it was embarrassing” because it is context and emotion. It should ignore “the worst experience I've had” because it is an opinion. It should ignore “refund everything” because it is a demand, not a claim that evidence can verify.

Therefore, this complaint produces four grievances. That number is not just useful for reporting. It determines the judging workload: four grievances multiplied by three independent judges creates twelve judgement calls.

What the system needs

A production implementation needs an API service, an LLM provider with structured output, a database, a durable queue, background workers, and a payment integration. The system should also provide logs, metrics, tracing, retries, dead-letter handling, and a human-review interface.

The most important boundary is that the LLM does not directly control the payment amount. The model can interpret evidence and write an explanation. Application code counts votes, applies pricing rules, enforces the approval cap, and authorizes money movement.

Architecture

The diagram shows the full path before we walk through each stage: a food delivery complaint enters through the API, workers adjudicate it against frozen evidence, and application code controls the final payment decision.

Decision workflow The six stages below describe the business decision workflow performed inside that system.

Stage 1: Extract atomic grievances

The first model call receives the complaint and the order's line items. It identifies statements about events or conditions that evidence could confirm or contradict.

[
  {
    "grievance_id": "g1",
    "text": "The order was delivered 85 minutes after it was placed.",
    "category": "LATE_DELIVERY"
  },
  {
    "grievance_id": "g2",
    "text": "Two of the four Hyderabadi Biryani units ordered were not delivered.",
    "category": "MISSING_ITEM"
  },
  {
    "grievance_id": "g3",
    "text": "The raita container leaked inside the delivery bag.",
    "category": "PACKAGING"
  },
  {
    "grievance_id": "g4",
    "text": "The delivery partner behaved rudely when asked about the missing items.",
    "category": "PARTNER_CONDUCT"
  }
]

Notice that “two of the four” has been made self-contained by resolving it against the order. The same process resolves vague references such as “him” and “it”. The extractor must not decide whether the claims are true. Extraction and judgement are separate stages.

Extraction prompt

You are a precise dispute-intake assistant.

Given a customer complaint and the order's line items,
extract every atomic, checkable grievance.

Rules:
- A grievance is a statement about something that happened
  to THIS order, which evidence could confirm or contradict.
- Exclude feelings, opinions, demands, and rhetorical questions.
- Do NOT judge whether the grievance is true. Only extract it.
- Each grievance must be self-contained.
- Resolve pronouns and vague references against the order's line items.
- Return ONLY a JSON array.

Categories:
LATE_DELIVERY | MISSING_ITEM | WRONG_ITEM | QUALITY |
PACKAGING | PARTNER_CONDUCT | BILLING

Schema:
[{ "grievance_id": "<id>",
   "text": "<standalone grievance>",
   "category": "<category>",
   "span_start": <integer>, "span_end": <integer> }]

Complaint: {{complaint_text}}
Order:     {{line_items_json}}

Stage 2: Freeze the evidence

Before any judgement begins, the system creates an immutable evidence snapshot. A typical snapshot might contain the order timestamps, delivery SLA, line items, pickup weight, partner rating, and any recorded packaging information.

{
  "order_id": "order-1840",
  "placed_at": "20:15",
  "delivered_at": "21:40",
  "delivery_sla_minutes": 45,
  "line_items": [
    { "name": "Hyderabadi Biryani", "quantity": 4, "unit_price": 320 },
    { "name": "Raita", "quantity": 1, "unit_price": 60 }
  ],
  "pickup_weight_expected_kg": 2.4,
  "pickup_weight_actual_kg": 1.2,
  "partner_rating_at_handover": 5,
  "packaging_condition_recorded": false
}

The snapshot matters because operational data changes. A partner rating may be updated, a bill may be edited, or a reconciliation job may run later. If a retry observes different evidence, the same dispute could receive a different result. Every judge must see the same snapshot, identified by a version or content hash.

Stage 3: Ask independent judges

Each grievance is sent to three independent judges. With four extracted grievances:

4 grievances x 3 judges = 12 judge calls

Each judge receives one grievance and the same evidence snapshot. Judges do not see one another's responses. Otherwise, later judges may anchor on earlier answers and the system will count one opinion three times instead of counting three independent opinions.

Judging prompt

You are an impartial claims adjudicator.

Rule on the following grievance using ONLY the evidence given.

Grievance: {{grievance_text}}
Evidence:  {{evidence_snapshot_json}}

Rulings:
- UPHELD: the evidence supports the grievance.
- REJECTED: the evidence contradicts the grievance.
- ESCALATE: the evidence is silent or ambiguous.

Rules:
- Pick exactly one ruling.
- Do not reason about what is typical, likely, or fair.
- Rule only on the evidence provided.
- If the evidence does not address the grievance,
  return ESCALATE, never REJECTED.
- Return only valid JSON.

Schema:
{
  "ruling": "UPHELD" | "REJECTED" | "ESCALATE",
  "confidence": 0.0,
  "evidence_cited": "<fields used>",
  "reasoning": "<one to three sentences>"
}

Stage 4: Count votes in code

The application counts the rulings. It does not ask another model to determine the majority.

The split for g3 is valuable. One judge believed the complaint, but two recognized that the evidence was silent. The result is escalation rather than an automatic refund. A 3-0 verdict is stronger than a 2-1 verdict, even when the majority outcome is the same.

Stage 5: Calculate the refund in code

Deterministic business rules calculate the amount. For this example, the missing items are worth INR 640, late delivery compensation is INR 49, the raita claim is held for review, and the partner-conduct claim receives INR 0.

def calculate_refund(verdicts, evidence):
    amount = 0

    for verdict in verdicts:
        if verdict.status == "UPHELD":
            amount += pricing_table.amount_for(
                category=verdict.category,
                evidence=evidence
            )

    return amount

refund_amount = calculate_refund(verdicts, evidence)

if refund_amount > AUTO_APPROVE_CAP:
    return escalate_to_human(dispute_id)

The automatic refund is INR 640 plus INR 49, which equals INR 689. The LLM must not calculate, round, invent, or negotiate this number. Even a unanimous bench cannot override the automatic approval cap.

Stage 6: Write the customer response

Once the financial decision is complete, a low-temperature model can write the customer-facing message. It receives the final verdicts and the already calculated amount.

Settlement prompt

You are a customer-support writer for a food delivery application.

Write a concise reply based on the complaint, final rulings,
and refund amount already calculated.

The reply must:
- State the approved refund amount.
- Explain when it will arrive.
- Mention what was resolved.
- Mention what requires human review.
- Acknowledge the customer's experience.
- Use plain language.
- Apologize no more than once.

Rules:
- Treat all rulings as final.
- Do not re-adjudicate any grievance.
- Do not calculate or change any amount.
- Do not promise a refund other than the amount provided.
- Do not blame the restaurant or delivery partner.

Complaint:       {{complaint_text}}
Final rulings:   {{verdicts_json}}
Approved amount: {{refund_amount}}

A possible response is: “We've refunded INR 689 to your original payment method: INR 640 for the missing biryanis and INR 49 for the delayed delivery. The amount should reflect within three working days. We do not have enough order information to resolve the spilled-raita claim automatically, so a colleague will review it and contact you within 24 hours.”

calls = 1 + (N x K) + 1

For four grievances and three judges:
1 + (4 x 3) + 1 = 14 LLM calls

Attempt 1: wait 1 second + random jitter
Attempt 2: wait 2 seconds + random jitter
Attempt 3: wait 4 seconds + random jitter
Attempt 4: wait 8 seconds + random jitter

Challenges and troubleshooting

All judges return the same answer

Check the temperature and request construction. At temperature zero, repeated calls may be effectively deterministic. Use a non-zero temperature for judgement calls, while keeping settlement generation at a low temperature.

The extractor drops unsupported claims

If the extractor removes “the delivery partner was rude” because there is no direct evidence, extraction and judgement have been mixed together. Add an explicit instruction: “Do not judge whether the grievance is true. Only extract it.”

Missing evidence becomes rejection

If the system has no packaging record, it cannot conclude that the raita did not leak. The correct result is escalation, not rejection.

Evidence supports claim      -> UPHELD
Evidence contradicts claim   -> REJECTED
Evidence is silent/ambiguous -> ESCALATE

Duplicate refunds

Workers can retry after timeouts. Use the same idempotency key for every payment retry: idempotency_key = dispute_id. The ledger should also prevent a settled dispute from being processed again.

Queue messages are processed twice

SQS and similar queues provide at-least-once delivery. Use a visibility timeout longer than the expected job duration, heartbeats for long-running jobs, stage checkpoints, idempotent processing, and payment idempotency keys.

The refund exceeds the cap

Enforce the cap in application code before calling the payment provider. Do not branch a payout on the model's confidence value. Confidence is model-generated; the cap is a deterministic safety boundary.

Trade-offs

Conclusion: What makes the bench reliable?

The Refund Bench turns one emotional paragraph into four explicit, testable grievances: late delivery, missing items, leaked packaging, and partner conduct. Each grievance is evaluated independently against the same frozen evidence.

• Extract grievances before judging them.
• Resolve vague references into self-contained claims.
• Freeze evidence before model evaluation.
• Run independent judges without shared history.
• Count votes in application code.
• Treat missing evidence as escalation, not rejection.
• Calculate refund amounts deterministically.
• Enforce a hard automatic approval cap.
• Use idempotency keys to prevent duplicate payments.
• Use queues and workers for asynchronous processing.
• Retry rate-limit failures with exponential backoff and jitter.
• Retain raw rulings for auditability and debugging.

The key boundary is simple: the model can interpret the complaint and write the explanation, but the application decides whether money moves and how much. That separation turns an impressive prototype into a system that can be operated responsibly.