Comparing enrolled and non-enrolled customers carefully

A spending gap between enrolled and non-enrolled customers describes the groups; it does not show what enrolment caused. People choose whether to join, and the reasons for that choice may also affect their purchases. A useful comparison starts with eligibility, a common starting point and a clear account of who could join.

Fix the starting point

For a descriptive comparison, choose a date when both groups were eligible and observed. Record membership status on that date and follow both for the same length of time.

If customers join later, show them separately or use a method that accounts for changing status. Do not classify someone as a member from the start merely because they enrolled later, or move their earlier orders into a post-enrolment result.

If the question is what happened after joining, give each joiner their enrolment date and choose non-joiners who were eligible and still under observation at comparable dates. Apply the same history and follow-up requirements to both. A checkout enrolment does not make the joining order evidence of a program-driven purchase.

Distinguish a customer who declined an offer from one who was never offered it. Note unmatched guest or store purchases and refunds processed after the order period. Report how many customers or orders are missing rather than silently restricting the analysis to easily linked buyers.

Show how the groups differed already

At the common starting point, compare prior orders and spend, time since last purchase, channel, location and customer tenure using the same look-back window. Separate first-time buyers from established buyers when their opportunities to return differ. Report the number of customers in each group and the records excluded for missing history or identity.

A member group that bought more before joining cannot have that earlier gap credited to the program. Equal follow-up time matters too: customers observed for longer have more opportunity to place orders.

Match the conclusion to the design

A basic member-versus-non-member table is descriptive. It can reveal who joins and where participation is weak. It cannot establish program lift.

If eligible customers can fairly be assigned an invitation or access before they decide whether to enrol, compare everyone by assignment. Include invitees who decline, and record any access or other offers reaching the comparison group. The result estimates the effect of offering the program under those conditions, rather than the effect of becoming a member.

Without assignment, matching recorded history and comparing changes before and after may make the comparison more informative. Check pre-offer trends and simultaneous promotions or stock changes. Matching cannot remove unmeasured differences. A difference-in-differences interpretation also relies on the groups having followed comparable trends without the program; similar observed pre-trends support scrutiny of that assumption but cannot prove it.

Label the finding accordingly: members spent more, matched groups changed differently, or assigned offer groups produced different outcomes. State uncertainty and data gaps beside the figure.