monival.org

01 · Evaluation designs and causal inference

Qualitative comparative analysis (QCA)

Qualitative comparative analysis identifies the combinations of conditions under which an outcome occurs across a medium number of cases — typically too many for case studies, too few for regression. Grounded in set theory rather than correlation, it tests which conditions are necessary or sufficient for the outcome, accepts that different recipes can produce the same result in different cases, and makes its analytic choices — especially calibration — explicit and checkable.

Last updated · Reviewed against 3 cited sources

The QCA niche

Between the single deep case and the large sample lies the territory most portfolio evaluators actually occupy: fifteen country programmes, thirty grants, forty facilities — each known in some depth, together too few for regression to say anything stable. Qualitative comparative analysis, developed by Charles Ragin in comparative politics and adopted in evaluation over the past two decades, is the systematic method for exactly this medium-N ground [1, 2].

QCA differs from statistical analysis in what it looks for, not just in sample size. Correlational methods estimate the average marginal effect of each variable, holding others constant. QCA is set-theoretic: it asks which combinations of conditions the outcome-achieving cases share, and formalises two features of causation that averaging obscures [1, 2]:

  • Conjunctural causation. Conditions work in packages: secure funding produces sustained outcomes when combined with a strong local partner, and not otherwise. No condition needs an effect “on average” for a combination containing it to be decisive.
  • Equifinality. Different cases can reach the same outcome by different routes — one programme sustains through government adoption, another through community ownership. QCA returns multiple recipes where multiple recipes exist, rather than blending them into a misleading mean.

Its causal vocabulary follows: a condition is necessary if the outcome does not occur without it (all outcome cases are inside the condition’s set), and a condition or combination is sufficient if, wherever it is present, the outcome follows [1]. These are precisely the claims programme managers make informally — “nothing sustains without a local partner” — and QCA is the machinery for testing them across a case set.

Crisp sets, fuzzy sets, and calibration

In crisp-set QCA every case is simply in or out of each set: the programme had a local partner, or it did not; the outcome was sustained, or it was not. Fuzzy-set QCA admits degrees — membership scores between 0 and 1 — so “partial sustainability” or “moderately secure funding” can be represented without forcing false dichotomies [1, 2].

Either way, calibration is where the analysis is won or lost. Calibration is the assignment of each case’s membership in each set, and it must be criterion-based, not mechanical: what, substantively, counts as “funding secured”? Where is the threshold between mostly-in and mostly-out, and why? Good practice anchors these decisions in theory and case knowledge, documents them in a calibration table, and tests whether conclusions survive reasonable alternative thresholds. Calibration done by quietly standardising whatever numbers were available is the most common way a QCA becomes uninterpretable — the algorithm will still run, and the recipes it returns will mean nothing [1].

Truth tables, consistency and coverage

The analysis pivots on the truth table: every logically possible configuration of the conditions gets a row, each case is assigned to the row matching its profile, and each row is examined for whether its cases achieved the outcome [1].

A stylised QCA data matrix: eight cases, three conditions, one outcome

A grid with eight cases labelled A to H as rows and four columns: the conditions funding secured, local partner and staff continuity, then the outcome sustained. Filled squares mark presence. Cases A, B and E are highlighted: all three have funding secured and a local partner, differ on staff continuity, and all show the sustained outcome. A callout beneath reads: candidate causal recipe — FUNDING and LOCAL PARTNER lead to SUSTAINED OUTCOME.

FundingsecuredLocalpartnerStaffcontinuitySustainedoutcomeABCDEFGHcondition / outcome presentabsentcandidate recipe: FUNDING × LOCAL PARTNER → SUSTAINED OUTCOME
Figure 1. Cases arrayed against three conditions and the outcome (filled square = present). The highlighted cases share funding and a local partner — with and without staff continuity — and all achieved the outcome, suggesting the recipe below. Real analyses then test that recipe's consistency and coverage across the full table.Stylised illustration; truth-table procedure per Befani (2016).

Two parameters, defined in Befani’s guide, discipline the claims the table can support [1]:

  • Consistency measures how reliably a configuration is associated with the outcome — the degree to which the cases displaying the combination also display the result. A configuration contradicted by several of its own cases has low consistency, and asserting it as sufficient would misdescribe the evidence.
  • Coverage measures how much of the outcome a configuration accounts for — of all the cases achieving the result, what share travelled this pathway. A perfectly consistent recipe covering one case in twenty is real but marginal; report both numbers, always.

The logical minimisation that follows — reducing the outcome-linked rows to their simplest expression — is handled by standard software. The evaluator’s contribution is everything around it: which conditions enter, how sets are calibrated, and how contradictory and empty rows (configurations no case exhibits) are handled and disclosed [1, 2].

Quality assurance for commissioners

QCA’s compact notation can intimidate commissioners into waving results through. Befani’s EBA volume — written for exactly this audience — insists the method is unusually auditable, because every step is documentable: the case set and its justification, the conditions and the theory behind them, the calibration table with thresholds and rationales, the raw truth table, the consistency and coverage of every reported recipe, and robustness checks under alternative calibrations [1]. A QCA report that shows only its final Boolean expressions should be sent back for the working. Transparency and replicability are the method’s distinctive virtues — an evaluator who cannot produce the intermediate tables has not done a QCA, whatever the report says [1, 2].

Limits, and the division of labour

QCA is theory-dependent at its foundations: conditions are chosen, not discovered. The method tests combinations of the conditions the analyst put in; omit the condition doing the real causal work and the recipes returned will rearrange the leftovers with unearned confidence. Garbage in, recipes out [1]. Case numbers bind too — with many conditions and few cases, most truth-table rows are empty and the analysis outruns its evidence; keep the condition set small and theoretically motivated [1, 2].

The recipes themselves are patterns of association across cases, disciplined by set logic. Their causal interpretation comes from the surrounding evaluative work: programme theory to select and interpret the conditions [3], and within-case analysis to verify the mechanism in the cases each recipe claims — which is why QCA pairs so naturally with process tracing (checking the pathway inside representative cases) and slots into a contribution analysis as its cross-case evidence engine. In a portfolio evaluation the division of labour is clean: QCA says which combinations travelled with success across the portfolio; process tracing says whether the claimed mechanism actually ran in the cases that matter; the contribution story assembles both.

Checklist before you commit to QCA

  • The case set is medium-N, and each case can be known well enough to calibrate honestly.
  • Conditions are few, theory-driven, and argued in writing before analysis.
  • Calibration criteria are substantive, documented case by case, and stress-tested against alternative thresholds.
  • The truth table, consistency and coverage figures, and treatment of contradictory and empty rows are all reported.
  • Reported recipes are backed by within-case verification in representative cases.
  • Conclusions respect equifinality: multiple pathways are reported as such, not averaged into one.

Sources

  1. Pathways to Change: Evaluating Development Interventions with Qualitative Comparative Analysis (QCA) — Expert Group for Aid Studies (EBA), Stockholm, 2016.Befani — the standard treatment of QCA for development evaluation, including step-by-step guidance and quality-assurance checks for commissioners.
  2. Qualitative Comparative Analysis (approach page) — BetterEvaluation (Global Evaluation Initiative), updated continuously.Practitioner overview, origins with Ragin, and applications in evaluation.
  3. Theory-Based Impact Evaluation: Principles and Practice. 3ie Working Paper 3 — International Initiative for Impact Evaluation (3ie), 2009.White — the theory-based frame within which cross-case configurational evidence carries causal weight.