13 Conclusion
Throughout this book, the same reference QCA has been examined through several distinct questions. How far can a qualitative anchor or truth-table cutoff move before a selected solution formula changes? What happens when several analytical choices vary together? Which case deletions alter the truth table or minimized solution? Does the same sufficient relation display comparable consistency and coverage across substantively meaningful groups? How do different selections of conditions reorganize the configurational space and the solutions obtained from it?
Each question extends the analysis to a particular part of the QCA workflow and evaluates a particular QCA mechanism or set relation. Their findings must therefore remain attached to the analytical uncertainty that motivated the comparison.
A robustness appraisal should enrich the research and leave both general readers and peers with a more precise understanding of the analysis. Although tables and plots can be relegated to the appendices, their heuristic value should be reflected throughout the work.
13.1 Robustness Is Not the Absence of Change
If we want to expose whether the same disjunction of prime implicants is obtained under a stated alternative analysis, exact preservation of a solution formula provides a clear and demanding criterion. Yet, alone, it does not establish that solution consistency, PRI, and coverage remain unchanged. Conversely, a different Boolean expression may still produce complete-solution memberships that overlap closely with those of the reference formula. These forms of preservation answer different questions and should not be compressed into one undifferentiated judgment (Oana and Schneider 2024).
More importantly, nor does every formula change carry the same meaning. An attention flag raised by moving an inclusion cutoff follows a different analytical route from one produced by raising the frequency cutoff, deleting a case, recalibrating the retained cases, or replacing a condition.
The first may alter the sufficiency assignment of an empirical truth-table configuration; the second may turn an observed configuration into a logical remainder; and the remaining operations may reconstruct the empirical distribution or the entire configurational space. The formula tells us that the minimization ended elsewhere. Interpreting why it ended elsewhere requires tracing the change through the relevant QCA objects, not merely stating that “something changed after X steps, so watch out!”
This is also why an unchanged formula should not be treated as a design victory. It is evidence of preservation under the alternatives that were actually tested, using the stated solution type, model position, intermediate branch, treatment of logical remainders, and comparison criterion. It does not certify every decision made elsewhere in the research design.
13.2 Let the Research Design Choose the Appraisal
A useful robustness appraisal begins with uncertainty that genuinely belongs to the study. If the placement of a qualitative anchor is debatable, a calibration diagnostic can show how far that anchor can move along the tested path before the selected solution formula changes. If several calibration and truth-table decisions are jointly plausible, an alternative-set diagnostic can examine their combined consequences. If the population or the influence of particular cases is uncertain, leave-one-out or subsampling diagnostics can reconstruct the analysis under changed case compositions. Cluster and theory comparisons become relevant when the substantive argument concerns variation across meaningful groups or the consequences of competing condition sets.
Running every available diagnostic is neither required nor inherently more rigorous. The alternatives must remain theoretically and empirically defensible, and the diagnostic must bear on the claim being made (Skaaning 2011; Thomann and Maggetti 2020). A case-oriented study with carefully justified scope conditions may learn little from indiscriminate case deletion. A wide calibration interval becomes uninformative if it extends into thresholds that no longer represent the concept. A sampled specification does not become substantively plausible merely because software can generate it.
The relationship between robustness appraisal and research design therefore runs in both directions. The research design identifies which analytical choices deserve examination; the diagnostic evidence then shows how much the QCA result depends on those choices. When a solution changes, the researcher returns to theory, measurement, the truth table, and case knowledge to determine whether that sensitivity modifies the substantive interpretation. When the selected solution formula and parameters of fit remain preserved, those same materials help establish whether the tested alternatives were substantively demanding enough to strengthen the original claim.
It’s almost as though QCA involves a “back-and-forth between theory and evidence”… Who would have thought?
13.3 What qcaERT Contributes
qcaERT contributes an integrated way to conduct and retain these comparisons. Its diagnostics reconstruct the affected stages of the QCA workflow, follow conservative, parsimonious, and intermediate solutions through a common family grammar, retain records of solution consistency, PRI, and coverage when the comparison requires them, and distinguish a changed formula from a comparison that could not be completed. That common grammar makes the separate diagnostics easier to combine without pretending that they ask the same question.
This contribution complements rather than replaces the robustness protocol implemented by SetMethods, which offers a set-theoretic interpretation of QCA robustness more closely aligned with its epistemological foundations. qcaERT concentrates on generating or organizing alternative analyses, tracing specified analytical changes through calibration, truth-table construction, minimization, and structured comparisons, and preserving the evidence produced along the way. Researchers may need either contribution or both, depending on where the unresolved part of their robustness appraisal lies.
Still, no package can decide which alternative calibration is conceptually defensible, which cases belong to the population, whether a logical remainder represents an acceptable counterfactual, or which condition set best expresses the theoretical argument. Those judgments belong to QCA as a research approach, where concept formation, case knowledge, scope conditions, and the movement between theory and evidence remain inseparable from the formal technique (Thomann and Maggetti 2020).
13.4 From Diagnostic Results to a Defensible Claim
By the end of a robustness appraisal, our readers should be able to identify the reference analysis, the analytical choices or structured comparisons examined, the criterion used to evaluate preservation, and the empirical scope of the conclusion. Formula preservation should remain distinct from changes in solution consistency, PRI, coverage, or complete-solution membership. Rates should retain the denominators of the comparisons that could actually be made, while failed and unavailable analyses should remain visible rather than being counted as either preservation or change. These elements make the analytical path reconstructable and keep the final claim within the evidence that produced it (Wagemann and Schneider 2015).
The final robustness argument do not need to portray the QCA result as immovable. It should show where the result changes, where it remains stable, which empirical or logical shifts produce those changes, and why the alternatives merit consideration. qcaERT can shoulder much of the repetitive reconstruction and record-keeping, but the interpretation remains ours. That division of labor is less glamorous—and, some would argue, more desirable—than inventing an alternative analytical technique that supposedly “solves” QCA’s dependence on researcher discretion.