vdem_subsample_unstratified_con <- subsample.test(
data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
sample_prop = 0.90,
reps = 20,
seed = 123,
calib = "fixed",
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "conservative",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)10 Subsample Diagnostics
The comparatively tame leave-one-out diagnostic presented in the preceding chapter shows whether the selected QCA result depends on any single observation. What it cannot do is assess dependence on several cases acting together: a solution formula may survive every individual deletion and still change when a particular collection of cases is removed simultaneously.
The admittedly punishing subsample.test() addresses that broader form of case-composition sensitivity by repeatedly retaining only part of the case universe and reconstructing the QCA analysis for each sampled composition.
Beginning from the same complete-data baseline, each replication draws the requested number of cases without replacement, rebuilds the truth table, repeats the requested minimizations, and compares the resulting solution formulas with their complete-data counterparts. The diagnostic then brings the replications together to show how often each baseline formula recurs, which prime implicants reappear, how much the prime-implicant sets overlap, and how the selected solutions’ parameters of fit vary across the sampled case compositions.
As before, with fixed calibration, removing several countries changes the empirical evidence represented in the truth table while leaving the retained countries’ set memberships untouched. The frequency, consistency of sufficiency, and PRI of truth-table configurations may change; a previously observed configuration may fall below n.cut and become a logical remainder; or another may cross incl.cut and receive a different outcome assignment. Logical minimization then proceeds from this changed truth table, so the selected solution formula may gain, lose, or simplify one or more prime implicants. Parsimonious and intermediate minimization may also receive a different collection of simplifying assumptions.
Unlike leave-one-out analysis, however, each result concerns the sampled case composition as a whole because repeated subsampling assesses whether the baseline result recurs across many joint perturbations of case composition.
This trial is a very specific diagnostic and, depending on the stablished fit tolerance, may be overkill for the average or even most QCA robustness workflows. A researcher must justify how many cases are retained, how many samples are drawn, and whether any observed group composition should be preserved.
So, if theoretically plausible boundary diagnostics and a leave-one-out analysis already answer the relevant robustness questions, dozens or hundreds of partial-sample tests may produce more output than insight. Subsampling earns its place when dependence on joint case composition is itself important to the research design, not because of a weird fetish.
The chapter continues with the 66-country V-Dem analysis, retaining the calibrated outcome, four conditions, truth-table cutoffs, and complete-data solutions established in the reference chapter. If at any point you find yourself wondering about what matters in a subsampling appraisal, check the Deciding What Matters in a Subsample Diagnostic section for some guidance.
10.1 Planning a Subsample Diagnostic
The baseline QCA inputs and solution-selection controls are the same ones used by loo.test(). The new research decisions concern the sampling design: how many countries each replication retains, how many case compositions are examined, and whether the allocation across an observed grouping should remain comparatively stable.
10.1.1 Deciding How Many Cases to Retain
Exactly one of sample_n or sample_prop must define the size of every subsample. Both default to NULL because the function cannot decide how severe the case-composition perturbation should be.
sample_ngives the number of retained cases directly. It must be an integer of at least two and smaller than the number of cases indata.sample_propgives a proportion strictly between zero and one. The function multiplies that proportion by the number of cases and rounds the result to obtain the realized sample size. The realized size must satisfy the same bounds assample_n.
This choice determines how much of the empirical basis of the truth table is removed in each replication. On the one hand, a sample close to the complete dataset tests whether the result survives comparatively modest joint deletions. On the other hand, a smaller sample imposes a more severe perturbation: more truth-table configurations may lose empirical representation, cross the frequency or inclusion cutoff, or cease to support the same minimization.
Because no subsample proportion is universally appropriate, the retained share should represent a degree of case removal meaningful for the study’s population and case-selection rationale.
10.1.2 Deciding How Many Case Compositions to Examine
reps gives the number of subsamples and defaults to 100. Sampling is without replacement within each replication, so a case cannot occupy two positions in the same sample. The same observation may, of course, appear in many different replications. More replications examine more case compositions of the chosen size and reduce the extent to which the observed formula frequencies, prime-implicant appearance rates, and parameter distributions depend on a very small collection of random draws.
seed defaults to NULL, which leaves the realized sampling plan dependent on R’s current random-number state. A reported diagnostic should supply an integer seed so that the same case compositions—and therefore the same numerical results—can be reconstructed, making the sampling plan reproducible.
10.1.3 Deciding Whether to Preserve Group Composition
The stratify argument determines whether every subsample preserves an allocation across observed groups. It defaults to "none", which samples from the complete case pool without constraining any group count. Two stratified alternatives are available:
"outcome"allocates the sample proportionally across the0and1values of a crisp outcome. It is unavailable for a fuzzy outcome because degrees of outcome membership do not define two observed sampling strata."user"allocates the sample proportionally across the groups supplied throughstrata. In this mode,stratamust contain one nonmissing group value for every case. The number drawn from each group remains fixed across replications, while the particular cases selected within each group continue to change.
An unstratified diagnostic allows observed group proportions to vary along with the cases, whereas a stratified diagnostic holds one selected aspect of composition comparatively steady and checks whether the QCA result persists as cases vary within that constraint. Stratification is thus a substantive part of the perturbation design, and neither approach is automatically preferable because they answer different questions.
The remaining analytical choices should reproduce the QCA specification whose case-composition sensitivity is being examined. The usual suspects are held constant across replications so that changes in the result are not produced by an unnoticed change in the comparison rules: truth-table cutoffs, solution selection, directional expectations, exclusions, and the parameter tolerance. Calibration handling, as before, might be fixed or recomputed.
10.2 What Each Replication Compares
For every solution type successfully obtained from both the complete data and a subsample, the diagnostic connects three levels of evidence.
At the formula level, a replication counts as an exact match only when it reproduces the complete disjunction of baseline sufficient terms for that solution type; retaining one baseline prime implicant is not enough if another disappears or a new one enters the formula.
At the prime-implicant level, subsample.test() records how often each baseline and nonbaseline prime implicant appears across successful replications and compares the set of prime implicants in each subsample formula with the corresponding baseline set using the Jaccard coefficient:
\[ J(A,B)=\frac{|A \cap B|}{|A \cup B|} \]
A Jaccard coefficient of 1 means that the two formulas contain exactly the same prime implicants, while 0 means that they share none. Intermediate values indicate partial overlap in the identities of the minimized terms, but do not measure overlap in the cases covered by two different prime implicants or in the fuzzy-set memberships represented by the complete solutions.
As in the leave-one-out diagnostic, the requested solution-level parameters of fit are compared whether or not the formula is preserved. When the formula is identical, differences in solution consistency, PRI, or coverage evaluate the same sufficient expression across different case compositions. When the formula changes, the values belong to different sufficient expressions produced by the same solution type. Formula recurrence and parameter movement must therefore remain separate results.
All three levels require explicit denominators. A replication may fail to produce one requested solution type even when another remains available, because the overall status is "ok" whenever at least one monitored solution type can still be compared with its baseline counterpart. The exact_solution and similarity summaries report n_successful separately for each solution type. Exact-match proportions and prime-implicant appearance rates are calculated from those solution-type-specific successful comparisons. The number of attempted replications and the number of comparisons available for each solution type must therefore be reported separately.
10.3 A First Subsample Diagnostic
Keeping things simple, we begin with a modest diagnostic so that the evidence from individual replications remains easy to inspect. Each replication retains 90% of the 66-country universe. Rounding produces a sample of 59 countries and a holdout of seven. The samples are unstratified, which allows any regional or other observed group count to vary along with the selected countries.
For now, calibration remains fixed so that the perturbation operates only through the 59-country case composition, the reconstructed truth table, and conservative minimization. Twenty replications are probably enough here, although far too few to treat the observed recurrence frequencies as settled. The diagnostic uses the same 0.005 tolerance adopted in the leave-one-out chapter and fixes the sampling plan with seed 123.
10.3.1 Reading the First Twenty Replications
vdem_subsample_unstratified_con
#> <subsample_test>
#> Outcome test: subsample robustness
#> Solution: conservative
#> Monitored solutions: conservative
#> which_M: 1
#> Replications: 20
#> Sample size: 59
#> Sample proportion: 0.8939394
#> Stratification: none
#> Fit measures: inclS, PRI, covS
#>
#> Result
#> rep n_sample n_holdout status exact_match_baseline term_jaccard_baseline solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 59 7 ok TRUE 1.0000000 <NA> CON 3 0.013292602
#> 2 59 7 ok TRUE 1.0000000 <NA> CON 3 0.016631413
#> 3 59 7 ok TRUE 1.0000000 <NA> CON 3 0.017414324
#> 4 59 7 ok TRUE 1.0000000 <NA> CON 1 0.005362976
#> 5 59 7 ok TRUE 1.0000000 <NA> CON 3 0.011403805
#> 6 59 7 ok TRUE 1.0000000 <NA> CON 3 0.027622280
#> 7 59 7 ok TRUE 1.0000000 <NA> CON 1 0.013499065
#> 8 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.088086564
#> 9 59 7 ok TRUE 1.0000000 <NA> CON 2 0.005596942
#> 10 59 7 ok TRUE 1.0000000 <NA> CON 3 0.023551104
#> 11 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.055963528
#> 12 59 7 ok TRUE 1.0000000 <NA> CON 3 0.018760648
#> 13 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.089967714
#> 14 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.062821763
#> 15 59 7 ok TRUE 1.0000000 <NA> CON 3 0.021279578
#> 16 59 7 ok TRUE 1.0000000 <NA> CON 1 0.015367530
#> 17 59 7 ok FALSE 0.6666667 CON:formula_changed CON 1 0.020955373
#> 18 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.069490677
#> 19 59 7 ok TRUE 1.0000000 <NA> CON 3 0.014164128
#> 20 59 7 ok FALSE 0.0000000 CON:formula_changed CON 3 0.065924042
#>
#> Summary
#> Successful runs: 20/20
#> Runs with any solution change: 7
#> Runs with any fit change: 20
#> Non-ok runs: 0
#> Mean combined term-set Jaccard vs baseline: 0.6833333
#>
#> Exact baseline matches by solution type
#> solution_type exact successful proportion
#> CON 13 20 0.65The printed header confirms the sampling and comparison design before presenting one record for each reconstructed 59-country QCA. Below it, n_sample and n_holdout identify the realized case split, while status shows whether at least one monitored solution type could be compared with its complete-data counterpart. This first application monitors only the conservative solution. All twenty replications have status = "ok", so the conservative minimization is available for every comparison that follows.
exact_match_baseline indicates whether the complete conservative formula is reproduced, while solution_change identifies the monitored solution type when it is not. Thirteen of the twenty replications reproduce the baseline conservative formula and seven produce another formula. The exact-match proportion is therefore 13/20 = 0.65, with all twenty successful conservative comparisons in the denominator.
In this single-solution application, term_jaccard_baseline compares the prime implicants in each conservative formula with the two prime implicants in the baseline conservative one. The mean coefficient is approximately 0.683, meaning that the prime-implicant sets retain that degree of overlap on average even though seven complete formulas differ from the baseline.
Looking at the parameter of fit fields, fit_changed_types identifies the solution types for which at least one requested solution-level parameter differs from its baseline value by, in this case, more than 0.005. n_fit_deltas counts how many distinct requested parameters cross that tolerance, and max_abs_fit_delta gives the largest absolute difference among the parameters classified as changed. All twenty replications cross the tolerance for solution consistency, solution PRI, solution coverage, or some combination of the three.
As stated, thirteen of those twenty replications nevertheless reproduce the conservative formula exactly. The Boolean expression can remain unchanged while its solution-level parameters move because the same sufficient expression is now evaluated across a different collection of countries. Formula recurrence and stability of solution consistency, PRI, and coverage are therefore empirically distinct results in this diagnostic.
10.3.2 Investigating Particular Replications
The overview above is normally all we need. But if we decide to conduct a forensic examination of a particular replication because it helps explain an interesting pattern—such as a recurring nonbaseline formula, unusually little overlap with the baseline prime implicants, a large difference in solution consistency, PRI, or coverage, or a failed minimization—we can select and inspect that replication in greater detail.
We will follow replication 13 because it has the largest absolute difference among the three monitored solution-level parameters of fit. Its conservative formula also changes, and its Jaccard coefficient of 0 indicates that the subsample and complete-data formulas share no prime implicants. The following code singles it out by number:
vdem_subsample_unstratified_con_df <- as.data.frame(
vdem_subsample_unstratified_con
)
subset(
vdem_subsample_unstratified_con_df,
rep == 13
)
#> rep n_sample n_holdout status exact_match_baseline term_jaccard_baseline solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 13 13 59 7 ok FALSE 0 CON:formula_changed CON 3 0.08996771As expected, the selected record reports a maximum absolute difference of approximately 0.090 among the solution’s parameters of fit. And now that the replication has been selected for an analytical reason, we can examine the omitted countries, the two conservative formulas, and the signed differences in solution consistency, PRI, and coverage:
vdem_subsample_rep_13 <- vdem_subsample_unstratified_con$by_run[["R13"]]
list(
omitted_countries = vdem_subsample_rep_13$holdout_labels,
complete_data_formula = vdem_subsample_rep_13$baseline$sig$conservative,
subsample_formula = vdem_subsample_rep_13$reduced$sig$conservative,
parameter_differences = vdem_subsample_rep_13$fit_delta
)
#> $omitted_countries
#> [1] "Ghana" "Burkina Faso" "Nepal" "Guinea" "Chad" "Republic of the Congo" "Kyrgyzstan"
#>
#> $complete_data_formula
#> [1] "FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $subsample_formula
#> [1] "FREEXP*GENCL+FREEXP*~PARTY*~JUCON"
#>
#> $parameter_differences
#> conservative.inclS conservative.PRI conservative.covS
#> -0.04109442 -0.04784207 0.08996771Replication 13 omits Ghana, Burkina Faso, Nepal, Guinea, Chad, the Republic of the Congo, and Kyrgyzstan. The complete-data conservative formula FREEXP*GENCL*JUCON + FREEXP*PARTY*GENCL is replaced by FREEXP*GENCL + FREEXP*~PARTY*~GENCL*~JUCON. Relative to the complete-data conservative solution, the subsample solution also has solution consistency approximately 0.041 lower, solution PRI approximately 0.048 lower, and solution coverage approximately 0.090 higher. Because the Boolean expression changes, the differences in the parameters of fit compare two different sufficient formulas.
To understand why the formula changes, we can compare the outcome assignments in the complete-data and subsample truth tables. There is no need to inspect every truth-table configuration. The following code retains only the configurations whose OUT assignment differs between the two analyses:
vdem_subsample_baseline_tt <- vdem_subsample_rep_13$baseline$tt$tt
vdem_subsample_rep_13_tt <- vdem_subsample_rep_13$reduced$tt$tt
vdem_subsample_changed_rows <- which(
as.character(vdem_subsample_baseline_tt$OUT) !=
as.character(vdem_subsample_rep_13_tt$OUT)
)
vdem_subsample_tt_columns <- c(
vdem_conds,
"OUT",
"n",
"incl",
"PRI",
"cases"
)
list(
complete_data = vdem_subsample_baseline_tt[
vdem_subsample_changed_rows,
vdem_subsample_tt_columns
],
subsample = vdem_subsample_rep_13_tt[
vdem_subsample_changed_rows,
vdem_subsample_tt_columns
]
)
#> $complete_data
#> FREEXP PARTY GENCL JUCON OUT n incl PRI cases
#> 4 0 0 1 1 0 2 0.728405430791534 0.23082177903784 Egypt,Burkina Faso
#> 9 1 0 0 0 0 5 0.758526742201357 0.400859177785658 Guinea,Mauritania,Central African Republic,Democratic Republic of the Congo,Somalia
#> 11 1 0 1 0 0 6 0.763868402990055 0.542457565776731 Angola,Cameroon,Chad,Kyrgyzstan,Madagascar,Togo
#>
#> $subsample
#> FREEXP PARTY GENCL JUCON OUT n incl PRI cases
#> 4 0 0 1 1 ? 1 0.786595063866544 0.321082113194993 Egypt
#> 9 1 0 0 0 1 4 0.858253747996411 0.599616602513265 Mauritania,Central African Republic,Democratic Republic of the Congo,Somalia
#> 11 1 0 1 0 1 4 0.848636499443891 0.694083459861076 Angola,Cameroon,Madagascar,TogoAll three truth-table configurations receive different assignments. The configuration ~FREEXP*~PARTY*GENCL*JUCON is empirically represented by Egypt and Burkina Faso in the complete data. Once Burkina Faso is omitted, its frequency falls from two cases to one, below n.cut = 2, so the configuration becomes a logical remainder in the subsample truth table.
The other two configurations become empirically sufficient for CIVSOC under the selected truth-table cutoffs. The consistency of sufficiency for FREEXP*~PARTY*~GENCL*~JUCON rises from approximately 0.759 to 0.858, while the consistency of sufficiency for FREEXP*~PARTY*GENCL*~JUCON rises from approximately 0.764 to 0.849. Both configurations retain enough cases to meet the frequency cutoff and now exceed incl.cut = 0.80, so their assignments change from OUT = 0 to OUT = 1. Conservative minimization consequently begins from a different collection of empirically sufficient configurations and returns the new two-term formula shown above.
10.3.3 Summarizing Recurrence Across Replications
Now, say we want to investigate formula recurrence across the complete collection of sampled case compositions. That is, how often the baseline complete-data conservative solution formula reappears when the QCA is reconstructed for each of the twenty subsamples.
Formula recurrence is an exact criterion: a subsample reproduces the baseline formula only when minimization returns the same complete Boolean expression, containing the same prime implicants. If one baseline prime implicant disappears, a new one enters, or the formula is otherwise simplified or expanded, the complete formula has changed—even when part of the baseline solution remains.
exact_solution begins with the complete solution formula. For each monitored solution type, it reports the baseline expression, the number of successful comparisons, the number and proportion of exact matches, and the frequencies of every selected formula observed in those comparisons.
vdem_subsample_unstratified_con$summary$exact_solution$conservative
#> $baseline_solution_expression
#> [1] "FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $baseline_key
#> [1] "FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $n_successful
#> [1] 20
#>
#> $n_exact
#> [1] 13
#>
#> $prop_exact
#> [1] 0.65
#>
#> $frequencies
#> keys
#> FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL FREEXP*GENCL FREEXP*GENCL+FREEXP*~PARTY*~JUCON FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL+FREEXP*~PARTY*~GENCL*~JUCON
#> 13 4 2 1Because an exact formula match is an all-or-nothing criterion, term_stability records the recurrence of individual prime implicants. Within each solution-type table, term identifies a prime implicant found in the baseline formula or in at least one successful subsample formula. baseline indicates whether that prime implicant belongs to the complete-data formula, while appearance_rate gives the proportion of successful comparisons in which it appears.
vdem_subsample_unstratified_con$summary$term_stability$conservative
#> term baseline appearance_rate
#> 1 FREEXP*GENCL FALSE 0.30
#> 2 FREEXP*GENCL*JUCON TRUE 0.70
#> 3 FREEXP*PARTY*GENCL TRUE 0.70
#> 4 FREEXP*~PARTY*~GENCL*~JUCON FALSE 0.05
#> 5 FREEXP*~PARTY*~JUCON FALSE 0.10The two baseline prime implicants, FREEXP*GENCL*JUCON and FREEXP*PARTY*GENCL, each appear in 14 of the 20 successful conservative comparisons, for an appearance rate of 0.70. Three prime implicants absent from the complete-data formula also appear in the subsample solutions. Among them, FREEXP*GENCL appears in six replications, whereas neither of the other two appears more than twice. The exact formula-match proportion of 0.65 therefore sits alongside an even recurrence of the two baseline prime implicants and a comparatively frequent alternative term.
Furthermore, two other summaries describe the numerical and structural variation across replications. fit_stability reports the distribution of each requested solution-level parameter. For each parameter, it gives the complete-data value, the median and interquartile range across successful subsamples for which that parameter of fit is available, and the corresponding distribution of signed differences from the baseline. prop_changed gives the proportion of those comparisons whose absolute difference exceeds fit_tol.
similarity remains at the level of prime-implicant identities. For each solution type, it reports the number of successful comparisons, the mean and median Jaccard coefficient relative to the baseline formula, and the average coefficient across all pairs of successful subsample formulas. The baseline coefficients show how closely the replications resemble the complete-data prime-implicant set, while the pairwise coefficient shows how closely the subsample formulas resemble one another, even when they do not reproduce the baseline.
vdem_subsample_unstratified_con$summary$fit_stability$conservative
#> measure baseline median q25 q75 median_delta q25_delta q75_delta prop_changed
#> 1 conservative.PRI 0.8590761 0.8497179 0.8339010 0.8644975 -0.009358191 -0.025175070 0.005421468 0.85
#> 2 conservative.covS 0.8226760 0.8348848 0.8175801 0.8803541 0.012208820 -0.005095893 0.057678087 0.95
#> 3 conservative.inclS 0.8877605 0.8793460 0.8651711 0.8913846 -0.008414453 -0.022589323 0.003624187 0.75
vdem_subsample_unstratified_con$summary$similarity
#> solution_type n_successful mean_baseline_jaccard median_baseline_jaccard avg_pairwise_jaccard
#> 1 conservative 20 0.6833333 1 0.5140351Exact formula recurrence, prime-implicant appearance, prime-implicant-set similarity, and the distributions of solution consistency, PRI, and coverage are related but noninterchangeable results. Appraised in tandem, they show whether the sensitivity concerns the complete solution formula, a particular sufficient term, the broader minimized structure, the consistency, PRI, or coverage of the sufficient relation, or several of these features at once.
10.4 Expanding the Diagnostic
The twenty-replication diagnostic was tiny by design and restricted to the conservative solution. A fleshed-out evaluation can include not only every solution type, but also enough replications to award us a trip to Monte Carlo. The expanded diagnostic therefore draws 100 samples and compares the conservative, parsimonious, and the sole C1P1 intermediate formulas with their respective complete-data baselines.
Every replication still retains 59 of the 66 countries, but the sampling is now stratified by V-Dem’s six-category politico-geographic classification, retained as region_pol:
vdem_subsample_region_composition <- data.frame(
country = rownames(vdem_raw),
region_pol = vdem_raw$region_pol
)
vdem_subsample_region_composition %>%
group_by(region_pol) %>%
summarise(
countries = paste(country, collapse = ", ")
)
#> # A tibble: 5 x 2
#> region_pol countries
#> <dbl> <chr>
#> 1 1 Kyrgyzstan, Tajikistan, Uzbekistan
#> 2 2 Bolivia, Honduras, Nicaragua
#> 3 3 Egypt, Yemen, Lebanon, Jordan, Morocco, Tunisia
#> 4 4 Ghana, Mali, Senegal, South Sudan, Ethiopia, Kenya, Nigeria, Tanzania, Uganda, Benin, Burkina Faso, Mozambique, Zambia, Zimbabwe, Guinea, Ivory Coast, Mauritania, Burundi, Cape Verde, Central African Republic, Lesotho, Liberia, Malawi, Sierra Leone, Angola, Cameroon, Chad, Democratic Republic of the Congo, Republic of the Congo, Djibouti, The Gambia, Madagascar, Rwanda, Somalia, Eswatini, Togo, Somaliland, Comoros, Sao Tome and Principe, Zanzibar
#> 5 6 Pakistan, Vietnam, India, North Korea, Philippines, Bhutan, Cambodia, Nepal, Timor-Leste, Papua New Guinea, Laos, Sri Lanka, Solomon Islands, VanuatuThe number of countries drawn from each observed region remains fixed across replications, while the identities of the countries selected within those regional allocations vary.
Under this design, region_pol fixes the number of countries retained from each politico-geographic group while allowing the identities of those countries to change. The retained countries are pooled into one subsample, from which a new truth table and new conservative, parsimonious, and intermediate solutions are produced; the resulting comparisons show whether the selected QCA results persist across joint case removals while regional composition remains comparatively similar to that of the complete case universe.
Regional stratification should not be confused with either statistical control or the cluster diagnostic introduced later in the book. It does not estimate separate regional QCA results or evaluate whether the pooled sufficient relations hold equally well within each region. Its only purpose is to prevent the regional composition of the random subsamples from varying freely while case composition is perturbed.
cluster.test(), by contrast, holds the selected pooled solution formula fixed and compares its consistency and coverage across the regional groups.
Calibration remains fixed, and the truth-table cutoffs, model and branch selection, directional expectations, and exclusion rule reproduce the all-solution specification established in the leave-one-out chapter. The case-composition perturbation and the larger replication set are therefore the new analytical ingredients.
vdem_subsample_stratified_all <- subsample.test(
data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
sample_prop = 0.90,
reps = 100,
stratify = "user",
strata = vdem_raw$region_pol,
seed = 123,
calib = "fixed",
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "all",
dir.exp = vdem_dir_exp,
exclude_mode = "recompute",
exclude_recompute = list(type = 2),
which_M = 1,
i_mode = "C1P1",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)10.4.1 Formula Recurrence Across Solution Types
Printing the complete object would display 100 replication records. The solution-specific exact-match summary gives the principal formula result more directly while retaining the successful-comparison denominator for each solution type:
vdem_subsample_stratified_all$summary$exact_solution
#> $conservative
#> $conservative$baseline_solution_expression
#> [1] "FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $conservative$baseline_key
#> [1] "FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $conservative$n_successful
#> [1] 100
#>
#> $conservative$n_exact
#> [1] 75
#>
#> $conservative$prop_exact
#> [1] 0.75
#>
#> $conservative$frequencies
#> keys
#> FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL FREEXP*GENCL FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL+FREEXP*~PARTY*~GENCL*~JUCON FREEXP*GENCL*JUCON FREEXP*GENCL*JUCON+FREEXP*~PARTY*GENCL FREEXP*GENCL+FREEXP*~PARTY*~JUCON
#> 75 10 7 6 1 1
#>
#>
#> $parsimonious
#> $parsimonious$baseline_solution_expression
#> [1] "FREEXP*JUCON+FREEXP*PARTY"
#>
#> $parsimonious$baseline_key
#> [1] "FREEXP*JUCON+FREEXP*PARTY"
#>
#> $parsimonious$n_successful
#> [1] 100
#>
#> $parsimonious$n_exact
#> [1] 54
#>
#> $parsimonious$prop_exact
#> [1] 0.54
#>
#> $parsimonious$frequencies
#> keys
#> FREEXP*JUCON+FREEXP*PARTY FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL FREEXP*GENCL FREEXP*JUCON FREEXP*JUCON+FREEXP*PARTY+FREEXP*~GENCL FREEXP FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL+FREEXP*~PARTY*~GENCL*~JUCON
#> 54 21 11 6 6 1 1
#>
#>
#> $intermediate
#> $intermediate$baseline_solution_expression
#> [1] "C1P1:FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $intermediate$baseline_key
#> [1] "C1P1:FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL"
#>
#> $intermediate$n_successful
#> [1] 100
#>
#> $intermediate$n_exact
#> [1] 75
#>
#> $intermediate$prop_exact
#> [1] 0.75
#>
#> $intermediate$frequencies
#> keys
#> C1P1:FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL C1P1:FREEXP*GENCL C1P1:FREEXP*GENCL*JUCON C1P1:FREEXP*JUCON+FREEXP*PARTY+FREEXP*~GENCL C1P1:FREEXP C1P1:FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL+FREEXP*~PARTY*~GENCL*~JUCON
#> 75 11 6 6 1 1All 100 replications produce comparable conservative, parsimonious, and intermediate results, so the denominator is 100 for every solution type. The conservative and intermediate formulas each reproduce their respective baselines in 75 replications, whereas the parsimonious formula does so in 54.
vdem_subsample_stratified_all_df <- as.data.frame(
vdem_subsample_stratified_all
)
table(
vdem_subsample_stratified_all_df$solution_change,
useNA = "ifany"
)
#>
#> CON:formula_changed | PAR:formula_changed | INT:formula_changed PAR:formula_changed <NA>
#> 25 21 54Of the 100 replications, 25 change all three solution formulas, 21 change only the parsimonious formula while preserving the conservative and intermediate formulas, and 54 reproduce all three baselines.
10.4.2 Prime-Implicant Recurrence and Similarity
Below, the solution-specific Jaccard coefficients measure the overlap between the prime implicants in each subsample formula and those in the corresponding complete-data formula, including the replications in which exact formula preservation fails:
vdem_subsample_stratified_all$summary$similarity
#> solution_type n_successful mean_baseline_jaccard median_baseline_jaccard avg_pairwise_jaccard
#> 1 conservative 100 0.8300000 1 0.7029798
#> 2 parsimonious 100 0.6100000 1 0.4303030
#> 3 intermediate 100 0.7866667 1 0.6337374The conservative solution has a mean baseline Jaccard coefficient of approximately 0.830, and the intermediate solution of approximately 0.787. The parsimonious solution has a lower mean of 0.610, while the median coefficient is 1 for all three solution types. Under this sampling design, with its case-composition perturbations, the selected parsimonious formula therefore has less average overlap with its baseline prime-implicant set than the conservative and intermediate formulas do, even though at least half of the replications reproduce each complete baseline set.
The term summaries identify which prime implicants account for those aggregate differences:
vdem_subsample_stratified_all$summary$term_stability
#> $conservative
#> term baseline appearance_rate
#> 1 FREEXP*GENCL FALSE 0.11
#> 2 FREEXP*GENCL*JUCON TRUE 0.89
#> 3 FREEXP*PARTY*GENCL TRUE 0.82
#> 4 FREEXP*~PARTY*GENCL FALSE 0.01
#> 5 FREEXP*~PARTY*~GENCL*~JUCON FALSE 0.07
#> 6 FREEXP*~PARTY*~JUCON FALSE 0.01
#>
#> $parsimonious
#> term baseline appearance_rate
#> 1 FREEXP FALSE 0.01
#> 2 FREEXP*GENCL FALSE 0.11
#> 3 FREEXP*GENCL*JUCON FALSE 0.22
#> 4 FREEXP*JUCON TRUE 0.66
#> 5 FREEXP*PARTY TRUE 0.60
#> 6 FREEXP*PARTY*GENCL FALSE 0.22
#> 7 FREEXP*~GENCL FALSE 0.06
#> 8 FREEXP*~PARTY*~GENCL*~JUCON FALSE 0.01
#>
#> $intermediate
#> term baseline appearance_rate
#> 1 FREEXP FALSE 0.01
#> 2 FREEXP*GENCL FALSE 0.11
#> 3 FREEXP*GENCL*JUCON TRUE 0.82
#> 4 FREEXP*JUCON FALSE 0.06
#> 5 FREEXP*PARTY FALSE 0.06
#> 6 FREEXP*PARTY*GENCL TRUE 0.76
#> 7 FREEXP*~GENCL FALSE 0.06
#> 8 FREEXP*~PARTY*~GENCL*~JUCON FALSE 0.01The two baseline conservative prime implicants, FREEXP*GENCL*JUCON and FREEXP*PARTY*GENCL, appear in 0.89 and 0.82 of successful conservative comparisons. The same prime implicants belong to the baseline intermediate formula and appear in 0.82 and 0.76 of the intermediate comparisons. No nonbaseline prime implicant reaches an appearance rate above 0.11 in either solution type.
The parsimonious baseline prime implicants, FREEXP*JUCON and FREEXP*PARTY, appear in 0.66 and 0.60 of successful comparisons. Two nonbaseline prime implicants recur much more often than the remaining alternatives: FREEXP*GENCL*JUCON and FREEXP*PARTY*GENCL each appear in 0.22. These recurring substitutions therefore account for the lower exact-match rate and mean Jaccard coefficient of the parsimonious solution.
10.4.3 Changes in Solution Consistency, PRI, and Coverage
Although formula recurrence tells us if a subsample produces the same minimized expression as the complete-data analysis, it does not identify whether that expression displays the same empirical strength and relevance once the case composition changes. For instance, two replications may reproduce exactly the same formula while assigning somewhat different values to its parameters of fit. If any of these values moves across sampled case compositions, we want to know which one moved, in which direction, and by how much.
fit_stability reports the parameter-specific distributions for each solution type:
vdem_subsample_stratified_all$summary$fit_stability
#> $conservative
#> measure baseline median q25 q75 median_delta q25_delta q75_delta prop_changed
#> 1 conservative.PRI 0.8590761 0.8576341 0.8454294 0.8693473 -0.001441933 -0.013646683 0.010271207 0.80
#> 2 conservative.covS 0.8226760 0.8259231 0.8168761 0.8404819 0.003247042 -0.005799916 0.017805836 0.73
#> 3 conservative.inclS 0.8877605 0.8866253 0.8781671 0.8951215 -0.001135165 -0.009593305 0.007361019 0.74
#>
#> $parsimonious
#> measure baseline median q25 q75 median_delta q25_delta q75_delta prop_changed
#> 1 parsimonious.PRI 0.8613564 0.8582797 0.8473175 0.8715575 -0.003076733 -0.01403897 0.010201070 0.78
#> 2 parsimonious.covS 0.8437745 0.8414141 0.8317520 0.8514743 -0.002360336 -0.01202246 0.007699889 0.74
#> 3 parsimonious.inclS 0.8892891 0.8872879 0.8790403 0.8966004 -0.002001136 -0.01024874 0.007311329 0.72
#>
#> $intermediate
#> measure baseline median q25 q75 median_delta q25_delta q75_delta prop_changed
#> 1 intermediate.C1P1.PRI 0.8590761 0.8576341 0.8452456 0.8693473 -0.001441933 -0.013830456 0.010271207 0.81
#> 2 intermediate.C1P1.covS 0.8226760 0.8259231 0.8168761 0.8412341 0.003247042 -0.005799916 0.018558092 0.73
#> 3 intermediate.C1P1.inclS 0.8877605 0.8866253 0.8781532 0.8951215 -0.001135165 -0.009607248 0.007361019 0.74For each solution type, the output reports complete-data and subsample values for every parameter of fit. baseline gives the value from the complete 66-country analysis, while median, q25, and q75 summarize the parameter across successful subsample replications; the corresponding *_delta columns summarize signed differences from the baseline, with positive values indicating higher values in the subsamples and negative values indicating lower ones.
The final column, prop_changed, reports, among the successful comparisons for which the parameter of fit is available, the proportion whose absolute difference from the baseline exceeds the fit_tol = 0.005 threshold selected for this diagnostic. All three parameters are available in all 100 replications here, so the denominator is 100 in every row of the three tables. This proportion allows us to count tolerance crossings, although it ignores their direction and magnitude. A change from 0.810 to 0.816 and a change from 0.810 to 0.900 both cross that threshold, even though they would hardly receive the same substantive interpretation. For that reason, prop_changed should always be read together with the observed distribution and its signed differences.
fit_stability shows that, depending on the solution type and parameter, the proportion of replications exceeding the 0.005 tolerance ranges from 0.72 to 0.81. That sounds a bit dramatic. Yet the median difference remains smaller than 0.005 in absolute terms for all nine comparisons. The largest median difference is approximately 0.00325 for conservative and intermediate solution coverage: their complete-data coverage is about 0.823, while the median across the subsamples is about 0.826. These findings show that the tolerance is crossed frequently in individual replications depending on the set value, but the movements occur on both sides of the complete-data value, and often narrowly, leaving the center of each subsample distribution quite close to its baseline.
The relevant conclusion is therefore not simply that the parameters of fit “changed” and QCA is inherently fickle. Rather, solution consistency, PRI, and coverage often differ from their baselines by more than the deliberately narrow tolerance of
0.005, while their typical values remain close to those obtained from the complete data.
10.5 Checking the Sampling Design
Every result in a subsampling diagnostic is conditional on how the samples were constructed. For example, in the appraisal presented above, each replication retains 90% of the observations while preserving the regional composition of the 66-country case universe reasonably well. Of course, the resulting recurrence rates do not describe what would happen under every plausible removal of cases, but simply what happens under this particular sampling design.
10.5.1 How Much of the Case Universe Should Be Retained?
sample_prop determines the severity of the perturbation: retaining 90% removes seven countries in each replication, whereas retaining 70% would remove many more and could produce much larger changes in configuration frequencies, parameters of fit, outcome assignments, available logical remainders, and minimized solutions.
The exact-match rate therefore depends on how much stress we are willing to put the analysis through, so researchers should choose a retention level that represents a substantively plausible challenge to their model.
10.5.2 How Many Replications Are Enough?
The reps argument controls how many sampled case compositions are examined; more replications provide a sturdier description of the recurrence pattern generated by the chosen sampling design, but there is no universally sufficient number that fits every analysis.
To appraise the number of replications, (1) run the planned diagnostic; (2) repeat it with more replications while leaving the retention level, stratification rule, and QCA specifications unchanged; and (3) compare exact formula recurrence, prime-implicant appearance, solution similarity, and the distributions of solution consistency, PRI, and coverage.
If those summaries continue to move appreciably, the smaller run was not yet giving a stable account of the sampling exercise; if they remain materially similar, the additional replications reinforce the observed pattern.
This check concerns the finite random exercise conducted by
subsample.test(). It does not create “confidence intervals” for a QCA solution, nor does it turn the observed countries into a probability sample from an external population.
10.5.3 What Should Stratification Preserve?
Stratification constrains which case compositions the diagnostic is allowed to generate. In the present example, regional stratification keeps the number of sampled countries from each region comparatively stable. The diagnostic consequently focuses on whether the QCA results recur across joint case removals that continue to represent the regional composition of the case universe reasonably well.
An unstratified diagnostic, then, would allow the regional composition to vary. Outcome stratification would instead preserve the allocation of cases across the crisp outcome groups. Another user-defined grouping would preserve whatever structure that variable represents. None of these alternatives is automatically more rigorous than the others, each one defining a different family of admissible subsamples and, thus, a different robustness question.
10.6 When Calibration Should Be Recomputed
All examples so far use calib = "fixed". The countries retained in each subsample therefore keep the set-membership scores assigned in the complete-data analysis. This is the appropriate design when the qualitative anchors are treated as fixed features of the research design. It also gives the perturbation a clear scope: cases are jointly removed, and the truth table and selected solutions are reconstructed from the remaining calibrated memberships.
Crucially, as in
loo.test(), recomputation is notcalib = "fixed"’s cleverer, more sophisticated cousin—the one its developer keeps comparing it to. It is most directly justified when data-guided threshold estimation withQCA::findTh()formed part of the original calibration procedure and the researcher wants to examine the stability of that complete procedure. Then, removing several observations may change those estimated anchors and, consequently, the calibrated memberships of cases that remain in the sample. Settingcalib = "recompute"allows both parts of that analytical sequence to vary: the case composition used to estimate the anchors and the case composition supplied to truth-table analysis.If the original memberships were calibrated from substantively established anchors, keeping those anchors fixed usually represents the original research design more faithfully.
The following example extends the four-group threshold search used in the reference analysis to the subsampling diagnostic.
vdem_subsample_recal_spec <- vdem_calib_spec_with_outcome
vdem_subsample_recal_spec$CIVSOC$findTh <- list(groups = 4)
vdem_subsample_recal_spec$FREEXP$findTh <- list(groups = 4)
vdem_subsample_recal_spec$PARTY$findTh <- list(groups = 4)
vdem_subsample_recal_spec$GENCL$findTh <- list(groups = 4)
vdem_subsample_recal_spec$JUCON$findTh <- list(groups = 4)
vdem_subsample_recomputed_con <- subsample.test(
data = vdem_calib,
raw.data = vdem_raw,
outcome = vdem_outcome,
conditions = vdem_conds,
sample_prop = 0.90,
reps = 100,
stratify = "user",
strata = vdem_raw$region_pol,
seed = 123,
calib = "recompute",
calib_spec = vdem_subsample_recal_spec,
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "conservative",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)As in the leave-one-out appraisal, a changed solution formula under this design may result from joint case removal, recalibration of the countries that remain, or both. Alongside the recurrence, similarity, and parameter-of-fit summaries introduced above, the diagnostic allows the recalibration itself to be examined directly:
vdem_subsample_recomputed_con$summary$calibrationTo assess the stability of the calibration stage, the summary reports one record for every anchor of every recalibrated set, including the complete-data value in baseline, the median and interquartile range estimated across successful replications, and prop_diff_baseline, the proportion of those replications in which the estimated anchor differs from its complete-data value.
10.7 Deciding What Matters in a Subsample Diagnostic
Okay… phew! By this point, you have been bombarded with a host of new parameters for subsample evaluation, including exact formula recurrence, prime-implicant appearance rates, Jaccard coefficients, and changes in solution consistency, PRI, and coverage. So you are probably wondering, “That’s all well and good, but which of these is really relevant?”
Honestly, the gist of it is that these measures do not compete to deliver a single verdict. Each one addresses a different part of the QCA result, and they should not receive equal weight merely because the function calculates them all.
In the usual subsampling appraisal, the principal question is whether each selected baseline solution formula reappears across the sampled case compositions. The exact-match count and its successful-comparison denominator should therefore come first. This result tells us how often the complete Boolean expression is reproduced for the conservative, parsimonious, or intermediate solution. It does not require a universal acceptance threshold: the recurrence rate becomes meaningful in relation to the severity and substantive plausibility of the sampling design.
When the complete formula is not reproduced, prime-implicant recurrence explains what changed. The term_stability summary shows whether the sufficient terms from the baseline formula continue to appear and which nonbaseline terms recur in their place. This evidence may be especially important when the substantive argument centers on one particular sufficient term, since it can recur even when another term enters or leaves the complete solution formula.
The Jaccard coefficient compresses some of this term-level evidence into one number, but its meaning is considerably narrower. It compares prime implicants as literal identities. Thus, FREEXP*GENCL and FREEXP*GENCL*JUCON share no identical prime implicant and receive a coefficient of 0, even though the latter is a more restrictive conjunction contained within the former. Jaccard similarity does not evaluate that logical relation, the overlap in cases covered, or the similarity between the complete solutions’ fuzzy-set memberships. It can serve as a supplementary description of literal prime-implicant overlap, but should not displace the exact formulas and term appearance rates.
Changes in solution consistency, solution PRI, and solution coverage form a separate part of the appraisal. When a subsample reproduces the baseline formula, the comparison evaluates the same sufficient expression across a different case composition. When the formula changes, however, the parameters of fit belong to different sufficient expressions obtained under the same solution type. The current fit_stability summary includes both kinds of successful replication. It therefore describes the distribution of the parameters attached to the selected subsample solutions, but it should not be presented as an apples-to-apples assessment of the baseline formula’s parameters of fit unless the comparison is restricted to exact formula matches.
The selected fit_tol provides a screening rule, not really a measure of robustness. prop_changed reports how often the absolute difference in a parameter of fit exceeds that tolerance, while max_abs_fit_delta identifies the largest absolute difference among the parameters that cross it in an individual replication. Neither quantity measures distance from the tolerance itself. Because the proportion of crossings depends directly on where the tolerance is placed, it must be interpreted alongside the signed differences and the observed distributions of solution consistency, PRI, and coverage.
The evidence can therefore be read in a practical order:
- Establish how many replications produce a comparable result for each solution type, since those successful comparisons provide the denominator for the remaining summaries.
- Report how often each complete baseline solution formula is reproduced.
- If formulas change, examine which baseline prime implicants persist and which alternative terms or complete formulas recur.
- Evaluate solution consistency, PRI, and coverage separately, distinguishing comparisons of an identical formula from comparisons involving different sufficient expressions.
- Use Jaccard coefficients and tolerance-crossing classifications only when their summaries add something useful to the formula-, term-, and parameter-specific evidence.
10.8 Reporting and Interpreting the Diagnostic
A useful report begins with the perturbation that was actually performed. In vdem_subsample_stratified_all, subsample.test() generated 100 reproducible, region-stratified samples, each retaining 59 of the 66 countries. Calibration remained fixed, and every sample was analyzed with the reference truth-table cutoffs, directional expectations, exclusion rule, and solution-selection settings. All 100 replications produced comparable conservative, parsimonious, and intermediate solutions.
The substantive results can then be reported at the level required by the claim. In the expanded example, the conservative and C1P1 intermediate formulas each recur in 75 of the 100 replications, whereas the parsimonious formula recurs in 54. The corresponding mean Jaccard similarities are approximately 0.830, 0.787, and 0.610, respectively. The baseline conservative prime implicants recur in 89% and 82% of the replications; the baseline intermediate prime implicants recur in 82% and 76%; and the baseline parsimonious prime implicants recur in 66% and 60%. These results show that the conservative and intermediate solutions are reproduced more frequently, and remain more similar to their baselines when they change, than the parsimonious solution.
Depending on the solution type and parameter, between 72% and 81% of the successful comparisons move solution consistency, solution PRI, or solution coverage by more than 0.005 from the corresponding baseline value, yet the median difference for every parameter remains smaller than 0.005 in absolute terms. Crossings of this narrow tolerance are therefore common even though the center of each parameter distribution remains close to the complete-data result.
All in all, these findings support a surprisingly rich and nuanced conclusion. The formulas do not recur equally often across solution types, and the parsimonious solution is especially sensitive to the sampled case composition. At the same time, the typical values of solution consistency, PRI, and coverage remain close to their baselines, even though individual replications often cross the selected tolerance. Neither part of this conclusion, by itself, should be collapsed into a single declaration that the analysis is—or is not—robust.
These results remain conditional on the observed countries, the 90% retention level, the regional stratification rule, the fixed calibration, the QCA specifications, and the 100 sampled case compositions. They do not, of course, estimate the probability that a solution is true in some external population, and they do not provide a confidence level for a QCA solution.