vdem_loo_fixed_con <- loo.test(
data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
calib = "fixed",
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "conservative",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)9 Leave-One-Out Diagnostics
Removing one observation from a QCA alters the empirical evidence represented in the truth table: beyond reducing the frequency of the Boolean configuration to which the country is assigned, the deletion usually changes the associated fit parameters and may alter an empirical outcome assignment or even turn a previously observed configuration into a logical remainder. Because minimization then proceeds from this changed truth table, the selected sufficient terms and the complete solution formula may change as well.
The label influential case therefore requires an explicit criterion. A country may be influential because its deletion changes a selected solution formula. Alternatively, it may be influential because of its effects on solution consistency, PRI, or coverage, pushing one or more of these measures below or above a threshold established by the researcher. In the worst-case scenario, the reduced model might not even be feasible above the usual consistency threshold of 0.8.
loo.test() organizes these possibilities as a series of one-case-at-a-time comparisons—also known as Jackknife resampling. It first reconstructs the complete-data baseline under the analytical settings supplied by the researcher. The function then deletes each selected case independently, rebuilds the truth table, repeats the requested minimization, and compares the reduced-data result with the complete-data baseline. Every country is thus evaluated against the same reference analysis; deletions do not accumulate from one run to the next.
The chapter applies this diagnostic to the 66-country V-Dem analysis. We will begin by holding the initial calibration and the remaining QCA specification fixed to assess which individual cases are relevant for the selected solution formula or its solution-level parameters of fit. We will then widen the comparison across the conservative, parsimonious, and intermediate solutions while continuing to hold the calibrated memberships fixed.
Finally, because the reference analysis estimated its qualitative anchors from the observed data, we will repeat the diagnostic with those calibrations re-estimated after every deletion. This third application asks a broader question about the dependence of the complete analytical workflow on each country.
9.1 Planning a Leave-One-Out Diagnostic
The ordinary workflow begins with the calibrated data used in the reference analysis. data supplies the condition and outcome memberships used to construct every truth table, while outcome names the outcome set and conditions the condition sets. Although conditions may be omitted when calibration remains fixed, naming the conditions explicitly keeps the diagnostic tied to the intended QCA specification and becomes mandatory if calibration is reconstructed.
The truth-table cutoffs, solution type, selected model or intermediate branch, directional expectations, and treatment of logical remainders retain the meanings established in the preceding chapters. loo.test() applies those choices to the baseline and to every reduced dataset. The new decisions concern the cases to be tested, whether the remaining memberships should stay fixed, and what amount of change will be treated as significant.
9.1.1 Deciding Which Cases to Test
By default, cases = NULL deletes every case in data, one at a time. This is the natural starting point when the number of cases is manageable because it produces a complete case-by-case scan of the study population. A character vector can instead name a theoretically relevant subset of cases, while a numeric vector can select their row positions. A restricted scan is perfectly legitimate, but its conclusions apply only to the deletions actually tested.
Readable case identifiers, as expected, are rather relevant here. case_labels can provide one label for each case, but it defaults to the row names of data when they are available and otherwise falls back to row numbers. Character values supplied through cases are matched against these labels. In practice, meaningful row names or an explicit case_labels vector save the reader from having to remember that case 37 was, say, Mauritania all along.
9.1.2 Holding Calibration Fixed—or Reconstructing It
calib determines whether deleting a case changes only case composition or also triggers a new calibration. Its default, "fixed", uses the membership scores already stored in data. After a country is removed, every remaining country’s membership in the condition and outcome sets stays exactly as it was in the complete dataset. This is the most direct design for identifying how one case contributes to the reference truth table and solution.
The alternative, calib = "recompute", reconstructs the calibrated sets from raw.data and calib_spec for the complete-data baseline and after every deletion. This option is useful when the qualitative anchors themselves were estimated from the observed cases and the researcher wants those anchors to be re-estimated after a country is removed. Because that broader design requires more preparation and changes the meaning of the diagnostic, we return to it after the two fixed-calibration applications.
9.1.3 Deciding What Counts as Influence
For every monitored solution type available in both the complete-data and reduced-data analyses, loo.test() first checks whether the two selected formulas are identical. Formula preservation is strict: changing a sufficient term or adding or removing a term means that the formula is not preserved. If the selected model or intermediate branch cannot be obtained in one of the analyses, no formula comparison is available for that solution type, and the missing comparison must not be counted as preservation.
A second comparison concerns the complete solution’s parameters of fit. fit_measures defaults to c("inclS", "PRI", "covS"), corresponding to solution consistency, solution PRI, and solution coverage. The researcher may select a different collection of available solution-level parameters or use fit_measures = NULL when only formula preservation matters.
fit_tol determines how much absolute movement in each requested parameter is tolerated before the deletion is classified as relevant. It must be non-negative and defaults to 0, an effectively exact comparison apart from a small allowance for floating-point noise. That default is, indeed, overly sensitive in fuzzy-set analyses, where removing a case may produce a tiny decimal movement without altering the practical interpretation of the set relation. A positive tolerance should therefore be chosen and reported as part of the diagnostic design, rather than discovered after looking at which countries happen to cross it.
Unlike the alternative-set diagnostic, loo.test() compares the selected solution-level parameters even when the formula changes. If the formula is preserved, a parameter difference evaluates the same sufficient expression under two case compositions. If the formula changes, by contrast, the reported difference compares the parameters of two different solution formulas and must be interpreted accordingly.
Finally, a deletion may prevent truth-table construction or a requested minimization from producing a comparable result. Such a run is not evidence of preservation. When only one solution type is monitored, status = "ok" means that its complete-data and reduced-data formulas are comparable. With solution = "all", however, the overall status is "ok" as long as at least one monitored solution type remains comparable; it does not guarantee that the conservative, parsimonious, and intermediate comparisons all succeeded. Any failure affecting only one solution type is recorded in the solution-type-specific statuses stored with the detailed baseline and by_case analyses.
9.2 A First Leave-One-Out Diagnostic
Our first application keeps the calibration fixed and tests all 66 countries because cases is left at its default. It monitors only the conservative solution while preserving the reference truth-table cutoffs and minimization settings. The requested parameters are solution consistency, PRI, and coverage, and an absolute change must exceed 0.005 to be classified as a parameter change. This tolerance is not a universal standard—as usual in QCA—but merely the criterion adopted for this didactic analysis.
9.2.1 Reading the First Case-Influence Scan
vdem_loo_fixed_con
#> <loo_test>
#> Outcome test: leave-one-out robustness
#> Solution: conservative
#> Monitored solutions: conservative
#> which_M: 1
#> Cases tested: 66
#> Fit measures: inclS, PRI, covS
#>
#> Result
#> row_index case_label status solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 Ghana ok <NA> CON 1 0.005118355
#> 2 Egypt ok <NA> <NA> 0 0.000000000
#> 3 Yemen ok <NA> <NA> 0 0.000000000
#> 4 Bolivia ok CON:formula_changed CON 3 0.045530955
#> 5 Honduras ok <NA> <NA> 0 0.000000000
#> 6 Mali ok <NA> CON 2 0.010153970
#> 7 Pakistan ok <NA> <NA> 0 0.000000000
#> 8 Senegal ok <NA> <NA> 0 0.000000000
#> 9 South Sudan ok <NA> <NA> 0 0.000000000
#> 10 Vietnam ok <NA> <NA> 0 0.000000000
#> 11 Ethiopia ok <NA> CON 1 0.008381286
#> 12 India ok <NA> CON 2 0.013796777
#> 13 Kenya ok <NA> <NA> 0 0.000000000
#> 14 North Korea ok <NA> <NA> 0 0.000000000
#> 15 Lebanon ok <NA> <NA> 0 0.000000000
#> 16 Nigeria ok <NA> <NA> 0 0.000000000
#> 17 Philippines ok <NA> <NA> 0 0.000000000
#> 18 Tanzania ok <NA> <NA> 0 0.000000000
#> 19 Uganda ok <NA> CON 2 0.016392674
#> 20 Benin ok <NA> <NA> 0 0.000000000
#> 21 Bhutan ok <NA> <NA> 0 0.000000000
#> 22 Burkina Faso ok <NA> <NA> 0 0.000000000
#> 23 Cambodia ok <NA> <NA> 0 0.000000000
#> 24 Mozambique ok <NA> CON 1 0.006976290
#> 25 Nepal ok <NA> <NA> 0 0.000000000
#> 26 Nicaragua ok <NA> <NA> 0 0.000000000
#> 27 Zambia ok <NA> CON 1 0.005014346
#> 28 Zimbabwe ok <NA> <NA> 0 0.000000000
#> 29 Guinea ok CON:formula_changed CON 3 0.029587467
#> 30 Ivory Coast ok <NA> <NA> 0 0.000000000
#> 31 Mauritania ok <NA> CON 1 0.012396165
#> 32 Burundi ok <NA> <NA> 0 0.000000000
#> 33 Cape Verde ok <NA> CON 1 0.005121705
#> 34 Central African Republic ok <NA> <NA> 0 0.000000000
#> 35 Timor-Leste ok <NA> <NA> 0 0.000000000
#> 36 Jordan ok <NA> CON 2 0.009525978
#> 37 Lesotho ok <NA> <NA> 0 0.000000000
#> 38 Liberia ok <NA> <NA> 0 0.000000000
#> 39 Malawi ok <NA> <NA> 0 0.000000000
#> 40 Morocco ok <NA> CON 2 0.013340333
#> 41 Papua New Guinea ok <NA> <NA> 0 0.000000000
#> 42 Sierra Leone ok <NA> <NA> 0 0.000000000
#> 43 Tunisia ok <NA> CON 2 0.012866719
#> 44 Angola ok <NA> CON 1 0.009449112
#> 45 Cameroon ok CON:formula_changed CON 3 0.063289509
#> 46 Chad ok <NA> <NA> 0 0.000000000
#> 47 Democratic Republic of the Congo ok <NA> <NA> 0 0.000000000
#> 48 Republic of the Congo ok <NA> CON 1 0.005601515
#> 49 Djibouti ok <NA> CON 1 0.008432301
#> 50 The Gambia ok <NA> CON 1 0.005084229
#> 51 Kyrgyzstan ok <NA> CON 2 0.007688506
#> 52 Laos ok <NA> <NA> 0 0.000000000
#> 53 Madagascar ok <NA> CON 1 0.010084062
#> 54 Rwanda ok <NA> <NA> 0 0.000000000
#> 55 Somalia ok <NA> CON 1 0.007644952
#> 56 Sri Lanka ok <NA> <NA> 0 0.000000000
#> 57 Eswatini ok <NA> <NA> 0 0.000000000
#> 58 Tajikistan ok <NA> <NA> 0 0.000000000
#> 59 Togo ok <NA> CON 1 0.013346709
#> 60 Somaliland ok <NA> <NA> 0 0.000000000
#> 61 Uzbekistan ok <NA> <NA> 0 0.000000000
#> 62 Comoros ok CON:formula_changed CON 3 0.048046732
#> 63 Sao Tome and Principe ok <NA> <NA> 0 0.000000000
#> 64 Solomon Islands ok <NA> <NA> 0 0.000000000
#> 65 Vanuatu ok <NA> CON 1 0.005088853
#> 66 Zanzibar ok <NA> CON 2 0.008297896
#>
#> Summary
#> Successful runs: 66/66
#> Solution changes: 4
#> Fit changes: 26
#> Non-ok runs: 0
#> Cases with solution changes: Bolivia, Guinea, Cameroon, ComorosAll 66 separate deletions were tested and received status = "ok". Each record represents the QCA reconstructed after removing the country identified by case_label; row_index simply records that country’s original position in vdem_calib. The case label is therefore the meaningful identifier for interpretation, while the row index is useful only when returning to the data.
Three groups of columns answer the diagnostic question. First, status indicates whether the reduced analysis retains at least one solution type that can be compared with its complete-data counterpart. Because this first application monitors only the conservative solution, status = "ok" means that the conservative comparison is available. Second, solution_change identifies any comparable monitored solution formula that differs from its complete-data counterpart. Third, fit_changed_types identifies the comparable solution types for which at least one requested solution-level parameter of fit moves by more than fit_tol.
n_fit_deltas counts the solution-type-and-parameter comparisons that cross the tolerance, whereas max_abs_fit_delta gives the largest absolute difference among them. A zero in these last two columns means that no requested parameter of fit crossed 0.005; smaller movements may still have occurred. An <NA> in either change column means only that the corresponding criterion was not met among the comparisons that were available. Reading it as formula preservation requires an available formula comparison for the solution type under consideration; reading it as preservation of the parameters of fit likewise requires the requested fit comparison to be available.
All 66 reduced analyses have status ok, so the denominator for both comparisons is the complete collection of 66 country deletions. Four of those deletions—Bolivia, Guinea, Cameroon, and Comoros—change the conservative solution formula.
9.2.2 Solution Changes Through Configuration Frequency
If needed, it is possible to take a closer look at the results. For that, let’s select only the yummy parts. The filter retains 26 of the 66 countries: the four whose deletion changes the conservative formula and the 22 additional countries whose deletion preserves that formula but changes at least one solution-level parameter by more than 0.005.
vdem_loo_fixed_con_df <- as.data.frame(vdem_loo_fixed_con)
subset(
vdem_loo_fixed_con_df,
!is.na(solution_change) |
!is.na(fit_changed_types)
) %>%
arrange(desc(max_abs_fit_delta))
#> row_index case_label status solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 45 Cameroon ok CON:formula_changed CON 3 0.063289509
#> 2 62 Comoros ok CON:formula_changed CON 3 0.048046732
#> 3 4 Bolivia ok CON:formula_changed CON 3 0.045530955
#> 4 29 Guinea ok CON:formula_changed CON 3 0.029587467
#> 5 19 Uganda ok <NA> CON 2 0.016392674
#> 6 12 India ok <NA> CON 2 0.013796777
#> 7 59 Togo ok <NA> CON 1 0.013346709
#> 8 40 Morocco ok <NA> CON 2 0.013340333
#> 9 43 Tunisia ok <NA> CON 2 0.012866719
#> 10 31 Mauritania ok <NA> CON 1 0.012396165
#> 11 6 Mali ok <NA> CON 2 0.010153970
#> 12 53 Madagascar ok <NA> CON 1 0.010084062
#> 13 36 Jordan ok <NA> CON 2 0.009525978
#> 14 44 Angola ok <NA> CON 1 0.009449112
#> 15 49 Djibouti ok <NA> CON 1 0.008432301
#> 16 11 Ethiopia ok <NA> CON 1 0.008381286
#> 17 66 Zanzibar ok <NA> CON 2 0.008297896
#> 18 51 Kyrgyzstan ok <NA> CON 2 0.007688506
#> 19 55 Somalia ok <NA> CON 1 0.007644952
#> 20 24 Mozambique ok <NA> CON 1 0.006976290
#> 21 48 Republic of the Congo ok <NA> CON 1 0.005601515
#> 22 33 Cape Verde ok <NA> CON 1 0.005121705
#> 23 1 Ghana ok <NA> CON 1 0.005118355
#> 24 65 Vanuatu ok <NA> CON 1 0.005088853
#> 25 50 The Gambia ok <NA> CON 1 0.005084229
#> 26 27 Zambia ok <NA> CON 1 0.005014346And, as a refresher, let’s summon the baseline truth table from the reference chapter:
vdem_tt
#>
#> OUT: output value
#> n: number of cases in configuration
#> incl: sufficiency inclusion score
#> PRI: proportional reduction in inconsistency
#>
#> FREEXP PARTY GENCL JUCON OUT n incl PRI cases
#> 16 1 1 1 1 1 23 0.927 0.905 Ghana,Honduras,Pakistan,Senegal,India,Kenya,Lebanon,Nigeria,Tanzania,Benin,Bhutan,Nepal,Zambia,Cape Verde,Timor-Leste,Jordan,Lesotho,Morocco,Sierra Leone,The Gambia,Sri Lanka,Somaliland,Zanzibar
#> 15 1 1 1 0 1 2 0.900 0.744 Bolivia,Comoros
#> 12 1 0 1 1 1 12 0.896 0.850 Mali,Philippines,Uganda,Mozambique,Ivory Coast,Liberia,Malawi,Papua New Guinea,Tunisia,Sao Tome and Principe,Solomon Islands,Vanuatu
#> 11 1 0 1 0 0 6 0.764 0.542 Angola,Cameroon,Chad,Kyrgyzstan,Madagascar,Togo
#> 9 1 0 0 0 0 5 0.759 0.401 Guinea,Mauritania,Central African Republic,Democratic Republic of the Congo,Somalia
#> 4 0 0 1 1 0 2 0.728 0.231 Egypt,Burkina Faso
#> 7 0 1 1 0 0 2 0.663 0.124 Vietnam,Uzbekistan
#> 3 0 0 1 0 0 5 0.607 0.108 Ethiopia,Zimbabwe,Burundi,Republic of the Congo,Djibouti
#> 5 0 1 0 0 0 3 0.545 0.074 North Korea,Rwanda,Tajikistan
#> 1 0 0 0 0 0 5 0.472 0.050 Yemen,South Sudan,Cambodia,Nicaragua,Eswatini
#> 2 0 0 0 1 ? 1 0.764 0.226 Laos
#> 6 0 1 0 1 ? 0 - -
#> 8 0 1 1 1 ? 0 - -
#> 10 1 0 0 1 ? 0 - -
#> 13 1 1 0 0 ? 0 - -
#> 14 1 1 0 1 ? 0 - -Bolivia and Comoros are the only two countries assigned to the truth-table configuration FREEXP*PARTY*GENCL*~JUCON, which receives OUT = 1 in the reference truth table. Deleting either country reduces the configuration’s frequency from 2 to 1. One country still occupies the configuration, but n.cut = 2 means that one is not enough for the configuration to remain empirically observed in this analysis. It is reclassified as a logical remainder.
This reclassification removes an empirically sufficient configuration from conservative minimization. And, since the conservative minimization does not employ simplifying assumptions, the sufficient term FREEXP*PARTY*GENCL can no longer be obtained. After either Bolivia or Comoros is deleted, the conservative solution retains only FREEXP*GENCL*JUCON.
9.2.3 Solution Changes Through Consistency of Sufficiency
The deletions of Guinea and Cameroon operate through a different part of the truth table. Guinea belongs to the five-country configuration FREEXP*~PARTY*~GENCL*~JUCON, and removing it raises that configuration’s consistency of sufficiency from approximately 0.759 to 0.802. Because the reduced configuration now reaches incl.cut = 0.8, its outcome assignment changes from OUT = 0 to OUT = 1. Conservative minimization retains it as the four-condition prime implicant FREEXP*~PARTY*~GENCL*~JUCON, which is added to the two sufficient terms in the baseline formula.
Cameroon belongs to FREEXP*~PARTY*GENCL*~JUCON, whose consistency of sufficiency rises from approximately 0.764 to 0.822 after its deletion. Likewise, this configuration changes from OUT = 0 to OUT = 1. Once it joins the other empirically sufficient configurations, conservative minimization can eliminate both PARTY and JUCON, producing the single sufficient term FREEXP*GENCL.
These two deletions are a useful reminder that, in set theory, the relevance of an observation often lies more in its relationship to its sets than in its sui generis idiosyncrasies. By changing the empirical evidence for the relevant subset relation, one deletion can move a configuration’s consistency of sufficiency across the inclusion cutoff and change what logical minimization can obtain from the truth table.
9.2.4 Solution Changes vs. Parameters of Fit Changes
The fit appraisal identifies more deletions than the formula comparison: 26 of the 66 country deletions change solution consistency, solution PRI, or solution coverage by more than 0.005. Four of those 26 deletions are Bolivia, Guinea, Cameroon, and Comoros, whose removal also changes the formula. The remaining 22 deletions preserve the conservative formula but move at least one of its solution-level parameters beyond the chosen tolerance.
The largest absolute difference is approximately 0.0633 for Cameroon, 0.0480 for Comoros, 0.0455 for Bolivia, and 0.0296 for Guinea. Because the formula also changes in these four reduced analyses, those numerical differences compare the parameters of different sufficient expressions. For the other 22 countries, the parameter differences evaluate the same conservative formula under the complete and reduced case compositions.
9.3 Expanding Across Solution Types
Now let’s enter the counterfactual realm and examine the consequences of deleting one case for the intermediate and parsimonious solutions as well. We keep the calibrated memberships fixed for one more application so that the expanded diagnostic remains a controlled comparison. The expanded application below retains the same 66 deletions and the 0.005 parameter tolerance. It monitors all three reference solution types, selects the C1P1 intermediate branch, and reapplies the established rule for excluding contradictory simplifying assumptions after each country is removed.
vdem_loo_fixed_all <- loo.test(
data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
calib = "fixed",
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "all",
dir.exp = vdem_dir_exp,
exclude_mode = "recompute",
exclude_recompute = list(type = 2),
which_M = 1,
i_mode = "C1P1",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)9.3.1 Reading the Expanded Scan
The same screening framework now identifies deletions that change at least one of the three monitored formulas or move solution consistency, solution PRI, or solution coverage beyond the 0.005 tolerance. In this application, the detailed records confirm that the conservative, parsimonious, and intermediate results remain comparable across all 66 deletions, so the denominator is 66 for each solution type. The case-level table identifies the countries that require attention:
vdem_loo_fixed_all_df <- as.data.frame(vdem_loo_fixed_all)
subset(
vdem_loo_fixed_all_df,
!is.na(solution_change) |
!is.na(fit_changed_types)
) %>%
arrange(desc(max_abs_fit_delta))
#> row_index case_label status solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 45 Cameroon ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.063289509
#> 2 62 Comoros ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.050304006
#> 3 29 Guinea ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.048296465
#> 4 4 Bolivia ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.047623915
#> 5 19 Uganda ok <NA> CON | PAR | INT 6 0.016392674
#> 6 12 India ok <NA> CON | PAR | INT 6 0.013796777
#> 7 59 Togo ok <NA> CON | PAR | INT 3 0.013778383
#> 8 55 Somalia ok PAR:formula_changed CON | PAR | INT 3 0.013453478
#> 9 40 Morocco ok <NA> CON | PAR | INT 6 0.013340333
#> 10 43 Tunisia ok <NA> CON | PAR | INT 6 0.012866719
#> 11 31 Mauritania ok PAR:formula_changed CON | PAR | INT 3 0.012396165
#> 12 53 Madagascar ok <NA> CON | PAR | INT 3 0.010657851
#> 13 6 Mali ok <NA> CON | PAR | INT 6 0.010153970
#> 14 44 Angola ok <NA> CON | PAR | INT 3 0.009849822
#> 15 36 Jordan ok <NA> CON | PAR | INT 6 0.009525978
#> 16 49 Djibouti ok <NA> CON | PAR | INT 3 0.008721696
#> 17 11 Ethiopia ok <NA> CON | PAR | INT 3 0.008612063
#> 18 66 Zanzibar ok <NA> CON | PAR | INT 6 0.008297896
#> 19 51 Kyrgyzstan ok <NA> CON | PAR | INT 6 0.007688506
#> 20 24 Mozambique ok <NA> CON | PAR | INT 3 0.007028247
#> 21 48 Republic of the Congo ok <NA> CON | PAR | INT 3 0.005601515
#> 22 33 Cape Verde ok <NA> CON | INT 2 0.005121705
#> 23 1 Ghana ok <NA> CON | INT 2 0.005118355
#> 24 65 Vanuatu ok <NA> CON | INT 2 0.005088853
#> 25 50 The Gambia ok <NA> CON | INT 2 0.005084229
#> 26 27 Zambia ok <NA> CON | INT 2 0.005014346Deleting Bolivia, Guinea, Cameroon, or Comoros changes all three solution formulas. As the preceding sections showed for the conservative solution, these four deletions alter either the frequency or the empirical outcome assignment of a truth-table configuration before minimization begins. The conservative, parsimonious, and intermediate minimizations thus receive a different truth table, even though they subsequently treat logical remainders in different ways. Deleting Mauritania or Somalia, however, changes only the parsimonious formula.
Once
loo.test()has identified such a case, the complete and reduced analyses stored inby_caseallow us, if needed, to inspect the formula and the treatment of logical remainders that produced the comparison.
For example, the following table extracts that evidence for Mauritania and Somalia:
vdem_loo_fixed_mauritania <- vdem_loo_fixed_all$by_case[["31:Mauritania"]]
vdem_loo_fixed_somalia <- vdem_loo_fixed_all$by_case[["55:Somalia"]]
data.frame(
analysis = c(
"Complete data",
"Without Mauritania",
"Without Somalia"
),
parsimonious_formula = c(
vdem_loo_fixed_all$baseline$sig$parsimonious,
vdem_loo_fixed_mauritania$reduced$sig$parsimonious,
vdem_loo_fixed_somalia$reduced$sig$parsimonious
),
excluded_remainders = c(
"None",
paste(vdem_loo_fixed_mauritania$exclude_reduced, collapse = ", "),
paste(vdem_loo_fixed_somalia$exclude_reduced, collapse = ", ")
)
)
#> analysis parsimonious_formula excluded_remainders
#> 1 Complete data FREEXP*JUCON+FREEXP*PARTY None
#> 2 Without Mauritania FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL 10, 13, 14
#> 3 Without Somalia FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL 10, 13, 14In the complete-data analysis, the recomputed exclusion rule does not identify any contradictory simplifying assumption. After either Mauritania or Somalia is deleted, however, logical remainders 10, 13, and 14 enter the exclusion zone. Their condition-set compositions can be recovered from the reduced truth table:
vdem_loo_fixed_mauritania_excluded_rows <- vdem_loo_fixed_mauritania$exclude_reduced
data.frame(
truth_table_row = vdem_loo_fixed_mauritania_excluded_rows,
vdem_loo_fixed_mauritania$reduced$tt$tt[
vdem_loo_fixed_mauritania_excluded_rows,
vdem_conds,
drop = FALSE
],
row.names = NULL
)
#> truth_table_row FREEXP PARTY GENCL JUCON
#> 1 10 1 0 0 1
#> 2 13 1 1 0 0
#> 3 14 1 1 0 1These are the logical remainders FREEXP*~PARTY*~GENCL*JUCON, FREEXP*PARTY*~GENCL*~JUCON, and FREEXP*PARTY*~GENCL*JUCON. Once they are excluded from parsimonious minimization, GENCL can no longer be cut from the two sufficient terms, so the parsimonious formula changes from FREEXP*JUCON + FREEXP*PARTY to FREEXP*GENCL*JUCON + FREEXP*PARTY*GENCL, which is the same formula obtained by the conservative and C1P1 intermediate minimizations. The conservative and intermediate formulas themselves remain unchanged.
Formula changes do not show how often solution consistency, solution PRI, or solution coverage moves beyond 0.005 for each solution type, so we tabulate those comparisons separately:
table(
vdem_loo_fixed_all_df$fit_changed_types,
useNA = "ifany"
)
#>
#> CON | INT CON | PAR | INT <NA>
#> 5 21 40Deleting 26 of the 66 countries moves at least one of the three solution-level parameters of fit by more than 0.005. For 21 countries, the tolerance is exceeded for every solution type. For five countries, it is exceeded for the conservative and intermediate solutions but not for the parsimonious solution. These countries are Ghana, Zambia, Cape Verde, The Gambia, and Vanuatu:
subset(
vdem_loo_fixed_all_df,
fit_changed_types == "CON | INT",
select = c(
case_label,
fit_changed_types,
n_fit_deltas,
max_abs_fit_delta
)
)
#> case_label fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 Ghana CON | INT 2 0.005118355
#> 27 Zambia CON | INT 2 0.005014346
#> 33 Cape Verde CON | INT 2 0.005121705
#> 50 The Gambia CON | INT 2 0.005084229
#> 65 Vanuatu CON | INT 2 0.0050888539.4 Recomputing Calibration After Every Deletion
As you may have noticed, the two applications above isolate each country’s contribution to the truth table constructed from the established reference memberships. After one case is deleted, the other 65 countries retain exactly the same memberships in the condition and outcome sets. This controlled design answers one kind of question, but it does not rebuild the complete workflow used in the reference analysis from the ground up. In that analysis, QCA::findTh() estimated the qualitative anchors from the empirical distribution of all 66 countries before their fuzzy-set memberships were calculated.
So, if a case’s presence helps determine those data-guided anchors, deleting that observation may change the calibration of the cases that remain. A leave-one-out diagnostic can therefore address a second question: does the selected QCA result survive each country deletion when the original calibration procedure is redone on the reduced data?
Keep in mind, though, that re-estimating the anchors after every deletion only reproduces your original calibration procedure if you were already using QCA::findTh() to estimate those anchors.
9.4.1 Preparing a Recomputable Calibration Specification
With calib = "recompute", the information given to the calib_spec argument constructs the complete-data baseline and every subsequent reduced analysis, and must cover both the outcome and every condition because all five sets have to be reconstructed from raw.data.
The existing vdem_calib_spec_with_outcome already identifies each raw indicator and records its fuzzy direct calibration. We copy that object and add the same four-group threshold search used in the reference analysis:
vdem_loo_recal_spec <- vdem_calib_spec_with_outcome
vdem_loo_recal_spec$CIVSOC$findTh <- list(groups = 4)
vdem_loo_recal_spec$FREEXP$findTh <- list(groups = 4)
vdem_loo_recal_spec$PARTY$findTh <- list(groups = 4)
vdem_loo_recal_spec$GENCL$findTh <- list(groups = 4)
vdem_loo_recal_spec$JUCON$findTh <- list(groups = 4)With the findTh entry, loo.test() estimates three thresholds from the complete raw dataset for the baseline and re-estimates them after each country is deleted. Because the complete-data search repeats the procedure used in the reference chapter, it reproduces the reference anchors and solution formulas before any deletion is tested.
9.4.2 Running the Recomputed Diagnostic
Recomputed calibration requires both the calibrated and raw datasets, which must represent the same cases in the same order so that each diagnostic run deletes the same observation from both objects.
vdem_loo_recomputed_all <- loo.test(
data = vdem_calib,
raw.data = vdem_raw,
outcome = vdem_outcome,
conditions = vdem_conds,
calib = "recompute",
calib_spec = vdem_loo_recal_spec,
incl.cut = vdem_incl_cut,
n.cut = vdem_n_cut,
solution = "all",
dir.exp = vdem_dir_exp,
exclude_mode = "recompute",
exclude_recompute = list(type = 2),
which_M = 1,
i_mode = "C1P1",
fit_measures = c("inclS", "PRI", "covS"),
fit_tol = 0.005,
complete = TRUE,
sort.by = c("incl", "n")
)Printing the object displays the 66 case-deletion comparisons and concludes with the principal counts:
vdem_loo_recomputed_all
#> <loo_test>
#> Outcome test: leave-one-out robustness
#> Solution: all
#> Monitored solutions: conservative, parsimonious, intermediate
#> which_M: 1
#> i_mode: C1P1
#> Cases tested: 66
#> Fit measures: inclS, PRI, covS
#>
#> Result
#> row_index case_label status solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 Ghana ok <NA> CON | PAR | INT 6 0.008673969
#> 2 Egypt ok <NA> CON | INT 2 0.019298200
#> 3 Yemen ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.098761229
#> 4 Bolivia ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.047623915
#> 5 Honduras ok <NA> <NA> 0 0.000000000
#> 6 Mali ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.096886874
#> 7 Pakistan ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 8 0.051339487
#> 8 Senegal ok <NA> <NA> 0 0.000000000
#> 9 South Sudan ok <NA> <NA> 0 0.000000000
#> 10 Vietnam ok PAR:formula_changed CON | PAR | INT 3 0.015394718
#> 11 Ethiopia ok <NA> CON | PAR | INT 3 0.010664242
#> 12 India ok <NA> CON | PAR | INT 6 0.010188229
#> 13 Kenya ok <NA> <NA> 0 0.000000000
#> 14 North Korea ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 6 0.032644720
#> 15 Lebanon ok <NA> <NA> 0 0.000000000
#> 16 Nigeria ok <NA> <NA> 0 0.000000000
#> 17 Philippines ok <NA> CON | PAR | INT 6 0.012212892
#> 18 Tanzania ok <NA> <NA> 0 0.000000000
#> 19 Uganda ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.097070240
#> 20 Benin ok <NA> CON | PAR | INT 9 0.025713337
#> 21 Bhutan ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.084374847
#> 22 Burkina Faso ok PAR:formula_changed CON | PAR | INT 9 0.032438746
#> 23 Cambodia ok <NA> <NA> 0 0.000000000
#> 24 Mozambique ok <NA> CON | PAR | INT 3 0.007028247
#> 25 Nepal ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.096784276
#> 26 Nicaragua ok <NA> <NA> 0 0.000000000
#> 27 Zambia ok <NA> <NA> 0 0.000000000
#> 28 Zimbabwe ok PAR:formula_changed CON | PAR | INT 8 0.019052179
#> 29 Guinea ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.083405122
#> 30 Ivory Coast ok PAR:formula_changed CON | PAR | INT 9 0.028984461
#> 31 Mauritania ok PAR:formula_changed CON | PAR | INT 3 0.012396165
#> 32 Burundi ok <NA> <NA> 0 0.000000000
#> 33 Cape Verde ok <NA> CON | PAR | INT 6 0.008677371
#> 34 Central African Republic ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 3 0.112748167
#> 35 Timor-Leste ok <NA> <NA> 0 0.000000000
#> 36 Jordan ok <NA> CON | PAR | INT 6 0.010469422
#> 37 Lesotho ok <NA> CON | PAR | INT 6 0.008403243
#> 38 Liberia ok <NA> <NA> 0 0.000000000
#> 39 Malawi ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 8 0.098741250
#> 40 Morocco ok <NA> CON | PAR | INT 9 0.053212054
#> 41 Papua New Guinea ok <NA> CON | INT 2 0.011891501
#> 42 Sierra Leone ok <NA> CON | INT 2 0.005115560
#> 43 Tunisia ok <NA> CON | PAR | INT 9 0.038554087
#> 44 Angola ok <NA> CON | PAR | INT 9 0.046356515
#> 45 Cameroon ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.063289509
#> 46 Chad ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.062290326
#> 47 Democratic Republic of the Congo ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.096839449
#> 48 Republic of the Congo ok <NA> CON | PAR | INT 3 0.017607146
#> 49 Djibouti ok <NA> CON | PAR | INT 3 0.009081353
#> 50 The Gambia ok <NA> CON | PAR | INT 6 0.007619987
#> 51 Kyrgyzstan ok <NA> CON | PAR | INT 9 0.034668911
#> 52 Laos ok <NA> <NA> 0 0.000000000
#> 53 Madagascar ok <NA> CON | PAR | INT 3 0.011152552
#> 54 Rwanda ok <NA> <NA> 0 0.000000000
#> 55 Somalia ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.096978434
#> 56 Sri Lanka ok <NA> CON | PAR | INT 9 0.046726805
#> 57 Eswatini ok <NA> CON | PAR | INT 9 0.035927897
#> 58 Tajikistan ok <NA> <NA> 0 0.000000000
#> 59 Togo ok <NA> CON | PAR | INT 9 0.034040313
#> 60 Somaliland ok <NA> CON | PAR | INT 9 0.045541638
#> 61 Uzbekistan ok PAR:formula_changed CON | INT 2 0.019501722
#> 62 Comoros ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.030526445
#> 63 Sao Tome and Principe ok <NA> CON | PAR | INT 5 0.008938666
#> 64 Solomon Islands ok <NA> <NA> 0 0.000000000
#> 65 Vanuatu ok <NA> CON | INT 2 0.005088853
#> 66 Zanzibar ok CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.097881934
#>
#> Summary
#> Successful runs: 66/66
#> Solution changes: 23
#> Fit changes: 49
#> Non-ok runs: 0
#> Cases with solution changes: Yemen, Bolivia, Mali, Pakistan, Vietnam, North Korea, Uganda, Bhutan, Burkina Faso, Nepal, Zimbabwe, Guinea, Ivory Coast, Mauritania, Central African Republic, Malawi, Cameroon, Chad, Democratic Republic of the Congo, Somalia, Uzbekistan, Comoros, Zanzibar9.4.3 Reading the Recomputed Diagnostic
Because deleting a country may also change the memberships assigned to the countries that remain, a changed formula or parameter of fit can arise from removing the country, re-estimating the qualitative anchors, or both. All three solution types remain comparable across all 66 deletions.
The printed vdem_loo_recomputed_all reports that 23 deletions change at least one solution formula. To see how those changes are distributed across the solution types—and which countries produce each pattern—we can tabulate and display the solution_change column:
vdem_loo_recomputed_all_df <- as.data.frame(vdem_loo_recomputed_all)
table(
vdem_loo_recomputed_all_df$solution_change,
useNA = "ifany"
)
#>
#> CON:formula_changed | PAR:formula_changed | INT:formula_changed PAR:formula_changed <NA>
#> 17 6 43
subset(
vdem_loo_recomputed_all_df,
!is.na(solution_change),
select = c(case_label, solution_change)
)
#> case_label solution_change
#> 3 Yemen CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 4 Bolivia CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 6 Mali CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 7 Pakistan CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 10 Vietnam PAR:formula_changed
#> 14 North Korea CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 19 Uganda CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 21 Bhutan CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 22 Burkina Faso PAR:formula_changed
#> 25 Nepal CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 28 Zimbabwe PAR:formula_changed
#> 29 Guinea CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 30 Ivory Coast PAR:formula_changed
#> 31 Mauritania PAR:formula_changed
#> 34 Central African Republic CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 39 Malawi CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 45 Cameroon CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 46 Chad CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 47 Democratic Republic of the Congo CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 55 Somalia CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 61 Uzbekistan PAR:formula_changed
#> 62 Comoros CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 66 Zanzibar CON:formula_changed | PAR:formula_changed | INT:formula_changedSeventeen deletions change the conservative, parsimonious, and intermediate formulas: Yemen, Bolivia, Mali, Pakistan, North Korea, Uganda, Bhutan, Nepal, Guinea, the Central African Republic, Malawi, Cameroon, Chad, the Democratic Republic of the Congo, Somalia, Comoros, and Zanzibar.
The remaining six—Vietnam, Burkina Faso, Zimbabwe, Ivory Coast, Mauritania, and Uzbekistan—change only the parsimonious formula, while the other 43 deletions preserve all three formulas.
We now examine how often recalibration moves solution consistency, solution PRI, or solution coverage beyond 0.005 for each solution type:
table(
vdem_loo_recomputed_all_df$fit_changed_types,
useNA = "ifany"
)
#>
#> CON | INT CON | PAR | INT <NA>
#> 5 44 17
subset(
vdem_loo_recomputed_all_df,
fit_changed_types == "CON | INT",
select = c(
case_label,
fit_changed_types,
n_fit_deltas,
max_abs_fit_delta
)
)
#> case_label fit_changed_types n_fit_deltas max_abs_fit_delta
#> 2 Egypt CON | INT 2 0.019298200
#> 41 Papua New Guinea CON | INT 2 0.011891501
#> 42 Sierra Leone CON | INT 2 0.005115560
#> 61 Uzbekistan CON | INT 2 0.019501722
#> 65 Vanuatu CON | INT 2 0.005088853Deleting 49 of the 66 countries while recalibrating the remaining observations moves solution consistency, solution PRI, or solution coverage by more than 0.005 for at least one solution type. Among those 49 deletions, 44 move at least one of the three parameters beyond the tolerance for every solution type. After deleting Egypt, Papua New Guinea, Sierra Leone, Uzbekistan, or Vanuatu, at least one parameter exceeds the tolerance for the conservative and intermediate solutions but not for the parsimonious.
Before we take a closer look at one of these deletions, a word of reassurance is in order: none of the additional appraisal from this point onward is required for the ordinary use of
loo.test(). In most applications, inspecting the main object is enough to identify which country deletions change a monitored solution formula or move solution consistency, solution PRI, or solution coverage beyond the chosen tolerance.We go further here for didactic reasons, showing how selected changes can be traced through calibration, truth-table construction, and minimization. When such an in-depth examination is of interest, evidence concerning formula preservation should be presented alongside evidence concerning changes in the solution-level parameters of fit, always identifying the solution type to which each comparison refers. But unless you are a nerd writing an entire chapter on leave-one-out robustness checks, that might be a bit excessive for the usual QCA workflow.
9.4.3.1 Following One Recalibration Mechanism
Yemen provides a nice, clear example of what happens inside one of these 66 recalibrated deletions. We can begin with the anchors estimated from the complete data and those re-estimated after Yemen is deleted:
vdem_loo_sets <- c(vdem_outcome, vdem_conds)
vdem_loo_recomputed_yemen <- vdem_loo_recomputed_all$by_case[["3:Yemen"]]
vdem_yemen_complete_anchors <- do.call(
rbind,
vdem_loo_recomputed_yemen$calibration_baseline$
thresholds_used
)
vdem_yemen_reduced_anchors <- do.call(
rbind,
vdem_loo_recomputed_yemen$calibration_reduced$
thresholds_used
)
data.frame(
set = vdem_loo_sets,
complete_E = vdem_yemen_complete_anchors[vdem_loo_sets, "E"],
complete_C = vdem_yemen_complete_anchors[vdem_loo_sets, "C"],
complete_I = vdem_yemen_complete_anchors[vdem_loo_sets, "I"],
without_Yemen_E = vdem_yemen_reduced_anchors[vdem_loo_sets, "E"],
without_Yemen_C = vdem_yemen_reduced_anchors[vdem_loo_sets, "C"],
without_Yemen_I = vdem_yemen_reduced_anchors[vdem_loo_sets, "I"],
row.names = NULL
)
#> set complete_E complete_C complete_I without_Yemen_E without_Yemen_C without_Yemen_I
#> 1 CIVSOC 0.1670 0.5275 0.7590 0.1670 0.4080 0.7590
#> 2 FREEXP 0.3055 0.4385 0.7450 0.3055 0.4385 0.7450
#> 3 PARTY 0.2385 0.4890 0.6840 0.2385 0.4890 0.6840
#> 4 GENCL 0.0900 0.3925 0.7155 0.0900 0.3925 0.7155
#> 5 JUCON 0.1370 0.3605 0.6965 0.1370 0.3605 0.6965The full-nonmembership and full-membership anchors for the outcome CIVSOC remain at 0.167 and 0.759, but its crossover moves from 0.5275 to 0.408. The four condition sets reproduce all three complete-data anchors, so their calibrated memberships—and therefore the retained countries’ assignments to truth-table configurations—remain unchanged. It follows that the recalibrated outcome membership changes the empirical evidence for the sufficiency of those configurations for CIVSOC.
The relevant complete-data and reduced-data truth-table configurations can now be compared directly:
vdem_yemen_complete_tt <- vdem_loo_recomputed_yemen$baseline$tt$tt
vdem_yemen_reduced_tt <- vdem_loo_recomputed_yemen$reduced$tt$tt
vdem_yemen_complete_rows <- with(
vdem_yemen_complete_tt,
FREEXP == 1 &
PARTY == 0 &
JUCON == 0 &
GENCL %in% c(0, 1)
)
vdem_yemen_reduced_rows <- with(
vdem_yemen_reduced_tt,
FREEXP == 1 &
PARTY == 0 &
JUCON == 0 &
GENCL %in% c(0, 1)
)
vdem_yemen_tt_comparison <- rbind(
data.frame(
analysis = "Complete data",
vdem_yemen_complete_tt[
vdem_yemen_complete_rows,
c(vdem_conds, "n", "incl", "PRI", "OUT"),
drop = FALSE
],
row.names = NULL
),
data.frame(
analysis = "Without Yemen",
vdem_yemen_reduced_tt[
vdem_yemen_reduced_rows,
c(vdem_conds, "n", "incl", "PRI", "OUT"),
drop = FALSE
],
row.names = NULL
)
)
vdem_yemen_tt_comparison
#> analysis FREEXP PARTY GENCL JUCON n incl PRI OUT
#> 1 Complete data 1 0 0 0 5 0.758526742201357 0.400859177785658 0
#> 2 Complete data 1 0 1 0 6 0.763868402990055 0.542457565776731 0
#> 3 Without Yemen 1 0 0 0 5 0.87337471447803 0.659797509271454 1
#> 4 Without Yemen 1 0 1 0 6 0.827477399503519 0.658052474914983 1The configuration FREEXP*~PARTY*~GENCL*~JUCON retains its five countries, but its consistency of sufficiency rises from approximately 0.759 to 0.873 and its PRI rises from 0.401 to 0.660. The six-country configuration FREEXP*~PARTY*GENCL*~JUCON likewise retains its frequency while its consistency rises from approximately 0.764 to 0.827 and its PRI rises from 0.542 to 0.658. Both configurations therefore change from OUT = 0 to OUT = 1 at the established inclusion cutoff of 0.8.
Finally, the selected formulas stored for the complete and reduced analyses show what the three minimizations obtain from those changed outcome assignments:
data.frame(
analysis = c("Complete data", "Without Yemen"),
conservative = c(
vdem_loo_recomputed_yemen$baseline$sig$conservative,
vdem_loo_recomputed_yemen$reduced$sig$conservative
),
parsimonious = c(
vdem_loo_recomputed_yemen$baseline$sig$parsimonious,
vdem_loo_recomputed_yemen$reduced$sig$parsimonious
),
intermediate = c(
vdem_loo_recomputed_yemen$baseline$sig$intermediate,
vdem_loo_recomputed_yemen$reduced$sig$intermediate
)
)
#> analysis conservative parsimonious intermediate
#> 1 Complete data FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL FREEXP*JUCON+FREEXP*PARTY C1P1:FREEXP*GENCL*JUCON+FREEXP*PARTY*GENCL
#> 2 Without Yemen FREEXP*GENCL+FREEXP*~PARTY*~JUCON FREEXP C1P1:FREEXPThe conservative formula becomes FREEXP*GENCL + FREEXP*~PARTY*~JUCON, while the parsimonious and C1P1 intermediate formulas both reduce to FREEXP. Yemen therefore demonstrates the additional mechanism introduced by calib = "recompute": deleting one country can change a data-guided qualitative anchor, which changes outcome memberships among the retained countries, changes the empirical sufficiency of truth-table configurations, and ultimately changes the selected solution formulas.
9.4.4 Taking a gander on fixed vs. recomputed loo
Now we can finally contrast the fixed and recomputed diagnostics. Because both examine the same 66 country deletions, their formula-change classifications can be compared directly. The first table below gives the four possible combinations; the second isolates the six countries whose deletion changes at least one formula while calibration is held fixed:
vdem_loo_fixed_formulas <- vdem_loo_fixed_all_df %>%
select(
case_label,
fixed_formula = solution_change
)
vdem_loo_recomputed_formulas <- vdem_loo_recomputed_all_df %>%
select(
case_label,
recomputed_formula = solution_change
)
vdem_loo_formula_comparison <- left_join(
vdem_loo_fixed_formulas,
vdem_loo_recomputed_formulas,
by = "case_label"
)
table(
fixed = !is.na(vdem_loo_formula_comparison$fixed_formula),
recomputed = !is.na(vdem_loo_formula_comparison$recomputed_formula)
)
#> recomputed
#> fixed FALSE TRUE
#> FALSE 43 17
#> TRUE 0 6
subset(
vdem_loo_formula_comparison,
!is.na(fixed_formula)
)
#> case_label fixed_formula recomputed_formula
#> 4 Bolivia CON:formula_changed | PAR:formula_changed | INT:formula_changed CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 29 Guinea CON:formula_changed | PAR:formula_changed | INT:formula_changed CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 31 Mauritania PAR:formula_changed PAR:formula_changed
#> 45 Cameroon CON:formula_changed | PAR:formula_changed | INT:formula_changed CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 55 Somalia PAR:formula_changed CON:formula_changed | PAR:formula_changed | INT:formula_changed
#> 62 Comoros CON:formula_changed | PAR:formula_changed | INT:formula_changed CON:formula_changed | PAR:formula_changed | INT:formula_changedAll six deletions that change at least one formula under fixed calibration also change at least one formula when calibration is recomputed. Bolivia, Guinea, Cameroon, and Comoros change all three formulas under both designs, while Mauritania changes only the parsimonious formula under both. As for Somalia, deleting the country changes only the parsimonious formula under fixed calibration but all three formulas when the anchors are re-estimated. The comparison also identifies seventeen additional cases whose deletion preserves all three formulas under fixed calibration but changes at least one formula under recomputed calibration.
The difference is just as pronounced for the solution-level parameters of fit. Under fixed calibration, deleting 26 of the 66 countries moves solution consistency, solution PRI, or solution coverage by more than 0.005 for at least one solution type. When the anchors are re-estimated after every deletion, that number rises to 49.
Yemen brings both contrasts together in a single case:
vdem_loo_yemen_designs <- bind_rows(
Fixed = subset(
vdem_loo_fixed_all_df,
case_label == "Yemen"
),
Recomputed = subset(
vdem_loo_recomputed_all_df,
case_label == "Yemen"
),
.id = "calibration"
)
subset(
vdem_loo_yemen_designs,
select = c(
"calibration",
"case_label",
"solution_change",
"fit_changed_types",
"n_fit_deltas",
"max_abs_fit_delta"
)
)
#> calibration case_label solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 Fixed Yemen <NA> <NA> 0 0.00000000
#> 2 Recomputed Yemen CON:formula_changed | PAR:formula_changed | INT:formula_changed CON | PAR | INT 9 0.09876123With calibration held fixed, deleting Yemen preserves all three solution formulas and keeps solution consistency, solution PRI, and solution coverage within the 0.005 tolerance for every solution type. When calibration is recomputed, however, deleting Yemen changes all three formulas, and the largest absolute difference among the monitored solution-level parameters of fit is approximately 0.099.
9.5 Interpretation and Reporting
In a nutshell, a leave-one-out report identifies the population and number of cases tested, states whether calibration was fixed or recomputed, names the monitored solution types, and specifies the criteria used to assess formula preservation and changes in solution-level fit parameters. Reporting the overall run status distinguishes attempted deletions from those that retain at least one comparable solution type; when several solution types are monitored, the report presents any type-specific failures separately rather than hiding them behind an overall status = "ok". It then identifies the countries whose deletion changes each solution formula, summarizes how many deletions shift solution consistency, PRI, or coverage beyond fit_tol, and—if a more in-depth analysis is warranted—explains the truth-table mechanism behind the formula changes that matter to the substantive argument. When calibration is recomputed, the report also makes clear that the diagnostic combines case deletion with recalibration of the retained cases.
Accordingly, the fixed-calibration evidence is local to 66 one-country deletions from the reference memberships, whereas the recomputed-calibration evidence is local to 66 combined perturbations in which each deletion is followed by the reference four-group threshold search.
As we have seen, a formula- or fit-changing deletion establishes only that removing the named country changes the selected formula or parameter of fit under the stated diagnostic design, but does not establish, by itself, that the country is deviant, improperly selected, or dispensable. The function has identified an observation worth reporting and, if necessary, understanding, not a case that should be treated as an outlier in the statistical sense.
The next chapter broadens the case-composition perturbation rather brutally. Instead of removing one case at a time, it repeatedly holds out several observations together and examines how often the complete-data formulas and their solution-level consistency, PRI, and coverage reappear across sampled case compositions. For most applications, this could be deemed overkill. Still, qcaERT offers it as an option for advanced robustness appraisal.