vdem_calib_spec
#> $FREEXP
#> $FREEXP$raw
#> [1] "FREEXP"
#>
#> $FREEXP$type
#> [1] "fuzzy"
#>
#> $FREEXP$method
#> [1] "direct"
#>
#> $FREEXP$thresholds
#> [1] 0.3055 0.4385 0.7450
#>
#>
#> $PARTY
#> $PARTY$raw
#> [1] "PARTY"
#>
#> $PARTY$type
#> [1] "fuzzy"
#>
#> $PARTY$method
#> [1] "direct"
#>
#> $PARTY$thresholds
#> [1] 0.2385 0.4890 0.6840
#>
#>
#> $GENCL
#> $GENCL$raw
#> [1] "GENCL"
#>
#> $GENCL$type
#> [1] "fuzzy"
#>
#> $GENCL$method
#> [1] "direct"
#>
#> $GENCL$thresholds
#> [1] 0.0900 0.3925 0.7155
#>
#>
#> $JUCON
#> $JUCON$raw
#> [1] "JUCON"
#>
#> $JUCON$type
#> [1] "fuzzy"
#>
#> $JUCON$method
#> [1] "direct"
#>
#> $JUCON$thresholds
#> [1] 0.1370 0.3605 0.6965
vdem_calib_spec_with_outcome
#> $FREEXP
#> $FREEXP$raw
#> [1] "FREEXP"
#>
#> $FREEXP$type
#> [1] "fuzzy"
#>
#> $FREEXP$method
#> [1] "direct"
#>
#> $FREEXP$thresholds
#> [1] 0.3055 0.4385 0.7450
#>
#>
#> $PARTY
#> $PARTY$raw
#> [1] "PARTY"
#>
#> $PARTY$type
#> [1] "fuzzy"
#>
#> $PARTY$method
#> [1] "direct"
#>
#> $PARTY$thresholds
#> [1] 0.2385 0.4890 0.6840
#>
#>
#> $GENCL
#> $GENCL$raw
#> [1] "GENCL"
#>
#> $GENCL$type
#> [1] "fuzzy"
#>
#> $GENCL$method
#> [1] "direct"
#>
#> $GENCL$thresholds
#> [1] 0.0900 0.3925 0.7155
#>
#>
#> $JUCON
#> $JUCON$raw
#> [1] "JUCON"
#>
#> $JUCON$type
#> [1] "fuzzy"
#>
#> $JUCON$method
#> [1] "direct"
#>
#> $JUCON$thresholds
#> [1] 0.1370 0.3605 0.6965
#>
#>
#> $CIVSOC
#> $CIVSOC$raw
#> [1] "CIVSOC"
#>
#> $CIVSOC$type
#> [1] "fuzzy"
#>
#> $CIVSOC$method
#> [1] "direct"
#>
#> $CIVSOC$thresholds
#> [1] 0.1670 0.5275 0.75908 Alternative-Set Diagnostics
The preceding diagnostics move one family of analytical choices at a time, which is useful when we want to locate a calibration or truth-table boundary and identify which decision produced a formula change. In an actual research project, however, things are a bit messier, and plausible alternatives do not always arrive so politely separated. A researcher may wish to reconsider several calibration anchors while also allowing the inclusion and frequency cutoffs to vary.
The altset.test() function examines these joint changes. Each draw defines a complete alternative specification: one admissible calibration for every selected condition or outcome set, one inclusion cutoff, and one frequency cutoff. The function then reconstructs the QCA analysis under that specification and compares the selected solution formulas and their solution-level fit parameters with those of the baseline analysis.
The analytical unit is therefore the sampled specification rather than a previously defined alternative solution. Each record identifies the analytical settings sampled and shows whether the corresponding QCA analysis produced formulas that can be compared with their baseline counterparts. Because some specifications produce only a subset of the requested solution types, the comparisons that follow must remain tied to the complete sampled specification.
8.1 From Combinatorial Explosion to a Sampled Alternative Set
The calibration chapter showed why an exhaustive joint search quickly becomes impractical. Once several anchors can each take several values, the complete Cartesian product can contain millions of calibration specifications before alternative inclusion and frequency cutoffs are added. altset.test() addresses that computational problem by sampling from a bounded domain chosen by the researcher instead of evaluating every possible combination.
Here, random sampling has a very specific purpose. It is not being used to manufacture comparable cases or to replace a substantive research decision! It selects a manageable collection of joint analytical specifications from a domain too large to enumerate. Sampling keeps the combinatorial kaiju under control; it does not, regrettably, decide which alternatives deserve admission.
That second responsibility remains with the researcher. The alternative set should be built from calibration movements and cutoff values that are plausible enough to constitute meaningful challenges to the baseline analysis. An excessively narrow domain can make preservation nearly inevitable, while an indiscriminately broad domain can generate dramatic changes from specifications that no informed analyst would defend. The match rates produced later in the chapter inherit the substantive quality—and the weighting—of the domain from which the specifications were sampled.
8.2 Preparing the Baseline and the Alternative Calibrations
The function needs both the raw empirical data and the calibrated analysis data, but the two objects have different roles. It is the memberships already stored in calib.data that define the baseline QCA analysis, and altset.test() constructs the baseline truth table directly from those memberships, using the supplied baseline inclusion and frequency cutoffs.
But the alternative calibrations are generated differently. For every selected set, the function takes the empirical values from raw.data, begins from the thresholds specified in calib_spec, moves the eligible anchors, and calculates new membership scores. Before constructing the baseline analysis, altset.test() therefore reconstructs each tested condition or outcome from raw.data and calib_spec and verifies that those memberships match the corresponding column in calib.data. A mismatch stops the diagnostic, preventing sampled calibrations centered on one set of qualitative anchors from being compared with a reference formula built from another. Conditions that are not selected for perturbation retain the memberships already stored in calib.data and are not included in this equality check.
As in the calibration diagnostic, raw.data and calib.data must represent the same cases in the same order. Informative row names must be unique and identical across the two objects. If both data frames use only the default row names, their alignment is positional, so they must not be reordered independently.
The reference analysis has already created the two calibration specifications used in this chapter:
vdem_calib_spec contains the calibration instructions for the four conditions. The second object adds the calibration of CIVSOC, which is required when the outcome anchors are also eligible to move. The calibration specification remains complete, even when only one condition is selected for perturbation. Within a draw, however, only the sets selected through test.conditions and test.outcome are recalibrated; the memberships of the untested sets remain as stored in calib.data.
The arguments test.conditions, test.outcome, and anchors_to_test retain the meanings established in the calibration section, but here they define the dimensions of a joint specification, since we are not interested in separate one-at-a-time paths. For each tested set, one draw samples a displacement for every eligible anchor. An anchor may remain at its reference threshold when its sampled displacement is zero, and a tested set may therefore remain unchanged in a particular draw.
8.3 Defining the Candidate Domain
The alternative set is built by selecting one admissible value from each of three candidate domains: calibration movements, inclusion cutoffs, and frequency cutoffs.
For calibration, unit_step and unit_step_divisor retain the scale-aware movement rules already taught. The new argument calib_max_steps sets the largest absolute number of step units by which an eligible anchor may move from its reference threshold. It must be a positive integer and defaults to 20. The function retains only calibrations whose thresholds remain in the order required by the calibration method and within the observed range of the corresponding raw measure.
The inclusion-cutoff candidates are constructed around the baseline incl.cut as
\[ incl.cut + k * incl\_step \]
where the integer \((k)\) runs from -incl_max_steps to incl_max_steps. incl_step defaults to 0.01, and incl_max_steps defaults to 20; values outside the admissible interval from 0 to 1 are removed. The resulting grid contains the inclusion cutoffs from which each draw samples one value.
The frequency-cutoff grid follows the same logic:
\[ n.cut + k * ncut\_step \]
Both ncut_step, which defaults to 1, and ncut_max_steps, which defaults to 20, are positive integers. Candidate values below 1 or above the number of cases in calib.data are removed, after which each draw samples one admissible frequency cutoff. The baseline inclusion and frequency cutoffs remain among the candidates whenever they fall within their respective admissible grids.
A draw may keep any one of these dimensions at its baseline setting. What altset.test() rejects is the complete baseline specification, in which all sampled calibrations and both truth-table cutoffs remain unchanged simultaneously. Sampling is performed with replacement, so different draw numbers may represent the same alternative specification. Accordingly, n_draws counts attempted draws, not necessarily distinct specifications.
n_draws must be a positive integer and defaults to 100. Increasing it provides more opportunities to cover the user-defined domain, but it also requires more complete QCA analyses and does not prevent repeated specifications. Because the specifications are sampled randomly, an ordinary research workflow should also supply seed. Its default is NULL, which follows the current state of R’s random-number generator; so a fixed integer is required to make the realized sample reproducible.
The resulting match rates summarize those realized draws. They are not exact proportions of the complete Cartesian product, and they certainly are not probabilities that the substantive argument is correct.
8.4 Deciding What Counts as Preservation
Every sampled specification is evaluated under the same solution-selection and minimization rules used for the baseline. The selected solution type, model position, intermediate branch, directional expectations, and treatment of simplifying assumptions are held constant so that an analytical perturbation is not confused with a changed comparison rule.
Formula preservation is assessed separately for each monitored solution type, meaning that exact Boolean identity is the criterion used here. The selected conservative formula in a draw is compared with the selected baseline conservative formula, the parsimonious formula with its own baseline counterpart, and the intermediate formula with the same selected intermediate branch in the baseline analysis.
The comparison of solution-level consistency, PRI, and coverage is conditional on a common selected formula. When the formula changes, its parameters of fit no longer evaluate the same sufficient expression, so altset.test() does not treat those values as a like-for-like comparison. Under the C1P1 branch selection used in this chapter, a parameter-of-fit comparison is therefore available when the corresponding selected formula is preserved.
fit_tol defines how much absolute movement in each solution-level parameter is accepted as preservation. It must be non-negative and defaults to 1e-6, which permits only negligible numerical differences. A tolerance of 0.05, used in the examples below, allows an absolute difference of up to five hundredths in solution consistency, solution PRI, and solution coverage.
The denominators require extra care. One sampled specification may successfully reproduce the conservative solution while exclusion recomputation or minimization prevents a parsimonious or intermediate comparison. For this reason, the evidence has several nested levels:
- Attempted specifications are all requested draws, including failures.
- Comparable specifications for a solution type are the draws in which that selected solution can be compared with its baseline counterpart.
- Formula-preserving specifications are the comparable draws that reproduce the selected baseline formula.
- Parameter-of-fit comparisons are the draws in which the same selected formula is available in both analyses.
The status ok means that at least one monitored solution type remains comparable. It does not indicate that every requested minimization succeeded, and it says nothing by itself about formula or parameter-of-fit preservation.
The printed summary provides the initial overview at both the draw and solution-type levels. It distinguishes draws that produce comparable formulas for every monitored solution type, draws that produce only some of them, and draws that produce no comparable formula. For each solution type, it then reports how many draws permit a formula comparison, how many preserve the baseline formula, how many permit a comparison of solution consistency, PRI, and coverage, and how many preserve all three parameters within the selected tolerance. The solution-type-specific fields retained in diagnostics become necessary only when we want to identify the particular draws behind those counts or investigate why a comparison could not be completed. The later chapter on returned objects shows how to perform that closer inspection without turning this chapter into an inventory of stored components.
8.5 A Focused First Application
The first diagnostic narrows the alternative set so that its draws remain easy to assess individually. Only the calibration of GENCL may change, and only the conservative solution is monitored. This remains a joint diagnostic because a draw may move several GENCL anchors and simultaneously select a different inclusion cutoff, frequency cutoff, or both.
Each GENCL anchor may move by at most five automatically calculated step units. The candidate inclusion cutoffs range from 0.70 to 0.90, and the admissible frequency cutoffs range from 1 to 4. From this domain, the function samples 25 specifications under seed 123.
The tolerance of 0.05 allows us to distinguish exact formula preservation from preservation of solution consistency, PRI, and coverage within a clearly visible margin.
vdem_alt_gencl_con <- altset.test(
raw.data = vdem_raw,
calib.data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
calib_spec = vdem_calib_spec,
# Joint calibration and cutoff domain
test.conditions = "GENCL",
anchors_to_test = c("E", "C", "I"),
unit_step = NULL,
unit_step_divisor = 10,
calib_max_steps = 5,
incl.cut = vdem_incl_cut,
incl_step = 0.01,
incl_max_steps = 10,
n.cut = vdem_n_cut,
ncut_step = 1,
ncut_max_steps = 2,
n_draws = 25,
seed = 123,
# Solution selection and comparison
solution = "conservative",
fit_tol = 0.05,
complete = TRUE,
sort.by = c("incl", "n")
)8.6 Reading Draw-Level Evidence
vdem_alt_gencl_con
#> <altset_test>
#> Outcome test: alternative-set robustness
#> Solution: conservative
#> Monitored solutions: conservative
#> which_M: 1
#> Draws: 25
#> Baseline incl.cut: 0.8
#> Baseline n.cut: 2
#> Fit tolerance: 0.05
#> Test conditions: GENCL
#>
#> Result
#> draw incl.cut n.cut status n_changed_sets changed_sets changed_roles solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 1 0.84 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 2 0.87 2 ok 1 GENCL condition <NA> <NA> 0 0.0107657449
#> 3 0.83 2 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 4 0.74 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 5 0.78 1 ok 1 GENCL condition <NA> <NA> 0 0.0348254555
#> 6 0.76 2 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 7 0.83 1 ok 1 GENCL condition <NA> <NA> 0 0.0014069338
#> 8 0.81 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 9 0.78 1 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 10 0.90 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 11 0.71 1 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 12 0.70 1 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 13 0.90 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 14 0.75 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 15 0.76 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 16 0.73 1 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 17 0.89 2 ok 1 GENCL condition <NA> <NA> 0 0.0215746354
#> 18 0.72 4 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 19 0.83 1 ok 1 GENCL condition <NA> <NA> 0 0.0008378074
#> 20 0.84 1 ok 1 GENCL condition <NA> <NA> 0 0.0039204003
#> 21 0.79 2 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 22 0.81 4 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 23 0.72 4 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 24 0.86 3 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#> 25 0.75 2 ok 1 GENCL condition CON:formula_changed <NA> NA NA
#>
#> Summary
#>
#> Draws requested: 25
#>
#> Draw status
#> status draws
#> Comparable solution 25/25
#> No comparable solution 0/25
#>
#> Solution-formula preservation
#> solution_type successful_draws formula_preserved match_rate
#> conservative 25/25 6/25 24.0%
#>
#> Parameters-of-fit preservation
#> solution_type successful_draws fit_compared fit_preserved match_rate
#> conservative 25/25 6/25 6/6 100.0%
#>
#> Fit match rates are conditional on fit being compared.
#> Draws without a matching solution formula are excluded from those rates.Each record in the compact table represents one sampled specification. incl.cut and n.cut identify its truth-table cutoffs. The calibration fields report how many tested sets changed, name those sets, and distinguish changed condition calibrations from a changed outcome calibration. In this first application, n_changed_sets = 1 and changed_sets = GENCL mean that at least one GENCL anchor moved, not that every eligible anchor moved.
The status field tells us whether the selected conservative analysis remains available for comparison. solution_change reports a changed conservative formula; a missing entry means that the formula actually compared was not classified as changed. The missing entry should never be interpreted without checking status, since the absence of a change label cannot rescue a failed comparison.
The last three fields concern solution-level consistency, PRI, and coverage for formulas present in both the baseline and the draw. fit_changed_types identifies the solution types for which at least one of those parameters exceeds the tolerance. n_fit_deltas counts the individual parameter differences that exceed it, while max_abs_fit_delta reports the largest absolute difference among all parameters compared, including differences that remain within tolerance.
These two numerical fields must be read together. When no selected formula is shared by the baseline and the sampled specification, no parameter-of-fit comparison is possible: both n_fit_deltas and max_abs_fit_delta are NA. When max_abs_fit_delta contains a number, by contrast, a value of n_fit_deltas = 0 means that the parameters were compared and none moved by more than 0.05. The largest observed difference may still be positive because preservation within the stated tolerance does not require numerical identity.
All 25 sampled specifications produce a comparable conservative solution, and every specification changes at least one GENCL anchor. Nineteen draws change the conservative formula. The remaining six—draws 2, 5, 7, 17, 19, and 20—reproduce the baseline formula and therefore permit comparisons of its solution consistency, PRI, and coverage.
Across those six formula-preserving draws, the largest absolute parameter difference ranges from approximately 0.00084 to 0.03483. None of the three parameters differs by more than 0.05, so every one of the six records has n_fit_deltas = 0. Notice the distinction: the parameters are not numerically identical, but the observed differences remain within the tolerance established for this diagnostic.
The printed summary keeps these denominators visible. The conservative formula is available for comparison in all 25 draws and is preserved in 6 of them, producing a formula match rate of \((6/25 = 0.24)\). Those same six draws permit a like-for-like comparison of solution consistency, PRI, and coverage, and all six preserve the three parameters within 0.05. The conditional parameter-of-fit match rate is therefore \((6/6 = 1)\). If we return to the complete collection of attempted specifications and ask how many preserve both the formula and its three parameters of fit, the answer is again \((6/25 = 0.24)\).
Under this particular sampled domain, exact formula preservation is uncommon, but every formula-preserving draw also preserves solution consistency, PRI, and coverage within 0.05.
8.7 Broadening the Alternative Set
The second diagnostic raises the stakes. It allows the calibrations of all four conditions and the outcome to vary, increases the maximum calibration displacement to ten step units, and expands the frequency-cutoff candidates from 1 through 7. The inclusion grid remains between 0.70 and 0.90, and the diagnostic now monitors the conservative, parsimonious, and C1P1 intermediate formulas under the reference model-selection and exclusion rules.
vdem_alt_sets_all <- altset.test(
raw.data = vdem_raw,
calib.data = vdem_calib,
outcome = vdem_outcome,
conditions = vdem_conds,
calib_spec = vdem_calib_spec_with_outcome,
# Joint calibration and cutoff domain
test.conditions = vdem_conds,
test.outcome = TRUE,
anchors_to_test = c("E", "C", "I"),
unit_step = NULL,
unit_step_divisor = 10,
calib_max_steps = 10,
incl.cut = vdem_incl_cut,
incl_step = 0.01,
incl_max_steps = 10,
n.cut = vdem_n_cut,
ncut_step = 1,
ncut_max_steps = 5,
n_draws = 100,
seed = 456,
# Solution selection and comparison
solution = "all",
dir.exp = vdem_dir_exp,
exclude_mode = "recompute",
exclude_recompute = list(type = 2),
which_M = 1,
i_mode = "C1P1",
fit_tol = 0.05,
complete = TRUE,
sort.by = c("incl", "n")
)Running vdem_alt_sets_all prints the complete table of 100 sampled specifications. Each record identifies the sampled inclusion and frequency cutoffs, the calibrations that changed, the status of the reconstructed analysis, any formula changes among the monitored solution types, and any changes in solution consistency, PRI, or coverage that exceed the selected tolerance. These are the same fields introduced in the focused application, so there is little value in reproducing all 100 records here. At the end of the table, qcaERT provides the following summary:
Summary
Draws requested: 100
Draw status
status draws
Complete for all solution types 81/100
Partial 13/100
No comparable solution 6/100
Solution-formula preservation
solution_type successful_draws formula_preserved match_rate
conservative 94/100 8/94 8.5%
parsimonious 81/100 7/81 8.6%
intermediate 81/100 8/81 9.9%
Parameters-of-fit preservation
solution_type successful_draws fit_compared fit_preserved match_rate
conservative 94/100 8/94 3/8 37.5%
parsimonious 81/100 7/81 2/7 28.6%
intermediate 81/100 8/81 3/8 37.5%
Fit match rates are conditional on fit being compared.
Draws without a matching solution formula are excluded from those rates.The summary makes the nested comparisons visible without requiring us to reconstruct their denominators from the complete table. Its first panel distinguishes complete, partial, and noncomparable draws. The second reports formula preservation separately for each solution type, while the third narrows the comparison to the draws in which the same formula permits a like-for-like comparison of solution consistency, PRI, and coverage. Together, these counts provide the overview, while the sections below identify the draws behind them and explain why some specifications produce only part of the requested analysis.
8.7.1 From Draw Availability to Formula Preservation
The Draw status panel divides the 100 sampled specifications into three groups. Eighty-one draws produce comparable conservative, parsimonious, and intermediate formulas and are therefore complete for all monitored solution types. Thirteen are partial because they produce a comparable conservative formula but no comparable parsimonious or intermediate formula. The remaining six produce no comparable formula for any of the three solution types.
The six complete failures occur in draws 2, 18, 37, 49, 88, and 99. In each case, the sampled calibrations and truth-table cutoffs produce no empirically sufficient configuration: the configurations either have a consistency of sufficiency below the sampled inclusion cutoff or fail to meet the frequency requirement in the first place. In other words, these six reconstructed truth tables contain no configurations with OUT = 1.1
At least the conservative solution is available in the remaining 94 draws. In 13 of those specifications, however, the procedure used to identify contradictory simplifying assumptions cannot be completed. The conservative formula remains available because conservative minimization does not depend on that exclusion procedure, whereas the parsimonious and intermediate formulas cannot be obtained under the requested recomputed exclusion rule.2 Thus, 13 draws provide only a conservative formula, while 81 provide conservative, parsimonious, and intermediate. An overall status of ok means only that at least one selected formula is available; it does not mean that all three solution types succeeded.
Once availability has been established, the Solution-formula preservation panel reports how often the comparable formula for each solution type reproduces its baseline counterpart. In this realized sample, all formula matches occur among the 81 draws in which all three solution types are available. Seventy-two of those 81 specifications reproduce none of the three baseline formulas. The remaining nine reproduce at least one: draws 6, 13, 16, 50, 53, and 84 reproduce all three formulas; draw 23 reproduces only the parsimonious formula; and draws 67 and 79 reproduce the conservative and intermediate formulas.
But there is more! Because solution consistency, solution PRI, and solution coverage are compared only when the corresponding formula is preserved, these nine specifications are also the only draws with at least one apples-to-apples parameter comparison. The following table isolates them:
vdem_alt_sets_all_df <- as.data.frame(vdem_alt_sets_all)
subset(
vdem_alt_sets_all_df,
!is.na(max_abs_fit_delta)
)
#> draw incl.cut n.cut status n_changed_sets changed_sets changed_roles solution_change fit_changed_types n_fit_deltas max_abs_fit_delta
#> 6 6 0.85 1 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> CON | PAR | INT 6 0.09403908
#> 13 13 0.87 1 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> CON | PAR | INT 9 0.10683996
#> 16 16 0.76 1 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> CON | PAR | INT 6 0.13744502
#> 23 23 0.79 1 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome CON:formula_changed | INT:formula_changed PAR 1 0.05870652
#> 50 50 0.72 2 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> <NA> 0 0.01549174
#> 53 53 0.84 4 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> CON | PAR | INT 9 0.11451494
#> 67 67 0.81 3 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome PAR:formula_changed CON | INT 6 0.10052542
#> 79 79 0.86 1 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome PAR:formula_changed <NA> 0 0.03335411
#> 84 84 0.81 2 ok 5 FREEXP, PARTY, GENCL, JUCON, CIVSOC condition, outcome <NA> <NA> 0 0.04498740The parameter comparisons add one final distinction to the formula pattern. Among the six draws that reproduce all three formulas, draws 50 and 84 keep solution consistency, solution PRI, and solution coverage within 0.05 for every solution type. At least one of the nine solution-level parameters exceeds the tolerance in each of the other four draws.
Draw 23 provides only one like-for-like comparison because only the parsimonious formula is preserved; one of its three solution-level parameters differs from the baseline value by more than 0.05. Draws 67 and 79 each provide six parameter comparisons because both the conservative and intermediate formulas are preserved. At least one parameter exceeds the tolerance for each preserved formula in draw 67, while all six parameters remain within tolerance in draw 79.
8.7.2 Keep the Denominators Attached to the Rates
The two solution-type panels use nested denominators because each comparison requires the preceding one to be available. In the formula-preservation panel, successful_draws reports how many of the 100 sampled specifications produce a formula that can be compared with the baseline formula for that solution type. formula_preserved then uses those comparable draws as its denominator. In the parameter-of-fit panel, fit_compared reports how many of the comparable formulas reproduce the baseline expression and therefore permit solution consistency, PRI, and coverage to be compared like for like. Finally, fit_preserved uses those parameter-of-fit comparisons as its denominator and reports how many keep all three measures within 0.05.
For the conservative solution, 94 of the 100 draws produce a comparable formula, and 8 of those 94 reproduce the baseline expression. The conservative formula match rate is therefore \((8/94 = 0.085)\). Those eight formula matches permit eight comparisons of solution consistency, PRI, and coverage, and three of them preserve all three parameters within tolerance. The conditional parameter-of-fit match rate is consequently \((3/8 = 0.375)\).
The parsimonious formula is comparable in 81 draws and is preserved in 7, giving a formula match rate of \((7/81 = 0.086)\). Two of those seven formula-preserving draws also keep parsimonious solution consistency, PRI, and coverage within tolerance, giving a conditional parameter-of-fit match rate of \((2/7 = 0.286)\). The C1P1 intermediate formula is likewise comparable in 81 draws, but is preserved in 8, giving \((8/81 = 0.099)\). Three of those eight formula-preserving draws keep all three intermediate solution-level parameters within tolerance, giving \((3/8 = 0.375)\).
The printed summary deliberately keeps these rates separate by solution type. If our question is stricter—how many of the 100 attempted specifications preserve all three formulas simultaneously?—the draw-level evidence supplies the answer: six. Two of those six specifications also keep all nine solution-level parameters within 0.05. Thus, joint preservation of all three formulas occurs in \((6/100 = 0.06)\) of the attempted specifications, while joint preservation of all three formulas and all nine parameters occurs in \((2/100 = 0.02)\). These are properties of the particular combination of matches observed across the seeded draws, not alternative denominators for the solution-type-specific rates.
8.8 Interpreting and Reporting the Alternative Set
Under the criterion of exact formula identity, the low match rates show substantial sensitivity to the combined analytical choices sampled from this alternative set. They do not tell us which individual anchor or cutoff produced a particular formula change, because several analytical choices may change within the same draw. When the research question requires attribution to one family of decisions, the calibration, inclusion-cutoff, or frequency-cutoff diagnostic remains the appropriate tool.
The sampling design also bounds the inference: the findings describe 100 fixed-seed draws from the stated calibration movements and cutoff grids. Because sampling occurs with replacement, the results depend on the domain, the discrete candidate values, and the realized draws. A different defensible domain—or even a different seed within the same domain—may produce different frequencies. The match rates neither exhaust all admissible specifications nor estimate the probability that the QCA solution or the underlying causal interpretation is correct.
A reconstructable report should identify the baseline calibration and truth-table cutoffs; the sets and anchors eligible to change; the calibration movement and cutoff grids; the number of draws and seed; the monitored solution types, model, intermediate branch, and exclusion rule; and the parameter-of-fit tolerance. The results should then report the numbers of complete, partial, and noncomparable draws; the formula-comparison denominator and formula matches for each solution type; and the number of formula-preserving draws in which solution consistency, PRI, and coverage remain within tolerance.
The six sampled pairs of inclusion and frequency cutoffs are
0.86/6,0.90/5,0.90/4,0.88/4,0.84/5, and0.87/7.QCA::minimize()reports that none of the values inOUTis explained. Recomputing the exclusions for the parsimonious and intermediate solutions also fails on these truth tables. Because none of the three solution types succeeds, the overall status inherits the conservativeminimize_error; the solution-type-specific statuses for the parsimonious and intermediate solutions areexclude_error.↩︎QCA::findRows(type = 2)identifies contradictory simplifying assumptions by obtaining parsimonious solutions for the outcome and its negation. In eleven of these thirteen specifications, one of those auxiliary minimizations reports that no value ofOUTcan be explained. In the remaining two, every truth-table configuration is used and every condition is minimized. The exclusion set cannot be constructed in either situation.↩︎