Skip to contents

Performance and reliability

Source: vignettes/performance-and-reliability.Rmd
performance-and-reliability.Rmd

compdid has two nuisance-estimation paths with different performance profiles. The default nuisance_method = "dml" path is intended for routine analysis. The local-polynomial path is more computationally intensive and is best used when the nonparametric mixed-kernel structure is substantively important.

Default Path

The default front door uses the formula interface and built-in custom DML learners.

set.seed(123)
dat <- simulate_comp_did(n = 1000, delta = 0.25)

fit <- comp_did(
  y ~ x1 + x2 + x3 + x4 + x5 + x6,
  data = dat,
  tname = "post",
  dname = "d",
  K = 2,
  seed = 123
)

summary(fit)
#> Compositional-change DiD summary
#> Observations: 1000 
#> Nuisance method: dml 
#> Nuisance backend: custom 
#> 
#>  term estimate std.error conf.low conf.high
#>   ATT    1.782     2.676   -3.462     7.026

Local-Polynomial Path

For mixed continuous, unordered, and ordered covariates, pass the covariate blocks explicitly. This avoids treating factor dummies as continuous kernel covariates.

fit_lp <- comp_did(
  yname = "y",
  tname = "post",
  dname = "d",
  data = dat,
  continuous = c("x1", "x2"),
  unordered = c("x3", "x4"),
  ordered = c("x5", "x6"),
  nuisance_method = "loo",
  list_control_ps = list(n_start = 1, ps_min = 1e-5, lp_order = 1),
  list_control_or = list(n_start = 1, bw_constrained = FALSE, or_order = 1)
)

The local-polynomial GPS fit returns clipped and normalized probabilities. Each row is a four-cell probability vector:

set.seed(123)
small <- simulate_comp_did(n = 80)
dp <- make_did_dp(
  data = small,
  y = "y",
  d = "d",
  post = "post",
  continuous = c("x1", "x2"),
  unordered = c("x3", "x4"),
  ordered = c("x5", "x6")
)

ps <- locpol_ps_fit(
  dp,
  bws = c(5, 0.5, 0.5),
  list_control = list(ps_min = 1e-5, lp_order = 1)
)

range(rowSums(ps$fitted.values))
#> [1] 1 1

Development Benchmarks

The development repository includes benchmark and guardrail scripts under dev/. They are excluded from package builds.

Rscript dev/benchmark_suite.R
Rscript dev/benchmark_suite.R --full --np --reps=3 --out=/tmp/compdid_benchmark_full.csv
Rscript dev/profile_locpol.R --modes --out=/tmp/compdid_locpol_profile.csv
Rscript dev/performance_guardrails.R

Use dev/performance_guardrails.R --strict-runtime only when comparing runs on the same machine. Runtime thresholds are intentionally opt-in because elapsed time depends on compiler flags, CPU load, BLAS, and OpenMP availability.

Release Gate

The local release gate combines correctness, documentation, stress, and performance smoke checks:

Rscript dev/release_audit.R

By default it runs unit tests, R CMD check with vignette rebuilding, the Sequeira saved-nuisance numerical verifier, local-polynomial regression fixtures, performance guardrails, stress scripts, and the quick benchmark suite. Use --skip-benchmark during fast edit cycles, and reserve --strict-runtime for same-machine performance comparisons.

The Sequeira full cross-validated application replication is intentionally not part of routine checks. It is a manual audit because it is slow and the historical GPS cross-validation path is toolchain-sensitive. The reproducible paper-number check is the saved-nuisance verifier.