samplics and svy: the comparison

Results of running samplics 0.6.1 and svy side by side on samplics’ bundled datasets: which pairs match, the largest differences, and where samplics was wrong.

Keywords

samplics svy comparison, samplics validation, samplics bugs

This page backs Migrating from samplics to svy.

Results

We installed samplics 0.6.1 and svy 0.31 side by side, ran each pair on samplics’ bundled datasets (NHANES II with its BRR and jackknife subsets, the PSU frame and samples, the auto data, the milk and county crop data) and compared every quantity both libraries report. “Yes” means equal to within 1e-8 relative. “Yes, with setting” means equal once the setting in Default differences is applied.

samplics	svy	Match	Max relative difference
TaylorEstimator mean, total, ratio	estimation.mean, total, ratio	Yes	2e-16
TaylorEstimator prop, as_factor=True	estimation.prop, mean(as_factor=True)	Yes	8e-15
TaylorEstimator with domain=	by=, where=	Yes (intervals: with setting)	3e-16
TaylorEstimator with fpc=	Design(pop_size=)	Yes	0
TaylorEstimator median	estimation.median(q_method="linear")	Yes for the estimate	0
SinglePSUEst.skip	singleton="skip"	Yes	6e-17
SinglePSUEst.certainty, SinglePSUEst.combine	singleton="self_representing", svy.Singleton("collapse", using=)	Pending an svy fix
ReplicateEstimator BRR, jackknife, Fay	method="replication" with BrrWgts, JackknifeWgts	Yes, with setting	5e-14
ReplicateWeight jackknife	create_jk_wgts()	Yes (same 62 columns)	1e-16
ReplicateWeight BRR, bootstrap	create_brr_wgts(), create_bs_wgts()	Valid, not identical: PSU order and random draws
SampleWeight adjust, normalize, poststratify	weighting.adjust, normalize, poststratify	Yes	7e-16
SampleWeight calibrate, rake	weighting.calibrate, rake	Yes	4e-15
SampleSelection SRS, PPS systematic, Rao-Sampford	sampling.srs, pps_sys, pps_rs	Yes for the probabilities	2e-16
SampleSelection Brewer	sampling.pps_brewer	Pending an svy fix
SampleSize, two-sample classes	estimate_prop, estimate_mean, compare_props, compare_means	Yes, with setting	0 (sizes)
allocate equal, proportional, Neyman	SampleSize().allocate()	Yes, with setting	0 (sizes)
Tabulation, CrossTabulation cells	categorical.tabulate	Yes	1.5e-14
CrossTabulation Pearson chi-square	Table.stats.chisq	Yes	4e-14
Ttest means, standard errors, t	categorical.ttest	Yes (df and p: with setting)	3e-16
SurveyGLM linear, logistic	glm.fit	Yes (intervals: with setting)	2e-12
EblupAreaModel REML	svy_sae.AreaLevel.fh(method="reml")	Yes	6e-8
EblupAreaModel ML	svy_sae.AreaLevel.fh(method="ml")	No: samplics stops before the optimum	2e-3
EblupUnitModel	svy_sae.UnitLevel.eblup()	Pending an svy-sae fix
EbUnitModel	svy_sae.UnitLevel.ebp()	Model parameters yes; predictions are Monte Carlo	1e-5 (parameters)

Where samplics was wrong

A few samplics results are wrong, and svy’s differ for that reason. If you are comparing against numbers produced with samplics, check these first.

• CrossTabulation reports a Rao-Scott F of 0 and a missing p-value whenever the covariance matrix inverts. svy’s F and p-value equal R’s svychisq.
• TaylorEstimator(PopParam.median) standard errors are near zero (9e-6 on NHANES II, against 0.49 from svy’s Woodruff interval).
• ReplicateWeight().replicate() attaches the replicate weights to the wrong rows. Created jackknife, BRR and bootstrap weights from samplics should not be reused.
• ReplicateEstimator with rep_weight_cls= loses the per-stratum jackknife coefficients and overstates the standard error.
• SampleSelection with pps_murphy reports \(2p\) rather than Murphy’s inclusion probability; pps_rs does not achieve the probabilities it reports; shuffle=True misaligns the probabilities with the units; pps_wr raises a TypeError.
• SampleWeight().calib_covariates() converts continuous variables to integers. rake() does not enforce ll_bound and up_bound, and poststratify(factor=) without a domain ignores the factor.
• power_for_one_mean reverses "greater" and "less". The two-sided power functions leave out the far tail.
• EblupAreaModel(method=FitMethod.ml) stops two iterations in, before the ML optimum, and its log-likelihood, AIC and BIC are computed with the wrong covariance matrix.
• EbUnitModel with a Box-Cox \(\lambda \neq 0\) back-transforms with the wrong inverse, and its bootstrap_mse() reuses the original fit in every replicate.
• SampleSizePropOneSample and SampleSize(method=SizeMethod.fleiss) ignore arcsin, continuity and pop_size.