A whirlwind of the survey workflow: load, wrangle, weight, estimate, and model
Tutorials
Getting Started
Survey Workflow
Python
A five-minute, hands-on tour of the svy survey workflow: load a bundled survey, wrangle it, adjust weights for nonresponse, and produce design-based estimates, tabulations, and regression models.
A five-minute, hands-on tour of the whole svy workflow — load a survey, prepare it, adjust its weights, and produce design-based estimates, tabulations, and models. Every block runs on a small example survey bundled with svy, so you can follow along offline. Each section links to a deeper tutorial.
Load and build a Sample
We use the individual-level records of a synthetic World Bank survey that ships with svy. Its bundled metadata already carries the sampling design, so building a Sample is two lines:
import svy# A bundled example survey (runs offline)data = svy.datasets.load("ind_sample_wb_2023", source="bundled")info = svy.datasets.describe("ind_sample_wb_2023", source="bundled")# Bind the data to its sampling designsample = svy.Sample(data, svy.Design(**info.design))print(sample)
╭────────────── Sample ───────────────╮│Survey Data││ Rows : 3246 ││ Columns : 15 ││ Strata : 7 ││ PSUs : 33 ││││Survey Design││ Case id None ││ Wave None ││ Stratum (geo1, urbrur) ││ PSU ea ││ SSU None ││ Weight hhweight ││ With replacement False ││ Prob None ││ Hit None ││ MOS None ││ Population size None ││ Replicate weights None │╰─────────────────────────────────────╯
A Sample bundles your data with its design and is the hub for everything that follows. (For the object model, see Getting Started; for the datasets themselves, see Datasets.) Peek at a few columns any time:
Every transformation returns a newSample, so an entire workflow reads as one fluent, immutable pipeline. Here we create a literacy indicator, simulate and adjust for nonresponse, trim extreme weights, and estimate the population literacy rate — start to finish:
import numpy as nprng = np.random.default_rng(42)literacy_rate = ( sample# 1. Wrangle: a 1/0 literacy indicator, plus a simulated response status .wrangling.mutate({"literate": svy.when(svy.col("literacy") =="Yes").then(1) .when(svy.col("literacy") =="No").then(0) .otherwise(None),# simulated for illustration — the example data has 100% response"resp_status": rng.choice( ["respondent", "non-respondent"], p=[0.85, 0.15], size=sample.n_records ), })# 2. Adjust the weights for nonresponse, within urban/rural classes .weighting.adjust( resp_status="resp_status", cells="urbrur", resp_mapping={"rr": "respondent", "nr": "non-respondent"}, wgt_name="nr_wgt", )# 3. Trim weights above 1.2 x the median; the excess goes to the others .weighting.trim(upper=svy.Threshold("median", 1.2))# 4. Estimate the population literacy rate, with a design-based SE .estimation.mean("literate", drop_nulls=True))print(literacy_rate)
╭────────── Estimate: MEAN (TAYLOR) ───────────╮│y: literate ││││ est se lci uci cv (%)││ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ││ 0.8366 0.0189 0.7977 0.8754 2.26 │││╰──────────────────────────────────────────────╯
The example data has full response, so we simulate a status just to show the adjustment. Each step has its own tutorial: Wrangling, Weighting, and Estimation.
More analysis, briefly
The same Sample reaches every other kind of analysis through an accessor.
Estimation
# A design-based mean by domain (urban vs rural)print(sample.estimation.mean("age", by="urbrur"))
╭──────────────── Estimate: MEAN (TAYLOR) ─────────────────╮│y: age ││││urbrur est se lci uci cv (%)││ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ││ Rural 27.4893 3.1681 20.5163 34.4624 11.53 ││ Urban 29.7557 2.5016 24.4236 35.0877 8.41 │││╰──────────────────────────────────────────────────────────╯
Means, totals, proportions, ratios, and medians all work the same way — see Estimation.
Tabulation
# One-way and two-way tabulations, with design-based standard errorsprint(sample.categorical.tabulate("educ_attain", units="percent"))print(sample.categorical.tabulate("educ_attain", "sex", units="percent"))
╭─────────────────────────────────── Table: educ_attain ────────────────────────────────────╮│││Row Estimate Std Err CV Lower Upper││ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ││ Less than primary 53.4272 1.8681 0.0350 49.5744 57.2394 ││ NIU (not in universe) or no education 9.7995 1.6227 0.1656 6.9328 13.6773 ││ Primary 21.1389 1.0922 0.0517 18.9810 23.4711 ││ Secondary 12.2317 1.3959 0.1141 9.6394 15.4023 ││ University 3.4027 0.6691 0.1966 2.2656 5.0809 │││╰───────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────────────────── Table: educ_attain × sex ─────────────────────────────────────╮│││Row Col Estimate Std Err CV Lower Upper││ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ││ Less than primary Female 27.4239 1.3103 0.0478 24.8142 30.1977 ││ Less than primary Male 26.0033 0.9584 0.0369 24.0822 28.0211 ││ NIU (not in universe) or no education Female 5.2608 0.8612 0.1637 3.7469 7.3396 ││ NIU (not in universe) or no education Male 4.5387 0.8214 0.1810 3.1197 6.5596 ││ Primary Female 10.3805 0.8081 0.0778 8.8330 12.1628 ││ Primary Male 10.7585 0.5718 0.0531 9.6384 11.9914 ││ Secondary Female 5.8735 0.7461 0.1270 4.5148 7.6084 ││ Secondary Male 6.3582 0.7495 0.1179 4.9808 8.0842 ││ University Female 1.4217 0.4035 0.2838 0.7917 2.5402 ││ University Male 1.9810 0.3738 0.1887 1.3424 2.9145 │││╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
More on tables and design-adjusted tests in Categorical.
Regression
# Design-based linear regression: years of schooling by age and sexmodel = sample.glm.fit(y="yrs_school", x=["age", svy.Cat("sex")], family="gaussian")print(model)