Serializing Survey Results to JSON in Python with svy

A stable, versioned contract for report templates, QA rules, wave comparisons, and caches

Tutorials
Reporting
Serialization
Python
Export survey estimates, hypothesis tests, tabulations, and model fits as stable, versioned JSON payloads in Python. Learn to bind report templates to estimate fields, automate CV-based quality checks, diff results across survey waves, and cache analyses using the svy library.
Author

Mamadou S. Diallo, Ph.D.

Published

August 6, 2026

Modified

August 7, 2026

Keywords

survey results JSON Python, serialize survey estimates Python, export survey results Python, versioned JSON schema survey data, survey report automation Python, coefficient of variation quality checks survey, survey estimate pipeline Python, wave over wave survey comparison Python, caching survey analysis results, design effect JSON export, complex survey data interchange Python

Every result in svy prints as a formatted table designed for human eyes. That is the right default for interactive work — and the wrong interface for everything downstream of it. A report template cannot bind to a field inside formatted text. A quality-assurance rule cannot compare a coefficient of variation it has to parse out of a string. Two survey waves cannot be diffed on real numbers when each wave is a text blob.

The svy.serialize module closes that gap. It converts every major result object — estimates, t-tests, tabulations, chi-square tests, GLM fits and predictions, data descriptions — into a stable, versioned, JSON-safe payload. Each payload is tagged with a kind discriminator and a schema_version, so downstream code can dispatch on the result type and know exactly which fields to expect, without importing anything from svy’s internals.

This tutorial covers the four public functions, the payload contract, and four practical workflows built on top of it: binding report templates to estimate fields, enforcing CV-based quality rules, computing wave-over-wave diffs, and caching results.

Setting Up the Sample

We’ll use the imaginary country household dataset from World Bank (2023), with the same design as in the Estimation tutorial, plus the derived poverty-status variable:

import svy

from rich import print as rprint

svy.Estimate.PRINT_WIDTH = 95

hld_data = svy.datasets.load(name="hld_sample_wb_2023", source="bundled")
hld_design = svy.Design(stratum=("geo1", "urbrur"), psu="ea", wgt="hhweight")
hld_sample = svy.Sample(data=hld_data, design=hld_design)

hld_sample = hld_sample.wrangling.mutate(
    {
        "hhpovline": svy.col("hhsize") * 1800,
        "pov_status": svy.when(svy.col("tot_exp") < svy.col("hhpovline")).then(1).otherwise(0),
    }
)

Four Functions

The entire public API is four functions:

from svy.serialize import from_json, serialize, to_dict, to_json
Table 1: The svy.serialize public API
Function Signature Use it when
serialize(result) result → typed struct You stay in Python and want typed, attribute-style access
to_dict(result) result → dict[str, Any] You want a plain dict — for templates, DataFrames, or json.dumps
to_json(result) result → bytes You persist, transmit, or hash the payload
from_json(data) bytes → typed struct You load a stored payload back into a typed struct

Start with a familiar result — the mean of total household expenditure, with the design effect:

mean_exp = hld_sample.estimation.mean(y="tot_exp", deff=True)

print(mean_exp)
╭─────────────────────── Estimate: MEAN (TAYLOR) ────────────────────────╮
│                                                                        │
│          est         se           lci           uci   cv (%)     deff  │
│  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  │
│  12,373.0892   744.3261   10,843.1051   13,903.0733     6.02   7.6658  │
│                                                                        │
╰────────────────────────────────────────────────────────────────────────╯

serialize() converts it to a payload struct. The struct is not the Estimate object — it is a frozen, JSON-safe snapshot of everything a report, a QA rule, or another system needs:

data = serialize(mean_exp)

print(f"kind:           {data.kind}")
print(f"schema_version: {data.schema_version}")
print(f"param:          {data.param}")
print(f"method:         {data.method}")
print(f"n_strata:       {data.n_strata}, n_psus: {data.n_psus}")

row = data.estimates[0]
print(f"estimate row:   est={row.est:,.1f}, se={row.se:,.1f}, cv={row.cv:.3f}, df={row.df}")
kind:           estimate
schema_version: svy-result/0.2
param:          Mean
method:         Taylor
n_strata:       7, n_psus: 33
estimate row:   est=12,373.1, se=744.3, cv=0.060, df=26

to_dict() produces the same content as a plain dictionary, and to_json() as compact JSON bytes:

d = to_dict(mean_exp)
print(sorted(d.keys()))

js = to_json(mean_exp)
print(f"{len(js)} bytes: {js[:80]}...")
['alpha', 'estimates', 'kind', 'method', 'n_psus', 'n_strata', 'param', 'q_method', 'schema_version', 'where_clause']
425 bytes: b'{"kind":"estimate","schema_version":"svy-result/0.2","param":"Mean","method":"Ta'...

from_json() decodes the bytes back into the typed struct. The round trip is exact — floats included — so equality holds:

restored = from_json(js)

print(restored.estimates[0].est == data.estimates[0].est)
print(restored == data)
True
True
NoteThe struct is the contract

The payload structs are the stable representation, deliberately decoupled from svy’s internal classes. Consumers render, compare, and store payloads — they do not reconstruct Estimate or GLMFit objects from them. If svy’s internals change, the serializers absorb the change and the payload contract stays put.

Kind-Tagged Payloads

Every top-level payload carries a kind field that identifies the result type, so consumers can dispatch without introspection or isinstance checks against svy classes:

match data.kind:
    case "estimate":
        print(f"{data.param} of {data.estimates[0].y}: {data.estimates[0].est:,.1f}")
    case "glm_fit":
        print("a regression table")
    case other:
        print(f"unhandled kind: {other}")
Mean of tot_exp: 12,373.1

Nine kinds cover the result types that serialize in svy 0.23.0:

Table 2: Kind tags and the results they serialize
kind Payload struct svy result Typical producing call
estimate EstimateData Estimate estimation.mean(), .total(), .prop(), .ratio(), .median(), .quantile()
estimate_list EstimateListData EstimateList any of the above with a list of variables or several probabilities
ttest_one_group TTestOneGroupData TTestOneGroup categorical.ttest(y=..., mean_h0=...)
ttest_two_groups TTestTwoGroupsData TTestTwoGroups categorical.ttest(y=..., group=...)
chi_square ChiSquareData ChiSquare tabulate(...).stats.chisq
table TableData Table categorical.tabulate()
glm_fit GLMFitData GLMFit (or fitted GLM) glm.fit(...)
glm_pred GLMPredData GLMPred glm.fit(...).predict(...)
describe DescribeResultData DescribeResult sample.describe()

The structs themselves (EstimateData, ParamEstData, GLMCoefData, …) are importable from svy.serialize when you want type annotations on the consumer side.

Let’s produce one of each family and confirm the tags. T-tests and tabulations come from the Categorical Data Analysis tutorial, models from the GLM tutorial:

ttest_urbrur = hld_sample.categorical.ttest(y="tot_exp", group="urbrur")
tab = hld_sample.categorical.tabulate(rowvar="urbrur", colvar="electricity")
logit_model = hld_sample.glm.fit(
    y="pov_status",
    x=["hhsize", "rooms", svy.Cat("urbrur")],
    family="binomial",
)
preds = logit_model.predict(hld_sample.data, y_col="pov_status")
desc = hld_sample.describe(columns=["tot_exp", "hhsize", "urbrur"], weighted=True)

for result in (ttest_urbrur, tab, tab.stats.chisq, logit_model, preds, desc):
    payload = serialize(result)
    print(f"{type(result).__name__:15s} -> kind={payload.kind!r}")
TTestTwoGroups  -> kind='ttest_two_groups'
Table           -> kind='table'
ChiSquare       -> kind='chi_square'
GLM             -> kind='glm_fit'
GLMPred         -> kind='glm_pred'
DescribeResult  -> kind='describe'

A few payload highlights, one per family. The two-group t-test carries the group specification, the difference with its confidence interval, and the test statistics:

tt = serialize(ttest_urbrur)

print(f"groups: {tt.groups.var} = {tt.groups.levels}")
print(f"diff:   {tt.diff[0].diff:,.1f}  [{tt.diff[0].lci:,.1f}, {tt.diff[0].uci:,.1f}]")
print(f"stats:  t={tt.stats.t:.3f}, df={tt.stats.df:.0f}, p={tt.stats.p_value:.4f}")
groups: urbrur = ['Rural', 'Urban']
diff:   6,513.9  [3,638.6, 9,389.3]
stats:  t=4.666, df=25, p=0.0001

The two-way table carries one CellEstData per cell plus the design-adjusted Rao–Scott statistics — a nested chi-square and its F transformation:

tb = serialize(tab)

print(f"type={tb.type!r}, rows={tb.rowvals}, cols={tb.colvals}, cells={len(tb.estimates)}")
print(f"chisq: value={tb.stats.chisq.value:.2f}, p={tb.stats.chisq.p_value:.2e}")
print(f"F:     value={tb.stats.f.value:.2f} on ({tb.stats.f.df_num:.1f}, {tb.stats.f.df_den:.1f}) df")
type='Two-Way', rows=['Rural', 'Urban'], cols=['No', 'Yes'], cells=4
chisq: value=105.67, p=8.08e-05
F:     value=15.54 on (1.0, 26.0) df

The GLM fit carries the coefficient table (with Wald tests) and the model-level statistics — everything a regression table in a report binds to. Note that serialize() accepts the fitted GLM wrapper returned by glm.fit() directly:

g = serialize(logit_model)

print(f"y={g.y!r}, family={g.family!r}, link={g.link!r}, n={g.stats.n}")
for c in g.coefs:
    print(f"  {c.term:15s} est={c.est:+.4f}  se={c.se:.4f}  p={c.wald.p_value:.4f}")
y='pov_status', family='Binomial', link='logit', n=825
  _intercept_     est=-1.5185  se=0.5515  p=0.0113
  hhsize          est=+0.6955  se=0.1149  p=0.0000
  rooms           est=-0.7604  se=0.1427  p=0.0000
  urbrur_Urban    est=-1.8544  se=0.4343  p=0.0003

And the prediction payload holds the fitted values with their uncertainty as parallel lists:

p = serialize(preds)

print(f"kind={p.kind!r}, df={p.df:.0f}, n={len(p.yhat)}")
print(f"first prediction: {p.yhat[0]:.4f}  [{p.lci[0]:.4f}, {p.uci[0]:.4f}]")
kind='glm_pred', df=23, n=825
first prediction: 0.0139  [0.0063, 0.0306]

What Does Not Serialize

Dispatch is by exact type, and anything without a registered serializer raises svy’s structured SerializationError. Catch it like any svy error and print it — it names the type it got, the types it supports, and where to read more:

try:
    serialize([1, 2, 3])
except svy.SerializationError as err:
    rprint(err)
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                 │
│  ✗ No serializer registered [UNSUPPORTED_RESULT_TYPE]                                                           │
│                                                                                                                 │
│  Objects of type 'list' cannot be serialized.                                                                   │
│                                                                                                                 │
│  where    serialize                                                                                             │
│  expected ['Estimate', 'EstimateList', 'TTestOneGroup', 'TTestTwoGroups', 'ChiSquare', 'Table', 'GLMFit',       │
│           'GLMPred', 'DescribeResult']                                                                          │
│  got      list                                                                                                  │
│  hint     Pass a svy result object (e.g. Estimate, Table, GLMFit).                                              │
│  docs     https://svylab.com/docs/svy/tutorials/serialization.html                                              │
│                                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

An unfitted model is caught early too, with the same ModelError that predict() would raise:

try:
    serialize(hld_sample.glm)
except svy.ModelError as err:
    rprint(err)
╭──────────────────────────────────────────────────────────────────────╮
│                                                                      │
│  ✗ Model not fitted [MODEL_NOT_FITTED]                               │
│                                                                      │
│  Cannot call 'serialize' because the model has not been fitted yet.  │
│                                                                      │
│  where serialize                                                     │
│  hint  Call .fit() on the estimator first.                           │
│                                                                      │
╰──────────────────────────────────────────────────────────────────────╯

Large internal fields are deliberately excluded from payloads: an Estimate’s covariance matrix and internal stratum bookkeeping, and a GLMFit’s covariance matrix and term metadata. Payloads carry what rendering, comparison, and QA need — not the full estimation state.

Inside an Estimate Payload

The estimate payload is the one you will consume most, so it is worth knowing its two levels. The top level records how the estimate was produced:

Table 3: Top-level fields of an estimate payload
Field Meaning
param What was estimated: "Mean", "Total", "Proportion", "Ratio", "Median", "Quantile"
method Variance method: Taylor linearization or replication
alpha Significance level behind the confidence intervals
n_strata, n_psus Design counts; n_psus - n_strata is the full-design degrees of freedom
where_clause The subpopulation filter, if any — audit trail for domain estimates
q_method Quantile interpolation rule (quantiles and medians)
estimates The rows: one ParamEstData per variable/level/domain combination

Each row in estimates is one estimate with its full uncertainty picture:

Table 4: Per-row fields of ParamEstData
Field Meaning
y, y_level Variable, and level for categorical outcomes
est, se Point estimate and standard error
lci, uci Confidence limits at the payload’s alpha
cv Coefficient of variation — the field QA rules live on
deff Design effect (None unless requested with deff=True)
df Design degrees of freedom backing this row’s \(t\)-quantile
prob The probability for quantile estimates (e.g. 0.9), else None
by, by_level Domain variable(s) and this row’s level(s)
x, x_level Denominator variable for ratios

The per-row df deserves a closer look. A domain or by-group is counted on its own active PSUs and strata, so grouped rows legitimately carry different degrees of freedom:

pov_by_region = hld_sample.estimation.prop(
    y="pov_status", by="geo1", deff=True, drop_nulls=True
)
pd = serialize(pov_by_region)

for r in pd.estimates:
    if r.y_level == 1:  # the "poor" level
        print(f"{r.by_level}: est={r.est:.4f}  cv={r.cv:.3f}  deff={r.deff:.2f}  df={r.df}")

print(f"\nfull-design df: n_psus - n_strata = {pd.n_psus} - {pd.n_strata} = {pd.n_psus - pd.n_strata}")
['geo_03']: est=0.2624  cv=0.371  deff=11.12  df=7
['geo_01']: est=0.1520  cv=0.630  deff=8.95  df=4
['geo_02']: est=0.1729  cv=0.359  deff=6.13  df=7
['geo_04']: est=0.2200  cv=0.283  deff=5.72  df=8

full-design df: n_psus - n_strata = 33 - 7 = 26

Quantile estimates (new in 0.23.0) populate prob, which is how a consumer distinguishes the 10th percentile from the 90th without parsing labels:

q90 = hld_sample.estimation.quantile(y="tot_exp", p=0.9)
qd = serialize(q90)

print(f"param={qd.param!r}, prob={qd.estimates[0].prob}, est={qd.estimates[0].est:,.1f}")
param='Quantile', prob=0.9, est=23,274.0

Multi-Estimate Results: estimate_list

Estimating several variables (or several probabilities) at once returns an EstimateList, which serializes under its own kind. Each member is exactly what a standalone Estimate would produce:

several = hld_sample.estimation.mean(y=["tot_exp", "pc_exp", "hhsize"])
ld = serialize(several)

print(f"kind={ld.kind!r}, members={len(ld.estimates)}")
for member in ld.estimates:
    r = member.estimates[0]
    print(f"  {member.kind}: {r.y:8s} est={r.est:,.2f}  se={r.se:,.2f}")
kind='estimate_list', members=3
  estimate: tot_exp  est=12,373.09  se=744.33
  estimate: pc_exp   est=3,810.07  se=216.54
  estimate: hhsize   est=3.87  se=0.28

The Versioning Contract

Every top-level payload embeds the schema version it was written under:

from svy.serialize import SCHEMA_VERSION

print(SCHEMA_VERSION)
svy-result/0.2

The contract, spelled out in the module’s design document:

  • Minor bump (0.1 → 0.2): fields were added. Backward-compatible — existing payloads still decode, and consumers should ignore unknown fields rather than reject them.
  • Major bump (0.x → 1.0): fields were removed or renamed. Breaking — consumers must be updated before accepting the new payloads.

Two corollaries for consumer code. First, check the major version, tolerate the minor:

def check_version(payload) -> None:
    schema, _, version = payload.schema_version.partition("/")
    major = int(version.split(".")[0])
    if schema != "svy-result" or major > 0:
        raise ValueError(f"unsupported schema: {payload.schema_version}")

check_version(data)
print("payload accepted")
payload accepted

Second, treat optional fields as optional. Additive evolution means a field your code reads may be absent in older archived payloads — from_json fills such gaps with None-style defaults, and your rendering code should do the same. prob, for example, was added in svy 0.23.0: payloads archived by svy 0.22 or earlier decode fine today, with prob = None on every row.

ImportantVersion history to know about

Schema 0.2 moved degrees of freedom onto each row (estimates[].df) and dropped the old top-level scalar. As shown above, a single number cannot represent grouped results, where each domain carries its own df. If you bind df in a template, bind the row’s df; for the full-design value use n_psus - n_strata, which stays at design level even under a where= domain filter.

Round-Tripping and Interop

to_json output is ordinary UTF-8 JSON. Consumers in Python that have svy installed get typed structs back with from_json; consumers without svy — a JavaScript dashboard, an R script, a database’s JSON column — read it with any JSON parser:

import json

chi_payload = json.loads(to_json(tab.stats.chisq))
print(json.dumps(chi_payload, indent=2))
{
  "kind": "chi_square",
  "schema_version": "svy-result/0.2",
  "df": 1.0,
  "value": 105.66799902701919,
  "p_value": 8.079301532348815e-05
}

A malformed or foreign payload fails loudly rather than decoding into the wrong shape:

try:
    from_json(b'{"no": "kind"}')
except svy.SerializationError as err:
    rprint(err)
╭──────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                          │
│  ✗ Payload missing 'kind' [PAYLOAD_MISSING_KIND]                                         │
│                                                                                          │
│  The JSON payload has no 'kind' discriminator, so its result type cannot be determined.  │
│                                                                                          │
│  where from_json                                                                         │
│  param kind                                                                              │
│  hint  Only decode JSON produced by svy.serialize.to_json().                             │
│                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────╯
NoteOne round-trip nuance

For the describe kind, the items entries are dicts rather than typed structs (the seven measurement-type variants are deferred), and tuple values inside them come back from JSON as lists — same values, different container. Compare describe payloads on their JSON bytes or with json.loads, not with struct equality. All other kinds round-trip to equal structs, floats included.

Pattern 1: Binding Report Templates to Fields

The original sin that serialization fixes: report generators binding to formatted text. With payloads, a template binds to named fields — est, se, lci, uci, cv, deff, df — and survives every cosmetic change to svy’s printing:

REPORT_LINE = (
    "Mean household expenditure was {est:,.0f} LCU "
    "(SE {se:,.0f}; 95% CI {lci:,.0f}–{uci:,.0f}; "
    "CV {cv:.1%}; deff {deff:.2f}; df {df})."
)

row = to_dict(mean_exp)["estimates"][0]
print(REPORT_LINE.format(**row))
Mean household expenditure was 12,373 LCU (SE 744; 95% CI 10,843–13,903; CV 6.0%; deff 7.67; df 26).

The same payload feeds tabular outputs. estimates is a list of flat dicts, which is exactly what a DataFrame constructor wants:

import polars as pl

pov_table = (
    pl.DataFrame(to_dict(pov_by_region)["estimates"])
    .filter(pl.col("y_level") == 1)
    .select("by_level", "est", "se", "lci", "uci", "cv", "df")
    .sort("by_level")
)

pov_table
shape: (4, 7)
by_level est se lci uci cv df
list[str] f64 f64 f64 f64 f64 i64
["geo_01"] 0.152 0.095833 0.022242 0.585469 0.630482 4
["geo_02"] 0.172949 0.062133 0.069654 0.368718 0.359256 7
["geo_03"] 0.262434 0.097264 0.09783 0.538636 0.370623 7
["geo_04"] 0.220011 0.062352 0.108759 0.394669 0.283406 8

From here it is one step to a formatted report table — a Markdown fragment for a briefing note, a styled sheet, or rows in a reporting database. The template never touches an svy object, so the reporting side of the pipeline needs no survey machinery at all.

Pattern 2: QA Rules over CVs

Statistical agencies commonly gate publication on estimate reliability — flagging estimates whose coefficient of variation exceeds one threshold and suppressing those above a higher one. With payloads, that rule is a loop over fields, not a parse of printed output:

FLAG, SUPPRESS = 0.20, 0.50

def qa_review(payload) -> pl.DataFrame:
    rows = []
    for e in payload.estimates:
        action = "suppress" if e.cv > SUPPRESS else "flag" if e.cv > FLAG else "publish"
        rows.append(
            {
                "y": e.y,
                "level": str(e.y_level),
                "domain": ", ".join(map(str, e.by_level or [])),
                "est": e.est,
                "cv": e.cv,
                "df": e.df,
                "action": action,
            }
        )
    return pl.DataFrame(rows).sort("cv", descending=True)

qa_review(serialize(pov_by_region))
shape: (8, 7)
y level domain est cv df action
str str str f64 f64 i64 str
"pov_status" "1" "geo_01" 0.152 0.630482 4 "suppress"
"pov_status" "1" "geo_03" 0.262434 0.370623 7 "flag"
"pov_status" "1" "geo_02" 0.172949 0.359256 7 "flag"
"pov_status" "1" "geo_04" 0.220011 0.283406 8 "flag"
"pov_status" "0" "geo_03" 0.737566 0.131872 7 "publish"
"pov_status" "0" "geo_01" 0.848 0.113011 4 "publish"
"pov_status" "0" "geo_04" 0.779989 0.07994 8 "publish"
"pov_status" "0" "geo_02" 0.827051 0.075126 7 "publish"

The same loop extends naturally to other fields: require df above a floor before quoting a confidence interval, or watch deff for signs of weight degradation. Because every wave’s payload has the same shape, the QA module is written once and applied to every run.

TipSet thresholds to your program’s standards

Reliability thresholds vary by statistical program — many agencies flag CVs somewhere between 15% and 33% and suppress beyond a program-specific ceiling. The pattern is the point here; encode your publication standard in one place and apply it to every payload.

Pattern 3: Wave-over-Wave Diffs

Serialized payloads make cross-wave comparison a numeric operation. The workflow: after each wave’s run, archive to_json bytes; next wave, load the archive with from_json and compare fields.

Archive this wave’s regional poverty rates:

import tempfile
from pathlib import Path

archive = Path(tempfile.mkdtemp()) / "pov_by_region_wave1.json"
archive.write_bytes(to_json(pov_by_region))

print(f"archived {archive.stat().st_size} bytes")
archived 2265 bytes

Our bundled dataset has a single wave, so we simulate the next one — nominal expenditure growth of 4% with household-level noise — and re-run the identical analysis:

import numpy as np

rng = np.random.default_rng(2024)

wave2_data = hld_data.with_columns(
    (pl.col("tot_exp") * (1.04 + rng.normal(0.0, 0.05, hld_data.height))).alias("tot_exp")
)

wave2_sample = svy.Sample(data=wave2_data, design=hld_design).wrangling.mutate(
    {
        "hhpovline": svy.col("hhsize") * 1800,
        "pov_status": svy.when(svy.col("tot_exp") < svy.col("hhpovline")).then(1).otherwise(0),
    }
)

wave2_payload = serialize(
    wave2_sample.estimation.prop(y="pov_status", by="geo1", drop_nulls=True)
)

Load the baseline and diff — joining rows on their identity keys, never on position:

baseline = from_json(archive.read_bytes())

assert baseline.kind == wave2_payload.kind
assert baseline.schema_version == wave2_payload.schema_version

def row_key(e):
    return (e.y, e.y_level, tuple(e.by or []), tuple(e.by_level or []))

baseline_rows = {row_key(e): e for e in baseline.estimates}

diffs = []
for cur in wave2_payload.estimates:
    prev = baseline_rows.get(row_key(cur))
    if prev is None or cur.y_level != 1:
        continue
    diffs.append(
        {
            "region": ", ".join(map(str, cur.by_level or [])),
            "wave1": prev.est,
            "wave2": cur.est,
            "delta_pp": (cur.est - prev.est) * 100,
            "outside_wave1_ci": not (prev.lci <= cur.est <= prev.uci),
        }
    )

pl.DataFrame(diffs).sort("region")
shape: (4, 5)
region wave1 wave2 delta_pp outside_wave1_ci
str f64 f64 f64 bool
"geo_01" 0.152 0.144 -0.8 false
"geo_02" 0.172949 0.149551 -2.339776 false
"geo_03" 0.262434 0.235133 -2.730101 false
"geo_04" 0.220011 0.203558 -1.645248 false
ImportantJoin on keys, not on row order

The order of by-group rows inside a payload is not guaranteed — two runs of the same analysis can emit domains in a different order. Always align rows on their identity fields (y, y_level, by, by_level) as above, and treat the outside_wave1_ci column as a screening device for where to look first — a proper test of change uses the two waves’ standard errors (and their covariance, if samples overlap).

Pattern 4: Caching and Change Detection

to_json is deterministic for a given result object, and re-computing an ungrouped estimate on unchanged data reproduces the payload byte for byte. That makes payloads safe to cache and cheap to fingerprint.

For caching, key on the request — data version plus the parameters of the call — and store the payload bytes:

import hashlib

cache: dict[str, bytes] = {}

def cached_mean(sample, y: str, data_version: str):
    key = hashlib.sha256(f"mean|{y}|{data_version}".encode()).hexdigest()
    if key not in cache:
        cache[key] = to_json(sample.estimation.mean(y=y, deff=True))
    return from_json(cache[key])

first = cached_mean(hld_sample, "tot_exp", data_version="hld_sample_wb_2023")
again = cached_mean(hld_sample, "tot_exp", data_version="hld_sample_wb_2023")

print(f"cache entries: {len(cache)}")
print(f"identical payloads: {first == again}")
cache entries: 1
identical payloads: True

For change detection — “did this quarter’s numbers move at all?” — fingerprint the payload content. Because grouped rows can arrive in any order, canonicalize before hashing:

def fingerprint(result) -> str:
    d = to_dict(result)
    if isinstance(d.get("estimates"), list):
        d["estimates"] = sorted(
            d["estimates"], key=lambda r: json.dumps(r, sort_keys=True)
        )
    return hashlib.sha256(json.dumps(d, sort_keys=True).encode()).hexdigest()

fp1 = fingerprint(hld_sample.estimation.mean(y="tot_exp", by="geo1"))
fp2 = fingerprint(hld_sample.estimation.mean(y="tot_exp", by="geo1"))

print(f"stable across recomputation: {fp1 == fp2}")
print(f"fingerprint: {fp1[:16]}…")
stable across recomputation: True
fingerprint: bdefa703e1fa3828…
NoteOne timestamp exception

describe payloads embed generated_at, so two otherwise-identical descriptions differ on that field. Drop it before fingerprinting descriptions, or fingerprint the items list alone.

Error Reference

Serialization failures raise svy’s structured exceptions — every error carries a stable code, the expected/got context, and a hint:

Table 5: Serialization errors
Situation Raises code
serialize() on a type with no registered serializer SerializationError UNSUPPORTED_RESULT_TYPE
serialize() on an unfitted GLM ModelError MODEL_NOT_FITTED
from_json() on JSON without a kind field SerializationError PAYLOAD_MISSING_KIND
from_json() on an unrecognized kind value SerializationError PAYLOAD_UNKNOWN_KIND

Both classes subclass svy.SvyError, so one except svy.SvyError: covers every svy failure. For programmatic handling, match on the code field — it is the stable contract, while messages may be reworded:

from svy import SerializationError

try:
    serialize({"raw": "dict"})
except SerializationError as err:
    print(f"code: {err.code}")
    print(f"got:  {err.got}")
    print(f"hint: {err.hint}")
code: UNSUPPORTED_RESULT_TYPE
got:  dict
hint: Pass a svy result object (e.g. Estimate, Table, GLMFit).

And fittingly for this tutorial, the errors themselves are structured data — err.to_dict() returns a JSON-safe payload, so a pipeline can log failures the same way it stores results.

Next Steps

You now have the full result pipeline: estimate with Estimation, test with Categorical Data Analysis, model with GLMs — and hand every result downstream as a typed, versioned payload.

Where do the numbers come from?
Revisit how the estimates are produced in Estimation →

References

World Bank. 2023. “Synthetic Data for an Imaginary Country, Sample, 2023.” World Bank, Development Data Group. https://doi.org/10.48529/MC1F-QH23.