Fit the Battese-Harter-Fuller (BHF) unit-level model in Python with svy-sae: EBLUP of district means from household records and census covariates, REML or ML, analytical and bootstrap MSE, and out-of-sample districts.
Author
Mamadou S. Diallo, Ph.D.
Modified
November 8, 2026
Keywords
Battese-Harter-Fuller model, BHF model Python, unit-level small area estimation, EBLUP Python, nested error regression model, bootstrap MSE small area
WarningAbout these examples
This tutorial illustrates the svy-sae API; it is not a rigorous small area analysis. The models are kept simple so each step stays readable, and in many places, from the choice of covariates to the variance model, better choices could be made to improve them.
The Battese–Harter–Fuller (BHF) model (Battese, Harter, and Fuller 1988), or nested error regression model, fits the survey records directly. Each household’s outcome is a regression on its covariates plus an effect shared by its district:
for household \(j\) of district \(i\). The EBLUP of the district mean adds the predicted area effect to the regression at the census mean of the covariates, \(\bar X_i\):
where \(\bar y_i\) and \(\bar x_i\) are the means of the \(n_i\) sampled households. The factor \(\gamma_i\) shrinks the district’s mean residual toward zero, the more so the smaller its sample. This is why census means are enough. It treats the sampled households as a negligible share of the district, as they are here: about 136 of 41,000 on average.
This tutorial demonstrates how to fit the model with UnitLevel.eblup() to estimate district mean expenditure per person, read its results, predict districts outside the sample, and measure uncertainty with the analytical and bootstrap MSE.
Rebuild the inputs from Preparing unit-level inputs
import polars as plimport svyimport svy_sae as saeyes_no = ["electricity", "car", "refrigerator", "tv", "cell", "bank"]covariates = ["urban","piped_water","flush_toilet",*yes_no,"hhsize","rooms",]def household_covariates(frame):return frame.with_columns( (pl.col("tot_exp") / pl.col("hhsize")).alias("exp_pp"), (pl.col("urbrur") =="Urban").cast(pl.Float64).alias("urban"), (pl.col("piped_water") !="No piped water") .cast(pl.Float64) .alias("piped_water"), (pl.col("flush_toilet") =="Flush toilet") .cast(pl.Float64) .alias("flush_toilet"),*[(pl.col(c) =="Yes").cast(pl.Float64).alias(c) for c in yes_no], pl.col("hhsize").cast(pl.Float64), pl.col("rooms").cast(pl.Float64), )hld_data = household_covariates(svy.datasets.load("hld_sample_wb_2023").data)hld_census = household_covariates(svy.datasets.load("hld_pop_wb_2023").data)census_units = hld_census.select("geo1", "geo2", *covariates)census_means = ( hld_census.group_by("geo2") .agg(*[pl.col(c).mean() for c in covariates], pl.len().alias("n_hh")) .sort("geo2"))truth = ( hld_census.group_by("geo2") .agg( pl.col("exp_pp").mean().alias("true_mean"), (pl.col("exp_pp") <1800).mean().alias("true_rate"), ) .sort("geo2"))
Fitting the model
UnitLevel takes the survey records. eblup() takes the outcome, the covariates, the district key, and the population data. With the census means, pop_size names the number of households:
model = sae.UnitLevel(hld_data)fit = model.eblup( y="exp_pp", x=covariates, area="geo2", pop_data=census_means, pop_size="n_hh",)print(fit)
The model is fitted to the 8,000 survey households and predicts all 61 districts of the census.
Census means or census records
The EBLUP of a mean is linear in the covariates, so the district means of the census are enough. Passing the census records instead, without pop_size, gives the same estimates:
Fixed effects, with standard errors and confidence intervals:
fit.to_polars("fixed")
shape: (12, 7)
term
est
se
t_score
p_value
lci
uci
str
f64
f64
f64
f64
f64
f64
"Intercept"
3221.008445
93.605018
34.410639
0.0
3037.542611
3404.474279
"urban"
621.470581
64.51355
9.633179
0.0
495.024023
747.917139
"piped_water"
-173.557733
49.974354
-3.472936
0.000515
-271.507467
-75.608
"flush_toilet"
689.008565
52.241042
13.189028
0.0
586.616123
791.401007
"electricity"
-219.091064
74.105807
-2.956463
0.003112
-364.338445
-73.843683
"car"
1949.365577
56.876879
34.273428
0.0
1837.886896
2060.844259
"refrigerator"
483.289073
61.837665
7.815448
5.5511e-15
362.08725
604.490897
"tv"
249.770099
66.27017
3.768967
0.000164
119.880566
379.659632
"cell"
960.257915
52.17809
18.40347
0.0
857.988859
1062.526971
"bank"
-117.416022
68.532228
-1.713296
0.086658
-251.739189
16.907145
"hhsize"
-594.004901
10.464816
-56.762097
0.0
-614.515941
-573.493862
"rooms"
340.917726
14.026426
24.305387
0.0
313.425931
368.409522
Variance components, the area variance \(\sigma^2_u\) and the residual variance \(\sigma^2_e\):
fit.to_polars("components")
shape: (2, 4)
name
kind
est
se
str
str
f64
null
"area"
"variance"
148686.240375
null
"residual"
"variance"
3.3292e6
null
Their standard errors are not estimated for unit-level models, and are reported as missing. The area variance is small next to the residual one: most of the variation in expenditure is between households of the same district, not between districts.
Predictions
pred = fit.to_polars("predictions")pred.select("area", "pred", "mse", "cv", "lci", "uci", "in_sample").head()
shape: (5, 7)
area
pred
mse
cv
lci
uci
in_sample
str
f64
f64
f64
f64
f64
bool
"geo_01_01"
4952.333874
12408.489589
0.022493
4734.002809
5170.664938
true
"geo_01_02"
4284.671918
47510.398354
0.050872
3857.453068
4711.890769
true
"geo_01_03"
4312.707174
10468.771801
0.023725
4112.165823
4513.248526
true
"geo_01_04"
4148.458611
35305.101023
0.045293
3780.181439
4516.735784
true
"geo_01_05"
4753.730365
27850.306259
0.035106
4426.637509
5080.823222
true
Districts outside the sample
The two districts the survey did not reach have no sampled households and no area effect to predict:
The two agree closely, within the Monte Carlo error of the bootstrap.
TipReplicates are cheap here
BHF bootstrap replicates run in batches and compilation dominates the cost, so 1,000 replicates take about as long as 100. Use a generous count: the Monte Carlo error of the MSE is what more replicates reduce.
How close did we get?
Compare the direct estimates, a model without covariates, and the model with all eleven, against the true district means:
The covariates cut the error of the direct estimates by more than half. Yet on the same districts the area-level Fay–Herriot model of Transformations does better, with an RMSE of 197, and the model’s MSE is well below the actual squared error: many intervals miss the true mean. The checks in the next tutorial show why: expenditure is far from normal around the regression, and the model assumes it is. The EBP tutorial fits the same model on a transformed scale.
Summary
This tutorial fitted the BHF model with UnitLevel.eblup():
Population data as census means with pop_size, or census records, which give the same estimates
Results: fixed effects, the area and residual variances, and predictions with MSE, CV, and intervals
Districts outside the sample predicted from their census means
Options: REML or ML, analytical or bootstrap MSE
Accuracy against the truth, and an MSE that understates it when the model’s assumptions fail
Battese, G E, R M Harter, and W A Fuller. 1988. “An Error-Components Model for Prediction of County Crop Areas Using Survey and Satellite Data.”J. Amer. Statist. Assoc. 83: 28–36.