Two waves of the LSMS-ISA General Household Survey-Panel reproduced in Python: the 2018/19 report under the Wave 4 weight, the 2023/24 report under the longitudinal weight, and the report’s change tables, with the standard errors a panel design earns.
The Nigeria General Household Survey-Panel is the LSMS-ISA panel: the same households, visited in 2010/11, 2012/13, 2015/16, 2018/19 and 2023/24. Each wave has a survey report, and the 2023/24 report also tabulates what changed since 2018/19, as percentage-point differences without standard errors.
In svy each wave is a Sample with its own weight, and the panel is their combination:
2018/19 reproduces under the Wave 4 weight. All 36 comparable cells of the 2018/19 report’s asset table (urban, rural, national) match to the printed decimal.
2023/24 reproduces under the longitudinal weight. All 63 cells of the 2023/24 report’s asset table match to the printed decimal, and so do all 36 food-insecurity cells across the report’s nine domains.
The changes reproduce, and get a standard error. The between-wave contrast equals the printed change on 20 of 21 assets (the rest by rounding) and on all 18 food-insecurity cells. Its SE carries the covariance of the households seen twice and is 0.42 to 0.85 times the independent-waves value; 15 of the 21 asset changes are distinguishable from zero.
svy’s estimators are validated separately against R’s survey package and against three producers’ published figures — see the case studies index.
Background
The GHS-Panel began in 2010 as a 5,000-household subsample of the General Household Survey, embedded in the LSMS-ISA programme. After a decade the sample was partially refreshed: Wave 4 (2018/19) kept a long panel of 1,590 of the original households and added a refresh of about 3,600 households in 360 new enumeration areas drawn from the same 2010 frame. Wave 5 (2023/24) revisited that combined sample. Each wave has two visits, post-planting and post-harvest, and the reports take the most recent visit for anything asked twice.
Eight enumeration areas in the North West could not be visited in 2023/24 for security reasons. Of the remaining 511, 4,771 households completed both Wave 5 visits, and 4,682 of them had also completed both Wave 4 visits.
Three files, two readers
Wave 5 ships as one Stata file per questionnaire section. The 2018/19 outcomes are read from the 2010–2019 Uniform Panel Data, a harmonised release that stacks the four earlier waves in one SPSS file per module with a wave column. That release carries no weights, so the Wave 4 weight comes from the cover file of the raw Wave 4 release, also Stata.
Wrangling the two waves
Everything in this subsection is data preparation, not survey statistics: svy reads the files and keeps their labels, and polars does the rest. The goal is one frame per wave with the household id, the design columns, that wave’s weight, and one column per outcome.
View code
from pathlib import PathDATA = Path.home() /"Data/surveys/lsms/nga_ghs_panel"W5 = DATA /"wave5/stata"W4 = DATA /"wave4/raw"NUPD = DATA /"wave4/spss"design_cols = [pl.col("hhid", "zone", "sector", "strata").cast(pl.Int64), "cluster"]cover4 = ( svy.read_stata(W4 /"secta_plantingw4.dta") .filter(pl.col("wt_wave4").is_not_null()) # both 2018/19 visits completed .select(*design_cols, "wt_wave4"))cover5 = ( svy.read_stata(W5 /"Post Planting Wave 5/Household/secta_plantingw5.dta") .filter(pl.col("wt_wave5").is_not_null()) # both 2023/24 visits completed .select(*design_cols, "wt_wave5"))print(f"2018/19: {cover4.height:,} households, weights sum to {cover4['wt_wave4'].sum():,.0f}")print(f"2023/24: {cover5.height:,} households, weights sum to {cover5['wt_wave5'].sum():,.0f}")
2018/19: 4,976 households, weights sum to 26,957,052
2023/24: 4,682 households, weights sum to 26,957,052
The asset module is one row per household and item in both releases, with the same item codes (301 = sofa set, 311 = kerosene stove, and so on). One function turns it into one row per household with an own_<code> column per asset; it runs on the Wave 5 file from the Stata reader and on the Wave 4 rows from the SPSS reader:
The 30-day food-insecurity items come from the post-harvest visit, where the two releases ask the same questions under different names. The reports tabulate post-harvest modules by where the household was at that visit, so the function keeps that location too:
Five of the 106 Wave 5 files refused to load at first. They are Stata 13 exports, which ReadStat decodes as Windows-1252, and some free-text answers contain characters that code page cannot hold:
The hint is new: the reader gained an encoding= option while this page was being written, so the file’s real encoding can be declared, and the offending strings turn out to be seed-variety names typed with emoji:
None of the five files is used in the analysis. The SPSS reader took all 65 uniform-panel files as they came.
Building the panel sample
The survey statistics start here, with two lines per wave. A wave is a Sample like any cross-section, with case_id naming the household and the wave’s own weight declared. combine_samples(kind="panel") stacks the waves in caller order, checks that households appearing in both keep their stratum and cluster, reports which households are in one wave only, and fills Design.wave from the wave column the frames already carry:
╭──────────── Design ─────────────╮│ Case id hhid ││ Wave wave ││ Stratum (strata,) ││ PSU (cluster,) ││ SSU None ││ Weight combined_wgt ││ With replacement False ││ Prob None ││ Hit None ││ MOS None ││ Population size None ││ Replicate weights None │╰─────────────────────────────────╯
The combined weight, combined_wgt, is each wave’s own weight copied through: wt_wave4 on the 2018/19 rows, wt_wave5 on the 2023/24 rows. The two are different kinds of weight, and that is what the rest of the page turns on.
NoteCross-sectional and longitudinal weights
wt_wave4 is a cross-sectional weight: it makes the 4,976 households interviewed in 2018/19 represent Nigeria’s households in 2018/19. wt_wave5 is a longitudinal weight: it makes the 4,682 households interviewed in both waves represent the same 2018/19 population, after handing the weight of the 294 households that dropped out to the survivors that resemble them. It is one number per household, and it sums to the same 26.96 million as wt_wave4.
The reports use them exactly this way. The 2018/19 report is the cross-section under wt_wave4; the 2023/24 report’s levels are the panel under wt_wave5; and its change tables subtract the one from the other. Declaring each wave’s own weight reproduces all three.
Reproducing the 2018/19 report
The Wave 4 report’s Table 3.16 gives asset ownership by zone and sector. One by= call on the combined sample gives every wave-by-domain level at once; the 2018/19 rows are the cross-section under the Wave 4 weight:
The report’s table pools some items the 2023/24 questionnaire lists separately (mattress, bed and mat; the stoves; washing machine and dryer), so the comparison covers the rows both questionnaires define the same way:
Asset, 2018/19
Urban, svy
Urban, report
Rural, svy
Rural, report
Nigeria, svy
Nigeria, report
Furniture (sofa set)
35.3
35.3
22.0
22.0
26.2
26.2
Furniture (chairs)
37.9
37.9
27.0
27.0
30.5
30.5
Furniture (table)
50.8
50.7
33.2
33.2
38.7
38.7
Sewing machine
12.3
12.3
9.3
9.3
10.2
10.2
Fridge
31.9
31.9
10.7
10.7
17.3
17.3
Freezer
16.7
16.7
5.0
5.0
8.7
8.7
Air conditioner
4.2
4.2
0.4
0.4
1.6
1.6
Bicycle
7.2
7.2
19.6
19.6
15.7
15.7
Motorbike
21.3
21.3
34.8
34.8
30.6
30.6
Cars and other vehicles
16.8
16.8
6.4
6.4
9.6
9.6
Generator
34.4
34.4
20.1
20.1
24.6
24.6
Fan
73.4
73.4
30.8
30.8
44.2
44.2
36 of 36 cells at the printed decimal.
Reproducing the 2023/24 report
Table 3.15 of the Wave 5 report gives 2023/24 ownership for every asset; the 2023/24 rows of the same estimate are the panel under the longitudinal weight:
Asset, 2023/24
Urban, svy
Urban, report
Rural, svy
Rural, report
Nigeria, svy
Nigeria, report
Furniture (sofa set)
33.2
33.2
16.3
16.3
21.6
21.6
Furniture (chairs)
42.5
42.5
22.8
22.8
29.0
29.0
Furniture (table)
42.0
42.0
22.9
22.9
28.9
28.9
Mattress
93.2
93.2
89.2
89.2
90.4
90.4
Bed
65.8
65.8
63.9
63.9
64.5
64.5
Mat
58.7
58.7
76.3
76.3
70.8
70.8
Sewing machine
13.5
13.5
6.2
6.2
8.5
8.5
Gas cooker
42.6
42.6
10.0
10.0
20.3
20.3
Stove (electric)
3.5
3.5
1.5
1.5
2.1
2.1
Stove gas (table)
13.0
13.0
4.8
4.8
7.4
7.4
Stove (kerosene)
18.9
18.9
11.2
11.2
13.6
13.6
Fridge
30.7
30.7
8.4
8.4
15.4
15.4
Freezer
16.2
16.2
3.6
3.6
7.6
7.6
Air conditioner
4.0
4.0
0.5
0.5
1.6
1.6
Washing machine
4.9
4.9
0.5
0.5
1.9
1.9
Electric clothes dryer
0.0
0.0
0.0
0.0
0.0
0.0
Bicycle
5.9
5.9
11.5
11.5
9.8
9.8
Motorbike
21.4
21.4
27.5
27.5
25.6
25.6
Cars and other vehicles
14.9
14.9
3.7
3.7
7.2
7.2
Generator
29.2
29.2
13.1
13.1
18.1
18.1
Fan
68.0
68.0
21.7
21.7
36.3
36.3
63 of 63 cells at the printed decimal.
Reproducing the changes
Table 3.16 of the 2023/24 report lists the change in the share of households owning each asset. On the combined sample that is a contrast between the two wave levels, and the contrast comes with what the report leaves out: a standard error, a confidence interval, and a test.
╭──────────────────────────── Contrast (TAYLOR, df=511) ────────────────────────────╮│││contrast est se cv (%) t p_value lci uci││ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ││ change -0.2560 0.0157 -6.12 -16.3513 1.208e-48 -0.2867 -0.2252 │││╰───────────────────────────────────────────────────────────────────────────────────╯
Kerosene stoves went from four households in ten to fewer than one in seven, the report’s −25.6 points. The table runs the same contrast for every asset in Table 3.16 and puts the report’s printed change in the last column.
Asset
2018/19
2023/24
Change (svy)
SE
95% CI
Change (report)
Furniture (sofa set)
26.2
21.6
-4.6*
1.34
[-7.2, -1.9]
-4.6
Furniture (chairs)
30.5
29.0
-1.5
1.59
[-4.6, +1.7]
-1.5
Furniture (table)
38.7
28.9
-9.8*
1.67
[-13.1, -6.5]
-9.8
Mattress
91.5
90.4
-1.0
0.98
[-3.0, +0.9]
-1.0
Bed
70.1
64.5
-5.7*
1.28
[-8.2, -3.1]
-5.6
Mat
73.5
70.8
-2.7*
1.28
[-5.2, -0.2]
-2.7
Sewing machine
10.2
8.5
-1.7*
0.72
[-3.1, -0.3]
-1.7
Gas cooker
11.0
20.3
+9.3*
1.15
[+7.0, +11.5]
+9.3
Stove (electric)
4.5
2.1
-2.4*
0.55
[-3.4, -1.3]
-2.4
Stove gas (table)
3.8
7.4
+3.6*
0.77
[+2.1, +5.2]
+3.6
Stove (kerosene)
39.2
13.6
-25.6*
1.57
[-28.7, -22.5]
-25.6
Fridge
17.3
15.4
-1.9*
0.80
[-3.5, -0.3]
-1.9
Freezer
8.7
7.6
-1.1
0.73
[-2.5, +0.3]
-1.1
Air conditioner
1.6
1.6
+0.0
0.26
[-0.5, +0.5]
+0.0
Washing machine
1.7
1.9
+0.2
0.29
[-0.4, +0.7]
+0.2
Electric clothes dryer
0.1
0.0
-0.1
0.07
[-0.3, +0.0]
-0.1
Bicycle
15.7
9.8
-6.0*
1.07
[-8.1, -3.9]
-6.0
Motorbike
30.6
25.6
-5.0*
1.23
[-7.4, -2.5]
-5.0
Cars and other vehicles
9.6
7.2
-2.5*
0.56
[-3.6, -1.3]
-2.4
Generator
24.6
18.1
-6.5*
0.99
[-8.4, -4.5]
-6.4
Fan
44.2
36.3
-7.9*
1.20
[-10.3, -5.5]
-7.9
* p < 0.05 for the change. Shares of households in percent; changes in percentage points. Each row is the contrast of the two wave levels. The 2018/19 column is the 2018/19 report’s Table 3.16 (national), the 2023/24 column the 2023/24 report’s Table 3.15, and the last column the 2023/24 report’s Table 3.16, which prints only the change; the SE and interval are svy’s.
20 of 21 changes equal the printed figure; the remainder differ by one unit in the last decimal, the rounding of the two levels before subtraction. With the standard error in hand, 15 of the 21 changes are distinguishable from zero at 5 percent; chairs, mattresses, freezers, air conditioners, washing machines and clothes dryers are not, which the report’s arrow glyphs cannot show.
Food insecurity, by domain
Tables 4.13 to 4.16 of the 2023/24 report track two 30-day food-insecurity items across nine domains. by= runs the domains in one pass, on the post-harvest location as the report does; the change is again a contrast per domain.
All 36 levels and all 18 changes at the printed decimal. Six households moved zone between the two 2023/24 visits; tabulating by the post-planting location instead moves three South East and South West cells off the printed values.
What the report cannot say
Standard errors on change
A change between two waves of a panel is estimated on households seen twice, in the same clusters. The two estimates are correlated, and a difference of correlated quantities is more precise than a difference of independent ones. svy’s contrast uses the pairing through the 4,682 households in both waves; the naive alternative adds the two variances as if the waves were separate surveys.
Asset
Change
SE, paired
SE, if independent
Ratio
Stove (kerosene)
-25.6
1.57
2.10
0.74
Gas cooker
+9.3
1.15
1.89
0.61
Fan
-7.9
1.20
2.84
0.42
Fridge
-1.9
0.80
1.73
0.46
Mattress
-1.0
0.98
1.31
0.75
The gain varies by asset, largest for fans and fridges, smallest for mattresses and kerosene stoves, because it depends on how strongly a household’s answer in one wave predicts its answer in the next.
Transitions
The same sample answers a question the report does not ask: which households changed. wrangling.lag brings a household’s previous-wave value onto its current row, and the four combinations of then and now are a categorical variable like any other, estimated on the households observed in both waves.
The 2018/19 rows have nothing to look back to, so their lagged value is null; the 2023/24 rows of households seen in both waves carry both answers. Crossing the two gives the four states. wrangling.mutate adds the column to the same sample, with svy’s own when and col, and the shares are a proportion estimated on the 2023/24 rows:
The same four states for the gas cooker, laid out as a table:
Gas cooker
Share of households (%)
SE
95% CI
Kept
6.9
0.82
[5.4, 8.7]
Lost
4.2
0.51
[3.3, 5.3]
Acquired
13.4
1.10
[11.4, 15.7]
Never
75.5
1.75
[71.9, 78.8]
The net change in kerosene stoves is almost entirely households giving them up, not new households failing to acquire them; gas cookers show the mirror image. The pairing of those two transitions, household by household, is the fuel-switching story behind the report’s two rows.
What this shows
checked against
result
2018/19 asset levels, 36 cells
2018/19 report, Table 3.16
36 at printed precision, under the Wave 4 weight
2023/24 asset levels, 63 cells
2023/24 report, Table 3.15
63 at printed precision, under the longitudinal weight
asset changes, 21 rows
2023/24 report, Table 3.16
20 at printed precision, the rest by rounding; 15 of 21 significant at 5%
food insecurity, 2 items × 9 domains
2023/24 report, Tables 4.13–4.16
36 of 36 levels and 18 of 18 changes at printed precision
paired vs independent SE
—
ratio 0.42–0.85
All results computed with svy 0.28.0 at render time from the microdata; nothing svy-side is transcribed. The scripts alongside this post reproduce the run given the public-use files from the World Bank Microdata Library.
References
National Bureau of Statistics (NBS) and World Bank (2024). Nigeria General Household Survey-Panel (GHS-Panel) Wave 5, 2023/24: Survey Report.
NBS and World Bank (2024). GHS-Panel Wave 5 Basic Information Document.
NBS and World Bank (2019). Nigeria General Household Survey-Panel, Wave 4 2018/19: Survey Report.
Data: NBS, General Household Survey, Panel 2023–2024, Wave 5 (NGA_2023_GHSP-W5_v01_M), General Household Survey, Panel 2018–2019, Wave 4 (NGA_2018_GHSP-W4_v03_M), and General Household Survey-Panel 2010–2019, Uniform Panel Data (NGA_2010-2019_NUPD_v01_M), World Bank Microdata Library.
This study uses public-use microdata from the Nigeria General Household Survey-Panel, produced by the National Bureau of Statistics with the World Bank’s Living Standards Measurement Study. The analysis and any errors are the author’s alone. The microdata is not redistributed; the code runs against files obtained from the World Bank Microdata Library.
TipCommunity signal
Help make svy the standard for survey analysis in Python
If rigorous, design-based survey inference in Python matters to you, starring the repository helps signal demand and prioritize validation and stability work.