From the Revd Ian Hill
Sir, — As a statistician before ordination, I still take an interest in statistical surveys. I am concerned that the NCT (National Churches Trust) survey (News, 26 March) may be less helpful than hoped.
One of the biggest challenges facing statisticians is non-response bias, which occurs when there is a less than complete response to a survey. Assumptions then have to be made about how those returns would have been completed. The problem is that if we knew how they would have been completed, we needn’t have sent them the form in the first place.
If those who fail to respond to the survey are random, then it is easy to gross up using other responses; but if those who fail to respond are concentrated in particular groups there will be a bias, unless it can be identified which groups they belong to and the results can be adjusted in line with the returns of others from that group.
It is for this reason that surveys (where a properly generated, much smaller random sample is used) are generally considered more statistically accurate than censuses, it is known how the sample in a survey was selected and which returns can be used to adjust for bias. With a census, that becomes significantly harder.
Non-response bias is likely to be significant in the proposed census for several reasons, but, I think, chiefly, because a 45-minute survey is excessive and unnecessary, particularly as filling in the forms is voluntary. Responses will be biased heavily towards those who have 45 minutes spare to fill in a form. I have no intention of spending more than two hours filling in the forms for my three churches: that would be a waste of my valuable time.
The NCT may think that I am wrong, and may have many good arguments why we should fill their forms in, but experience tells statisticians that those arguments will fall on deaf ears.
I would be very surprised if the NCT had enough metadata (specific information on a particular non-respondent) to enable it to allow accurately for non-response. I know churches that would carry similar metadata in terms of demographics, congregation size, denomination, and churchmanship, and yet which are in a significantly different state of repair.
I also suspect that a minister with a church in a poor state of repair is more likely to fill in the forms in the hope that it may lead to a grant. Such a supposition cannot be proved either way without a separate survey of responding and non-responding churches after the initial survey, which the timescales given in the article would not permit. This is how the Office for National Statistics adjusts for non-response in the UK Census, for which it does get a good response rate, because it is illegal to refuse to fill it in. The NCT has no such luxuries.
This concerns me, because the NCT will analyse the results, publish results, and probably get media headlines. But the underlying statistics are likely to have such a wide margin of error that the headline-grabbing results will be statistically meaningless and potentially misleading. I calculate that 48,000 forms each taking 45 minutes to complete would cost the church about 15 years of clergy or other-person time. That is a huge cost to the Church.
To put it another way, would we employ 15 people in Church House for a whole year just to gather this information? If there is only a 50-per-cent response rate (which would be very good for a voluntary survey), that would be 7.5 years wasted.
I would very strongly urge the NCT to conduct instead a properly sampled and stratified statistical survey. The costs to the Church would be significantly less, and the statisticians will be able to get significantly more accurate results than they will from the census.
IAN HILL
The Vicarage, Vicarage Road
Buntingford
Hertfordshire SG9 9BH