*** DEBUG END ***

National Churches Trust survey has flawed basis

30 March 2010


From the Revd Ian Hill

Sir, — As a statistician before ordination, I still take an interest in statistical surveys. I am concerned that the NCT (National Churches Trust) survey (News, 26 March) may be less helpful than hoped.

One of the biggest challenges facing statisticians is non-response bias, which occurs when there is a less than complete response to a survey. Assumptions then have to be made about how those returns would have been completed. The problem is that if we knew how they would have been completed, we needn’t have sent them the form in the first place.

If those who fail to respond to the survey are random, then it is easy to gross up using other responses; but if those who fail to respond are concentrated in particular groups there will be a bias, unless it can be identified which groups they belong to and the results can be adjusted in line with the returns of others from that group.

It is for this reason that surveys (where a properly generated, much smaller random sample is used) are generally considered more statistic­ally accurate than censuses, it is known how the sample in a survey was selected and which returns can be used to adjust for bias. With a census, that becomes significantly harder.

Non-response bias is likely to be significant in the proposed census for several reasons, but, I think, chiefly, because a 45-minute survey is excessive and unnecessary, particularly as filling in the forms is voluntary. Responses will be biased heavily towards those who have 45 minutes spare to fill in a form. I have no intention of spending more than two hours filling in the forms for my three churches: that would be a waste of my valuable time.

The NCT may think that I am wrong, and may have many good arguments why we should fill their forms in, but experience tells statisticians that those arguments will fall on deaf ears.

I would be very surprised if the NCT had enough metadata (specific information on a particular non-respondent) to enable it to allow accurately for non-response. I know churches that would carry similar metadata in terms of demo­graphics, congregation size, denomination, and churchmanship, and yet which are in a significantly different state of repair.

I also suspect that a minister with a church in a poor state of repair is more likely to fill in the forms in the hope that it may lead to a grant. Such a supposition cannot be proved either way without a separate survey of responding and non-responding churches after the initial survey, which the timescales given in the article would not permit. This is how the Office for National Statistics adjusts for non-response in the UK Census, for which it does get a good response rate, because it is illegal to refuse to fill it in. The NCT has no such luxuries.

This concerns me, because the NCT will analyse the results, publish results, and probably get media headlines. But the underlying statistics are likely to have such a wide margin of error that the headline-grabbing results will be statistically meaningless and potentially misleading. I calculate that 48,000 forms each taking 45 minutes to complete would cost the church about 15 years of clergy or other-person time. That is a huge cost to the Church.

To put it another way, would we employ 15 people in Church House for a whole year just to gather this information? If there is only a 50-per-cent response rate (which would be very good for a voluntary sur­vey), that would be 7.5 years wasted.

I would very strongly urge the NCT to conduct instead a properly sampled and stratified statistical survey. The costs to the Church would be significantly less, and the statisticians will be able to get significantly more accurate results than they will from the census.

The Vicarage, Vicarage Road
Hertfordshire SG9 9BH

Letters to the editor

Letters for publication should be sent to letters@churchtimes.co.uk.

Letters should be exclusive to the Church Times, and include a full postal address. Your name and address will appear alongside your letter.

Forthcoming Events

6-7 September 2022
Preaching as Pilgrimage conference
From the College of Preachers.

8 September 2022
Church Times Cricket Cup: North v. South
Join us to watch the match at the Walker Cricket Ground, in Southgate, north London.

26 September 2022
What am I living for? God
Sam Wells and Lucy Winkett begin the St Martin-in-the-Fields autumn lecture series in partnership with Church Times.

More events

The Church Times Archive

Read reports from issues stretching back to 1863, search for your parish or see if any of the clergy you know get a mention.

FREE for Church Times subscribers.

Explore the archive

Welcome to the Church Times

​To explore the Church Times website fully, please sign in or subscribe.

Non-subscribers can read four articles for free each month. (You will need to register.)