The nonsense of the NOS ‘sex’ survey

In that parallel universe of Daily Mail and Daily Telegraph readers there was a mass explosion of relief this morning. Curtains will have twitched across the nation as the worst fears of many were assuaged with the Daily Mail’s social affairs correspondent (sic) categorically assuring the threatened masses that they were not surrounded by a huge percentage of gay neighbours: it’s official, only one man in 10 is gay and less than a whole woman is a lesbian. What relief then.

Daily Mail story

Today's Daily Mail reassurance to its readers Credit: http://twitter.com/catherine_mayer

I’ll quickly pass by the unquestioning, slovenly and uncritical journalism of the Mail since that is what we must expect, sadly. A more considered appraisal of the National Office of Statistics survey is in the Guardian. But, even there, the report needs to be taken with a massive dose of common sense and comments (I didn’t dare look at the DM’s) raise many of the issues that concern this report.
As someone who has commissioned and helped design some pretty groundbreaking market research in my time, I hesitate to tell the NOS how to do its work. This survey is, as many point out, deeply flawed. I have no issue – albeit by a government paid-for quango –seeking to go where others fear to tread, indeed that maybe their role. But given that every homophobic newspaper would glibly misinterpret unquestioningly its results, it really should have been more careful.
When teaching questionnaire design I always proposed two rules:

  • only seek answers that you can do something with (useful, not just ‘interesting’);
  • only ask questions to which there is a definitive answer and that can actually be answered.

I am not at all sure what purposes the NOS is intending to use its data for: predicting levels of ‘hate crime’ perhaps; managing growth of same sex households and their demands for schools? Mmm.
The question itself is perhaps more interesting: is there a valid answer to the question “What is your sexual group?” Given a list (and the methodology employed by NOS seems to me wholly flawed – why didn’t they use self-administered computer based trade-offs, for example?) first of all assumes that people accept the idea of a group definition. Many would not, yet they might engage in behaviour that is a characteristic, supposedly, of such a defined group (all ‘gay’ men like “Mama Mia”). The behaviour itself simply defines an action at a particular time: so a man who marries a women and has children, then lives with a male partner is what? Depends when you ‘measure’ what behaviour – the relationship, having children, sex with a man or a women. And at what point does the behaviour characterise the person as being part of a ‘sexual group’?
In such complex circumstances people will always moderate their answer to the most favourable light – as they see it – at that particular moment: doctors have rules of thumb for what people say their smoking/drinking/sex habits are. They may also have personal qualifiers that shape their answers: some men who have sex with other men who are the ‘active partner’ do not regard themselves as homosexual at all, but regard their sexual partner as such. That may be a valid description for that person but it wouldn’t have helped in this survey to arrive at a useful, valid answer.
The safest, most reliable way of getting data is to ask about actual events that can be reliably recalled. When did you last have sex with another person? What gender was that person? Do you have sex regularly (monogamously even) with that person? Are you in a long-term relationship? What sex is that person? And so on, building a volume of behavioural information that is more likely to be accurately recalled – but not necessarily truthfully told unless the methodology is secure. Take a group of answers together and you might, only might, be able to lump them into a group behavioural description: gay, bisexual, lesbian, whatever. I’m not going to even attempt to do that!
So, while this survey is interesting, it’s far from useful and probably not valid. If it gives comfort to homophobes then perhaps that can’t be bad. But watch out. Whatever the DM and Telegraph might tell you, the chances are you have a gay, lesbian, transgendered – or just plain heterosexual kinky – neighbour. Keep twitching those curtains.

Advertisements