Data analysis and experiments - John C. Kirk — LiveJournal
Oct. 15th, 2006
05:12 pm - Data analysis and experiments
This post is motivated by a recent discussion in nou's journal, which some of you may have read. However, I hope that it will also make sense in isolation. Basically, this is an attempt to explain the way that I discuss issues, in the hope that it will reduce future misunderstandings. This is a fairly long post, because I'm trying to explain this as clearly as I can.
When I was at school, I watched the film Soul Man. This was fifteen years ago, so my memory is a bit hazy, but it struck me as a very good film. The basic premise is that a young white guy is ready to go off to university, but he can't afford it (because his father is rich but refuses to pay), and then he finds a scholarship for black people. So, he takes a massive overdose of suntan pills, thus appearing to be black, and gets the funding. The snag is that he then has to maintain this illusion once he gets to university, so he gets to walk a mile in someone else's shoes. Anyway, I'd recommend watching it if you get the chance; I've added it to my Amazon DVD rental list so that I can see it again.
A few years later, I saw an episode of The Oprah Winfrey Show which involved an experiment along similar lines. Basically, they arranged for a young black man to go around asking people to tell him the time, while he was followed by a hidden camera. They then put some make-up on him so that he appeared to be white, sent him out again, and compared the results. In theory I think that this is a good idea, but I also think that the way they actually did the test was so flawed that it invalidated their results.
Thinking back to my science lessons at school, one important concept we were taught about is the "control experiment". This means that if you want to test the effect of one particular factor, you should try to keep all other factors as similar as possible. It's not always possible to achieve that, but it does represent an ideal to aspire to. For instance, one surprising fact in physics is that if you drop two objects at once, and one is heavier than the other, they will both reach the ground at the same time; most people would intuitively expect the heavier object to fall faster. That said, there are other factors that can affect this, such as air resistance. For instance, a skydiver in freefall will travel a lot faster than someone hanging from a parachute. So, if you wanted to test this theory, a good experiment would be to take two sealed boxes, one filled with feathers and the other filled with ball bearings, then drop them off a fairly tall building. A bad experiment would be to drop a pillowcase and a paperweight at the same time, since their shape/size could skew the results.
Coming back to the Oprah experiment, ideally you would want all conditions to be the same, including the same people who were initially approached. In practice, that's not feasible, and even if it was there might still be other factors, e.g. a person who is in a hurry to get to a meeting on one of the days. Still, even if you can't get an exact match, it still makes sense to do the best you can with the factors that are under your control. Unfortunately, they didn't do this. When their volunteer was in "black mode" (for want of a better phrase), he was dressed casually, clean shaven, and he approached people by swaggering around, knocking on car windows, and saying things like "Hey, yo, what time is it?" When he was in "white mode", he was dressed in a suit, given a beard, and approached people far more politely, e.g. "excuse me, could you tell me what time it is, please?"
Now, it's possible that these changes were irrelevant. In fact, some of them could even have made it less likely that people would help him; I've heard some people say that they don't trust men with beards, on the basis that those men have something to hide. The problem is, we don't know. I am not trying to deny that racism exists - far from it. In fact, I acknowledge the possibility that all of the people who refused to talk to the black man were solely motivated by the colour of his skin. My point is simply that the results were inconclusive, so they should be ignored.
There is a related issue here of source evaluation, which is a technique I learnt about when I did GCSE History. I applied this in my recent post about news coverage of two events. There are two key aspects to this: how direct is the source, and how reliable are they? The first aspect means that you should identify a primary vs secondary source; for instance, an eyewitness would be a primary source, whereas a newspaper article written by someone who spoke to various eyewitnesses would be a secondary source. For reliability, you should consider whether the person/organisation has anything to gain by misleading you. For instance, there are various tabloid newspapers which benefit from high circulation figures (so that they can sell advertising space), and this gives them a motivation to make their headlines as dramatic as possible. Now, just because someone has a motivation to lie, that doesn't automatically mean that they will, but I do think that it's prudent to maintain a certain scepticism about the information they give rather than accepting it all at face value.
Taking another example, CRISIS is a charity that deals with homelessness, particularly in London. I think that this is a good cause, and that they do important work - I helped them out as a volunteer about ten years ago, and I'm intending to set up a direct debit payment for them soon. Having said that, I also suspect that they have deliberately misled people in order to attract more funding, based on their definition of the word "homeless". If you look at the definition on their website (PDF), they say that: "Whilst rough sleepers are the most visible homeless population, the vast majority of homeless people live in hostels, squats, bed and breakfasts or in temporary and insecure conditions with friends and family." However, they don't mention that on their posters, and I can only speculate about the reasons for this decision.
I've mentioned a further example on my main website, where I discuss being a vegetarian. If you read the penultimate paragraph (just before the conclusion), this mentions the theory that someone who's been vegetarian for a few years would have trouble digesting meat. I'm not entirely convinced by this, since I know of a couple of people who've resumed meat eating after a period of vegetarianism, and I didn't have any trouble when I accidentally ate a small amount of chicken in July, so I suspect that this is just propaganda. I understand the agenda, and it's actually one I share, but I'm still cautious about accepting all the claims at face value.
So, the key point here is that just because I'm suspicious about a summary that comes from an organisation with their own agenda to promote, that does not mean that I have any grudge against that organisation or that I disagree with their agenda.
Turning to the field of mathematics, you may be familiar with the four colour theorem. The basic idea is that if you have a map of England, showing each county, you don't want two counties that touch each other to be the same colour, otherwise you wouldn't easily be able to tell where one ends and the next begins. The theorem states that you can do this with four different colours; you will almost certainly use the same colour for two different counties (e.g. West Sussex and Durham), but you won't need more than four to colour the whole map. In fact, this can be generalised to say that you only need four colours for any conceivable map (not just real world ones).
My personal experience of this theorem is mainly from computer science rather than maths. During my undergrad degree, my second year functional programming project (using Miranda) was to assign colours to an arbitrary map, assuming that the theorem was correct. During my MSc I took a module on Algorithmic Graph Theory, and this involved learning the proof for the six colour theorem (relatively simple, taking up about one page of A4) and the five colour theorem (a bit more complicated, building on the six colour theorem, and taking up about three pages of A4).
The proof for the four colour theorem is a lot more complicated, and even now there is some doubt about it. It has been generally accepted that the theorem is true for the last 150 years, but there have also been various failed proofs along the way (see the linked Wikipedia page for more details). So, when mathematician A said "I have a proof!", and mathematician B said "Actually, no, you don't - there's a flaw in it here", that didn't (necessarily) mean that B disagreed with the theorem, just that he/she disagreed with the proof.
The reason I mention this is by way of analogy. If someone says "I believe in X because of Y", and I disagree with Y, that doesn't automatically mean that I disagree with X.
One trivial example of this is a Usenet discussion from 1999, regarding an issue of Young Justice, archived here. It's a massive thread, so I don't expect you all to wade through it; the relevant comments are #49 and #50. Basically, there was a debate about which character had spoken a particular line, since the speech bubble was pointing off-panel. Elayne Riggs and I both agreed that Arrowette had said this (as did Peter David, the guy who actually wrote the comic, in comment #297), rather than an anonymous member of the crowd, but I pointed out that her reason for believing this was incorrect. Unfortunately, this meant that she then switched over to the opposite point of view; this wasn't my intention, but I don't regret saying what I did - I think it's important to maintain intellectual honesty.
I've recently been reading Carl Sagan's book "The Demon-Haunted World: Science as a candle in the dark". One idea that he's very keen on is "scientific literacy", which basically means training people to be able to make proper assessments of what they see and hear. I think that this is a good idea, so I hope that the proposed changes to science GCSEs will help with this.
This ties back into statistics. I studied that as half of my Maths A Level (and took it further during the first year of an undergrad Maths degree), and I have found it useful in everyday life. For instance, suppose that there is a debate about legalising cannabis, and someone says "X% of heroin addicts started out with cannabis, therefore it's a gateway drug". Whatever the percentage is, it's irrelevant, and the best way to understand that is to substitute a common item that most people use. For instance, "Y% of heroin addicts started out by drinking coffee, therefore coffee leads to drug abuse". It doesn't matter how high Y is, because there are plenty of people who drink coffee and don't go on to take illegal drugs. (Personally, I don't drink tea or coffee at all.) It's more useful to ask "What percentage of coffee drinkers go on to take heroin?", and compare this to the percentage from the general population, then decide whether coffee drinkers are more or less likely to progress to heroin than non-coffee drinkers. You can then substitute in cannabis, nicotine, alcohol, or the item of your choice, and the same logic will apply. I don't have a citation to hand for anyone who's made this mistake (I'll update this post later if I find one), so we can just consider this as a hypothetical situation for now. If such a situation arose, I wouldn't be denying a link between cannabis and heroin just by pointing out that the quoted statistic is irrelevant, I would simply be asking for a more useful figure.
Edit: In 1951, a congressional committee heard that over 50% of young heroin addicts had started with marijuana.
Moving on to my final case study, I've recently been involved in a discussion about unequal pay between genders, i.e. the claim that men are paid more than women. I use the word "claim" in the same way that lawyers use the word "alleged" - I acknowledge that other people believe this, but I am neither making that assertion myself nor denying it. In this context, someone referred me to the EOC (Equal Opportunities Commission), and I found a document on their website (PDF) summarising pay and income in 2003; their first key fact is that "Women working full-time in Britain earned 81 per cent of the average full-time earnings of men in 2002." When I looked through this document, I saw some problems with it.
For instance, the report doesn't give a breakdown by age. This means that the average could be skewed if salary tends to increase with age, but if women are likely to leave the workforce (full-time) at an earlier age than men. For instance, my salary has gradually increased over the last ten years, and I hope that it will continue to do so in the future. However, I have female friends from university (i.e. the same age as me) who have given up work in order to raise a family, so if they don't return to full-time employment then that means that their maximum salary has basically peaked already. I think it's reasonable to compare two recent graduates in the same job, and see whether they're paid the same. I don't think it's reasonable to compare a 50 year old man with 30 years of experience to a 20 year old woman with none.
In a similar way, I remember doing temp work for the post office when I was a student, delivering the Christmas post. My younger sister was doing the same job, but she was paid less than me. Is that gender discrimination? I'd say no, because it was based on age - I was 18 and she was 16, so I was allowed to carry a heavier sack of letters than her, therefore I'd (theoretically) get more work done in a given round. That may count as age discrimination, but I assume that it was brought in as some kind of child protection law.
This reminds me of a line from Astro City: "My father was the original Jack-in-the-box. He was a designer for Whamco Toys. A black designer in the early sixties was unusual, but he was talented - so they hired him, and paid him half what less productive white designers made." Granted, this is fictional, and comes from a biased source, but taken at face value I would say that this situation is clearly wrong. Similarly, I am aware that the large accountancy firms tend to hire new graduates each year; if one company had employed a group of people with equivalent degrees, but the men were paid more than the women, then I would say that this was also wrong, or that at the very least it would warrant further investigation. My point is that you need to compare like for like.
So, I shall now try to clarify my objections, bearing in mind what I've previously said about my methods. I apologise to anyone who thinks this is blatantly obvious and is getting bored.
Firstly, I am criticising the way in which the data has been analysed. I haven't seen the raw data (although rjw1 has since pointed me at this site), and I'm not challenging its accuracy or the conclusion, I'm simply saying that the analysis they've given doesn't necessarily support that conclusion.
Secondly, I often use analogies and anecdotes, when I think that they will help to illustrate a particular point. However, that does not mean that I am saying "if there's one example that's justifiable then every other example is also justifiable"; what I'm trying to do is demonstrate the need for a more detailed analysis.
Thirdly, I am suspicious of the summary because it supports the organisation's agenda. That means that I think it's prudent to check the report myself, but it doesn't mean that I disagree with their agenda. In particular, it does not mean that I believe in a "feminazi conspiracy"! I have only ever used the word "feminazi" when I've been quoting someone else, never to express my own opinion.
I make an effort to choose my words carefully, so it annoys me when people misrepresent me by taking a strawman approach, i.e. by criticising something that I didn't actually say. This is partly due to a quote from The Last Emperor, which has stuck with me ever since I saw that film: "If you do not say what you mean then you will not mean what you say, and a gentleman should always mean what he says."
This fits into my general trend of developing an "exo-brain". In the Science of Discworld books, the authors refer to "extelligence" (as compared to "intelligence"), i.e. the ability to store knowledge externally. In the long term, I like to think that my views are of interest to others, and that there will be a record of them after I'm dead and gone. In the short term, I find that writing is a skill which improves with practice (like many others), and that the exercise of gathering my thoughts into a coherent form actually helps me, so this blog is primarily for my own benefit.
I don't really want to get into a discussion about the specific examples I've mentioned here (e.g. racism, homelessness, or sexism). The point of this post is to explain the way I debate issues, so if you see any logical flaws then I'd be happy to hear about them.