I was sitting in a focus group facility somewhere in northern Europe, watching a presentation about a proposed integrated resort.
Anyone familiar with this genre of research will know the setting: the two-way mirror, the plastic chairs around a Formica-topped table, the slightly stale sandwiches, the sense of shared purpose that descends when a beautifully produced video begins to play. The screens showed the renderings: hotel towers, entertainment venues, a dozen cuisines, the casino floor, the pools. The respondents were leaning forward in their chairs. The enthusiasm was palpable. I could have sold them tickets right then and there.
The problem is that I wasn’t there to sell tickets. I was being paid to work out what those respondents would actually do, not what they said in the excitement of a well-produced concept presentation.
Over several proposed integrated resort projects in southern Europe, with research run across the main European feeder markets, I sat through enough focus groups to relearn something that ought by now to be a settled principle of consumer research: People’s stated intentions and their revealed behaviour are not the same thing and with aspirational leisure, the gap between them is measured not in percentage points, but in multiples. The focus-group industry, to its credit, has been learning this afresh every decade since such groups were invented.
The methodology is predictable in its flaws. You gather a group of consumers, show them a compelling vision of a world-class resort, and ask whether they would visit, how often, and what they would spend. Without a plane ticket to book or a hotel room to pay for and with no money at stake, the exercise is hypothetical and people, in the hypothetical, are generous with their own future behaviour. The version of you that exists in a focus-group room is a more adventurous and free-spending individual than the one who later checks the bank balance.
The numbers that came back were, in every dimension, too good. Both visitation intent and stated gambling spend per visit were inflated. Set against actual propensity and gambling spend per head in each of those countries, where real people make real decisions with real money, the discrepancy was large.
I should be fair about one thing. An integrated resort creates its own demand in a way that a simple extrapolation from existing behaviour cannot fully capture. There is also what I would call the airport retail effect: People spend far more in the heightened aspirational setting of a holiday than in their ordinary lives. Have you noticed how airports have become shopping centres? A traveller who would nurse a single coffee in their local Starbucks will happily buy a handbag or something in the departure area, which they could have picked up on their own high street. The setting changes the behaviour, accelerates spending, and an integrated resort is built to produce exactly that effect.
Even allowing for all of that, the stated preference data was too optimistic by a wide margin. Cyprus is a case in point. I worked on the feasibility of an integrated resort there. The project, now operated by Melco, may be performing in line with Melco’s own expectations, but it has come in well below what our consumer research suggested the demand would support. It is not alone in that, which is some comfort, though not much if you were one of the investors in the group I represented, had they won and gone ahead.
Paul Samuelson, an American economist and the first American to win the Nobel Memorial Prize in economics, addressed this in 1938, so it has been understood for nearly 90 years. His argument was simple: The only reliable signal of preference is revealed preference, what people choose when both the choice and the consequences are real. Asking people what they would do, in the relative comfort and safety of a focus group, tells you something, but not what they will do when the moment arrives and the decision involves real money, time, and commitment. The gambling industry, like many others, has been paying consultants, myself included, to relearn this on a regular basis ever since.
A sociologist, Richard LaPiere, made the same point earlier still. In 1934, he drove across the U.S. with a Chinese couple. More than 250 hotels and restaurants served them without difficulty. Some months later, he wrote to the same establishments asking whether they would serve Chinese guests. Ninety-two per cent said they would not. The gap between the stated attitude and the actual behaviour was almost total. His study was about racial prejudice, not casino visitation, but the dynamic is identical: What people say, on socially loaded or aspirational topics, is a poor guide to what they actually do.
Environmental economics has its own name for this, the “green gap.” Survey after survey shows that around 50 per cent more people identify as environmentally conscious consumers than actually buy green products. Only about 16 per cent of those who say they are concerned about the environment act on it at the point of purchase. Researchers have found that social-desirability bias inflates stated valuations of environmental goods by nearly three times against the revealed preference data. People say they care, but the till roll says otherwise.
Which brings me to the point I have been circling. A focus group is one way of collecting stated preferences. An online survey is another. The two look very different — the mirror and stale sandwiches on one side, a questionnaire on a phone screen on the other. But they are collecting the same thing: what people say, not what people do. Changing the delivery does not fix the defect. If anything, the questionnaire gives up the one small advantage the room has, which is that you can watch people.
And people do not answer surveys truthfully. A large research literature on this goes under the heading of “social desirability bias.” Respondents overstate the behaviour they think is approved of or expected in the survey.
Online surveys add faults of their own. A meaningful share of respondents are not really answering. They straightline down a grid ticking the same box, they speed through to collect the incentive if one is offered, and a professional minority sit on several panels answering surveys for a living. An online survey is not a more honest instrument than a focus group. It is a cheaper one, with a different set of problems.
The specific danger in gambling research, and this is where the focus group and the regulator meet, is what I would call “quantification bias.” Once a number has been produced by something that looks like a scientific method, it takes on an authority that is hard to challenge. Stated gambling spend from a focus group becomes a line in a financial model, which goes to investors, who commission independent reports that cite it; thus, the number has been laundered through several layers of apparent rigour into being treated as fact. Pointing out, at that stage, that the original figure came from a room full of people who were not spending any actual money tends not to go down well in a board presentation.
The same thing happens in regulation — with an extra twist. A survey produces a problem-gambling prevalence rate; the rate finds its way into policy documents, ministerial and parliamentary speeches, and advocacy press releases. Everything I have said about truthfulness applies here too. But a gambling survey carries another problem that a focus group does not and it arrives before anyone has answered a single question: topic salience. Call a survey a gambling survey and you over-recruit gamblers, the more engaged among them in particular, in the same way that a survey called “Fast Cars” would fill up with petrolheads. The sample is undoubtedly skewed.
The evidence for this is not subtle. When people opt into a survey, the rate goes up. The Gambling Commission’s Gambling Survey for Great Britain, which recruits by pushing people to a web questionnaire, puts problem gambling at 2.5 per cent. The NHS Health Survey for England, run face to face and using the same PGSI threshold, has put it at well under one per cent, nearer 0.3 to 0.5 per cent. Measuring the same thing in the same country differs by several times.
The Commission, which once cautioned against reading across from one survey to the other, now stands behind the higher figure. That has not settled the point; it has just sharpened it. A gap of that size is far more likely to come from the recruitment methodology than from a real jump in problem gambling of that order. That is not necessarily a measure of how much problem gambling exists; it is more likely a measure of who bothered to click.
The figure of 400 annual gambling-related suicides in Britain is certainly a case of quantification bias. It comes from clumsily and lazily applying a ratio taken from a Swedish hospital study to UK problem-gambling estimates and the Swedish researchers themselves concluded that gambling did not appear to be a significant independent risk factor once other mental-health conditions were properly controlled for. None of which has stopped the figure being cited in parliamentary debates as established fact. A specific number carries an authority the original underlying research cannot support; 400 sounds like someone sat down and counted, but nobody did.
The Commission’s handling of the GSGB follows a familiar pattern. The independent statistician it commissioned to review the method identified a real risk that it substantially overstates the true level of problem gambling. That warning sat in the technical documents, while the headline figures were put in front of policymakers without it.
A separate Commission survey, in which 77 per cent of respondents opposed affordability checks, was described in the published summary as reflecting “a wide range of views”, which, in the immortal words of former Cabinet Secretary Sir Robert Armstrong, is being “economical with the truth”. Seventy-seven per cent against is a wide range, in the sense that it is a long way from support for the proposition.
I am not arguing that stated-preference research is worthless. A focus group that shows genuine enthusiasm for a concept is a useful signal, even when the numbers it throws off are too high. It tells you the direction of preference, if not the magnitude. A good feasibility analysis uses stated-preference data as one input, suitably discounted, but not as a firm foundation. The same goes for a survey, read with its recruitment and caveats in full view.
The problem is the same in both, stated preference treated as though it were revealed preference. In commercial development, when enthusiastic focus-group answers go into a financial model without a scepticism discount, the error costs investors money. In regulation, when a survey showing elevated problem-gambling rates drives policy without anyone admitting the method probably overstates the position, the error costs operators their competitive position and consumers their freedom of choice. It is the same mistake. It differs only in who pays for it and how long the bill takes to arrive.