Yes.

Not necessarily in their design, but certainly through the use we make of them.

It is simply a matter of numbers. You have a panel of 100,000 people, 10% qualify for a study (geography, other criteria, whatever), we have only 10,000. We get maybe a 5% response rate because we’re in a real hurry to get done. Hmmm 500 people. The client wants a sample of 750. You end up reaching out to any possibly qualified person. Period. Sample of convenience.

Now, what many people will not acknowledge is that many phone studies do much the same thing. You start with RDD (or more likely an RDD list that was filtered to exclude dead numbers , cell blocks, etc). Great. The call center is supposed to “work” the numbers (7-8 times? I forget the exact number)> But to keep up the flow rate, most operators of a call center will tell you new numbers are always more productive than re-dials. If they didn’t answer last time, or you got voicemail, the odds are high they not answering this time either. You try, but in the end when costs start climbing you buy or generate more sample.

This in itself doesn’t not convert RDD into convenience sample. It’s just the start.

You have do-not-call lists (not the gov’t one which MR is exempted from but private ones maintained by research companies where respondents say don’t call me again)

You have cell phones which you’re not supposed to call & a population going mobile & cutting the landlines at alarming rates.

You have an enormous number of people that refuse to respond to any survey by phone. Ever. (mega-non-response bias)

So when a HUGE percentage of your sample can never be contacted by “random” selection, what the hell happened to the sample frame? Randomly selecting numbers that you KNOW can never be contacted is RDD fiction.

In the end, phone studies have the same criteria as opt-in panel studies. Get the job done, keep the costs down.

To get those last couple of quota groups fills, methodological short cuts are taken.

Any one you can get on the phone to do it, you get.

Is this perfect? No.

Is it science? No.

Can we get actionable data with which to make business decisions? Yes.

Absolutely.

Quoting the apparent divergence of  “accuracy” of say 10% -assuming we can measure this in OPINION research - would you change a business decision if 37% of the respondents hated something, and  suddenly it was found that really 27% people who did (this is an over simplified example for illustration).

All studies have sampling error of some sort.

The deciding factor should be “can the client, who is paying for the study, make a good business decision based on the data supplied”. If the answer is yes, I can live with whatever sampling methodology (or lack thereof) used.

Even with the most rigorous design, and perfect execution (possible maybe in the 1970’s) you have statistically valid sample. It is still OPINIONS of PEOPLE we are asking. Opinions change, with mood, with weather, time of the month, the economy, whether your kids are screaming or the dog ate your shoes that day.

I have never believed sampling to be science when it comes to the opinion part of a survey. Demographics? Absolutely. Telling us about things you own. Sure. But about how you FEEL about something? I don’t think so.

Bottom line. Most sample is sample of convenience for practical reasons and this is not a crime. This is not a science lab and we are not using the same chemicals to replicate a formula. THAT is science. If I ask you what you like about a brand of toothpaste, that is the capture of an opinion true at that time (we hope) but nothing more.

All things being equal, and not supposing a grand conspiracy among respondents to lie, I think a snapshot in time is a good thing. We are capturing a moment in history, and grouping popular sentiment. And the law of large numbers kicks in even in samples of convenience.

I think it is more than accurate enough to tell good from bad. And really that is all the client needs, most of the time.

Is this a good decision or a bad one? If you can answer that question, we’ve done our job.

I have rambled on enough.

This was typed on the spot with no thought of composition. Stream of consciousness rant.

My thoughts are certainly not replicable at a 95% confidence level.

But tomorrow I’m still going to feel this way. So its a valid opinion. As good as we can get in opinion research.

Cheers