I travelled to attend a key event in the Make Opinion calendar in Japan this year (my favourite country to visit). I usually join a handful of events every year for what I call "business pillow talk". What trips like these also give me is the rare opportunity to see our clients face to face; whether it's the fresh pacific air or the rush from 30-hours of go-go-go, I always find these conversations extremely compelling. Sometimes they don't even end, but continue over shabu-shabu at dinner. By the first few days in Tokyo, three of these talks came down to the same question and by midnight I was up in my hotel room writing a product brief.
It started with a client of ours, a market research agency in Japan, asking us a question I couldn't quite answer: can we run a screener, three to five questions with the same respondent, to get an incidence rate estimate in minutes without breaking the bank before bidding on a project? As an agency, they needed a scalable way to quickly understand the proportion of respondents that would qualify for a given study before they commit to it.
As our client explained their situation, I realised that, while our custom screeners could do the job, there's a tonne of room for optimisation when it comes to doing it fast, and almost for free. So, over the next week, we discussed their challenges together to develop a solid understanding of their needs. Then we built and tested a custom feature on our market research platform that would do the job. From talks to tech, it was the most memorable (and shortest) product sprint of my career.
Why do agencies in the region experience friction around incidence rate (IR) estimation? In Japan, it's the market research agency's problem to get IR right, not the research buyer's. Without a feasibility check upfront, you simply don't get invited to pitch. To make the estimation problem worse, there isn't deep omnibus infrastructure to lean on the way European agencies do; the large firms running weekly nat-rep omnibuses in Germany or the UK simply don't exist in the same form there.
Why what we already had wasn't good enough
After twenty years in this industry, I've seen first hand how poor IR estimation can eat through research buyer budgets in the background. When I started my career in market research, fieldwork already took weeks, so a wrong guess on incidence coming from the client's side wasn't that painful for market research agencies to absorb. They'd simply go back mid-relationship and ask for more budget or time.
But online panels changed everything about how market research agencies operate, most notably that their projects gain momentum a lot faster. Fieldwork today is measured in days, not weeks. Agencies are quoting projects on tighter timelines and thinner margins and an incorrect IR from the client's side simply takes the wind out of a project's sails right when it picks up. It's a solvable issue, but the time and budget strain simply adds unnecessary friction which clients remember the next time a brief goes out.
We were already using free custom screeners that clients could run through our research panel before launching an online survey as a solution (since we trust who's responding from our 95% profiled proprietary panel). In practice, however, our custom screeners weren't perfect for the bidding use case. First, they don't have quota control. If a client needed to follow national representativeness on age, gender and region, a custom screener can't follow that distribution. Second, they can't share a logical sequence of questions with a single respondent.
The concept we built around
What our clients in Japan needed wasn't to screen people out, it was to quickly find out how many people would qualify, how long that would take and what it would cost, all during project negotiations. So we reverse engineered the problem: no one gets screened out. The survey's screening logic is still defined upfront, but every respondent answers every question in sequence. As for the quota distribution problem, all respondents follow the distribution that is set for all the audience screening variables. Incidence rate is calculated by retracing the screening criteria that is defined.
Result? An incidence number based on real responses that arrives in minutes (and at under about €50). It also means zero rejection messages and fewer mid-survey dropoffs for respondents. Everyone gets fully compensated, but data we collect from the people who wouldn't have qualified if this were simply a screener is counted toward the ultimate IR estimate retroactively. For 200 completes on three to five questions at a few cents a response, that wasn't bad!
The outcome for our client in Japan
After a week of development and testing, we launched a new research type on our market research platform called Quick Incidence Check (QIC). The first test we ran with QIC in Japan came back with 211 completes in under 9 minutes with a CPI of ¥38.92, or about 20 cents (quota criteria, age and gender, were built in). Once launched, QIC shows incidence rates live, broken down by question stage, indicating at which question the drop-off in qualification really happens.
How does it work? Instead of stopping the respondent from finishing the survey by screening them out, we compute a live question level incidence rate based on all data recorded. For example, assuming respondent A would have been screened out at q1, they are allowed to complete the survey. Because they would have been screened out if this were a normal project, their data is not taken into account for any computations after q1. This applies to all answers across all questions for all respondents.
What's next?
Thank you to every Make Opinion client spending time to share their problems so we can have a real opportunity to solve them. Perhaps the biggest challenge to solve next is this: if we call understanding a client's problem "business pillow talk" (I want to trade mark this expression), what do we call solving that problem in a week? I'm open to suggestions!
If you'd like to try out QIC on an upcoming project, reach out to [email protected] for more information.
