Statistics culled from voluntary poll data, collected primarily online, are routinely provided to Greater Sudbury city council members in reports to help inform various decisions.
The problem is, this data is statistically irrelevant to the municipality’s population as a whole.
“When there’s no random selection and people get to volunteer for a survey, it’s basically just a glorified online suggestion box,” Nanos Research chief data scientist Nik Nanos told Sudbury.com. “It’s only representative of the people who actually decided to fill it out.”
This voluntarily provided data follows the idiom, “The squeaky wheel gets the grease,” because only those interested in the topic at hand generally respond, which skews results in their favour.
“You never make important decisions using something like that,” Nanos said.
A recent example is the French Language Services Policy review, for which an online poll was posted on the city’s Over To You public consultation page online.
The page received 632 visits, from which 243 people completed an online survey.
Statistics Canada shows that 62.3 per cent of the Greater Sudbury population speaks English only, 0.9 per cent speak only French, 36.6 per cent are bilingual and 0.3 per cent speak neither official language.
The city’s Over To You public consultation found 38 per cent of respondents self-identify as Francophone, 30 per cent identify as Anglophone, 25 per cent identify as bilingual and five per cent identify as “other.”
As such, the poll data disproportionately represents Greater Sudburians who speak French.
This is typical of voluntary polling, Laurentian University Social Sciences professor Dr. Parveen Nangia told Sudbury.com, who echoed Nanos’ opinion of the city’s methodology.
“It is not representative of the population,” Nangia said, adding that the survey only represents the 243 people who completed it.
“People who are participating in the survey are the ones with some kind of interest in it, and in that sense, the results cannot be generalized to the entire population.”
The municipal report that included French Language Services Policy polling data includes some context, such as the fact the polling technique was voluntary and not random. This is also the case with other municipal reports containing voluntary polling data.
Even so, Nangia said additional context regarding the data’s relevance should also be included.
“They should be more clear in speaking about the drawbacks of these surveys, or the shortcomings of these surveys — what is covered and what is not covered, and how it could bias their research,” he said.
This context, he added, “should be mentioned in the report,” which Nangia said “would make things more clear to everyone, whether it be the general public or the people working in the city.”
Similar low-quality voluntary polling data has been used to supplement municipal reports to city council on various other things in recent months, including those informing such major decisions as the city’s overhaul of emergency services infrastructure and their ongoing review of aquatics facilities.
Data collected through voluntary polling isn’t entirely without merit “as a way to collect ideas and hear a diversity of views,” Nanos said, but “no one should look at these results and say, ‘This is what people in Sudbury think,’ because that’s not the case.”
To find statistically relevant results, Nanos said he would conduct a randomized telephone survey and check respondents’ profiles against a few variables to ensure they parallel the general population base, which might include weighting data to make it fit the target community.
“You don’t necessarily need a massively large survey to be statistically reliable or representative, you just need to make sure the people you’re interviewing are representative of the community,” he said.
For Greater Sudbury’s population, LU prof Nangia said he’d aim for a sample size of approximately 385 people.
City CAO Ed Archer defended the use of the statistically irrelevant polls to Sudbury.com, arguing the polling data — despite being low quality and not truly representative of the general opinion of the Sudburians — is just one of many tools staff provide to city council members to aid in decision-making.
The data drawn from voluntary polling and provided to city council members “isn’t intended to be definitive or declarative about the views of the community as a whole,” he said. Their goal, he added, is always about gaining insights from members of the public and providing them with an opportunity to engage.
“I could understand the concern about the non-random polls, but that’s not the world we live in,” he said.
“There’s a rational element to all our decisions, but there’s also a judgment, and council’s responsible for exercising their judgment. Our job is to try and help them feel comfortable doing that, and (voluntary polls) are just one more tool in that process.”
It has been a few years since the city enlisted a third-party to conduct a randomized survey, he said, clarifying that municipal staff will always work to provide the tools city council requests, which is continually evolving.
“If the answer (to a decision point of city council) was available from a statistically relevant piece of data, that would mean there’s one right answer,” Archer said, adding that this is not the case.
“The data will get you to a point, but at the end of the day you need judgment, and council’s ability to use their judgment is most important.”
Tyler Clarke covers city hall and political affairs for Sudbury.com.
