Saturday, February 14, 2009

Surveys: Movies, McCune and NM Community Capital

Earlier today into the email inbox there popped a survey that claimed to "take approximately 20 minutes of your valuable time." The survey was said to be a joint effort of the McCune Foundation, which sometimes does odd stuff, and New Mexico Community Capital, which, in my limited experience, lies among the good guys. The response here comes from a long, if peripheral, association with survey research. the movie comment is t the end.
The McCune/NMCC survey claimed to be "market research targeted at a statewide examination of the Strengths and Gaps/Opportunities in the New Mexico entrepreneurial ecosystems."
The "20 minute" part wasn't true. Whether it was a lie or an error, I can't say. But completing the survey, all ten pages or so, would have required at least an hour. The survey began with, "Which of the following categories best describes your role in the New Mexico Ecosystem?" One category was "C-level manager." Define please.
The next four asked the respondent to "rank the top 5 current industries that you associate with New Mexico." One "industry" was "food Services." Define please. Restaurants? Then the request was to rank the top five fastest growing, fastest declining and "most strategic industries that you associate with future growth in New Mexico." Food services only made the first of the four questions.
These questions are hard. Thought is required, even for someone like me with knowledge of the numbers.
It got better.
The survey went on to a detailed investigation of the respondent's knowledge of New Mexico Community Capital and of technology transfer and capital structures in the state. No regular person could provide meaningful answers without having spent a fair amount of time in the middle the topic. I suspect that few people have heard of New Mexico Community Capital.
There was no indication that the survey was going to a sample. I suspect the audience consisted of a bunch of email addresses accumulated over time. If that guess is correct, the responses will mean only that they are the responses to the survey. Nothing may be imputed from the survey. I look forward to seeing the news release about the results.
Recently the New Mexico Film Office released a study of the economic impacts of filming in New Mexico that purports to counter last year's report from New Mexico State University saying the film subsidies were a big-time loser. The new study, produced by an outfit with a huge vested interest in favorable news, totals data gleaned from a variety of sources and surveys, which I think is statistically dubious, to claim a positive return on the subsidy. The problem is that differing surveys have differing things happening with the probability theory and therefore the results can't be added. I may be wrong as to whether this notion applies to the new movie survey.
In any case, the survey summary has a wonderful line. It says "the length of the average tourist’s stay in New Mexico increased by 1.2% due to interest in" film attractions. OK, so what is "the average tourist’s stay in New Mexico?" And if the stay is 100 hours, the average increases an hour, a statistically insignificant length of time.

No comments: