This will pr’bly only make sense to a few of you out there, but it was amusing to my cold-addled brain, so I thought I’d share.
Part of my job involves some nominal oversight for creating samples for electronic surveys. It’s not a big part, but because it’s something that we’ve been seeing a lot more of, it’s sort of been getting a lot of attention lately, so I’ve been more or less muddling my way through the process of trying to create a way to figure out how much electronic surveying we’re actually doing on our campus.
In the process, we’re trying to provide guidance to those conducting surveys about how large of a sample they reall need. It’s very easy to just say “Send it to everyone!” because with an email distribution, there’s no additional marginal cost for increasing the sample size. Unfortunately, when everyone starts doing that, recipients get fatigued from constantly being asked to respond to surveys and stop responding. So it’s one of those “tragedy of the commons” type problems where it’s really in everyone’s best interest to use an appropriately sized sample, even if any individual *could* (and usually does) take the easier option of just hitting up everyone.
So, the question of sample size is one that is covered in every basic research methods class, right? So it should be easy to figure out – run a little power analysis, figure out expected response rate, and you’re good to go. So I decided I needed to actually put myself through the paces yesterday for an institutionally sponsored survey and immediately realized that I couldn’t do a power analysis because I didn’t know what comparisons would be made with the data. So I emailed a bunch of friends in the faculty, some from our Psychology department and some from our Sociology department, and explained. And the Psych folks were scratching their heads along with me, but the Soc folks got it immediately and straghtened me out.
In summing up for all of us this morning, a dear friend made the following statement (edited somewhat to remove names and such): we were talking apples and oranges.. the question was how many apples do we need to represent the orchard within a confidence level. The Psych folks were talking about how many apples and oranges do you need to compare them to each other.” My response (again, remember – cold is eating my brain): “Having been brought up in an orange orchard, my thinking was more in line with the Psych folks, so I had to learn how to make fruit salad.. or.. something.”