A 50th Reunion Survey? What’s So Hard?

When the guys on the 50^th Reunion Committee asked if I could help out with a class survey, I readily agreed.  I’ve had decades of experience as a survey researcher, and it didn’t seem like this would be a difficult assignment.  It soon became clear, however, that we would have some challenges to overcome.

In my trade, the aim is to produce accurate, objective, relevant, timely, survey-based information that helps the client achieve their goals.  It that’s the aim, it kinda helps if the client knows what their goals are.  “Let’s do a survey” isn’t quite enough.  So our first challenge was to work through what the survey would really be about.

Any of you who’ve served on boards of non-profit organizations know the challenge — getting consensus on a precise description of what starts as a diffuse wishlist.

Setting Goals

After some extended discussions with the reunion team, it gradually became clear that the Reunion Survey would need to serve more than one purpose.  Several of our classmates are going to be preparing essays for the 50th Reunion ClassBook, and they hoped that the survey would give them insight into where the class as a whole has been and what—collectively—we’ve accomplished.   Some of the essayists were looking for deeper insight into “the meaning of it all.”  Others wanted interesting individual stories that would shed light on our various life journeys.

Yale ’69 is by no means the first class to do a 50^th Reunion Survey, and other classes have sometimes taken a fairly light-hearted approach, asking fun questions that sometimes delve into matters usually kept private.  We all know, too, that our class is a kind of reference group for many of us as we go through life; many of us would be curious to compare our individual life experiences with those of our classmates. Survey statistics—averages and ranges—would surely be of interest: how much we make, which fields we worked in, how many kids, how many marriages, etc.

After much discussion we eventually reached consensus that we would design the survey so as to partially meet all of these goals.  But how could we do that without making the survey too long?  And without violating the confidentiality of private information?

Avoiding Systemic Sources of Error

Defining the goals wasn’t the only challenge. We who design surveys professionally worry about several types of potential error that can affect the survey results.  First, we worry about coverage error, which occurs when the list that we sample from does not include all the members of the population of interest.  Of course, Yale knows exactly who is affiliated with the Class of 1969, and they’ve done fairly well in tracking deaths, so I thought we’d have fairly complete coverage.  Then I saw the actual list: many of our classmates have no email address on file, or no phone number; a few even have no known addresses.   If we want to reach everyone in the class, we won’t be able to rely on email alone.

We survey researchers also worry about sampling error, but that’s a non-issue in this case, since we won’t actually sample from the class list; we’ll just invite everyone to participate.

Then there’s non-response error, which happens when those who respond to the survey are substantially different from those who don’t respond.   There is no way to address this risk completely, for we can be dead sure that men less engaged with Mother Yale will be less likely to respond.  But we can minimize non-response effects by being persistent and creative in soliciting responses.  (Sorry, guys: we ruled out the $100 gift certificates to responders.)

Tied to this is the issue of response rate.  We worried that some of our classmates wouldn’t respond if the questions were too personal.  And we worried that a long questionnaire would be a turn-off.  On the other hand, we learned that previous classes got very good response rates even though their questionnaires were fairly long and did get personal.  So, we followed their example, so as not to sacrifice breadth of measurement for the sake of avoiding a small number of survey refusals.  We also tried to steer between depth and frivolity in our questions.

Finally, we considered the problem of measurement: how do we word our questions so that they elicit accurate responses?  Here the solution is two-fold: work carefully on question wording, and where possible, use standard questions that derive from reputable, national sample surveys that have been stripped of biases.

Privacy, Anonymity and Validity

As for that troublesome issue of privacy, we agreed early on to set up the survey to use a fully anonymous protocol.  Our survey invitations will make clear how anonymity is provided and how we will secure the collected data, which will not be shared with Yale at all.  We won’t issue individual passwords to classmates, but will allow everyone to access the survey using the same URL.  To increase the response rate, we’ll issue multiple invitations, including outreach through regular mail, email, and by phone as needed.  As each classmate completes the online survey, they will be switched to a separate webpage where they can tell us that they’ve done the survey and thus opt out of further reminders.  We’ll have no way of connecting these completion notices to specific completed questionnaires.

Thanks to advanced features in Qualtrics, our chosen survey tool, it will still be possible for survey takers to interrupt their work and resume later (as long as they don’t switch to a different device).  That mechanism is based on Quatlrics putting temporary “cookies” on your computer.  (The more paranoid among us can elect to turn off cookies, which is fine as long as they finish the survey in one sitting.)  And for the ultra-paranoid, you can always skip questions you don’t wish to answer, or you can just choose not to respond.

So, it’s not been as easy as I first thought.  But thanks to the hard work of the committee and classmate Mike Baum (TD), who pulled together the initial survey draft in consultation with me, we’ve gotten to a pretty good design for this survey.

I do hope everyone in our class will take the time to respond. I can promise that when you click on the link we’ll send you, you’ll find some fun things and some thought-provoking  questions about your life, your experiences (good and bad), your feelings about Yale (then and now), and even a short assessment of your basic values.  We’ll be launching the survey soon, and I can’t wait to see everybody’s (anonymous) responses!