LibQUAL+ or Why it is important to understand statistical significance… August 24, 2007
Posted by Ian in QEII, academia, libraries.trackback
Yesterday a session was held to go over the results from our most recent iteration of the LibQUAL+ library quality survey. For those of you not in the know LibQUAL+ is a massive survey put out by the Association of Research Libraries (ARL). The aim is to measure perceptions of the library in three broad categories: Service, Collections/Technical Services, and Library as Place (aka facilities). In the survey users are asked about a variety of functions and for each to give a number between one and ten to represent the minimum level of that service that the user would expect, then to give a number between one and ten to represent how the library is actually doing and finally, a number between one and ten to represent the users desired level for the item in question. Statistics are then gathered about the gaps between minimum and perception, as well as between perception and desired. Libraries are then supposed to go out and use these stats as indicators for what needs fixing, as well as use them to compare their situation with other similar sized universities. The survey is very broad and its usefulness is dubious at best but what really gets me is the sample size chosen when we completed our survey.
Memorial has about 17000 students add in faculty and administration and there are probably closer to 19000 users (probably even more than that). Now considering that most surveys get between 20-30% response rate one would think that you would want to send the survey to as many users as possible. But no, we sent the survey to 5500 users… that’s about 29% of our users in total. Our response rate was 18% or about 890. 890 out of 19000… That is 5% roughly of our user base and yet this is a tool that is supposed to be meaningful? You cannot base any decision on what 5% of your user base thinks.
Here’s something really depressing for the math nerds out there. For the most part the numerical data that is gathered in this survey is mostly ignored – or people will say “look our users gave us a score of 8 in [insert service thing here]” when the actual score is meaningless for comparison purposes.
Now some say that this tool is a good basis for historical comparisons. I say hogwash! For one thing you are not surveying the same people over time. The users you have now may have very different expectations from the users you have 5 years from now. Even if it were possible to survey the same cohort every year, I think you might find that the results would not be consistent as user needs evolved and changed.
If libraries and librarians want to do serious research more of us need to be more conversant in the language of statistics and we need to invest more time in thinking about experimental design.
Ian, I could quote homer’s quote about statistics, but you know as well as I do, Stats are useless and dont really mean anything, because with any bias you can change their meanining. One more thing about surveys is that I feel most people give evaluations that are much higher than they should give. They dont want to be mean. Another point is that they people who respond to things like this are usually the ones who have a strong feeling. The middle of the ground people usually dont fill them in.
Unfortunately in the social sciences surveys are about the only way we can collect data that gives us any kind of qualitative feedback.
Funny you should mention the bias inherent in statistical interpretation. I just bought a book for general interest mathematics on how to present statistics to show what you want (not necessarily what you found). I think it will be quite popular
The biggest issue is that there really is no way to collect data with bias.
As Professor Farnsworth from Futurama so eloquantly stated is regards to a photofinish at a race: “changed the outcome by measuring it”.
Feedback and statistics are for bean counters. The way I like to evaluate myself is how happy each customer I deal with is. I am only as good as my last customer.