One of the recent trends of the last few decades has been to measure things and hold people accountable for those measurements. As a culture, we measure everything and then punish people for not living up to some arbitrary standard. Or worse, we base a person’s income on those performance measures. Anyone who thinks about it for a minute will tell you that’s a bad idea.
In his book, The Dilbert Principle, Scott Adams (whom I generally don’t agree with) printed a letter he got from a reader. In order to encourage beta testers to find bugs and programmers to fix them, a monetary amount was attached to both finding and fixing these bugs. In short order, a ‘conspiracy’ quickly developed in which the programmers would introduce bugs, tell the testers, who would report the bug and then the programmer would quickly fix the bug that he had inserted into the code.
When you tie performance measures to something, then people will game the system. It’s the what we do.
When I was teaching, there were two performance measures, test scores and graduation rates. So, instead of teaching, we spent nearly three months preparing students for the test. And there were many programs for helping students graduate. For example, the state I taught in had a law that students missing more than 9 days in a semester could not get credit for the semester. Thus harming their chances of graduation.
So the school set up a program in which students could “make up” hours by attending an after school program at the end of the year. This program was advertised to students early in the school year, with predictable results. Student attendance plummeted. I was asked to help students gain their science credit (for a not-insignificant amount of extra pay).
The end result was that students didn’t attend class, but by attending an after school ‘class’ for three weeks they could get that credit. That’s three weeks total, for all classes, not just one. During this process, I asked several times to ‘tone down’ the work I required of students in this program. Considering the entire program was worksheets and end of chapter texts, I thought it was ridiculously easy anyway.
Most of these students went on to graduate, in spite of missing well over 18 days of school over the year.
Another example is the performance measure of learning assessments. I am an advocate of assessments. Everyone likes their teachers, nurses, dentists, dental hygienists, plumbers, electricians, cosmetologists, etc certified. Those are performance assessments. But few people like high school students tested. I understand why. Because the tests are rarely used for the purpose for which they were designed.
They are a tool to determine if a student has learned the curriculum. Just like determining if an electrician knows all the rules and safety procedures for his profession. Unfortunately, these tools are often used as a measure of teacher or school performance. Students are rarely, if ever, held accountable by the tests that determine if they know what they are supposed to. Instead teachers are held accountable for those students. Which is stupid.
I’m reminded of the old saying “You can lead a horse to water, but can’t make him drink.” The best teacher in the world cannot make every student interested in their course. In my old school, I had the highest pass rate for the biology test, almost 33%. That’s the highest pass rate out of six biology teachers and 12 science teachers.
You can’t teach students who are not present (for reasons ranging from skipping class to jail time), who are stoned, who don’t have the foundations for the class, or who have decided that they make more money selling drugs.
But teachers and students aren’t the only ones with this performance assessment issue. Recent news is filled with stories about people (usually black) who are being released after wrongful convictions. This story, is particularly heart wrenching. Jonathan Fleming, convicted of a crime 25 years ago was released. According to the story he was in Florida and convicted of a murder in New York. He had a receipt for the phone calls he made while in Florida, but that evidence was never presented in court. It was found in his police file 25 years later. Police, apparently, coerced a woman into giving eye-witness testimony that he was the murder. But she recanted just weeks after the trial.
This isn’t an uncommon case. Police have performance measures: number of tickets issued, arrest rates, drug busts, and the like. District attorneys also have performance measures, usually convictions. But there’s no performance measure for correct convictions. Nope, just like the software bugs in the Dilbert an ‘economy’ seems to have sprung up among some police and some DAs.
There are other examples, probably in almost every industry that exists. Big chain stores and restaurants often use performance measures that are directly tied to employee pay and/or rating. There are stories of these performance measures being effectively impossible to meet. Of course, these performance measures can be made difficult in order to drive out employees that are paid too much. This lowers costs and reduces overhead for the company (examples).
I know, anecdotes are not evidence. I’m not a social researcher, but these are not hard to find stories. They are common.
A performance measure is just that. It is a tool used to determine the knowledge, skills, and/or abilities of a person or organization. As we have seen, there are some serious problems with commonly used performance measures. Either they are being used incorrectly (student assessments determining teacher or school performance) or they are easy to game (programming bugs and police/DA convictions) or they don’t measure the right things anyway (police/DA convictions ).
A performance measure has to be something that is both objective and meaningful. But it also must be applied correctly to be of any value. For example, the purpose of a high school test is to verify that a student has the knowledge, skills, and abilities to move to the next level or to graduate. If the student doesn’t, then the assessment should be used to inform teachers, parents, and the student where the student is lacking and be used to develop a personalized plan that can help the student.
For work that I’ve done, if a student misses a question, then I can almost always tell you why that student missed the question. It’s not just that it’s wrong, but that the student has a particular misunderstanding that resulted in that question being wrong.
There are also cases where performance measures are actually harmful. While conviction rates are an objective measure, they are not meaningful. Because one must assume that DA got the answer right. What if they got the wrong answer? We don’t know… we may never know. Using an unknowable measure just isn’t valid. I would much rather deal with a DA with a low conviction rate, but one who is honest and shows all the evidence (for or against).
While performance measurement is a staple of our society, in most cases it is either not effective or detrimental to societal good. Like most such systems, figures don’t lie, but liars can figure. It means that the development, use, and analysis of a performance measure are all subject to the problems inherent to humans anyway. We game the system to get the most with the least effort. Performance measures that are easy to game or have great rewards for minimal risk or that are inherently flawed anyway just make things more difficult for people and our society.
