Skip to Content

What I Learned about Higher Ed Assessment in a Small Village in South America

A few years after I began teaching undergraduates, I was informed that our program needed to be doing continual assessment. But we assess students all the time, I responded. No, I was told, that is not assessment. I dutifully went through a couple of professional development workshops: writing student learning outcomes, program outcomes. I learned to use action verbs, and struggled with the definitional difference between goals and outcomes and formative and summative. I wrote student learning outcomes for all of my courses, and worked with colleagues to develop outcomes for our program. I attended meetings where I was told our institution was building a culture of assessment. I was appointed to be the assessment coordinator of our multidisciplinary department, and obediently collected our results and passed them along to a larger faculty assessment committee, which reported them to our administration for inclusion in our accrediting reports.

As I collected data, collated information, and presented reports, I could not find any evidence that all of this assessment actually led to its ostensible goal: improving student learning. Perhaps, I thought, I simply wasnt doing it correctly. I experimented with what I felt were more authentic tools, ones that provided a more holistic appraisal of student workrequiring students to submit portfolios, or collecting extensive writing samplesbut it quickly became apparent that these required an enormous amount of time, and relied on what I felt were subjective standards for evaluation. These experiments were frustrating to me. While I wanted my assessment techniques to provide reliable and genuine measures of what I was sure my students gained from my classes and program, actually doing this kind of in-depth assessment did not seem to add anything. More manageably, I fell back on simpler alternatives, using what I felt to be rather superficial mechanismspre-tests and post-tests, a few questions on an examto make it possible to report a measurable outcome.

As an anthropologist who studies economic development, I found myself pondering a question that echoed one asked by well-known anthropologist James Ferguson.1 In looking at economic development, Ferguson suggested that it was well documented that development was not very effective at accomplishing its stated goals: to improve the material conditions of the people that it was supposed to help. Yet governments persisted in creating new projects on top of old failed ones. Fergusons analysis revealed that the discourse of development was very effective in reducing issues of poverty to problems that were apparently technical. Issues faced by people without access to resourcesshortages of food, poor education, inadequate employmentare not easily solved through simple technical fixes. The problems are interrelated, complex, and contextual. At its base, in Fergusons analysis, poverty is a political problem: resources are not evenly distributed, but accrue to those with power. By redefining poverty as a limited problem that can be addressed through standardized programs and projects, the state disguised its own political interests, thus de-politicizing both its own actions and the initial problem. While development was ineffective in creating economic change, it was highly effective in extending the reach of a bureaucratic state into remote rural regions, in a process that Ferguson famously dubbedthe anti-politics machine. The discourse of development adeptly camouflaged a political problema particular distribution of resourcesunder a cloak of teaching to a technical technical and bureaucratic interventions.

In contemplating assessment of student learning in the liberalarts and social sciences, I find myself feeling as though I am on familiar ground.2 The discourse of assessment is that it about enhancing education. The discourse operates from the belief that assessment is nothing more than collecting, analyzing, and acting rationally on information about student learning and faculty effectiveness.3 While professors have long tested their students on their mastery of skills or knowledge of content, the process of learning itself goes much deeper. We rely on a liberal arts education to develop human beings as citizens who, certainly, have skills and knowledge that are valuable and valued in the world, but who also appreciate beauty, have a sense of reflection and self-reflection, are capable of empathizing with others, who strive toward a greater understanding of the human condition.4 Many of the impacts of a college education may not be observable to an outsider at all or amenable to objective measurement. Yet assessment insists on reducing the qualitative processes of learning and teaching to a technical problemdefining and measuring objectives. After 20-odd years of assessment as a dominant practice stretching from kindergartens through universities, the problem certainly has not been fixed. Just as Ferguson asked, if development isnt very good at doing what it says it is doing, why does it continue? I ask, if assessment isnt doing what it says it is doing, what is really happening?

A discourse of development serves to obscure the political agendas and processes of transformation that accompany development projects, hiding them behind a neutral, technocratic veil. Similarly, the discourse of assessment serves to obscure the bureaucratization of higher education. It uses a neutral, technocratic language to reduce what is a tremendously complex processlearninginto a set of inputs and outputs. Yet, just as in economic development, the purposes and aims of the discourse can be subverted and reworked to serve the needs of those on the ground. By understanding the discourse as a discourse, we may be able to repurpose it to meet our own needs, even while complying with its bureaucratic requirements.

LIES, DAMN LIES, AND OUTCOMES

The discourse of assessment, like any discourse, shapes how we think about its domain, in this case the possibilities of teaching and learning. A discourse provides definitions and values, outlines the limits of acceptable behavior and response, and excludes other kinds of options and thinking. A discourse represents, metaphorically, the rules of the game, a shared vocabulary and set of assumptions about how things work. How, then, does the discourse of assessment shape our approach? There are certainly many dimensions to this, but I want to explore two here that stand out to me.

We measure the measurable. Curricular assessment requires faculty to document student learning by measuring achievement toward ostensibly objective outcomes or results. By defining learning as measurable, the discourse of assessment ignores the aspects of learning that are difficult, if not impossible, to measure. Instead, it leads us to focus on that which can be measured. In my courses and program, for example, one of the things that I surely want to do is to create student empathy with people in other cultures and tolerance for different cultural practices. I have resisted, however, including this as an outcome, because I am not convinced that it is measurable in any meaningful sense. I suspect that anthropologists like myself, whether by nature or by training, are hesitant to engage in what feel like reductionist and simplistic measurements about a subject that we know is complex and multidimensional.

Another component of assessment is that it happens over a relatively short time frame, within the context of a course over a single semester, or perhaps more ambitiously in a single student over the four year or so trajectory of their undergraduate studies. It eliminates, then, that much of the information in a course may be forgotten once the final is doneor, more to my point, that a student may be influenced by a course or a program many years after it was completed, in ways that I cannot begin to foresee, never mind go back and measure later. Moreover, it does not produce replicable results over time: one of my problems is that I change my course every time it is taught. An assessment mechanism, to produce valid, comparable data, should be more static.

Assessment, however, requires us to produce relatively immediate results that are quantitative rather than qualitative. As an anthropologist, I have been trained to understand the key benefits and drawbacks of each of these. Like any statistics, the public purpose of assessment data is to give an accurate, true description of the world. But statistics, of course, are constructed to support particular views. Numbers are created and repeated because they supply ammunition for political struggles, and this political purpose is often hidden behind assertions that numbers, simply because they are numbers, must be correct. People use statistics to support particular points of view, and it is naive to simply accept numbers as accurate, without examining who is using them and why, wrote Joe Best in Damn Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists.5 There is a tendency to treat assessment data as a straight- forward fact that cannot be questioned, without scrutiny of whether we were asking the right question. As faculty, we find ourselves struggling with the difficult questions around whether an assessment mechanism fits with whats actually being taught, writes Matt Reed in his blog, Confessions of a Community College Dean.6 But as with any quantitative measure, suggests Best, [t]his ignores the way statistics are produced. All statistics, even the most authoritative, are created by people. This does not mean that they are inevitably flawed or wrong, but it does mean that we ought to ask ourselves just how the statistics we encounter were created.7

We measure for an audience. Much of the literature on assessment claims that the audience for our measurements is neutral, that we are our own audience, producing numbers to close the loop. In fact, faculty regularly change courses and programs based on their feelings about what is working and what isntadjusting readings and assignments, adding or removing courses, and so on. Most faculty that I know, however, produce and document these changes as assessment-driven for an external audience: usually our own administration, and beyond that, generally speaking, accrediting agencies and, increasingly, state lawmakers.8 We are demonstrating that we measured, not in a way that is useful for ourselves, or even our students, but in a way that makes the process visible and transparent to an external observer.

ASSESSMENT AS GOVERNMENTALITY

Assessment, then, is a management technique that purports to be a pedagogical technique. Just as the discourse of development extended the power of the state into remote rural areas, the discourse of assessment has extended a neoliberal logic of management into one of the last bastions of non-corporate space, higher education. It refashions faculty from unique experts in their fields into interchangeable deliverers of content, or, increasingly, mere evaluators of student competencies.

When universities become corporatized, as has been happening quite systematically over the last generation as part of a general neoliberal assault on the population,their business model means thatwhat matters is the bottom line,Noam Chomsky told adjunct faculty during a 2014 speech.9Assessment is part of this. It compartmentalizes the process oflearning, transforming an essentially qualitative and unmeasurable processlearning!into discrete, ostensibly measurable outcomes. The role of the professor becomes that of the mechanical assessor, who applies a universalized, objective rubric. It imposes a logic of standardization inputs and outputs. It overrules the decisions that are made within the classroomand even more so as the majority of professors now are untenured and precarious. Assessment becomes a disciplinary technique, a form of governmentality.

Does assessment change what is being taught? In my field, and at the university level, this is not reducible to the simplistic teaching to the test notion that we see in K12. But let me use an analogy here. When I was conducting research on economic development in a remote rural area of the Argentine Chaco, a community had received some public funds to construct a new community center. They discussed where to place the center. The middle of the village seemed like a logical, common sense choice. But an NGO employee involved in the process advised them differently. Make it visible from the road, she suggested. The village itself was not visible from the unpaved rural route that traversed the province. It sat over a mile back in the bush. But the people who sent the money, she sagely noted, would want to see the building; they would not get out of their car, risking mud and snakes, trekking to the village itself. The strategic choice was to make the building visible, even though that largely defeated the purpose of a community center, and made the structure far less useful to the villagers themselves. A process of assessment similarly seeks to make our work visible to those casual monitoring agencies, to those who wont get out of their car. And, as with the Argentinian village, it makes it less useful, defeating its own purpose. Yet, the need to conduct an assessment that will be legible to those external agents does shape what we do. Assessment not reported to the administration meets the requirements of neither campus assessment procedures not accreditation standards, and is thus indistinguishable from non-assessment.10 Just as James Scott documented in economic development, assessment requires legibility, that outsiders (in this case, administrators) are able to understand and make comparisons, in order to facilitate management by those outsiders.11

MAKING ASSESSMENT WORKDIFFERENTLY

From a faculty perspective, the question becomes one of repurposing assessment. It is the recognition that assessment is a management technique, not a pedagogy, which has shaped my response to its demands. At various points, pondering my options, I toyed with what James Scott called the weapons of the weakminimal compliance with external obligations, foot-dragging and delays, creating intentionally low bench- marks to demonstrate continuous improvement.12 For various reasons, these strategies were not satisfying for me. My more recent solution has been to try to manage the technique myself: to comply with the requirements of assessment, becoming proficient in its own discourse, to try to bend it to the needs of my courses and curriculum, or, at the least, to try to prevent it from doing harm. Again, I see similarities with economic development. Recognizing the discourse of development for what it is, a discourse that privileges certain kinds of processes and outcomes over others, does not make the issues of poverty go away, and does not excuse a lack of action to try to remedy those problems. Once we recognize the discourse for what it is, and understand that it is a management technique, we are liberated, at least to some extent, from a need to internalize it, and can, instead, treat it as a tool.

Learning the jargon, words like formative and summative, inquiry-centered, artifacts, and benchmarks, and mastering the process of assessment creates a space in which to try to effect positive change.13 For example, a few semesters ago, I decided I needed to add a prerequisite to one of my upper level courses. Because the course met a general education requirement, it was attracting too many students who were not terribly interested in the subject, turning it into a frustrating class for all concerned, including me. While this was a common-sense solution to my intuitive understanding of the problem, I included it in my annual curricular assessment report. I counted the number of majors and non-majors in the course, represented their grades on a rubric, and announced that the addition of the prerequisite was the response. I reported what was, to me, an ordinary (and meaningful) change to the program as a response to an assessment (demonstrating that we closed the loop).

IF YOU CANT JOIN THEM, BEAT THEM

Within our program, we defined a couple of goals that we thought could be served through the strategic use of assessment, supporting particular courses, for example, by including an outcome that could best be met there. In a climate of budget cuts, faculty retrenchments, and pressures to eliminate small programs, some of the decisions we made were aimed primarily at not losing any ground.

One example of our approach is that even though we are a small program, we have a strong philosophical commitment to maintaining what anthropologists call a four-field approach, including (1) cultural anthropology, (2) biological anthropology, (3) linguistic anthropology, and (4) archaeology. Our permanent faculty include a cultural anthropologist and an archaeologist, and our curriculum includes courses taught by a linguist in another department. Yet several of our program outcomes explicitly reference biological anthropology. We cover this in our four-field introductory course, and in one course for majors, taught by the archaeologist. We have, to some extent, over-represented biological anthropology as an outcome so that we can reference it when we ask, on a regular basis, to hire an additional faculty member. It demonstrates that there is a need. While we havent been able to hire a third regular faculty, we did successfully deploy this argument to gain approval to hire a replacement faculty to cover a sabbatical leave at a point when not many replacements were being funded, because we could argue that some of our graduating students would not be able to meet the outcome. At the same time, although we strongly wanted each student to have field experience of some type before graduating, we did not include this in our outcomes because we knew that we could not support the courses required; we were concerned that if we offered them regularly, they would be low-enrolled, which could be used to justify cutting the program.

In another strategic decision, we chose to repurpose assessment by keeping our outcomes broadly defined, so that they would allow us to assess them in a number of different ways and in several different courses. While they are certainly an expression of what we want for our students, retaining some flexibility makes them less invasive in each course, and in the program as a whole.

A FINAL NOTE

The pedagogical advantages of assessment are unclear, at least to me. Years after this kind of an approach became dominant in K12 education, the educational system is perhaps even more profoundly in crisis. The solution? More assessment, more standardized tests. It is a technocratic fix, a discourse that limits the potential solutions. At the same time, the discourse of assessment has extended into universities. While it is not successful at improving education, it is highly successful at increasing bureaucratization. My comparison of assessment to earlier analyses of economic development isnot just fortuitous or coincidental: they have both been shaped byidentical processes of neoliberalism, or what Robert McChesney calls capitalism with the gloves off.14 Although it began as a set of economic principles advocating more liberal trade policies and lower rates of inflation, neoliberalism mushroomed into an ideological force that mandated decreases in social spending by governments at all levels and a reflexive disdain for public institutions, and even more, a set of assumptions that places economic profit above other values and assumes that the primary function of individuals is as rational economic actors and entrepreneurs. For higher education, and particularly public higher education, neoliberalism has set the stage for significant spending cuts, even as it recast a college education as an individual good, rather than a social good, which served to further justify austerity measures.

A lack of engagement with assessment, however, may lead to our absence from the dialogue and a tacit acceptance of allowing it to be (over simplistically) defined by others. This is my primary argument for engaging in assessment at all. Recognizing assessment as a discourse and a management technique is not a knee-jerk, reflexive dismissal of it. Rather, it is bringing our theoretical tools to bear in a way that can provide us the room to maneuver within this externally imposed regime.

Laurie Occhipinti is the dean of Liberal Arts, Education, Humanities, and Communications at Mount Wachusett Community College in Massachusetts. Previously, she was a professor of anthropology at Clarion University of Pennsylvania. Occhipinti received her Ph.D. and masters degrees from McGill University. Since the mid-1990s, her research has focused on faith-based organizations working on issues of poverty and development in Latin America. In addition to numerous articles, she is the author of Acting on Faith (2005) and Making a Difference in a Globalized World: Short Term Missions that Work (2014).

ACKNOWLEDGEMENTS

I presented a version of this paper during a panel discussion at the American Anthropological Association Annual Meeting in Washington, D.C., in December 2014. I am grateful to Amber Clifford for organizing that session, which sparked off a fruitful discussion. I am also indebted to Susan Prezzano, my former colleague in the anthropology program at Clarion University; the work that I did on assessment for our program has been in partnership with her. I would also like to thank Megan Miller and Joseph Occhipinti for their comments and support.

ENDNOTES

  1. Ferguson. The Anti-Politics Machine.
  2. My experience, and discussion here, refers to assessment of student learning in the liberal arts and social sciences, not certification exams, licensure exams, or similar instruments that serve to evaluate specific skills within a narrowly defined professional or technical field.
  3. Johnson, Administrators Should Work with Faculty to Assess Learning the Right Way.
  4. In his 2011 essay The University Besieged, Lustig put it beautifully: The liberal arts arent bodies of knowledge that can be ladled out. They cant be set down on a study sheet (though developing them requires the mastery of specific bodies of knowledge). They are abilities, like the ability to see beauty or do critical inquiry, and are cultivated or brought out (e-duced) of students latent powers....
  5. Best, Damn Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists, p. 13.
  6. Reed, Assessment Done Well and Badly.
  7. Best, op cit., p. 22.
  8. At least 32 states allocate some funds to their public colleges and universities based on performance indicators, such as student progression to degree or even salaries earned by graduates. This financial incentive, particularly in times of shrinking budgets, provides a powerful incentive for institutions to turn their efforts towards improving their metricsat the expense of other possible values and agendas.
  9. Chomsky, Thinking like Corporations is Harming American Universities.
  10. Johnson, op cit.
  11. Scott, Weapons of the Weak: Everyday forms of peasant resistance.
  12. Ibid
  13. The jargon associated with assessment is extensive, and has even inspired an online phrase generator, at http://www.sciencegeek.net/lingo.html.
  14. McChesney, Noam Chomsky and the Struggle Against Neoliberalism.

WORKS CITED

Best, Joel. 2001. Damn Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists. Berkeley: University of California Press.

Chomsky, Noam. 2014. Thinking Like Corporations is Harming American Universities. Transcript of remarks made to the Adjunct Faculty Association of the United Steelworkers, Pittsburgh, October 8, 2014. Retrieved from: http://www.alternet.org

Ferguson, James. 1990. The Anti-politics Machine. New York: Cambridge University Press.

Hutchings, Pat. 2010. Opening Doors to Faculty Involvement in Assessment. National Institute for Learning Outcomes Assessment. Occasional Paper # 4, April 2010. Retrieved from: http://www.learningoutcomeassessment.org/documents/PatHutchings.pdf

Johnson, Jeffrey Alan. 2014. Administrators Should Work with Faculty to Assess Learning the Right Way. Inside Higher Ed (November 7). Retrieved from: https://www.insidehighered. com/views/2014/11/07/administrators-should-work-faculty-assess-learning-right-way-essay

Lustig, Jeff. 2011. The University Besieged. Thought & Action. Washington DC: National Education Association.

McChesney, Robert W. 1999. Noam Chomsky and the Struggle Against Neoliberalism. Monthly Review (April). Vol 50, no. 11. Retrieved from: https://www.questia.com/ magazine/1G1-54517444/noam-chomsky-and-the-struggle-against-neoliberalism

Reed, Matt. 2014. Assessment Done Well and Badly. Inside Higher Ed (November 10). https:// www.insidehighered.com/blogs/confessions-community-college -dean/assessment-done- well-and-badly

Scott, James C. 1987. Weapons of the Weak: Everyday Forms of Peasant Resistance. New Haven, CT: Yale University Press.

 


download the article:

What I Learned about Higher Ed Assessment in a Small Village in South America (PDF)