Evaluating Teaching and Learning in Museums

Presented at the Leicester University Museum Studies Conference, April, 1993, and published as Chapter 17 (pp. 189-203) in E. Hooper-Greenhill, editor, (1995) Museum: Media: Message, London: Routledge.

This chapter provides an overview of the methodology we have used for the past fifteen years to evaluate programmes and exhibits in museums. Our work has all been carried out within the framework of an educational theory that places major emphasis on the role of the learner. The first section of this chapter describes this constructivist educational theory to provide a background for our evaluation methodology. The second section outlines three examples of our evaluation work, emphasizing the methods we use and illustrating each with a single outcome from that work. The third section discusses the components of our evaluation approach which make them appropriate for the educational view to which we subscribe. Finally, our work is placed in a larger context of visitor research in museums.

The last two decades have seen a tremendous growth in museum education: we now have major departments specifically devoted to this activity in most museums, the literature is expanding, students are graduating with degrees in museum education, and professional publications increasingly focus on aspects of museum education. [See Note 1 at right.] 

Simultaneously, our ideas about learning theory, about what it means to learn, have undergone a sea change, not so much in that there are dramatic new ideas, but in that a coherent and interrelated set of ideas advocated by a steady steam of thoughtful commentators from Dewey and Piaget to Vigotsky (as well as a wave of current writers) now receives wide acceptance. These ideas cluster around the notion that the most important issues involved in understanding learning are derived from analyzing the actions of the learner rather than in probing the nature of the subject to be learned. We now talk about constructivism, how the learner constructs meaning out of experience. [See Note 2 at right.] 

Broadly conceived, museums have always had an educational function. The Harvard Museum of Comparative Zoology was built to refute Darwin (Gould 1981), the Louvre was opened to the public to demonstrate the glories of French imperialism, and other museums were designed to show off national aspirations, ranging from the demonstration of imperial wealth and prestige, to preservation of a disappearing culture. These examples are all of educational missions for museums which are primarily didactic in nature. The intention of the museum founders was to instill a specific message in the visiting public. A striking recent example of such a didactic function for a museum is provided by an article in The New York Times (1993), which includes the following paragraph:

References & Figures
in this Paper:

Note 1:
I have recently compiled a bibliography of literature on museum education (Hein 1993b). Without stretching beyond the field — into literature on learning theory that is relevant to museums, for example — I was able to describe almost 70 items, not counting journal articles, in the United States alone. This is particularly impressive, since there is almost no literature published by trade publishers; all of it is put out by professional organizations and university presses.

Hein (1993b) ‘Museum education; a bibliographic essay’, Choice, Middleton, CT: American Library Association, vol. 30, 1733-41.

Gould, S. J. (1981) The Mismeasure of Man, New York: Norton.

Note 2:
The discussion in this chapter represents an education perspective on a parallel issue discussed in this volume by E. Hooper-Greenhill (see Chapter 1) from a communications perspective and by I. Moroevic (see Chapter 3) from a theoretical, semiotic analysis. Prof. C. Dufresne-Tassé uses a similar approach in her work described elsewhere in this volume (see Chapter 22).

New York Times (1993) ‘Hussein rebuilds Iraq's economy undeterred by the U. N. sanctions’, 24 January.

There can be no doubt of the didactic intention of this museum! 

When an exhibit is built with a specific didactic function, it is also reasonable to ask whether that didactic function is achieved. We can ask: do visitors learn about the voyage of the Mayflower after they view an exhibit mounted on the dock where the Mayflower replica is berthed? Does the school group know some of the principles that underlie art deco design after it has participated in a programme based on an exhibition of American design in the early 20th century? Do visitors know more about sexual reproduction in animals after a visit to the exhibition about sexual reproduction and gender roles? These are all didactic questions we were asked to address in the evaluations discussed below. 

These questions, typical of the kind that evaluators are often asked to address, make sense within a model of learning and teaching that is based on the teacher's perspective, where the theory of education stipulates that the teacher decides what is to be learned and the task of education is to organize the material and present it in such a way that it is transmitted to the student. [See Note 3 at right.] The teacher (i.e. the museum) has chosen to teach something about the voyage of the Mayflower, about early 20th century design, about gender and reproduction, etc., and it is legitimate to ask whether what was taught by the teacher was learned by the student. 

Note 3:
In this paper the language of teachers and students is used to emphasize that the discussion is based on writings about formal education. In the museum setting this language needs to be modified. In our context the ‘teacher’ is the museum, its exhibit designers and other staff, while the ‘learner’ is the visitor, regardless of his or her intention on entering the museum. An issue, which I will not address in this chapter, is that the motivation of many visitors who come to museums is not to learn but to be entertained. Does this make teaching impossible? It certainly does not interfere with learning as defined by constructivist theory, since visitors will still make meaning out of their experience.

But constructivist theory requires that we ask a very different question. What we intend to teach is not the central concern in this theory; rather the focus is on what people learn, that is, on what meaning they make out of whatever it is that we do and exhibit. In describing constructivism (Hein 1993a) we ask, what does the visitor (learner) make of our museum exhibit? What does she understand? What is the meaning for the visitor of the ship replica floating in Plymouth Harbor, of the pictures of skyscrapers and photographs of the modern factory, or the video of foetal development? Our intentions are not the beginning of our investigation, nor do they frame the questions we ask or structure the analysis of the answers visitors give; we have to start with the visitors and tell a story based on their experience. One can go so far as to say that our intentions are irrelevant! [See Note 4 at right.]

The two approaches — seeing the museum as teacher and seeing the museum as a place to learn — are not only two aspects of the educational role of museums, but they are not logically connected with each other. That is, if we accept constructivist theory, we also have to accept the view that there is no necessary connection between learning and teaching. In fact, the conditions that favor learning are such that if we maximize them, we cannot predict with certainty what will be learned. I don't mean just how much will be learned, but in a very fundamental sense, we cannot predict what meaning learners will make of the experiences we provide for them. The more we construct a situation that allows and encourages learning, the more likely we are to construct something that is open, ambiguous and able to be manipulated in a variety of ways by the learner; thus, the less likely we are to be able to predict precisely what has been learned. Conversely, the more we structure a situation so that it will provide very specific teaching of content, principles or skills that we have predetermined, the less likely we are to fulfill the conditions under which learning can take place most fruitfully. [See Note 5 at right.]

This is the paradox of education, the dilemma that teachers have struggled with for centuries. Once we accept the idea of constructivism, the notion of the active role of the learner, then we realize that teaching and learning are not necessarily connected. [See Note 6 at right.] If I want to assure that my student focuses on a specific task, attends only to it and achieves a precisely specified goal, then I am likely to shape an environment in which concentration on that task is assured, and use all my verbal and other skills to ‘keep the student on task.’ On the other hand, if I concentrate on the conditions of learning more broadly (for example, by patterning my education on what we know about how young children learn) then I am likely to provide a rich and varied environment, a number of ways to interact with the resources that are offered, and to prescribe to the attributes that developmental, constructivist theoreticians suggest to us. [See Note 7 at right.] But in doing so, and thus maximizing the opportunity to learn, I also increase the possibilities that the learner will focus on aspects of the situation that are not of interest to me. Precisely because the environment is rich, and multiple interactions are possible, I cannot limit the attention of the learner or assure that the focus is on a specific concept/ fact or skill.

The Evaluation of Teaching and Learning in Museums

We have struggled with the problem of documenting the results of teaching and of probing the consequences of environments for learning (nota bene the distinction!) for a number of years; the two issues are present in every evaluation study we carry out. 

On the whole, our clients come to us because they know we use certain methods, methods that are variously called qualitative, naturalistic or interactive. [See Note 8 at right.] So we start out with a certain community of ideas in our work. This makes it likely (although not necessary) that they are interested in addressing the issue of learning as well as determining the specific consequences of their teaching.

But we still face the dilemma posed by any effort to determine both the outcome of teaching and the outcome of learning. The primary reason, but not the only one, why clients come to any evaluator, especially in the United States, is that their funding agency demands an evaluation. The next most common reason is that they want to know the results of their work. [See Note 9 at right.] And when we have the inevitable discussions about assessing the exhibition or the educational programme, there is usually a strong interest in determining the outcome of teaching and a less strong, sometimes vague, interest in finding out what visitors 'make' of the exhibit, what meaning they gain from their experience. The challenge for us is that we have to, and want to, address both of these concerns, and needed to develop a system that provides data relevant for both of them. 

We have found that the primarily qualitative methods we employ, coupled with some simple quantitative measures (counting number of visitors, how long they stay in the gallery, etc.) can provide us with information that addresses both topics adequately, so that we feel we have learned something, and our clients appear to be satisfied since a significant fraction of them ask us to continue to work with them. 

The key to organizing all this work is to be clear at the outset what questions we are addressing (both questions about the results of the direct teaching that takes place, and questions about what the visitors make of their experience), and then to follow the plan we have devised. We have found a matrix that links the issues of concern with our methods a particularly useful tool to guide our work. [See Note 10 at right.]

Hein (1993a) ‘The significance of constructivism for museum education’, in Museums and the Needs of the People, Jerusalem: Israel ICOM, p. 89—94.

Note 4:
Proponents of so-called 'radical constructivism' (von Glasersfeld 1991) insist that there is no meaning besides the meaning attributed to something by the learner. That makes the teacher's intentions irrelevant.

von Glasersfeld, E. (1991) "An exposition of constructivism: why some like it radical’, in R. B. Davis, C. A. Maher and N. Noddings, (eds) Constructivist Views of the Teaching and Learning of Mathematics, Washington, DC: National Council of Teachers of Mathematics..

Note 5:
That is not to say there is anything wrong or inappropriate in advocating didactic teaching. There are a myriad of life conditions where the best approach is simply to ‘teach’ and not worry about the meaning making of the learner. Parents use this direct didactic approach in teaching children to take care in crossing the street (we don't wait to find out what meaning the child makes of the experience, or what cars signify to the child, we directly and forcefully instill in the child's mind the necessity of holding daddy's hand, and simply don't worry about the cognitive (or emotional) consequences of this action. Many other examples, from school days, military 'indoctrination' (note we use a different word!) and other situations could be cited.

Note 6:
To put this in perspective we can contrast the view described here with a simple stimulus-response (S-R) theory. In S-R theory, it is argued that any stimulus produces a response and learning is simply the aggregation of responses to appropriate stimuli. Thus, given the correct stimuli, the organism must learn. A forceful argument against behaviorist theory is provided in the pioneering work of Feuerstein (1979) in diagnosing and educating children with severe biological limitations to their ability to learn. He emphasizes the active participation of mediators between stimulus and response in order for learning to take place.

Feuerstein, R. (1979) The Dynamic Assessment of Retarded Performers, Baltimore, MD: U. Park Press.

Note 7:
One model for this type of learning in museums is found in the increasingly popular ‘discovery’ rooms. These are often rich learning environments without specific learning objectives, places where the learner can decide what to pursue.

Note 8:
The terminology for the kind of evaluation we carry out varies considerably among practitioners. Additional adjectives applied to it include holistic, illuminative, responsive, field-based, ethnographic and authentic. Opponents use other descriptive terms.

Note 9:
These two reasons are, of course, not mutually exclusive.

Note 10:
We have been using this approach for some time. See Engel and Hein (1981) and Hein (1982a, 1982b).

Engel, B. S. and Hein, G. E. (1981) ‘Qualitative evaluation of cultural institution/school education programs’, in S. N. Lehman and K. Inge, (eds) Museum School Partnerships: Plans and Programs, Washington, DC: Center for Museum Education.

Hein, G. E. (1982a) ‘Evaluating museum education programs’, Art Galleries Association, (U.K.), Bulletin, September, 13—16.

Hein (1982b) ‘Evaluation of museum programs and exhibits’, in T.H. Hansen, K.-E. Andersen and P. Vestergaard (eds) Museum Education, Copenhagen: Danish ICOM/CECA.

Examples of evaluation studies

What follows are three examples of evaluation designs we developed for three very different museum exhibits or programs in history, art, and science museums respectively. In each case the evaluation methodology is described to give some details of our approach, and then a single conclusion is presented which illustrates a way that visitors made meaning of the exhibition or program independent of the didactic intention of the museum's staff.

Plimoth Plantation ‘Mayflower’ dockside exhibit

Some years ago, Plimoth Plantation, a first-person interpretation museum in Plymouth, Massachusetts, decided to place an explanatory exhibit on the dock in Plymouth Harbor where they had berthed a replica of the Mayflower. Visitors came on board the Mayflower, where they were greeted by costumed staff who provided first person (17th century) interpretation of the ship, its crew and passengers. Staff at Plimoth Plantation wanted to improve the experience of visitors by providing a dockside exhibit that would serve as an introduction to the visit. We developed a matrix to guide our evaluation and agreed to carry out some evaluation activities to see whether the objectives of the exhibit were achieved.

Figure 17.1 shows the objectives we agreed to explore at the request of Plimoth Plantation education staff. These objectives included some that are directly intended to assess the results of teaching (‘Visitors will know more facts about the Mayflower people’) as well as others that are vague, but directed more towards assessing the results of the experience (‘The exhibit will enhance the visitor experience.’) The four means we decided to use to assess these evaluation issues are illustrated in the matrix which combines means and methods, shown in Figure 17.2. We proceeded to gather data at three different times: before any exhibit was up on the dock and people just milled around until they were admitted on the ship; a few months later, when a temporary exhibit was mounted; and finally, a year later, when the permanent exhibit was mounted. It was clear after the evaluation that the exhibit had enhanced the visitor experience. Visitors knew more about the Mayflower and its voyage and could engage in more interesting discussions with the staff based on this increased knowledge. We were able to answer the didactic questions posed by the museum staff. 

We also noted some other consequences of the visitors' interaction with the exhibit. I remember especially the older couples who stood before the map of England which showed from where the Mayflower passengers had departed. In carrying out our observations, we never heard visitors commenting that the Aldens had come from such and such a town or the Winthrops had come from another. What many of them did do was to use the map to discuss where their families originated.

Brooklyn Museum school group visits to
‘The Machine Age in America’

The second example I want to discuss is an evaluation we did at the Brooklyn Museum (of Art) of a school visit programme. The Museum had mounted a major exhibit, The Machine Age in America, 1917-1941, and in conjunction with that installation, they had developed a school visit programme so that classes could get a guided tour of the exhibit followed by optional workshops where they did art work related to the exhibit. 

The Museum staff had some objectives for the evaluation, in this case a rather ambitious and extensive evaluation agenda, which is illustrated in Figure 17.3. We developed an equally elaborate set of sources of data, shown in Figure 17.4 and combined them into a matrix, illustrated in Figure 17.5. Again, the situation is such that the issues we are investigating are broader than would be included if we were only searching for evidence of the results of teaching specific topics. We carried out the evaluation and answered most of the Museum staff's questions. As in the previous example, some conclusions from the evaluation involved visitor meaning-making that emerged from the data. 

One example involves the wide difference between the culture of art museums and the expectations of schools. Many school groups signed up for the programme and happily participated in the guided visits. But there was a dissonance between the interests of the school groups and the interests of the curators who had mounted the exhibit. The schools came essentially because the exhibit had images (skyscrapers, bridges, paintings of workers and machinery, and examples of objects, streamlined trains, art deco radios, etc.) that fit into their teaching of social studies in general and their mandated teaching about New York City in particular in the fourth grade. But the exhibit curators had mounted the exhibit to make an aesthetic statement. The exhibition was not about social history, but about design. Their concern was not New York City in the 1920s and 1930s but the relationship between form and function. 

We found out about this problem inadvertently. A hidden agenda for our evaluation was the desire by the Education Department to make a political case for themselves within the museum. So we designed part of the evaluation to answer this need. Since the education group wanted their work better known within the Museum administration and staff (and this school programme was novel enough for various museum staff to wander by to see what was going on) we enlisted the help of any such staff and asked them to fill in evaluation forms especially designed for this purpose. [See Note 11 at right.] We managed to get various administrative staff, and even one or two trustees of the Museum to observe an entire school group visit through this device. Among the people who observed the visits were curators, and they were not happy to find that their aesthetic exhibit was being used as background for a social studies lesson. Again, this instance illustrates a situation where the meaning attributed to the museum experience by the visitors is important and different from the intended meanings of the museum curators.

Boston Museum of Science: ‘Two of Every Sort’

The third evaluation was a modest study of visitor responses to a temporary exhibition on gender and sexuality at the Boston Museum of Science. In this case our brief was not as specific; we were asked to get a general idea of how visitors viewed this exhibition, and again were concerned with what they learned from it. The Museum was particularly interested in visitors' responses to the controversial nature of the exhibition, since its content included sexuality, reproduction and social issues related to these topics. The exhibition consisted primarily of wall text with some illustrations, a few traditional exhibit cases and a small number of video screens that showed 3- to 5-minute film loops on topics such as animal reproduction, foetal development, or early infant response.

In this case, since our brief was less specific than in the previous examples, we did not develop a matrix. We combined some simple quantitative measures with a visitor questionnaire to obtain information about where visitors stopped and what they thought about the exhibit. Our information on visitor behavior was obtained through a tracking study. We followed visitors through the exhibition and indicated where they stopped and what they did when they stopped.

It was no surprise to us that visitors overwhelmingly stopped at the video screens and seldom at panels with texts or photographs only, although the extent of the difference was quite dramatic. But most surprising was visitor response to our carefully worded question about the controversial nature of the exhibition. As good social scientists, we couched our question in a neutral, accepting format and asked: ‘Some people think exhibits like "Two of every sort" should not be displayed in a museum of science, other people think it is a good idea to have such an exhibit here. What do you think, and why?” Visitors overwhelmingly responded not only that they had no objection to the exhibit in the Museum, but that they were even puzzled by our question. They looked at us with a surprised look as if to say, ‘Why do you ask such a question? We all know that it is the job of museums to present this kind of material.’ This answer even came from visitors who did not particularly agree with some of the messages in the exhibit. They indicated that although they might restrict their own children's access to the exhibit, they did not question the Museum's right to show it.

From other responses to open-ended questions, in which we asked them what they liked the most and the least, we found that the one controversial component of the exhibit was the display of an animal head, and other animal material which offended animal rights activists. This aspect of the exhibit, and its effect on one group of visitors, had not even entered the minds of the exhibit developers.

Mager, R. F. (1975) Preparing Instructional Objectives, 2nd edn, Belmont, CA: Fearon.

Note 12:
There is a whole literature on life in classrooms, dating back to Jules Henry's (1963) pioneering application of anthropology to familiar settings. For example, see Armstrong (1980).

Henry, J. (1963) Culture Against Man, New York: Random House.

Armstrong, M. (1980) Closely Observed Children, The Diary of a Primary Classroom, London: Writers and Readers.

Note 13:
We once had great success in finding out about visitor behavior by talking with the teen-age boy scout volunteer guides at the National Museum of the Boy Scouts in Murray, KY.

On reflection, it seems to me that the methods we use are adequate to respond to the questions from museum staff concerning their teaching objectives. They are not adequate to explore fully all the meaning making that takes place (can take place) in museums. This is not so much a limitation of the methods we use as of the limitations which usually circumscribe evaluation studies. For any reasonable question that is posed to us by the staff, for any objective they can imagine obtaining as a result of the exhibit, we can make a good estimate of whether and to what extent that has been achieved. But on the whole, the means at our disposal when carrying out an evaluation are usually inadequate to explore fully the learning possibilities of exhibitions, the myriad ways in which visitors may make meaning out of the experiences. In order to achieve the latter objective, we would need to broaden our evaluation methods and carry out more open-ended studies, using the techniques of field-based research. We need to provide “thick descriptions” (Geertz, 1983) of visitors' experiences.

Some examples of the kinds of activities which might be particularly fruitful for the understanding of visitors' making of meaning out of exhibits that have appeared in the museum literature include:

Geertz, C. (1983) Local Knowledge: Further Essays on Interpretive Anthropology, New York: Basic Books.

1. careful analysis of visitor's conversations (Carlile 1985);
2. analysis of visitors overheard conversations, using microphones (Lucas and McManus 1986);
3. detailed ethnographic studies of the kind first carried out in museums by Laetch's students (Laetch et al. 1980);
4. retrospective interviews about museum memories (Falk and Dierking 1992);
5. asking visitors to 'think out loud' as they view exhibits. Dufresne-Tassé and coworkers (Dufresne-Tassé and Lefèbre 1994) have done such work, following the model of protocol analysis (Ericcson and Simon, 1984); developed by cognitive psychologists;
6. the expansion of modalities for visitors to respond, as well as expansion of modalities for visitors to learn from; [See Note 14 at right.]
7. in depth interviews (McDermott-Lewis, 1990);
8. Focus Groups in which visitors discuss their museum experiences (Getty Center 1991).

Museums as teaching institutions (or more accurately exhibits and programmes with educational objectives which intend to teach their visitors/participants something) need to decide what they what to impart and how they plan to do it. This is hardly revolutionary. The problem in all this is the often implicit assumption that this task of deciding on educational goals requires a focus on the topic or subject. How shall we arrange the artists to get across our message? How shall we guide the visitor through the museum so he or she will understand what we want to impart? What label will be most understandable (that is, from which label will the visitor best get the knowledge we wish to supply?) 

I argue that most evaluation work in museums has been based on the premise that we need to modify our exhibits so as to maximize what visitors learn of the content we want to teach. This also assumes a close causal relationship between a particular way of installing an exhibit or devising a programme and the quantity and quality of learning for a majority of visitors. ‘I tried this label and no one read it. I put up a different one and seven out of ten visitors stopped and could tell me what it said.’ Therefore the second label accomplishes what I want. 

But there is another whole world of learning that goes on in museums, the learning that is constructed by the visitor out of the experience and is not necessarily correlated closely with our teaching efforts. In order to understand the museum visitors and find out what they have learned, we need a broad approach to museum evaluation which includes a rich infusion of qualitative, naturalistic research into the museum field.

Carlile, R.W. (1985) ‘What do visitors do in a science center?’, Curator 28: 27—33.

Lucas, A.M. and McManus, P. (1986) ‘Investigating learning in informal settings’, European Journal of Science Education, 8, 343—53.

Laetch, W. M., Diamond, J., Gottfried J. L. and Rosenfeld, S. (1980) ‘Naturalistic studies of children and family groups in science centers’, Science and Children, March, 14—17.

Falk, J. and Dierking, L. (1992) The Museum Experience, Washington, DC: Walesback Books.

Dufresne-Tassé, C. and Lefèbvre, A. (1994) ‘Some data on the psychological functioning of the adult visitor at the museum and their Implications for museum education’, International Review of Education, in press.

Ericsson, K. A. and Simon, H. (1984) Protocol Analysis: Verbal Reports as Data, Cambridge, MA: M.I.T. Press.

McDermott-Lewis, M. (1990) The Denver Art Museum Interpretive Project, Denver, CO: The Denver Art Museum.

Getty Center for Education and the Arts (1991) Insights, Museums, Visitors, Attitudes, Expectations, a Focus Group Experiment, Los Angeles, CA: The J. Paul Getty Trust.

Note 14:
When we asked visitors about their knowledge of animal adaptation from viewing an enhanced diorama exhibit at the Museum of Science (Davidson, Heald and Hein 1991) they mentioned things they had learned from touching, from listening, from smelling and from conversation, not only items they could have learned from seeing or reading. In this exhibit we had people who told us they had not read the labels, but could then recite them. the exhibit included recorded material that gave the label information. Visitors did not perceive that listening to the information counted as reading the labels (maybe it doesn't).

Davidson, B., Heald, L. and Hein, G. E. (1991) Increased exhibit accessibility through multi-sensory interaction’, Curator 34: 273—90.