Museums and Impact

How Do we Measure the Impact of Museums?

Carol Scott
President Museums Australia
Manager- Evaluation and Audience Research / Powerhouse Museum Sydney

Carol Scott is the President of Museums Australia, the national professional association representing
museums and galleries, and the Manager of Evaluation and Audience Research at the Powerhouse
Museum in Sydney. She has worked for the Australia Council and the national arts industry training body,
Create. Her work and research are concerned with the value of culture within an environment of change.

Abstract
Over the last two decades a combination of increased public accountability, the growing cost of sustaining
collections and competition for funding within reduced public sector budgets has forced museum
performance to come under close scrutiny. Various monitoring systems, tied to funding agreements, have
been implemented to demonstrate that museum activity justifies public investment.
The most recent monitoring mechanism to be introduced, ‘impact evaluation’, foreshadows a radical
departure. Impact evaluation sets targets beyond the existing activity of museums and asks whether
museums ‘make a difference’ in terms the longer term.
Within this context, articulating and demonstrating the value and impact of museums has never been so
important for, in this closely monitored and increasingly competitive environment, Guetzkow (2002) and
Weil (1994; 1997) believe that museums are being required to prove the very worth of their existence.
This paper reports a study that explores impact of museums from the perspectives of professionals with
expert knowledge of the field and the general public. It has specifically examined whether it is possible to
develop valid indicators, shared across these cohorts and substantiated by evidence, to prove that
museums ‘make a difference’ in terms of long term social impact.

Keywords
Museums, Impact, Evaluation, Evidence Based Policy.

Background

Attempts to assess museums raise fundamental questions about why we have museums, what
value they are to society and how we can effectively determine the extent and breadth of their
contribution. These were not dominant questions for most of the last century when funding was
based on the notion of museums serving the “public good”. But from the 1980’s, the emergence
of economic rationalism radically altered the fiscal contract with governments with the result that
museums have had to progressively prove their worth to maintain funding.

While economic rationalism determined public policy until the mid 1990’s, the last decade has
witnessed challenges to the dominance of the economic paradigm as governments have been
forced to address major societal changes related to work, leisure, family and community
structure, values, globalisation, and technology. The result has been a re-assertion of social,
environmental and cultural factors in developing healthy and sustainable communities and a
corresponding shift towards more public-centred policy. The contribution that the public sector,
including museums, can make to the realisation of greater equity, access and social inclusion is
acquiring greater prominence as a result.

1
Added to policy shift are specific issues related to the future of museums. The financial
sustainability of the museum sector is a cause of concern as the cost of maintaining growing
collections increases. While there has been development in the number and profile of museums
in the last thirty years, the situation is now one where supply is exceeding demand and where a
declining audience base is shared amongst a growing number of museums. Alison and Coulter
note that

...In this context of increasing competition and financial pressure, many

museums have been forced to give greater attention to their ‘product’ and how
it meets the requirements of individuals and communities (Alison and Coulter,
2001:6)

In addition, greater public accountability and transparency demand evidence of service

provision,

The need to measure impact comes from a general climate of transparency

and accountability to stakeholders in all public sector organizations and has
increased awareness of service provision in relation to client needs….(Wavell,
Baxter, Johnson, Williams 2002: 5)

Within this context, articulating and demonstrating the value and impact of museums has never
been so important for, in this closely monitored and increasingly competitive environment,
Guetzkow (2002) and Weil (1994; 1997) believe that museums are being required to prove the
worth of their existence.

As crunch time approaches, ….. and as the demands that are made on the
public and private resources available to the non-profit sector continue to grow
at a faster rate than those resources themselves, virtually every museum may
find itself faced with several much tougher questions- ….without disputing the
museum's claim to worthiness, what these questions will address instead is its
relative worthiness. Is what the museum contributes to society commensurate
with the annual cost of its operation? Could some other organization (not
necessarily a museum) make a similar or greater contribution at lesser cost?"
(Weil, 1994: 42)

However, museums, long used to being funded as a ‘public good’, do not have a tradition of
planning activities with social impact in mind and the result is that the sector itself has been slow
to articulate the value and impact of museums on its own terms.

Impact Evaluation

Impact evaluation has emerged as an outcome ‘evidence based policy’ and is the newest
mechanism developed to assess the public worth of museums. But as the following discussion
reveals, impact evaluation itself raises a whole new suite of challenges.

At the most fundamental level, Matarasso (2000a and 2000b), Appleton (2002) and Ellis (2002)
question whether it is appropriate to measure the impact of museums in terms of their capacity
to be socially inclusive. They argue that social benefits tend to occur as by-products of cultural
programmes rather than being the primary raison for them and question ‘how well-suited

2
museums are to solving deep-seated socio-economic problems, and how appropriate it is for
them to seek to do so’. Matarasso suggests that a

….serious, sustainable response by museums to the challenge of social

exclusion must be a cultural one arising from the heart of their values and
purpose, rather than an additional, project-based approach which can only
address symptoms (Matarraso, 2000b: 5).

Similarly, Bennett (1989) questioned whether the social objectives of access and equity are
realistic goals by which to measure the impact of museums. He notes that the complex range of
highly differentiated demands placed on museums by a multi-differentiated public is enormous
and perhaps unachievable given that museums cannot compensate for structural deficiencies in
society which deliver an already differentiated population to the museum’s door.

But, by its very nature, impact evaluation requires clear intentions of what is to be achieved and
through which means so outcomes can be assessed. The absence of clear social objectives
within museums has been noted in several recent studies (Alison and Coulter, 2001; (Wavell,
Baxter, Johnson, Williams 2002; Bryson, Usherwood and Streatfield 2002).

There is a lack of clearly stated purposes and objectives for museums,

accompanied by a failure to monitor the extent to which objectives are being
met…(Alison and Coulter, 2001: 8)

Given this context, there is, not surprisingly, ‘…no broad consensus as to how impact could be
measured, and no clear view of the timescales that should be involved’ (CHC, 2002: 20), no
generic outcome and impact indicators developed (Alison and Coulter 2001) and no models for
using valid and reliable qualitative indicators (Wavell et al 2002).

There is consensus the individuals and communities should be part of impact assessments as
they are major stakeholders who place value on the programs and services that museums
provide (Wavell et al , 2002 and Resource, 2002). Not only is community involvement important
from the perspective of their role as cultural consumers, but they serve to counterbalance the
potential for impact claims to be skewed from within the professional museum sector itself as
the pressure builds to provide evidence as a basis for securing on-going funding (Selwood
2002).

Finally, though claims of impact may be made, issues of proving causality and providing
substantiating evidence remain substantially unresolved.

Some Recent Australian Research

Taking these factors into account, a recent Australian study sought to test whether it was
possible to develop generic outcome and impact indicators ‘from the heart’ of museums’ value
and purpose. To do this required the participation of professionals working with the museum
sector to articulate impact from the perspective of their knowledge and expertise. Mindful also of
the need to explore the impact of museums from the perspective of end-users, a public cohort
was convened with the wider aim of comparing its findings with those of the professional sector
to determine whether it would be possible to develop a set of generic impact indicators that had
credibility both within the profession and the community. A further stage of the research involved
examining the case for evidence to substantiate the indicators.

3
The public cohort, consisting of 70 visitors and non-visitors to social history, technology and
science museums, was recruited from two cities and regional and rural centers (table 1).

Table 1:
Public Cohort: Museum Visitors and Non-visitors

Large City Medium city Regional centers

Pop: 3-4 million Pop: 750k-1million Pop: 10-50k
Visitors *18-24 years *18-24 years *18-24 years
*Parents with *Parents with *Parents with
children under 12 children under 12 children under 12
years years years
*Adults 35-50 years *Adults 35-50 years *Adults 35-50 years
without dependent without dependent without dependent
children children children
*Seniors *Seniors *Seniors
Non-visitors *18-24 years *18-24 years *18-24 years
*Parents with *Parents with *Parents with
children under 12 children under 12 children under 12
years years years
*Adults 35-50 years *Adults 35-50 years *Adults 35-50 years
without dependent without dependent without dependent
children children children
*Seniors *Seniors *Seniors

The professional cohort involved 34 participants including directors, senior managers and staff
working in social history, technology, science and natural history museums at each of the three
tiers of Australian government—federal, state, local. Also included were people associated with,
but not working directly in, museums such as academics, consultants, local government and
tourism officers.

The chosen methodology was an online Delphi Panel, a method for generating ideas and
facilitating consensus among individuals who have knowledge and opinions to share but who
cannot physically meet due to geographic dispersion (Linstone and Turoff 1975; 2002).

For both cohorts, the Delphi panel was implemented through three rounds of questions. In the
first round, three-four open-ended questions were asked. In the second and third rounds,
responses were submitted for comment and rating on a 5 point Likert scale.

In the first round, the professional cohort were provided with definitions of impact and asked to
respond to three questions, the first of which was In your opinion, what are the long-term
impacts of museums on communities? Give concrete examples where museums effect social
development, personal development and economic development.

For the public cohort, a similar question was asked: In your opinion, how do museums
contribute to the social development of a community? How do museums contribute to the
personal development of the individuals who visit them? How do museums contribute to the
economic development of communities?

The two panels were run separately. Neither the professional not the public cohorts were aware
of the questions or the results emerging from the other panel. The professional cohort

4
generated 95 impact statements. Of these 95 items, 74 received a rating of agree/strongly
agree by 65 percent or more of the respondents on a 5 point likert scale when subjected to a
second round. The public cohort generated 62 impact statements of which 53 achieved an
agree/strongly agree rating from 65 percent of respondents when subjected to a second round
and rating on a 5 point likert scale.

Developing a Set of Generic Indicators

Following the completion of three rounds of questions for each panel, the impact indicators that
received an agree-strongly agree rating from 65% of the respondents in both cohorts were
compared. Considerable consensus emerged across a range of indicators although the degree
of emphasis varied across the cohorts. As can be seen from the following tables, the public
generated similar impacts related to economic benefit, contributing to a community’s intellectual
capital, fostering social cohesion, providing distinctive leisure venues and public amenities and
developing personal learning, perspective and inspiration (Scott, 2003a).

1. Museums build social capital—Both professional and public cohorts agree that museums
contribute to the intellectual capital of communities by supporting the formal education sector,
providing opportunities for learning by people of all ages and presenting information through
meaningful selectivity. Further, the learning experiences offered by museums are perceived by
the public cohort to be unique. This cohort commented on the features that characterize the
distinctive learning experience in museums including its visual dimension, its informality, its
reflective atmosphere and the opportunities provided to extend horizons.

Table 2:
Museums Build Social Capital

Professional Cohort Public Cohort

Social capital: Education and learning Education and learning
Contribution to the - Museums reinforce the formal - Museums are a graphic and
communication of ideas, educational system through physical way of educating
information and values, helping extension programs. people about their own history
improve participant’s skills in - Museums provide support for and heritage and other
planning and organizing, community education societies’ history and heritage.
improving understanding of programs. - Museums give schools and
different cultures and lifestyles, - Museums build knowledge in special interest groups a place
partnership building (Kelly and the community by conducting where they can see, rather
Kelly 2000). formal educational programs. than have to read about, items.
- Museums provide education
without the formality of a
classroom.
- Museums make available to
the community the means to
learn and “see” outside its own
environment.
- Museums provide a place for
people to be educated about
the world and the country they
live in.

5
 Professional Cohort Public Cohort
Museums build social capital Museums provide a unique type Museums provide a unique
continued of learning experience. type of learning experience.
- Museums provide a free - The visual representation of
choice environment to pursue facts makes museums more
individual interests and accessible and interesting to
discovery learning. visit for many more people.
- Museums create a great
“hands on” learning
experience.
- Museums provide
information for all five senses.
- Museums provide an
environment that is quiet, not
intrusive yet offering
assistance when required.
- In museums, people can
spend as much time as they
want pondering over an item.

2. Museums develop communities—Both cohorts separately generated similar categories of

responses in relation to the role of museums in building identity through interpreting heritage,
contributing to sense of place and reinforcing community identity, values and worth. Moreover,
the opportunities provided by museums for meeting and working with people contribute to social
cohesion in valued public leisure spaces.

Table 3:
Museums Develop Communities

Professional Cohort Public Cohort

Building and developing Building identity Building identity
communities: - Museums provide the - Museums can make people
Contribution to developing opportunity to experience a sense feel that they belong to a
sense of community of belonging around shared common heritage.
identity, social cohesion, collective values. - Museums give communities
recreational opportunities, -The “permanency” of museums a sense of place.
development of local endows them with significance as - Museums provide a place to
enterprise, improvement the guardians and repositories of learn more about the shared
of public facilities and that part of the collective memory experiences that matter to all
amenities, and help to most valued by communities. of us.
convey history and - Museums enable us to - Museums contribute to a
heritage of an area (Kelly understand the present and sense of local pride as they
and Kelly 2000). foreshadow the future by say “we have something
reflecting on evidence and which is unique and valued by
experience from the past. others.”
- Museums reflect, construct, - Museums contribute to the
define and explore national social development of the
values. community by showing its
- Museums reinforce a sense of progress and achievements
community identity, values and and improvements over time.
worth.

6
 Professional Cohort Public Cohort
Social cohesion Social cohesion
- The development of a museum - Museums generate social
can spark community spirit and interaction between out-of
awaken civic consciousness and town visitors and people who
co-operation. live in the local community.
- Museums provide a focus for - Museums allow different
communities to celebrate people from different
significant cultural events and backgrounds to come together
rituals. around common interests.
- Museums provide the focus for - Museums contribute to the
active commemoration and development of a community
memorialization of significant local if they have committees or
and national events. “friends groups” who work
- Museums provide the focus for together.
forging new community networks. - Communities can become
involved in museums and
contribute to their collections.

Building and developing Heritage Heritage

communities: - Museums develop a sense of - Museums are important
Contribution to developing community identity through sources of history to help
sense of community conveying the unique history and guide us into the future.
identity, social cohesion, heritage of an area. - Museums construct a
recreational opportunities, - Museums that interpret historic common awareness of
development of local buildings and places give history.
enterprise, improvement authority to the events that took - Museums trace the progress
of public facilities and place there. from the past to present: They
amenities, and help to - Museum research extends and can show what a town looked
convey history and expands a community’s view of like a hundred years ago and
heritage of an area (Kelly itself as it learns more about its the changes that have
and Kelly 2000). own history, heritage and sense occurred.
of place.
Public spaces Public spaces
- Museums are important civic - As public spaces, museums
spaces like opera houses, promote social development
libraries and universities. through providing interesting
activities and places to
visit with family and friends.
Leisure Leisure
- As other leisure attractions - Museums are a source of
become more commercialized entertainment.
museums occupy a unique role as - Museums are for all ages;
a cultural counterpoint. that means that they can be a
- Museums provide a focus for great family day out.
family excursions. - Museums are good
excursions where we can take
our friends at a low cost.
Table 3: Museums Develop Communities continued

3. Museums contribute to social change and public awareness—In this category, there was
a major difference in the degree of importance between the two cohorts related to the role of
museums and as agents of reconciliation with Indigenous communities.

7
 Table 4:
Museums Contribute to Social Change and Public Awareness

Professional Cohort Public Cohort

Social awareness Social awareness
- Museums highlight issues of - Museums make individuals
contemporary social concern. more conscious of global
- Museums hold up a mirror to events.
communities and reflect them
back to themselves.
Social Change and Public Museums as agents of Museums as agents of
Awareness: reconciliation reconciliation
Contribution made to - Museums assist Indigenous - Museums remind us how
stimulating and developing communities to reclaim lost recent the history of white
public awareness of heritage through repatriation of people is compared to
important issues and material culture. Aborigines.
changing people’s attitudes - Museums reflect cultural shifts in
on political, ethnical, a nation’s relationships with
religious or moral issues Indigenous communities.
(Kelly and Kelly 2000). - Working on museum projects
together changes the social
relationships between Indigenous
and non-Indigenous members of
local communities.
- Indigenous communities lay
claim to mainstream cultural
terrain through the inclusion of
their stories and cultural material
in museum exhibitions.
- Museums affirm Indigenous
culture through including
Indigenous cultural material in
museums.

Social Change and Public Museums as agents of social Museums as agents of social
Awareness: inclusion inclusion
Contribution made to - Exhibitions and specific - Museums help people
stimulating and developing museums, dedicated to reflecting understand difference and
public awareness of cultural diversity, extend the approach difference with more
important issues and definition of “culture”. respect and interest.
changing people's attitudes - Migrant communities experience - The community can have
on political, ethnical, pride, confidence and a greater better social interaction if
religious or moral issues sense of belonging to the wider people understand each other.
(Kelly and Kelly 2000). community through presenting - Museums are catalysts that
their stories in museums. enable other cultures to view
- In engaging under-represented and understand one another.
minority groups, museums
contribute to social inclusion.
- Museums extend the identity of
cultural groups through revealing
unknown aspects of the culture.

8
4. Museums build human capital—Both cohorts perceive that museums contribute to human
capital through building social networks and relationships, providing personal inspiration and
developing a sense of perspective.

Table 5:
Museums Build Human Capital

Professional Cohort Public Cohort

Human Capital: Building social networks and Building social networks and
Contribution to improving relationships relationships
participant's human and - Working together on - Museums foster a volunteer
communication skills, exhibitions enhances personal element that allows people of
analytical and problem- and social relationships among common interest to make or share
solving skills, creative community members. a topic of interest.
talents, and social - Museums provide the focus
awareness (Kelly and Kelly for forging new community
2000). networks.
Providing perspective Providing perspective
- Museums help people see - Museums bring home to us the
destinations beyond their local size and appearance of many
horizons. animals (some extinct) that we will
probably never see in the flesh.
- In museums, seeing how people
interacted in the past may better
help us to understand ourselves in
the present.
- Museums make you realize you
do not live in a cocoon and that
you are part of a bigger picture.
- Museums give people an insight
into others’ lives and the
opportunity to see things from
others’ perspectives.
- Museums help to put the
process of change into
perspective.
Providing personal inspiration Providing personal inspiration
- Museums provide inspiring - Museums help to stimulate
examples of other people’s individual curiosity and interest in
lives, disappointments and seeking further information on a
achievements. subject.
- Museum collections provide a - When I get the rare chance to
unique source of creative visit a museum, it’s worth the time
inspiration for artists and and the effort as it’s fascinating to
designers. learn about the world.
- Museums inspire and validate - Museums raise questions about
personal interests why, how and where without
- Museums provide personal traveling all around the world.
validation through reflecting - Museums provide a place for
experiences relating to one’s people to be educated about the
family, community, work or world and the country they live in.
individual meaning. - Museums provide personal
inspiration.

9
5. Economic benefits—Finally, both cohorts recognize the potential for museums to attract
tourism, stimulate the local economy, create employment and attract and generate income.

Table 6:
Museums Generate Economic Benefits

Professional Cohort Public Cohort

Economic impact: Employment Employment
Work created; role of the - Museums create - The upkeep and
organization in the promotion employment for staff and maintenance of museum
of a positive image for contractors. buildings creates
city/town, contribution to - Museum capital works and employment.
tourism and local economy. projects stimulate - Museums provide
employment and investment. employment for staff.
Attracting tourism Attracting tourism
- The presence of a museum - Museums can be an
in a community results in economic lifeline in regional
more spending by tourists on towns where they are the
accommodation, food, main attraction for visitors.
transport and supplementary - Museums attract tourists
items such as souvenirs. who spend more time and
money in the community.
Economic impact: Stimulating the local Stimulating the local
Work created; role of the economy economy
organization in the promotion - The development of new - Museums use local
of a positive image for museums stimulates the businesses and, thus,
city/town, contribution to development of local stimulate the local economy.
tourism and local economy. infrastructure.
- Museums extend the trade
skills purchased from their
own community by requiring
them to be applied in a
different context.
- Museums stimulate the
local economy through the
purchase of services and
products.
- The presence of major
exhibitions can inject
additional funds into the local
economy.

The preliminary findings from this study indicate that there are commonly held views across
professional and the public cohorts regarding the impact of museums.

But there were differences as well. The professionals were also aware that museums contribute
to community capacity through partnership building and that they have an important role in
building the intellectual capital of communities through research on collections with often long
term consequences for society.

Other differences emerged from the public cohort. The major difference is the importance that
the public places on having ‘access to the past’ through visiting museums. Access to the past
emerges as important for several reasons. There is a strongly held perception within the

10
community that the lessons from the past can help us evaluate the present and guide us into the
future. The respondents associated knowledge of the past with providing a yardstick to measure
and confirm human progress.

Common sharing of the past is also perceived to enable socialisation to occur around common
values and contribute to a community’s identity. Knowing about the past also contributes to a
‘sense of place and belonging’: Museums give a perspective of history and a sense of where we
came from and what we did.

And significantly, access to the past is perceived as contributing to increased understanding and
tolerance: By giving all people a place that has their local history stored, almost regardless of
what background they have. In a sense they could bring a feeling of ‘belonging’ and
understanding about other people in their community. I feel that this could lead to people being
more accepting of different cultures and wanting to learn more.

A further significant impact that arose from the public cohort is that the museum experience
builds ‘perspective’. Perspective can be the ability to see the present in relation to the past,
insight into new worlds and the opportunity to reflect on the human condition, our relationship to
ourselves and to others. It also makes you realise you do not live in a cocoon and you are part
of a bigger picture; It gives people an insight into others lives. It can make them realize that they
should be happy with their lot;

Providing Evidence

One of the key features of ‘evidence based policy’ is the need to provide substantiating proofs of
impact claims- to demonstrate that museum programs and activities have ‘made a difference’ in
social terms that makes them worthy of public funding. The availability of evidence is a
problematic area with Wavell et al (2002) reporting that

While there is an abundance of anecdotal evidence and descriptions of best

practice in the sector, extensive hard evidence of impact, gathered
systematically, is often lacking (2002: i)

A further stage of this study required the professional cohort to back up their claims of impact
with examples of evidence. From the commentary provided, it became evident that this
mechanism was exceptionally useful in generating a self-reflective dimension that effectively
separated ‘advocacy’ claims from those that are defensible. In all, the following impacts
generated by this study could be supported by examples of evidence. With the exception of the
impacts related to indigenous cultures, comparisons between the Australian and British contexts
reveal that these categories of impact are defensible in both countries (CAMD, 2003/4; NMDC,
2003; 2004).

• Museums build the intellectual capital of communities through (a) supporting the formal
education system (b) building knowledge partnerships with other educational providers
and research agencies (c) contributing new knowledge through research on collections;
• Museums contribute to the human capital of communities by providing opportunities for
skill building and team work through volunteer programs;

11
 • Museums contribute to social cohesion by (a) providing a focus for civic engagement
and co-operation around shared projects (b) becoming sites for the celebration of
significant cultural events and rituals
• Museums contribute to the social capital of communities through providing important
leisure amenities;
• Museums contribute to the reduction of social exclusion through (a) providing
opportunities for interaction amongst community members (b) providing public spaces
within which people from a community can meet and interact with visitors to the
community (c) engaging migrant and culturally diverse communities and (d) presenting
indigenous history, culture and contemporary social issues;
• Museums contribute to community identity through presenting the unique history and
heritage of an area.;
• Museums contribute to social change and awareness through supporting indigenous self
management of material cultural heritage
• Museums contribute to the social capital of communities through partnership building;
• Museums build cultural capital for communities (a) through their role as repositories for
receiving bequests and donations of material cultural valued by individuals and
communities and (b) through acquiring significant material for the collection
• Museums have an economic impact on communities through (a) creating work (b)
promoting a positive image for city/town that attracts tourists and investment (c)
attracting revenue via grants and sponsors (d) the multiplier effect of special programs

Reflections for the Future

Outcomes are the short to medium term results of applying outputs. New or renewed interest in
a subject might be the outcome of a museum visit. (CHC, 2002: 13)

Outcomes are the positive or negative engagement with planned outputs by an

intended or unintended user. Outcomes can be short or medium term. (For
example: books read, visitor interaction with a website, user satisfaction with
answer to enquiry, recollection of a memorable event). (Wavell et al, 2002:7)

The difference between outcomes and impacts is that impact is associated with long term
changes in people or a community

……resulting in a change in state, attitude or behaviour of an individual or

group after engagement with the output and is expressed as ‘Did it make a
difference?’ (Wavell et al 2002: 7)

An example here might be the growth in literacy (or even improvements in

mental well-being) as a result of reading library books, or an improved
understanding of history and heritage as a result of visiting a museum.(CHC,
2002: 14)

It is the author’s opinion, that the list of indicators generated by this study goes some
considerable way to demonstrating that museum activities have intermediate outcomes that go
beyond the immediate visitor experience and which contribute to the individual, social and
economic life of communities in the longer term. What they do not provide is sufficient evidence
that these contributions have made a difference in terms of changed attitudes, behaviour or

12
knowledge amongst individuals or in terms of the social and economic life of the communities in
which they live.

This takes us back to three fundamental areas: (a) the issue of causality (b) the necessity for
programs planned with intentional social outcomes if impact related to causality is to be
measured and proved and (c) the importance of benchmark data against which change can be
monitored.

Australia is fortunate to have made considerable headway in producing benchmark data over
the last ten years through the Council of Australian Museum Directors’ Annual Survey and
through the work of departments of evaluation and audience research permanently located
within major museums over the last 14 years. Moreover, two impact studies related to
measuring change directly related to museum experiences have been undertaken in Australia
(Scott et al, 2003; Scott et al, 2004).

But more needs to be done. Recent work undertaken by the Australian Expert Group in Industry
Studies (AEGIS, 2004: 33), for which the author was an industry advisor, identifies a range of
issues that need to be considered in developing impact studies if direct causality is to be
identified and impact confirmed.

Determining targets before the project begins in order to conduct appropriate evaluation of the
outcomes, is the starting point. In other words, direct intention to create an impact is essential if
impact is to be measured and proved. AEGIS recommends identifying the expected effects to
make causality easier to establish and to spell out these anticipated effects in a testable
manner. Differentiating long-term from short and medium term impacts and determining whether
different approaches and measures are required to appropriately measure the causalities
sought is crucial. Identifying and refining the measures to be used and creating a framework by
which these can be applied in a consistent manner across the different types of impacts is
necessary.

Further, both Creigh-Tyte and Mundy (2003) AEGIS (2004) recommend the use of more
experimental research particularly using control groups to isolate effects and to ensure that the
impact can be evaluated and measured.

Direct planning for intended social impact is new to museums. Is this to become part of the
‘heart’ of our future values and purpose?

References

Australian Expert Group on Industry Studies AEGIS. 2004. Social Impacts of Participating in the Arts and
Cultural Activities: Report on Stage Two- Evidence, Issues and Recommendations, University of
Western Sydney, Sydney (unpublished report to the Cultural Ministers Council via the Department
of Communication, Information, Technology and the Arts, Canberra).
Alison, M. and Coulter. F. 2001. Realizing the Potential of Government Services: the Case for Museums,
London, Centre for Leisure Research for the Local Government Association.
Appleton, J. 2002. “Distorted Priorities Are Destroying Museums”. The Independent. 29 May p16.
Bennett, T., 1989. “Museums, and Public Culture: History, Theory and Politics” Media Information
Australia, N0. 53, August, 57-65.
Creigh-Tyte, S., Mundy G. 2003. A Research Strategy for DCMS 2005 /06 (Department of Culture, Media
and Sport) Technical Paper No 3 June.

13
Culture Heritage Consortium, 2002. Impact Evaluation of Museums, Archives and Libraries: Quantitative
Time Series Data Identification Exercise. London: Resource- The Council of Museums, Archives
and Libraries.
Department of Communications, Information Technology and the Arts. 2004. The Current State of Play
2004: report on the information economy website accessed 4/04/05
http://www.dcita.gov.au/__data/assets/pdf_file/23426/CSP_2004.pdf
Ellis, A. 2002. Planning in a Cold Climate. Lecture at the Getty Leadership Institute, July. Los Angeles:
The Getty Center.
Enterprise LSE Cities 2004. Museums and Galleries Creative Engagement, a Report Commissioned by
the National Museum Directors’ Conference: United Kingdom, the National Museum Directors'
Conference.
Ferguson, L. 2004, CAMD Survey results 2002/3, a Report Prepared for the Council of Australian
Museum Directors, September 2004, Canberra.
Guetzkow, J. 2002, How the Arts Impact Communities: An Introduction to the Literature on Arts Impact
Studies, paper prepared for the Taking the Measure of Culture Conference, Princeton University
June 7-8, 2002.
Kelly, A. and A. Kelly. 2000. Impact and Values, Assessing the Arts and Creative Industries in the South
West. Bristol: Bristol Cultural Development Partnership.
Linstone, H. and M. Turoff. 1975. The Delphi Method: Techniques and Applications. Reading, MA:
Addison-Wesley.
——— 2002. The Delphi Method: Techniques and Applications. www.is.njit.edu/pubs/delphibook
accessed March 2002, April 2003 and November 2004.
Matarasso, F. 1996 Defining Values: Evaluating Arts Programmes, Social Impact of the Arts Working
Paper 1, Stroud, Comedia.
Matarasso, F. 2000a. Cultural Indicators: A Preliminary Review of Issues Raised by Current Approaches,
a meeting held at the Arts Council of England in September 2000.
Matarasso, F. 2000b. Opening Up the China Cabinet: Museums, Inclusion and Contemporary Society. A
paper by François Matarasso for the Museums Association conference, Jersey 16 October 2000.
Scott, C.A. 2003a, “Museums and Impact” Curator- The Museum Journal, Vol. 46, no. 3, July 2003, pp.
293-310.
Scott, Carol A., G. Caban, J. Falk and L. Dierking. 2003b. Museums and Creativity: The Role of Museums
in Design Education, Sydney, Powerhouse Publishing.
Scott, C.A. J. Falk, L. Dierking, L. Rennie and M. Cohen-Jones. 2004. “Interactives and Visitor Learning”
Curator -- The Museum Journal, vol. 47, no. 2, April, pp. 171-192.
Selwood, S. 2002. Measuring Culture. Paper presented at the symposium, Statistics in the Wake of
Challenges Posed by Cultural Diversity in a Globalization Context, October 21-23, Montreal,
Canada.
Wavell, C., G. Baxter, I. M. Johnson, D. A. Williams. 2002. Impact Evaluation of Museums, Archives and
Libraries: Available Evidence Project. Resource: The Council for Museums, Archives and Libraries.
Aberdeen, Scotland: The Robert Gordon University.
Weil, S. 1994. “Creampuffs and Hardball: Are you Really Worth what you Cost?” in Museum News
(U.S.A.), vol. 73, pt.5, Sept- Oct: 42-3, 60, 62-3.
Weil, S. 1997. “The Museum and the Public” Museum Management and Curatorship, Vol. 16, No. 3: 257-
271.

14