|  | | | |
What tips can you give me?
- Evaluation is best integrated into a project right from the planning stages.
- Collect information at the beginning to establish your baseline, then during and at the end of the project.
- Involve all partners as each may have different aims and share the workload if possible.
- Budget time and money for the evaluation process.
- Outline clear aims, objectives and measures of success.
- Collect both quantitative and qualitative information.
- Use a range of methods to collect information and have fun by being creative in the way you do it.
- Test out questionnaires with a sample group.
- Ask permission if you want to quote people at a later date.
- Schedule in evaluation meetings during, as well as after, the project to enable you to reflect and adapt in response to your findings.
top
What's evaluation?
Evaluation involves making judgments, based on evidence, about quality, value and effect during and after completion of a project.
Related terms:
'Monitoring' = the gathering of information about the progress of a project at any given time, measuring things, actions, numbers etc.
'Documentation' = the physical evidence you might gather of your project e.g. examples of work, photographs or video, visual diaries, written comments etc.
top
What's special about evaluating the arts?
The arts have a powerful impact on our sense of identity, meaning and our values. This is difficult to measure.
Also, peoples' views and tastes regarding the arts vary enormously which can make it difficult to decide on measures of quality.
However, as in all evaluation processes, when we evaluate an arts project we compare what is happening/ has happened against benchmarks that we have set.
top
Why evaluate?
| To affirm |
- clarify to yourself and others the value of the work as you go along and after the end of a project, celebrating the strengths and defining good models of practise. |
| To learn & improve |
- benefit from the learning opportunities that arise from aspects that don't work so well, improve professional practise during and for future projects. |
| To establish a track record |
- convince others you run well managed projects, which hit key targets and fulfil aims and objectives. |
| To communicate |
- create a stock of visual images, objective statements, quotes, qualitative and quantitative information that enables you to communicate the scope of a project to a wider audience. Include reports, marketing materials or funding bids for future projects. |
| Quantitative information |
- objective factual information including statistics, numbers etc |
| Qualitative information |
- subjective views, particularly from observers and participants, giving a sense of what happened and quality judgements. |
| Aims |
- why is the project taking place? |
| Objectives |
- the specific things partners want to achieve. Think of these as steps towards the overall aims. Ideally these should be: |
| SMART |
- Specific, Measurable, Achievable, Realistic and within a Timescale. |
| Measures of success |
- how you can identify if an objective has been achieved. These are also called "performance indicators", "success criteria", or "outcomes". | (In depth illustrative examples of projects aims, objectives and measures of success can be found in "Partnerships for Learning - a guide to evaluating arts education projects" by Felicity Woolf , published by The Arts Council of England- a downloadable pdf, 600K, which can be accessed from http://www.evaluationforall.org.uk/
top
How to evaluate?
Evaluations can be done on any scale. They can be made quickly or they can be the result of systematic and wide ranging data collection exercises over time. Whatever the scale of your evaluation it will follow these key stages:
- Planning
- Collecting information
- Assembling and interpreting information
- Reviewing, reflecting and making changes
- Reporting and sharing information
top
1) How to plan?
Plan in the evaluation right from the start
|
Whatever your circumstances, evaluation is most effective integrated from a projects initial planning stages. |
| Understand your context |
The scale of the project, geography, participants, resources, budget, your expectations and the particular aims and objectives of all partners |
Establish aims, objectives and measures of success
|
This enables all partners to clarify what it is they hope the project will achieve and how to assess if these have been achieved. |
| Involve all partners |
Different partners may have different aims and objectives to evaluate. They can also share the workload. |
| Assess what information you want |
Assess what information you will need to collect, and what methods to use to collect it?
Use your aims, objectives and measures of success to plan what information you need to collect. Remember it is to help you judge if you have achieved what you set out to, not just to document the project. Aim to collect both quantitative and qualitative information. See example, definitions and "Collecting information" below. |
| Decide who does what |
Who will have overall responsibility for evaluation? Who else will have a role in collecting evidence? Spread the workload as much as possible, it will also encourage ownership of the process. |
Budget
|
How much will evaluation cost in time and money? (Remember, artists will expect to be paid to attend evaluation meetings). |
| Schedule review meetings |
Review sessions during and after the project to enable partners to share, reflect, respond and adapt as things progress and inevitably do not run to plan. |
Expect the unexpected
|
There are invariably additional unexpected outcomes to arts projects. Be prepared to record and/or respond to these when they arise.
|
|
Example:
A youth club analysed the responses to a questionnaire about a drama project during the summer holidays. Each question used a combination of scoring and commenting. In the analysis the comments were given a context through the quantitative information.
Q: What did you think of the activities during the week?
A: 88% thought that the activities were good and gave them 5/5.
Comments included:
- "Great - the first day I wasn't sure but I soon got into it"
- "Can I do this next year?"
- "I've enjoyed it all" | top
2) How to collect information?
General Points:
- Use existing material
It is perfectly legitimate to draw on existing material, e.g. reports to management committees, extracts from annual reports, publicity materials, project briefs etc.
- Collect information at the beginning, during and end of project
- Establish your baseline
Establish your baseline at the beginning of your project in order to be able to assess later what has changed, e.g. participants previous experience, knowledge, skill levels, personal and socio-economic situations.
- Use a range of methods to collect information
Different materials and methods lend themselves to collecting different types of information.
- Consider the best methods to enable your participants to communicate their opinions
Be aware of the abilities, personalities or special needs of your participants. Some forms of evaluation require literacy, confidence etc.
top
a) How to collect written and statistical information?
| Reports and records from artists and workers |
Evaluation questionnaires |
| Direct written feedback from participants |
Written diaries |
| Comment boxes |
Graffiti walls | Written material will provide quotes you can use at a later date. Personal recommendation is a powerful tool and people do read comments especially if made by someone they consider a peer.
Always explain where information will be used and ask permission if you want to quote people specifically.
There is in depth advice on creating questionnaires on
http://www.evaluationforall.org.uk/
Key points to remember about questionnaires
- Avoid leading questions and ones which can be answered with a yes/no.
- Numerical scoring can be useful as it allows statistics to be calculated. e.g.. "Between 1 (poor) - 10 (excellent), how would you rate...?" enables you to calculate the average rating.
- It is useful to test out a questionnaire with a sample group enabling you to weed out questions that don't work.
- Leave space for people to comment
- Keep them as short as possible
| top
b) How to collect verbal information?
Group discussions
These tend to reflect how well participants have engaged with the task, how appropriate the plan and methods were and how the group have worked together. There is in-depth advice on running group feedback sessions on http://www.evaluationforall.org.uk/
One-to-one interviews
These provide more personalised microscopic views as to how well your project allows for individuals development, creativity and learning.
Audio recordings / diaries / interviews
Sometimes people find recorders inhibiting. It can be fun to get your participants to interview and record each other's feedback, or create audio diaries.
Consider how you will record verbal information: write a summary of your findings and include quotes; use a flip chart to write up peoples comments; make an audio recording; have somebody take notes or transcribe a recording.
top
c) How to collect visual information?
| Documentation of work produced |
| Documentation of working processes |
| Visual comment's made by participants (see creative tools below) |
| Video recordings/diaries/interviews |
| Photo-stories | The better the quality of your documentation, the better you will be able to represent your project. You may even consider budgeting for a professional to do some of this for you to provide high quality photographs and video.
Video is a useful way of recording an experience providing it does not intrude on the work in progress. Playing back a video at the end of a session can illicit interesting feedback from the artists and participants. Setting up a 'video box' can give participants the opportunity to make private comments.
Beware - if you want to use your video material to publicise or document your project publicly you will probably need to budget time and/or money to edit it.
top
d) How to use creative tools?
E.g. painting, diagrams, collage, drawing, photography, sound, music, role play, acting, dance, movement, poems, word play, storytelling etc.
Using creative tools to generate evaluation material is a powerful method of representing the quality of an experience. It generates insightful, surprising and emotive material.
How you approach it will depend on your resources and skill base - it might be using the art form(s) in your project, or you may have a store of art materials you can use. Be brave and have fun, you'll be amazed and amused at the results.
Try these
| 1) |
Provide your group with coloured paper, felt tips, pritt stick, newspapers (for text collage), pencils, glitter, and scissors.
Either as individuals or in small groups, ask your participants to make something using these materials that expresses their experience of the project. Tell them they can use shape, colour, diagrams, glitter, collaged text etc. Set them a time limit. |
| 2) |
Ask participants to go into groups of two. Provide each group with either a Polaroid camera or digital camera. Give them 20 minutes to devise and take 3 - 6 pictures that express their experience of the project. Encourage them to be creative and expressive e.g. portraits of facial expressions, a bird in the sky, an overflowing rubbish bin etc.
Once they have done this, ask the group to interpret each other's images without the photographers giving an explanation first. Encourage them to compare the interpretation with their intentions. | top
3) How do I assemble and interpret information?
You will want to interpret and present your information convincingly, showing if aims and objectives are being, or have been, achieved. Until you have interpreted your information it does not always tell you very much. Interpreting information can be time consuming so make time for it. It is useful to assemble and interpret information for review meetings during your project, as well as at the end of your project.
- Transform data into information
- Include unexpected outcomes
- Try to create an overview of the project
- Illustrate if aims and objectives are being achieved | About percentages:
When turning numerical information into percentages it is important to state how many 100% would be e.g. 7 out of 10, and 70 out of 100 are both 70%. It is important that the basis of the information you are offering is clear.
About interpreting qualitative information:
This can be difficult and it is useful to:
- Refer to your measures of success to help choose what information is useful, usually that which shows change, progress or development.
- Sort comments into positive and negative groups
- Look for unexpected outcomes
- Try to establish a majority view
- Make sure you have a range of voices including participants, artists, group leaders etc.
top
4) How to review and make changes?
Reflecting on the results of your evaluation and using this to improve your practise and strengthen decision-making is one of the key benefits of evaluating your work. Do this during your project you as well as at the end.
Using review meetings:
Use review meetings during the project to update partners on progress, share information and make decisions about any changes.
At the end of the project it may still be most useful to reflect with your partners, especially if you are continuing to work together. Alternatively, it may be more useful for partners to reflect on the evaluation individually and consider how it will affect their individual future work.
You should also consider if the project is likely to have longer-term outcomes, as yet unknown. If so, how will partners find out about them and are they likely to be significant for future work?
In considering what are the key findings from the evaluation for the partnership and/or for individual partners you may find it useful to do a SWOT analysis - Strengths, Weaknesses, Opportunities and Threats
top
5) Reporting and sharing
Sharing your evaluation and documentation creates the opportunity for wider access, education and appreciation of your work. You can
- Present models of best practise for the benefit of all
- Show others what you are doing and how well
- Support future project development and funding applications.
Reports
Written reports are useful and usually necessary. If you have been evaluating on an ongoing basis you will have most of the information you need.
- When writing a report consider:
- Who it is for, what do they need to know and what else would you like them to know
- Keep it as short as possible
- Combine qualitative and quantitative information to be most convincing
- Remember that pictures and diagrams save on words
- Try to avoid long, unbroken passages of text
- Use tables and bullet points to summarise information.
Other
As well as an opportunity to focus and present your work, the following will also generate awareness, support, understanding and publicity.
You can use these to invite and/or inform participants, partners, potential partners, workers, artists, carers, funders, potential funders, bosses, arts officers, councillors, local press etc.
|
A celebration event
Exhibition
Performance
Website
Publicity campaign
Publications / cd / cd rom / video
Training sessions, seminars, conferences |
| EVALUATION |
Comments |
| Planning |
|
| Have you included all your partners? |
|
| Have you established aims, objectives and measures of success? |
|
| Have you agreed who does what? |
|
| Have you budgeted time/money? |
|
| Collecting information |
|
| Have you checked if there is existing information you can use? |
|
| Are you collecting information at the start, during and at the end of the project? |
|
| Have you established your baseline? |
|
| Are you collecting both quantitative and qualitative information? |
|
| Are you using a range of methods to collect information? |
|
| Assembling and interpreting information |
|
| Have you transformed any data into information? |
|
| Have you included unexpected outcomes? |
|
| Have you got an overview of the project? |
|
| Are you illustrating if aims and objectives have been/are being met? |
|
| Reviewing and reflecting |
|
| Are you making time to present, review, reflect and respond to your findings during as well as at the end of your project? |
|
| Is the project likely to have longer-term outcomes and how will you find out about them? |
|
| Reporting and sharing |
|
| Have you combined qualitative and quantitative information? |
|
| Are you showing how you have met the aims and objectives, as well as describing what happened? |
|
| Have you considered all the possibilities for sharing your work and your findings with others? |
| |
|
