Social_Programme

Essay CSV 3102 Social Program Evaluation What is a program A program is an objective based set of inter-related activities that intend to meet a number of specific aims and objectives (McDavid, Hawthorn, 2006). Programs are organised activities, “a series of planned actions designed to solve some problem”, according to Royse, Thyer, Padgett, and Logan (2006). So programs are interventions or services that are expected to have some kind of an impact on the program participants. (Royse et al). They are user-defined. They can vary a great deal in scale, structured simply or be complex. A good program would have adequate resources in staffing, stable funding, professional expertise and tangible items such as equipment, as well as intangible items such as a “service philosophy” (Royse et al). The evaluator There is sometimes a perception of threat from the presence of an outsider. (Owen, 2006, p.140). Therefore, an ‘outsider’ should become familiar with the needs of the internal audiences and ensure that the evaluation is responsive to the needs of the agency. The more the evaluator can work with the agency personnel, the more clients will come to trust the evaluator and use the findings to make changes suggested by the evaluation (Owen, 1990) as cited in Owen (2006). An ‘insider’, on the other hand, knows the organisation and the key staff, so would have a better understanding of the program. However, his dilemma would be to be truthful or be loyal to the organisation and not disclose the true picture. The program I have been called in by Star Secondary School (SSS) to conduct an evaluation on a program that has been in operation for three years. The program, STAR is one of the initiatives in education techniques at SSS. This program came about in 2006 to address the rising number of failures mainly because of failing in English, Science or Mathematics. A hundred and thirty-six students have been on this program since. Through this program, SSS aims: • to help the secondary four students who have failed or done badly in their examinations improve in their grades. • to check the rising number of failures Objectives: • students to improve in their grades in their weakest subject by at least 5% by end of year examination. • students to improve their grades in other weak subjects by at least 10% by end of year examination. • students to attain at least a credit for English, Mathematics and Science in the final examination. • school to attain a hundred per cent pass in at least three subjects in the ‘O’ level examination. • to prepare the students well for their GCE ‘O’ Level Examinations. Though funding - from the Ministry of Education (MOE) - is available, is the time ripe for evaluation, Weiss (1998) would ask. Through a procedure called “evaluability assessment” (Weiss, 1998), the evaluator could establish if the program warrants a full evaluation. As cited in Weiss (1998), Wholey (1979, 1983, 1994) states that a program has to satisfy three criteria: “a) The program should operate as intended; b) it should be relatively stable; and c) it should seem to be achieving positive results”. There are two models we can use for planning and conducting an evaluation; one is the linear model, a step by step approach which consists of 15 steps and the other, the cyclic model, an alternative view in planning an evaluation by Carol Weiss. According to Weiss (1998), planning is based on constant reflection, a to-and-fro questioning and therefore, cyclic. Conducting an evaluation of a program consists of two phases according to Rutman (1984) as cited in McDavid & Hawthorn (2006, p.25). The first is the planning stage known as evaluation assessment which has ten steps and the second stage, the evaluation study which has five steps. A very important step to take before all steps is ensuring that there is a fully articulated contract (if the evaluator is an outsider) or a memorandum of understanding (if the evaluator is an insider). This forms a written agreement to protect the evaluator. I shall use these steps to help me determine if a full program evaluation is warranted. Phase 1: Evaluation Assessment Step One – who are the clients for the evaluation' Program evaluations are “user-driven” (McDavid, Hawthorn, 2006). Intended users must be identified early in the process and must be involved in the evaluation assessment. My clients here are the key personnel of SSS, the STAR committee comprising the vice principal, the heads of department of Pastoral Care and Career Guidance (PCCG) and English, the clients (students) of the program and also the funding agency, the MOE , Step Two – what are the questions and issues driving the evaluation This scenario calls for an outcome (summative) evaluation. Program stakeholders often have differing views about the purpose of an evaluation. “….it is essential for the evaluator to be aware of these …” (McDavid, Hawthorn, 2006). The vice principal who is new to the school wants to know if the program outcomes meet the intended program objectives. ‘Are we getting the results that we hope to achieve'’ Step Three – what resources are available to do the evaluation Yes, the STAR committee and I have agreed on the resource requirement throughout the duration of the evaluation. Step Four – what has been done previously There has been a small scale evaluation done by the school in the first year of the implementation of the program. I would take advantage of this pre-program data. Step Five – what is the program all about The program objectives are spelt out earlier in this text (page three). The program is stable as I was told. There are proper the key causal linkages among the main parts of the program' If the program has not been implemented the way it was intended, hence, its success or failure, may be the result of the administration rather than of its design (Weiss, 1998) as cited in McDavid and Hawthorn (2006). Step Six – what kind of environment does the program operate in and how does that affect the comparisons available to the evaluator. As open systems (McDavid and Hawthorn, 2006), programs interact with the environments in which they are embedded. Hence, as an evaluator, I have to take this into consideration. The program is stable and there has been previous data I can use. Step Seven – which research design alternatives are desirable and appropriate. Program evaluations involve multiple research designs; evaluators have to consider the strengths and weaknesses of different designs and work on combinations. Step Eight – what information sources are available/appropriate, given the evaluation issues, the program structure and the environment in which the program operates. What data are currently available; are they reliable; will it be necessary for the evaluator to collect additional information to measure key constructs in the program logic. Step Nine – given all the issues raised in steps one to eight, which evaluation strategy is least problematic. Evaluators would do well to anticipate criticisms and be able to respond appropriately as there is no one perfect evaluation strategy. Step Ten – should the program evaluation be undertaken. After having looked at the various issues and constraints, the evaluator may not recommend an evaluation just yet. If that is so, the assessment can tell funders and program staff about obstacles and drawbacks so as to be able to make improvements for favourable outcomes. However, in this particular case, I see the program as worth evaluating. Phase 2: The Evaluation Study As mentioned earlier in this essay (on page four), Weiss (1998) has her own idea about planning. Hers is a cyclical process where questions are asked resulting in the collection of data, then determining the type of data – qualitative or quantitative; that then leading to the selection of the design of the evaluation. Her emphasis – the design is selected based on the answers to questions. The design should be able to explain that the outcomes were due to the program itself and not to other variables or reasons that it should be able “to rule out rival hypothesis” (handout, 2009). I shall incorporate some of her methods into the second phase of the program evaluation, the steps in designing and data-collection. However, I will still list out the remaining steps of the linear model. At this point actually, we are not touching on steps twelve to fifteen, except for a few lines in “disseminating the report” in step fourteen. Step Eleven – develop the measures and collect the data. Data-collection instruments may need to be designed, pretested then implemented. Step Twelve – analyse the data. We would employ the least complex method that will fit the situation Step Thirteen – write the report. The key part is usually the recommendations that are made. The recommendations must be backed up by evidence and appropriate. Rely on visual representations of findings where possible; use clear, simple language; use more headings and subheadings; prepare a clear, concise executive summary; solicit feedback on drafts of the report before finalising it. Step Fourteen – disseminate the report. After the program evaluation, I would prepare and produce a report and make presentations of the findings to key stakeholders, including the clients of the study. Step Fifteen – make changes, based on the evaluation. Possible changes would include improving the existing program; increasing the size and scope of the program; downsizing the program; replacing or eliminating the program. Critique: Ethical issues during planning of an evaluation “Evaluation deals with real people in real programs, often people in serious need of help”. (Weiss, 1998). Therefore, all the more, evaluation has an obligation to pay even more attention to ethical questions than most other areas of social science research. The following are some ethical issues that surface during the planning phase. Ethical issues not mentioned during the planning of an evaluation: • Working with people is not the same as carrying out a laboratory experiment with experimental and control groups. When there is limited access to programs, who and how do we select to be in the program or no-program (control) group' In human services, we need to be fair to all potential participants. We could strategise to provide different versions of the program (planned variation) to all eligible participants so no one is excluded. In this way, the control group would have access to some form of the program. • Another scenario – say, we are testing if a certain drug helps control cancer cells in afflicted patients. If the program outcome turns out as intended, are we being fair to those in the control group, who would then be deprived of the chance to ‘survive the cancer’. • Protecting individual rights: Informed consent – respondents should have leeway to decide whether to consent or refuse to participate. They should never be coerced into participation. We have to respect their rights to self-determination. Confidentiality and anonymity – we have to ensure that all information collected be held in strict confidence. Only the evaluation team can have access to any information about any particular individual. Honesty – we should not lie to respondents; sometimes we are tempted to not to reveal the purpose of the study for we fear that respondents may be not take well to the word ‘evaluation’ (Weiss, 1998). Conclusion We have seen that there are two phases in program evaluation. First, there is the evaluation assessment process followed by the evaluation study. The two processes can be done separately so that stakeholders can decide whether to proceed with the evaluation from the assessment results. Conducting a program evaluation is no easy task. It is a complex process - there are many questions to ask, many human issues to tackle, and in that the evaluator should be able to come up with the best design possible. Amidst all these, the evaluator should never forget to consider the ‘human factor’. References McDavid, J. C., Hawthorn, L. R. L. (2006). Program evaluation & performance measurement: an introduction to practice. (pp.1-160). Beverly Hills, CA: Sage Owen, J. (2006). Program evaluation: Forms and approaches. (3rd ed). Australia: Allen and Unwin Royse, D., Thyer, B., Padgett, D. K. and Logan T. K. (2006). Program evaluation: An introduction (pp.1-32). Australia: Thomson Weiss, C. (1998). Program evaluation: Methods for studying programs and policies (pp.72-96). Prentice Hall, New Jersey.