for finding more information when little information is readily available.
Evaluating Programs and Interventions
There are two major ways we characterize an evaluative research study with regard to scope. The scope of some evaluative studies is broad, and they could be labeled as program evaluations. The scope of other evaluative studies is narrower, and they will be labeled as evaluations of interventions. More detail on this distinction will be offered in a later chapter. In this chapter, you will be given an introduction to this distinction.
A program evaluation may entail the examination of various aspects of a program. An example could be the child abuse treatment and prevention program, which has services designed to review complaints of child abuse and take necessary legal action, services designed to offer treatment for abused children or abusive caregivers, and services designed to prevent child abuse in the community. This evaluation might examine the effectiveness of the intake service in the initial assessment of complaints of child abuse, the extent to which good standards of service are being implemented, or the degree to which repeat offenses of child abuse have gone down in the past year. Each of these could be viewed as an evaluation of success.
An intervention evaluation is more narrow. An example could be your evaluation of your parent education intervention for a group of 11 parents confirmed for child abuse. You want to measure the extent to which they have increased their knowledge of good parenting practices. Another example could be your evaluation of the effectiveness of your treatment for a single abused child for the improvement of self-esteem. For each evaluation of an intervention, you have a designated client or group of clients, and you will measure each client on the same outcome measures.
You might say that your evaluation of your practice as an individual social worker is an evaluation of an intervention, while your participation in the evaluation of a major program is an example of a program evaluation. In the latter case, the outcomes are not significantly due to your own practice. This book places emphasis on the evaluation of your practice as a social worker, so we will focus more on the evaluation of interventions.
The Program Evaluation
When you participate in a program evaluation, you may examine any aspect of the service system. The service system will show all the parts of a program that contribute to success in one way or another. We can view some of the aspects of the system as follows:
1 A depressed woman asks your agency for therapy.
2 This woman is eligible for your service because she is a resident of your county and this is the only eligibility criterion that is employed for this service.
3 The client receives cognitive–behavioral therapy.
4 The therapist is a licensed clinical social worker, so she is qualified to offer the service.
5 The therapy is in the form of eight 1-hour treatment sessions.
6 The client’s level of depression is reduced by 35%.
7 The estimated cost of this therapy is $800 ($100 per hour of therapy).
In a later chapter, you will see the above in the systems terminology of input, process, output, and outcome. You will also see that efficiency is determined by the cost per output (e.g., an hour of therapy). For now, you will review the complexity of the aspects of service that could be the subject of a program evaluation.
Let’s suppose you are evaluating the system for the Hampton Behavioral Health Practice. Each of the clients treated for depression by this agency is given a pretest for depression using a given depression scale where higher scores represent higher depression. This scale has the score of 20 as the cut point for a level of depression that indicates that treatment is needed. Let’s suppose that you examined the mean pretest depression score for the clients of this agency and found that it was below 20. This suggests that the typical depressed client did not have a level of depression that suggested a need for therapy. These data are contrary to the major goal of the agency, which focuses on the rehabilitation of persons who are severely depressed. You have evaluated one aspect of the service system for this agency and learned that it is not doing a good job of reaching the intended target population.
You may learn from a program evaluation that a given agency is not adhering to the standards of good service with regard to things such as the credentials of the staff or the size of caseloads. Adherence to established protocols for the delivery of service is another possible theme for a part of a program evaluation. For example, the Rape Crisis Program of your hospital may have a list of seven things that must be done for each victim of rape who is served. You might examine the cases for this service to see if all these elements of service were offered. What percentage of clients had all the elements provided?
The most critical aspect of an evaluation of a service system is the measurement of client outcome. Did the clients get better? Is the level of depression lower? Are the school grades higher? Is the recidivism rate for delinquency lower? Here you can see some conceptual overlap between the program evaluation and the evaluation of an intervention. You might want to characterize a program evaluation as having, among its many components, data on various interventions.
The Intervention Evaluation
For the part of this book that deals with evaluation research, the evaluation of an intervention is the priority. An intervention is a set of activities designed to achieve an objective with a client or group of clients. The focus is on outcomes. Did this group of 18 discharged hospital patients adhere to the discharge plan at a level higher than that for patients of other hospitals? Were the treatment scores for Ms. Jones for self-esteem higher in the treatment period than in the baseline period (i.e., before the treatment started). Did this group of eight at-risk middle school students have higher grades during the semester of the intervention than in the prior semester?
Quantitative and Qualitative Measurement
There are two major types of measurement in social work research—quantitative and qualitative. With quantitative measurement, you measure each variable either as a category (e.g., male or female) or as a number (e.g., age measured in years, score on the anxiety scale, etc.). When you measure your variables in a qualitative manner, on the other hand, you will normally have words to examine. This would be the case if you have responses to an open-ended question on a survey. For example, you may confront the following question on a questionnaire: “In your own words, how would you describe your feelings about being a parent?” In this case, you put down words on the page. These words will be examined in a qualitative study. Sometimes you make notes from your observations of an environment you are studying. In this case, you are also analyzing words, the words you wrote down.
For quantitative measurement, the researcher decides how to categorize things and wishes for the study subject to respond according to these categories. As a study subject, you may be asked to select one of the following categories that best reflect your opinion about a theme: strongly agree, agree, undecided, disagree, or strongly disagree. With qualitative measurement, the study subject has the flexibility to determine the words that will characterize his or her thoughts on the theme of the question being posed. Because of this flexibility, the qualitative method of measurement is more often found in exploratory studies.
While you can measure your phenomenon of interest either qualitatively or quantitatively with regard to any of the four types of research, we will use examples in this book of qualitative measurement
