of research designs tend to go best with different types of EIP questions.)
Wanting to start with a very broad search before attempting to narrow it down, Rubin used the keywords: treatment and traumatized children. Alternatively, he could have used filters to narrow things down regarding research designs or his client's characteristics. When you conduct a search, it might be best to try it both ways. Then you can get a sense of what kinds of evidence you'll find using either approach. In his broad search on PsycINFO, however, he found only four references that seemed to be relevant to his EIP question.
Consequently, he returned to Google Scholar to see what would happen if in addition to the main search term, effective interventions with children who witness domestic violence, he added the term experimental design in the next box. In other words, he searched for links to sources that had all of the words effective interventions with children who witness domestic violence anywhere in the reference and with the exact phrase experimental design anywhere in the reference. This time, 153 references appeared. Thus, by adding the filter term experimental design, he came up with a more manageable list of articles. The vast majority of them, however, were still not adequately relevant to his EIP question. For example, the irrelevant ones addressed things like the prevention of violence among adult perpetrators, the treatment of battered women, dating violence among older children, and so on. Moreover, by stipulating that he was interested only in studies employing experimental designs, he was filtering out studies using other designs that might have been more relevant to his EIP question and the particular client characteristics in it.
Finally, he repeated the last step, but this time entering the words battered women in the box labeled without the words. (See Figure 2.1 for a display of the window for the advanced search option in Google Scholar.) Doing so, however, cut the number of references to only 19, and none of them addressed empirical support for the effectiveness of interventions germane to his client.
So, what lessons can be drawn from his various searches? One is that there is no pat answer – no panacea – for simplifying your Internet database search for studies relevant to your EIP question. Even if you use the and conjunction with filter terms, you might have to wade through many irrelevant references to find the ones that are pertinent to your EIP question. Moreover, by narrowing your search that way, you might miss some studies that are pertinent. Instead of expecting to find what you need easily with one set of search terms, you'll probably have to play around with a variety of terms to be sure you aren't missing something relevant and valuable. Things can get pretty complicated very quickly as you try searching different combinations of terms, synonyms, and databases. Some databases track the search terms that you have used, while others do not. If the database you are using doesn't automatically track your search terms, it's likely worth the effort to record the search terms that you are trying. Because searching can be a bit of an art form, and you will likely try several combinations of words and phrases, it can quickly become easy to forget which terms you have tried. Keeping a sheet of paper nearby to jot down your search terms along the way can help you avoid repeating your efforts and wondering, “Did I remember to try searching for the terms family violence, domestic violence, and interpersonal violence?” Also, you'll probably have to wade through many irrelevant references to be sure to find the real gems.
When working with similar clients, you may not need to repeat this process each time. What you find the first time might apply again and again. This is especially true if many of your clients experience similar problems and needs. However, keep in mind that the evidence might change over time. Therefore, if several months or more elapse after your search, you might want to repeat it to see if any new studies have emerged supporting different interventions. Moreover, some newer studies might be more applicable to your client's unique characteristics or your unique practice situation.
2.2.4 A Time-Saving Tip
When conducting your own search, you don't have to read every study that you find. You can examine their titles and abstracts to ascertain which ones are worth reading.
For example, many years ago, when Rubin conducted a review of the effectiveness of EMDR versus exposure therapy in treating PTSD, he encountered an abstract depicting a study that concluded that EMDR helps “bereaved individuals experience what they believe is actual spiritual contact with the deceased” (Botkin, 2000, p. 181). He could tell from the title of the study that it was not relevant to his review regarding PTSD. (But given its bizarre claim, he read it anyway!)
You can decide which studies to read based on the relevance of the study to your practice question as well as any mention in the abstract of attributes that might tip you off about the quality of the study. For example, if one abstract tells you that the study offers a practitioner's anecdotal account of providing exposure therapy to one client, and another abstract depicts a large, multisite experiment evaluating the effectiveness of exposure therapy with many clients, you might be more predisposed to read the latter study. Much of this book is devoted to giving you the information and understanding you'll need to appraise the quality of the various studies you'll find. This brings us to the next step in the EIP process: critically appraising studies and reviews.
2.3 Step 3: Critically Appraising Studies and Reviews
As we've already intimated, the individual studies and reviews that you'll find in your search might vary greatly in regard to their objectivity and rigor. The journal peer review process offers a level of quality assurance. In the peer review process, typically two or three other researchers offer critical feedback and help journals decide whether or not an article is appropriate for publication. Therefore, published articles in peer-reviewed journals have at least been exposed to some process of review and critique. However, the rigor of the review process in journals can vary greatly. Some very strong research studies do not appear in journal articles, while some relatively weak studies do get published. Some studies and reviews, whether in journal articles or other sources, for example, will be conducted and reported by individuals or groups with vested interests. But reviews and studies can be flawed even when no vested interests are involved. Some objective investigators do the best they can with limited resources to overcome some practical obstacles that keep them from implementing their study in a more ideal manner. A while back, for example, Rubin conducted an experiment evaluating the effectiveness of EMDR in a child guidance center (Rubin et al., 2001). He had no funding for the study and conducted it simply because – as a professor – he was expected to do research and was quite curious about whether EMDR was really as effective with children as its proponents were touting it to be. The administrative and clinical leaders in the center projected that in a year's time over 100 clients would participate in his study. They were wrong. It took three years for them to refer 39 clients to his study.
Some flaws are egregious and fatal. That is, they destroy the credibility of the study's findings. To illustrate a fatally flawed fictitious study, suppose Joe Schmo invents a new therapy for treating anxiety disorders. He calls it psyschmotherapy. If it is effective, he will be rich and famous. To evaluate its effectiveness, he uses his own clinical judgment to rate client anxiety levels – on a scale from 0 to 100 – before and after he provides psyschmotherapy to 10 of his clients. His average before rating is 95, indicating extremely high anxiety. His average after rating is 10, indicating extremely low anxiety. He concludes that psyschmotherapy is the most effective intervention available for treating anxiety disorders – a miracle cure, so to speak. You probably can easily recognize the egregious bias and utter lack of trustworthiness evident in Joe Schmo's study.
Other flaws are more acceptable. For example, suppose instead of using the foregoing evaluation approach, Joe Schmo proceeds as follows. He collaborates with a colleague who works in his mental health clinic and who specializes in cognitive-behavioral treatment for anxiety disorders. Joe provides psyschmotherapy to the first 10 new clients referred to him for treatment of anxiety disorders.
