Dr Gerald Koocher, Norcross' colleague on both surveys, after seeing a draft copy of my evaluation of their work, has complained that:
"You appear to expect your readers to blindly substitute your personal review criteria (emanating from your own admittedly biased perspective as an NLP advocate) for the opinions of 6+ anonymous experts selected by two different journal editors to review the work blind prior to accepting the papers for publication. As written, unsophisticated readers of your blog [sic] will remain ignorant of the rigorous peer review process to which our work was subjected."
(Personal e-mail communication)
Note the apparent implication that "NLP advocates" are "admittedly biased" - though this is not an admission I am aware of having made to Koocher or anyone else - but psychologists who characterise whatever it is they think of as "NLP" as a "pseudoscientific, unvalidated or 'quack' psychotherap[y] [sic]" (Norcross et al, 2006. Page 515), are (presumably) to be seen as straight-shooting, clear-headed and totally bias free and well-informed.
Just like the Easter Bunny!
Now, I could contest that last suggestion by telling you about my understanding of the peer review system. That is, the process whereby articles submitted for publication are (usually anonymously) submitted to a panel of "experts" in the subject concerned, who will advise the editor of the relevant learned journal what they think of the article. Then it's up to the editor to decide whether an article gets published, or not. But then I came across what seemed, to me, to be a better option. The statement below was written by Richard Horton, the editor-in-chief of the internationally renowned British medical publication The Lancet, (a peer reviewed journal):
"The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."
(Horton, R. (2000), Genetically modified food: consternation, confusion, and crack-up. In the Medical Journal of Australia, 172: 148–9.
So there you are. Now you have a genuinely expert opinion of the peer review process and hopefully will not be mislead by my alleged bias.
John C. Norcross is Professor of Psychology and Distinguished University Fellow at the University of Scranton, a clinical psychologist in part-time practice, and editor of the Journal of Clinical Psychology: In Session.
He has authored over 300 publications and has co-written or edited 20 books, principally in the areas of psychotherapy, clinical training, and self-change.
(From Professor Norcross' home page: http://academic.scranton.edu/faculty/norcross/
accessed on June 1, 2010)
He also has a string of honours and awards, posts, editorships, etc. that would turn many academics a none too delicate shade of 'envious green'.
Yet Norcross has also headed up what are, in my opinion, two of the most cack-handed surveys I've ever come across. Not least because they seem to be designed to manipulate the results in a none too subtle manner.
(This review includes what I believe to be concrete evidence, from the two articles, to support this contention.)
Before going any further it is only fair to point out that both articles include a section of "Cautions and Caveats" (pages 519-520 (2006) and 12 (2010)) in which the authors themselves admit to several reasons why the results should be regarded with a great deal of suspicion (see Cautions and Caveats).
*** The Short Version ***
Critic(s):
(academic role at the time of publication) Norcross, J.C., PhD: Distinguished University Fellow at the University of Scranton. Koocher, G.P., PhD: Dean, Graduate School for Health Studies, Simmons College, Boston.
with, in alphabetical order: Fala, N.C., B.S. (2010): Post Graduate student? Garofalo, A. (2006): Doctoral candidate in clinical psychology at Nova Southeastern University. Wexler, H.K., PhD (2010): National Development and Research Institutes Inc.
Critical Material: Discredited Psychological Treatments and Tests: A Delphi Poll, Norcross, J.C., Koocher, G.P., and Garofalo, A (2006). Professional Psychology: Research and Practice, Vol. 37, No. 5. Pages 515-522. What Does Not Work? Expert Consensus on Discredited Treatments in the Addictions, Norcross, J.C., Koocher, G.P., Fala, N.C. and Wexler, H. W. (2010) Journal of Addiction Medicine, Vol. 4, No. 3. pages 174-180.
Nature of criticisms:
Based on two sets of poll results: "Neuro-Linguistic Programming (NLP) [is possibly discredited] for treatment of mental/behavioral disorder" (2006) and "certainly discredited ... [as a treatment] for drug and alcohol dependence" (2010).
Original/derivative:
The first poll allegedly follows the established pattern for Delphi polls, and methodologically speaking, the second poll seems to be little more than a re-run of the first, but with a different focus and different interviewees.
Flaw(s):
First and foremost, at no point in either article is there any indication at all that the pollsters provided their "experts" with clear definitions of what was meant by any of the treatment/test labels. Thus both polls totally lack any standardization of what respondees are being asked to assess on.
The profound weakness of this approach can be judged by the fact that in regard to Neuro-Linguistic Programming the "experts" were asked to rank two treatments which don't actually exist.
Second, Norcross et al claim to be seeking support for an EBP - evidence-based practice - approach to psychotherapy, yet their own polls/articles are opinion-based rather that evidence-based.
Indeed, it is rather ironical that these "researchers" should choose to attack the credibility of the FoNLP at all. Because by so doing they are attacking a field of study and practice which, when the processes are carried out according to the genuine NLP-related principles, is evidence-based in a far more practical sense than most academically-oriented psychological research-based practices (see Conclusions, below).
These articles share several basic flaws, any one of which would be enough to call the poll results into question. Taken together they invalidate the polls, the results and even the underlying rationale in general, not just for the question about "NLP":
Neither NLP nor the FoNLP are forms of psychotherapy, which is why there is no standard procedure in the FoNLP relating to the "treatment of mental/behavioral disorder" (note, by the way, the lack of any indication as to what "mental/behavioral disorder(s)" is/are being referred to).
Likewise there is no standard procedure for using any part of the FoNLP for the treatment of "drug and alcohol dependence".
Likewise in numerous cases a variety of loosely related "treatments" were bundled together as though respondents were being asked to rate a specific procedure: "Treatments for mental disorder resulting from Satanic ritual abuse" (2006), "Hypnosis for alcohol dependence" (2010), etc.
No effort was made to discover whether any of the participants were genuinely and sufficiently familiar with the "treatments" and "tests" they rated to offer accurate evaluations.
In practice, any "expert" who rated the credibility of either of the allegedly NLP-related "treatments" automatically DIScredited themselves as an "expert" the subject insofar as they were claiming to have knowledge of something that doesn't exist.
Both polls are based on the highly dubious proposition that being an "expert" in one area of psychology automatically makes one an expert in all matters psychological, unless a respondent says otherwise. No evidence is provided to substantiate this implied claim.
In practise even the basic concept behind the two polls appears to be a logical error, an "appeal to authority". That is to say, the underlying assumption is that because the poll respondents are allegedly "experts", if their averaged opinion holds that a particular test or treatment is "discredited", then it is.
There is at least prima facie evidence that the results of both polls were affected by a form of manipulation likely to evoke psychologically-induced compliance in those taking part (see Priming the Pump, Stacking the Deck below).
The argument offered in support of the negative emphasis in the two polls implies that it is more useful to know which therapeutic processes should be avoided than it is to know what process(es) should be used.
In practice, neither poll has been designed in a way likely to elicit accurate/reliable results. Nor is it possible, on the evidence supplied in the two articles, to derive any sensible idea as to what the results actually mean.
On the contrary, both polls seem to be seriously lacking in the kind of safeguards needed to ensure that the results were based on something more than sheer guess work and/or personal prejudice. "Expert opinions may become widely held either because they are correct or because most experts simply share the same heuristic biases" (Norcross et al (2010, page 179)).
"... the standard deviations decreased on 57 of the 59 ratings. The mean difference was -.213." (page 176)
"The mean ratings evidenced similar change: ratings on 56 of the 59 items increased (in the direction of more discredited). Mean difference from round 1 to round 2 was .197." (page 176)
Taking the average of the reported mean scores for the treatments, (based on 2010, Table 2, pages 177-178), 56 results shifted by an average of 0.21 in a negative (upward - more discredited) direction, whilst 3 scores shifted by an average 0.05 in a positive (downward - less discredited) direction.
The spread of negative shifts was between 0.42 (Chloradiazepoxide for alcohol dependence) and 0.01 (Group process psychotherapy for alcohol dependence). The spread of positive shifts was between 0.08 (Providing transitory substitute gratifications for treatment of alcohol dependence) and 0.01 (Twelve-step facilitation for alcohol dependence).
95% of the scores moved in a negative direction between Round 1 and Round 2, and by an average that was approximately three times the size of the average positive shift.
This is arguably evidence of the extent of the influence of using a strongly negative word in all of the rating descriptions and in the participant instructions. That is to say, respondents, might be negatively influenced on Round 1, and when receiving the feedback on the first round, might unconsciously be further persuaded to comply with the implication that the treatments are less than fully creditable (especially since not one treatment has received a rating lower than 1.97) and thus make their own ratings more negative in Round 2.
Moreover, if Loftus and Palmer's deductions were correct, respondents might literally, "remember" giving somewhat higher (more negative) ratings first time around, and imagine that the later ratings are similar to those earlier ratings.
An interesting demonstration of the authors' lack of accurate knowledge (in both articles) appears when they present their readers with these statements:
Recently, several authors have attempted to identify pseudoscientific, invalidated, or "quack" psychotherapies ..."
(Norcross et al, 2006. Page 515)
Most assuredly, select investigators have attempted to identify pseudoscientific or ineffective treatments applied to a variety of mental disorders and addictions ..
(Norcross et al, 2010. Page 5)
Both versions are followed by a list that includes:
Carroll, 2003; Della Sala, 1999; Eisner, 2000; Lilienfeld, Lynn, and Lohr, 2003; Singer and Lalich, 1996.
It is interesting to note, then, that:
Carroll (Robert Todd) is a retired teacher of philosophy, not a psychologist. And his article on "NLP" in his website/book The Skeptic's Dictionary is notable for a string of errors from the first paragraph onwards.
See HERE.
Della Sala's criticisms of "NLP", in both the cited work and in the Introduction to his 2007 book Tall Tales About the Mind and Brain, seem to be based on Sharpley's deeply flawed "review" of 1984 (via the Druckman and Swets report of 1988), and Heap's similarly inaccurate review of 1988, 1989, etc.
See HERE.
Eisner is a bit of a two-edged sword in this context. On the one hand he certainly claims to be weeding out the rubbish, and on the other he basically states that all psychotherapy is pseudoscience. Which presumably isn't much comfort to Professor Norcross who is, amongst his various roles, a psychotherapist!
On that basis it seems the lead author of these two articles attacking "quackery" and "pseudoscience" is himself a pseudoscientist!
(Eisner has little of any consequence to say about "NLP" since he imagines it to be a form of psychotherapy.)
Lilienfeld, Lynn & Lohr have very little to say about "NLP" and gave little sign that they know much about it. In the publication cited here the main commentary on "NLP" (a single paragraph on one page, and a reprise of two or three sentences from that paragraph later on) was actually part of a chapter written by Associate Professor Nona Wilson. Ms Wilson seems to have had little or no idea what "NLP" is about.
See HERE
Singer and Lalich's book Crazy Therapies contains some interesting material - mainly directed at deviant forms of psychotherapy. As far as "NLP" is concerned, however, her seven page treatment is little more than a rant, remarkable for its lack of authoritative quotes (just 1), its preference for non-authoritative material, a pair of thoroughly underwhelming "case studies", and its assumption that the Druckman and Swets report was accurate.
See HERE
In both articles the authors worry that these "pioneering efforts" have not "systematically relied on expert consensus to determine their contents", and "have provided little differentiation between credible and noncredible treatments" (2006, page 515; 2010, pages 175).
Nevertheless, Eisner (2000), Lilienfeld et al, (2003) and Singer & Lalich (1996) are listed as being amongst the 'journal articles and books discussing discredited, potentially harmful, and "crazy" therapies' (page 6) the authors consulted when looking for candidates for their second (2010) poll.
From a FoNLP standpoint it would have been interesting if all of the authors concerned had bothered to do the research needed to reach an accurate understanding of the subject before making their pronouncements.
It is almost incomprehensible, under the circumstances, that the authors of the first article actually closed by giving themselves a big pat on the back:
Our Delphi study systematically compiled clinical expertise on credibility, based perhaps on the best available research. ... The consensus emerging on this Delphi poll on potentially discredited treatments and tests leaves us feeling encouraged.
(Norcross et al, 2006. Page 522)
Likewise, at the close of the second article they wrote:
We believe that this study, as did its parallel on mental health treatments (Norcross et al, 2006) offers a cogent, positive first step in consensually identifying the "dark side" or "soft underbelly" of modern addiction treatments and in providing a more granular analysis of the continuum of discredited procedures."
(Norcross et al, 2010. Page 179)
As a famous tennis star was so fond of saying: "You cannot be serious!"
But of course they were.
References
Carroll, R.T., neuro-linguistic programming (NLP) (2003). In The Skeptic's Dictionary [sic], R.T. Carroll (ed.). John Wiley & Sons, Inc. pages 252-260
Della Sala, S. and Beyerstein, B.L. (2007). Introduction: The myth of 10% and other Tall Tales about the mind and the brain. In Tall Tales about the Mind and Brain: Separating fact from fiction, Sergio Della Sala (ed.), Oxford University Press, Oxford. Pages xx-xxii.
Loftus, E. E., and Palmer, J.C. (1974) Reconstruction of automobile destruction. In Journal of Verbal Learning and Verbal Behavior, 13. Pages 585-589.
Norcross, J.C., Koocher, G.P. and Garofalo, A. (2006), Discredited Psychological Treatments: A Delphi Poll. In Professional Psychology: Research and Practice, Vol. 37, No. 5. Pages 515-522.
What Does Not Work? Expert Consensus on Discredited Treatments in the Addictions, Norcross, J.C., Koocher, G.P., Fala, N.C. and Wexler, H. W. (2010) Journal of Addiction Medicine, Vol. 4, No. 3. pages 174-180.
Singer, M.T. and Lalich, J. (1996) Crazy Therapies. Jossey-Bass Publishers, San Fransisco. Pages 168-176.
Wilson, N. (2004) Commercializing Mental Health Issues: Entertaining, Advertising and Psychological Advice, in Lilienfeld, S.O., Lynn, S.J. and Lohr J.M., Science and Pseudoscience in Counseling Psychology. Pages 446 and 455.