If at First You Don't Succeed...

Conclusions:
Both polls seem to be intended to "push" what the first article refers to as "The burgeoning evidence-based practice (EBP) movement in mental health" (Norcross, Koocher and Garofalo, 2006.  Page 515).

Just how "Neuro-Linguistic Programming" got dragged in, other than the unfocused searches for anything that someone, somewhere has criticised, is not made clear, which makes its presence even more inexplicable since the various authors involved in the two polls seem to know little or nothing about the FoNLP.  On the contrary, both questionnaires appear to refer to Neuro-Linguistic Programming (NLP) as a single instrument.  Indeed, the authors of both articles seem to have shared the erroneous belief that there is something called "NLP" which is a form of psychotherapy hence, for instance, the reference to finding out: "Which psychotherapies are effective?" (Norcross, Koocher and Garofalo, 2006.  Page 515).

*** End of Short Version ***

*** 'Director's Cut' ***

'Away From' is a Poor Way to Set Goals

Most experienced NLPers will know about "meta programs" - ways in which we consciously or unconsciously filter incoming information.  One such meta program is known as "Towards/Away from", meaning that some people tend to take notice of information which will help them to achieve some goal, whilst others are more likely to look for ways to avoid what they don't want.  For example, one office worker may be willing to do unpaid overtime because they are looking for ways to move on to a more rewarding position, whilst another shows the same willingness, but only because they see it as a way to avoid getting fired.
In the NLP-related goal-setting process it is claimed that people are more likely to be successful if they set positively-framed goals rather than negatively-framed goals.  Because setting a goal based on what you don't want is like trying to drive a car in which the only view of the outside world is through the rear-view mirror.

It is interesting, then, to note how the authors of these two articles have framed their goal:

The ... evidence-based practice (EBP) movement in mental health ... has provoked enormous controversy within organized psychology, and with the exception of the general conviction that psychological practice should rely on empirical research, little consensus currently exists among the various stakeholders on either the decision rules to determine effectiveness or the treatments designated as "evidence-based" ...
    We believe that it might prove to be as useful and probably easier to establish what does not work - discredited psychological treatments and tests.  Far less research and clinical attention have been devoted to establishing a consensus on ineffective procedures as compared to effective procedures.
(Norcross et al, 2006.  Page 515.  Italics added for emphasis)

And in the second article:

The focus of EBP falls squarely on what works ...
    But EBP largely ignores what does not work.  Far less research and clinical attention has focused on establishing a consensus on ineffective methods as compared to effective methods ...
    We believe it will prove useful and perhaps easier to establish a professional consensus on discredited treatments for addictions.  Doing so may counter the widespread tendency for professionals to practice (or repeat) what they have been taught by their mentors or authorities.
(Norcross et al, 2010.  Page 174.  Italics added for emphasis)

Is there any link, I wonder, between the negativity embedded in the two polls and the fact that in 2006 only 29.3% of the potential participants completed both rounds of the survey?  A figure which dropped to 22.8% in the second poll.

Two for the Price of One

I have combined the evaluations of the two articles by Norcross et al (describing polls conducted by Delphi), because, from the NLP-related point of view they both have the same flaws and are both equally lacking in credibility
Indeed, both polls are so badly designed (as even their authors admit) that there is really not much else of any substance to comment on except the errors.

Starting as They Meant to Go On - 2006

The credibility of the 2006 article disappears in a puff of metaphorical smoke in the second sentence of the Abstract.  Thus we are told that:

A panel of 101 experts participated in a 2-stage survey, reporting familiarity with 59 treatments and 30 testing techniques ...
(Norcross et al, 2006.  Page 515)

But when we check the reported results, in Table 2 (pages 518-519), and Table 3 (page 520) we find that the figures aren't entirely accurate.
In the section of the article headed "Expert Panel" we learn that:

In October 2004, we mailed the five page questionnaire to 290 doctoral-level mental health professionals
(Norcross et al, 2006.  Page 516)

Of the 290 recipients only 130 people (about 45%) returned the questionnaire, but 37 were unusable (26 were from retirees, the other 11 were rejected for other, unspecified) reasons.  So those taking part in round 1 represent only 35% of the original mailout.  Which in and of itself seems to seriously question the usefulness of the poll in the eyes of those of the pollsters' peers who were initially approached.
Then came "Round 2" (of which more in just a moment):

The same instrument was then redistributed to the 101 panelists in February 2005 ... 85 of the original 101 (84%) panelists responded to the second round.
(Norcross et al, 2006.  Page 516)

Hang on a moment.  This means that only 85 of the alleged experts "participated in [the whole of the] 2-stage survey".  And whilst that may be 84% of the people who participated in the first round, it's only 29.3% of those who received the initial questionnaire (though we don't know how many retirees were included in that initial mail out).

Moreover, whilst there were 59 treatments and 30 tests in the first questionnaire, 4 treatments and 5 tests were unfamiliar to over 75% of the Round 1 respondents and were therefore eliminated in Round 2.  On this basis the statement quoted above would have been a lot more accurate if it had read:

A panel of 85 experts participated in a two-stage survey, each of whom reported some (unspecified) degree of familiarity with 55 treatments, and 25 tests.

And as far as the voting on the mythical "Neuro-Linguistic Programming (NLP) for treatment of mental/behavioral disorders" was concerned, just 74 people thought they were able to rate the non-existent treatment in Round 1, down to only 65 people in Round 2.

Starting as They Meant to Go On - 2010

And again, in 2010, we find exactly the same kind of inaccuracies.  In the third sentence of the Abstract we are told that:

A panel of 75 experts participated in a two-stage survey, reporting familiarity with 65 treatments ...
(Norcross et al, 2010.  Page 174)

And again, when we check the text, and the reported results (in Table 2 (pages 177-178)), we find that the figures show a somewhat different picture.
In the section of the article headed "Expert Panel" we learn that:

In January 2007, we mailed a 5-page questionnaire and a 1 page personal information survey to the 250 potential participants.
(Norcross et al, 2010.  Page 176)

Of the 250 recipients only 113 people returned the questionnaire but 38 of these returned blank questionnaires leaving just 75 Round 1 participants.
Then came "Round 2":

We then redistributed the same instrument to the 75 panelists in April 2007 ... 57 of the original 75 (76%) panelists responded to the second round.
(Norcross et al, 2010.  Page 176)

Surely this means that only 57 of the alleged experts "participated in [the entire] 2-stage survey"?  And whilst that may be 76% of the people who participated in the first round, it's only 22.8% of those who received the original mailing (though again we don't know how many retirees were included in that initial mail out).

Furthermore, whilst there were 65 treatments in the first questionnaire, 6 of them were unfamiliar to over 75% of the Round 1 respondents and were therefore eliminated in Round 2.  On this basis the statement quoted above would have been a lot more accurate if it had read:

A panel of 57 experts participated in a two-stage survey, each of whom reported some (unspecified) degree of familiarity with 59 treatments

And as far as the voting on the mythical "Neuro-Linguistic Programming for drug and alcohol dependence" was concerned, it seems that only 32 of the alleged "experts" thought they were able to rate the non-existent treatment in Round 1, down to only 27 "experts" in Round 2.

Were these Articles Written by "Experts"?

Despite the heavy emphasis on the "expert" nature of the participants in the two polls, a rather critical error appears in the first sentence of the main body of the first article:

Which psychotherapies are effective?
(Norcross et al, 2006.  Page 515)

The article allegedly covers "psychological treatments and tests" (page 515).  But "NLP" is a label for a specific form of modelling, and the wider field of NLP (FoNLP) is primarily concerned with communication techniques - which can be used in a wide variety of contexts.  It is certainly not a form of psychotherapy, nor a "treatment [for] mental/behavioral disorders" (see Table 2 on page 518, 2006), nor yet a treatment for drug and alcohol dependence (Table 2, page 177, 2010).

As the entry on the National Center for Biotechnology Information, U.S. National Library of Medicine website correctly states:

Neurolinguistic [sic] Programming
A set of models of how communication impacts and is impacted by subjective experience. Techniques are generated from these models by sequencing of various aspects of the models in order to change someone's internal representations. Neurolinguistic programming is concerned with the patterns or programming created by the interactions among the brain, language, and the body, that produce both effective and ineffective behavior.
(Accessed at: http://www.ncbi.nlm.nih.gov/sites/entrez?db=mesh&term="Neurolinguistic Programming" on July 8, 2011)

There is also evidence in both articles that suggests that the authors viewed Neuro-Linguistic Programming (NLP) as a single instrument.  In the second article, for example, interventions such as "Past-life therapy", "Scared Straight" and "Synanon-style boot camps" each appear twice - once linked to substance abuse, once linked to alcohol dependence (Norcross et al, 2006.  Pages 19-20).  Presumably this is to take account of two separate regimes.  But "Neuro-Linguistic Programming" is listed only once, "for drug and alcohol dependence", presumably based on the assumption that there is a single procedure for dealing with both situations.

Of course it is true that some NLP-related communication techniques can be used "in a psychotherapeutic context", as Bandler and Grinder put it in the Forward to Neuro-Linguistic Programming, Volume 1 (1980, no page number), but that doesn't make NLP and/or the FoNLP a kind of therapy.

Priming the Pump, Stacking the Deck

But should we really be surprised by this information?  Probably not.  After all, the authors themselves indicate, in both articles, that they compiled the lists of "treatments and tests" from other sources:

We searched broadly and collected nominations for discredited mental health treatments and tests [sic] via literature searches, electronic mailing lists requests, and peer consultations
(Norcross et al, 2006.  Page 515)

The 65 potentially discredited treatments were compiled through an extensive literature review that included electronic database searches, listserv requests, and peer consultations.  We searched electronic databases (e.g., PubMed, PsychINFO, Cochrane Collaboration, Google Scholar) for published literature using the keywords "discredited," "quack," and "harmful" placed with the words "treatment" and "addiction".  We examined journal articles and books discussing discredited, potentially harmful, and "crazy" therapies (e.g., Eisner, 2000; Lilienfeld et al, 2003; Singer & Lalich, 1996).
(Norcross et al, 2010.  Page 175)

Which may seem to be commendably thorough, until we consider the implications of such a search.

In the first of the two articles the authors wrote:

A surprising finding to us was the large percentage of experts unfamiliar with the listed practices.  For example, in the first round, 56% were not sufficiently familiar with Thought Field Therapy to render a rating and 37% were not familiar with Erhard Seminar Training.  These numbers may help us to understand one important and perhaps unappreciated reason for the relative apathy of many mental health experts toward discredited practices: many experts simply do not know much about them.
(Norcross et al, 2006.  Page 521)

Where, then, would the "experts" - being members of the same community as the authors of the two articles - look for information (and we know that at least some of them did look for information - see below), other than in the same places that the authors had looked?
And what results would they be likely to get?  Presumably, the same results that the authors got.

Yet just as the authors seem to have been unaware of the potential effect of using such pejorative wording in their survey, they also seem to have been totally unaware that they were potentially "priming the pump".  Far from being impressed by the negative results of the poll one can only wonder that there were some items which got off relatively lightly.

And on a similar note, weren't the authors somewhat ill-advised when they commented (Norcross et al, 2006.  Page 521) on the 37% of their colleagues' who didn't know about "est" (Erhard Seminar Training), given that the organization ceased operating over 20 years ago?  (It was bought out by the organisers of the Landmark Forum, now known as the Landmark Education Corporation (LEC).)

This leads us directly into yet another concern that is relevant to both surveys as far as any alleged connection with the authentic field of NLP is concerned.

What Makes an "Expert" Expert?

Norcross et al define the term "expert" as follows:

Expert in our study was defined by status as journal editor or association [i.e. APA] fellow, which predictably produced a disproportionate percentage of academics and cognitive-behavioral proponents.
(Norcross et al, 2006.  Page 520)

Well that's clear enough, isn't it?
No, actually it isn't.
On the contrary it appears to depend on a major, and highly dubious, assumption.

To be specific, the authors of the 2006 report initially contacted, via random selection (page 516):

• 100 fellows of the APA's Division 12 (Clinical Psychology)
• 45 fellows of the APA's Division 17 (Counseling Psychology)
• 23 fellows of the APA's Division 17 (School Psychology)
• 46 fellows of the APA "with a major field in clinical psychology"
,
• 57 "current and former editors of scholarly journals in mental health
• 14 "members of the APA Presidential Task Force on Evidence-Based Practice"
• 5 chairs or editors of the Diagnostic and Statistical Manual of Mental Disorders

The second "expert" panel (2010) potentially consisted of (page 8):

• 62 fellows of the American Society of Addiction Medicine
• 63 fellows of the APA's Division of Addictions
• 25 editorial board members of Addiction
• 25 editorial board members of Psychology of Addictive Behaviors
• 25 editorial board members of Journal of Studies on Alcohol
• 25 editorial board members of Journal of Substance Abuse Treatment
• 25 professional members of the advisory council of NIAAA (National Institute of Alcoholism and Alcohol Abuse) and NIDA (National Institute of Drug Abuse)

(In neither case are we given any indication as to how this information breaks down in Round 2.)

And here's the big question: On what basis can we safely assume that clinical psychologists, counseling psychologists, school psychologists or fellows of any other APA division or editorial board members are, per se, "experts" in any of the methodologies listed as "treatments" in either survey?

It is true that respondents could indicate that they were not familiar with any given methodology, but this is meaningless without a clear definition of either the methodologies or the word "familiar".  And even if such definitions had been supplied, how many of the respondents are likely to be genuinely "familiar", to any significant degree, with so many non-conventional treatments and/or tests?

In the 2006 article we are told that, according to the given answers, only 26.7% of the Round 1 respondents, and 24.1% of the Round 2 respondents, were "not familiar" with the use of "Neuro-Linguistic Programming (NLP) for treatment of mental/behavioral disorders".  But what about the other 75% (approx)?  They may have thought that they were genuinely familiar with this "treatment", but what evidence, other than notoriously unreliable self-reporting, do we have on this score?

In the 2010 article the results are even more significant - in Round 1, the 71% of respondents who acknowledged that they knew nothing about "Neuro-Linguistic Programming for drug and alcohol dependence" were the largest such group, with the 67% who were not familiar with "Metronidazole for alcohol dependence" coming second.  Likewise, in Round 2, though the figures had dropped to 59% and 51% respectively, the "NLP" and "Metronidazole" groups were still largest and second largest as far as "not familiar" was concerned.

To repeat an earlier observation: there is no definitive/standard "NLP treatment for mental and/or behavioral disorders" or for "drug and alcohol dependence".  Nor, as far as I can see from the two articles, do the authors themselves know what they mean by these labels.
So how likely is it that any of the respondents would be "familiar" with these undefined, non-existent treatments?  And how can we make any sense of the ratings of "possibly discredited" (2006, page 518) and "probably discredited" (2010, page 177) evoked by the two polls?

In practice, just to dot the "i's" and cross the "t's", we would do well to consider the contents of the FAQ #28 Project as they relate to academic psychologists.  At the latest count (December 2011 - 18 entries) 100% of those critics whose work has been reviewed have demonstrated that even though they may present themselves as being knowledgeable about "NLP", it is entirely possible, or even probable, that they know nothing of any consequence about the subject.  As we saw in the previous section, Norcross et al, who apparently imagine that "NLP" is a form of psychotherapy, etc., and who presumably (?) think they know what "NLP" is about, are themselves examples of this kind of seriously flawed "knowledge" in action.

Compliance Rules OK - 1

Way back in 1974, Professor Elizabeth Loftus co-authored an article which described an experiment designed to measure how far people's memory of a traffic accident might be influenced by the specific wording of questions about the event.

In that experiment subjects were shown short film clips of a two-car accident and asked a number of questions about what they had seen, including: "About how fast were the cars going when they hit each other?"
What the subjects didn't know was that there were actually five different versions of the questionnaire, identified by the wording of this one question, which in its various versions described the meeting of the two cars as "contacted", "hit", "bumped", collided" and "smashed".

When averaged out, the subjects who had been asked what speed the cars were doing when they "contacted" was 31.8 mph.  The other four groups gave increasingly faster estimates (in the order shown above) with "smashed" producing an average estimate of 40.8 mph.  In other words, the more "sensational" the description, the higher the estimated speed; the more neutral the description, the slower the perceived speed.

In a second, related experiment Loftus and Palmer showed that the variability of the answers is likely to be due to the question altering the subjects' actual memory of the event they had witnessed.
(This result might be counted as evidence in support of the NLP-related claim that our actual neurology is affected by the language we hear.)

So what is the likely effect of the constant emphasis, in these two polls, on the highly judgmental word "discredited", rather than some neutral word or phrase?  It appears in all of the rating definitions, and is the focus of the first two paragraphs of the instructions to the participants (according to the 2006 article).
No matter whether the use of such language was intentional or just careless, the likelihood of it biasing the poll results seems irrefutable.

And that is only part of the story.

Compliance Rules OK - 2

In the "Second Round" section of both articles we are told that:

the second round questionnaire presented the pooled responses from the first round, which is the standard procedure in Delphi polls.
(For example, Norcross et al, 2006.  Page 517)

This element of the procedure was supposedly included because:

In this manner, the panel of experts will exchange opinions and arrive at a greater consensus.  Your individual responses will not be identified.
(For example, Norcross et al, 2006.  Page 516)

Maybe I'm missing something here, but how does any "exchange [of] opinions" take place without the parties involved being able to engage in any kind of discussion?  In what sense, precisely, does being shown an anonymous set of statistics qualify as any kind of meaningful "exchange [of] opinions"?

To be honest, this seems more like a second form of subtly enforced compliance.

Given the heavy emphasis on the word "discredited", as described above, it is surely relevant to note what actually happened to the allegedly "expert" opinions in the two rounds (2006):

But the argument has another dimension, which again reduces the poll's results to meaningless data.
To be precise, in both polls the Round 1 and Round 2 results are treated as if they had come from the same sample.  But in both cases they obviously did not .
In 2006, Sample 1 (101 useable responses) includes sample 2 (85 responses), but sample 2 is 16 people/16% smaller than sample 1.
In 2010, Sample 1 consisted of 75 responses, and Round 2 consisted of only 57 responses, a drop of approximately 25%.

In must further be noted that in neither case are we given information about Round 2 "dropouts" which would allow us to determine what difference their absence made to the results.

The "Homework" Factor"

Another factor which differentiated Sample 1 from Sample 2, at least in the 2006 poll, was a change in participants' knowledge.  No, let me be more accurate - a change in some participants' knowledge.

Thus in the 2006 article we are told that:

... a large number of expert panelists informed us that completing the initial questionnaire prompted them to secure and read critical reviews of the treatments and tests on the questionnaire.
(Norcross et al, 2006.  Page 521.  Italics added for emphasis)

So how many of the "experts" did this?  And for how many of the treatments.  Which sources did they use?  And were their sources always accurate?

In the case of "NLP" we can pretty much guarantee that anyone searching academic sources would find the kind of material gathered by the FAQ #28 Project.  In other words, poorly-researched, denigratory misinformation.

Is this really the way to go about running a survey of such probity that the authors can say, of the second poll:

As a field, we have made progress in differentiating science from pseudoscience, credible from discredible [sic] in addictions treatment.
(Norcross et al, 2010.  Page 179)

Cautions and Caveats

If this assessment of the two polls seems unreasonably critical it should be noted that the authors of the polls also listed a number of errors they detected in their studies.

This first tranche of shortcomings is made up entirely of direct quotes from the 2006 article (page 519):

• Firstly, many of our experts lacked familiarity with the many "fringe" therapies or "unusual" assessment techniques.
   
• Second, some might challenge the relatively modest size of our panel of experts.
   
• Third, our reliance on traditionally trained and academically vetted experts with a disproportionate number of cognitive-behavioral therapists might be too narrow.
   
• Fourth, the robust rating differences due to theoretical orientation ... indicate that the epistemological commitments of the expert panel materially influence the results and thus the conclusions of what is discredited.
   
• Fifth, several panel members noted that a single item assessing the credibility of an omnibus assessment for a given purpose was insufficient.

And these are from the 2010 article (page 178):

Picking Your Friends

An interesting demonstration of the authors' lack of accurate knowledge (in both articles) appears when they present their readers with these statements:

Recently, several authors have attempted to identify pseudoscientific, invalidated, or "quack" psychotherapies ..."
(Norcross et al, 2006.  Page 515)

Most assuredly, select investigators have attempted to identify pseudoscientific or ineffective treatments applied to a variety of mental disorders and addictions ..
(Norcross et al, 2010.  Page 5)

Both versions are followed by a list that includes:

Carroll, 2003; Della Sala, 1999; Eisner, 2000; Lilienfeld, Lynn, and Lohr, 2003; Singer and Lalich, 1996.

It is interesting to note, then, that: