Article #13
Newsletter list 


Previous Article 

Next Article 


Evaluation of Expertise

James Shanteau
Manhattan, KS

With support received from the FAA, our project team (see below) has successfully developed and tested a new approach to the evaluation of expertise where there is no independent criterion.

Labeled the Cochran-Weiss-Shanteau (CWS) measure, the index provides a high degree of predictive accuracy in the identification of expertise. To date, CWS has been successful in a variety of contexts, including research of experts in auditing, livestock judging, and personnel selection. In addition, CWS has been applied to data collected by FAA researchers in low-fidelity simulations of air traffic control (ATC). Indeed, we have yet to find a study of expertise where CWS has not done well in identifying who is, and who is not, an expert. (Of course, we have only examined a small fraction of the total number of studies done on experts. As more studies become available to us for reanalysis, we will continue to apply our approach retroactively.)

Our ongoing research both extends the work-to-date and expands the applications of CWS into new domains. Specifically, the purposes of our present efforts are four-fold. First, we plan to apply the index to the results of high-level simulations involving experts. This will allow us to "ramp up" the application of the index to more realistic settings using dynamic, real-time task environments.

Second, we will continue to develop the conceptual and theoretical underpinnings behind the CWS approach. While the index has worked remarkably well at the empirical level, there is much to be done to explain how and why the measure works as well as it does. In addition, it is necessary to explore both alternative definitions and boundary conditions for the approach.

Third, we will be conducting a wide-ranging set of empirical studies using both laboratory and field studies of experts. These studies will allow us to better understand the advantages and limitations of applying CWS to real-time data. For instance, we are using CWS to track the development of skills from naive to novice to expert. This will allow us to examine the usefulness of our approach for training and selection of experts.

Finally, a postdoctoral associate will be analyzing CWS using simulation methods. Through these methods, we will be able to examine in detail the conditions under which CWS works well and works poorly. The simulations will also allow for the evaluation of CWS under conditions that cannot be studied empirically, for example, unfriendly or hostile environments.

The CWS research team consists of James Shanteau, Rick Thomas, and Jack Windhorst at Kansas State University, David Weiss at California State University in Los Angeles, Ward Edwards at Wise Decisions in Los Angeles, and Julia Pounds at the Federal Aviation Administration in Oklahoma City.

Contact James Shanteau

Previous Article                Next Article

Newsletter list