Lindell, M. K. (2001). Assess and test an interdisciplinary agreement on a single objective using evaluation scales with several individual values. Appl. Psychol. Meas. 25, 89-99. doi: 10.1177/01466216010251007 Meade, A. W., and Eby, L.
T. (2007). Use group agreement cues in validating construction in several stages. organ. Res. Methoden 10, 75-96. doi: 10.1177/1094428106289390 Cohen, A., Doveh, E., and Nahum-Shani, I. (2009). Test agreement for multi-item scales with rwg (j) and ADm (j) indices. organ. Res. Methods 12, 148-164.
doi: 10.1177/1094428107300365 Lindell, M. K., and Brandt, C. J. (1999). Evaluation of the interdisciplinary agreement on the importance of test commands: a comparison of the CVI, T, rwg and rwg (j) indices. J. Appl. Psychol. 84, 640-647. doi: 10.1037/0021-9010.84.4.640 Bliese, P.
D., and Halverson, R. (2002). Using random re-stamping groups in multi-step search. Leadersh. Q. 13, 53-68. doi: 10.1016/S1048-9843 (01) 00104-7 Brown, R. D., and Hauenstein, N.M A.
(2005). The Interrater agreement is envisaged: an alternative to rwg indices. organ. Res. Methods 8, 165-184. doi: 10.1177/1094428105275376 Biemann, T., Cole, M., and Voelpel, p. (2012). Within the group: on the use (and abuse) of rWG and rWG (J) in leadership research and some guidelines for good practice.
Leadersh. Q. 23, 66-80. doi: 10.1016/j.leaqua.2011.11.006 Harvey, R. J., and Hollander, E. (2004, April). « Benchmarking rWG interrater agreement indices: let`s drop the.70 rule-of-thumb, » in Paper Presented at the Meeting of the Society for Industrial Organizational Psychology (Chicago, IL). Newman, D. A., and Sin, H.-P. 2009. How does the lack of data bias estimates of group agreements? The sensitivity of SDwg, CVwg, rwg (j), CCI and CCI to systematic non-response.
organ. Res. Methods 12, 113-147. doi: 10.1177/109442810629896999 For data aggregation at the individual level, most researchers applied direct consensus composition or reference-layer-consensus models when they aggregated data at the individual level to a higher level of analysis. The composition of the consensus assumes that there is sufficient consensus within the group regarding the management structure of interest; in the absence of agreement, the entire governance structure is unsustainable. At the same time, guidelines to help leadership researchers make decisions about data aggregation have received little attention. In particular, a discussion of how data aggregation decisions can improve or conceal the theoretical contribution of a study – a central priority of this article – has not been addressed in depth. Recognizing that empirical generalization depends on the accuracy with which aggregation decisions are applied, we examine the often overlooked assumptions associated with the most common consensus statistic used to justify data aggregation – rWG and rWG (J) (James, Demaree and Wolf, 1984). Based on a dataset published as part of a « Leadership Quarterly Special Issue » (Bliese, Halverson, Schriesheim, 2002), we highlight the potential abuse of rWG and rWG (J) as the only statistic justifying aggregation at a higher level of analysis. We conclude with prescriptive implications for promoting consistency in the way leadership research is conducted and reported at multiple levels.
Lindell, M. K., and Brandt, C. J. (1997). Measure of interdisciplinary agreement for the evaluation of a single objective. Appl. Psychol. Meas. 21, 271-278. doi: 10.1177/01466216970213006 George, J.M. (1990). Personality, affect and group behaviour.
J. Appl. Psychol. 86, 1075-1082. doi: 10.1037/0021-9010.75.2.107 Burke, M. J., and Dunlap, W. P. (2002). Estimate of the Interrater agreement with the average variation index: a user manual. organ. Res. Methods 5, 159-172.
doi: 10.1177/1094428102005002 Keywords: Interrater agreement, rwg, multilevel, data aggregation, group agreement, reliability James, L. R., Demaree, R.G., and Wolf, G. (1984). Assess reliability within the interrater group, with or without a reaction distortion. J. Appl. Psychol. 69, 85-98. doi: 10.1037/0021-9010.69.1.85 George, J.M. and James, L. R.
(1993). Personality, affect and behaviour in revisited groups: Comment on aggregation, analysis levels and a common application of analysis and analysis