Scale Observer Agreement
Reasons for disagreement among the 41 studies for which the observer pairs disagreed on the standardized averages calculated The status of the information in the audit minutes is shown in Table 1⇓. None of the audit minutes contained any information on the criteria to be preferred. Three protocols indicated the date to be chosen and four indicated whether a change in baseline or values should be preferred after treatment. Nine described the type of control group to be selected, but none reported a hierarchy between control groups or similar intentions to combine these groups. The distribution of differences of opinion is presented in Figure 2⇓. Ten percent fully agreed, 21% disagreed under our intersection of 0.1, 38% disagreed between 0.1 and 0.49 and 28% disagreed with at least 0.50 (including 10% with disagreements of ≥1). The last 18 couples (4%) quantifiable, as one observer excluded all trials from two meta-analyses. The median disagreement was SMD-0.22 for the 432 quantifiable pairs with an interquartile range of 0.07 to 0.61. There was no difference between methodologists and PhD students (Table 2⇑). Conclusions Differences of opinion were frequent and often more important than the effect of commonly used treatments. Smd meta-analyses are subject to variations of observers and should be interpreted with caution.
The reliability of meta-analyses could be improved through more detailed verification protocols, more than one observer and statistical expertise. The goal of the MDS is to provide physicians and policy makers with the most reliable summary of available test results when the results have been measured on different continuous or numerical scales. Surprisingly, the procedure has not yet been the subject of a detailed review of its own reliability. Previous research was rare and focused on data extraction errors.2 4 5 In one study, the authors found errors in 20 of Cochrane`s 34 evaluations, but because they did not provide numerical data, it is not possible to estimate how often they were important.4 In a previous study involving 27 meta-analyses, it is not possible to estimate how often they were important.4 In a previous study involving 27 meta-analyses , 16 of which were Cochrane,2 evaluations, we were unable to replicate the MDS result for at least one of the two studies. that we had chosen for the audit. meta-analysis within our 0.1 intersection point in 10 meta-analyses. In trying to replicate these 10 meta-analyses, including all attempts, we found that seven of them were defective; One was then removed and a significant difference disappeared or appeared in two ends.2 This study complements the research conducted to date by also emphasizing the importance of different decisions in the selection of meta-analysis results. The results of our study are broader than for meta-analyses using MDS, as many of the reasons for disagreement are not related to the MDS method, but would also be important for data analysis with the weighted average difference method, which is the method of choice if the results were measured on the same scale. H.
J. A. Schouten. Statistical measure of the Interobserver agreement. Unpublished dissertation, Erasmus University Rotterdam. J. Cohen: Cohen. A coefficient of agreement for nominal scales. Educational and psychological measure, 20, 37-46. H. J. A.
Schouten. Measure in pairs of correspondence between many observers. Biometric Review, 22, 497-504. Design: For example, a second class of the ICF, d430 lifting and carrying objects, was used. Clinically useful definitions have been given to qualifiers in this category. The data was collected in the case of a cross-sectional survey with repeated measurements. We report on raw, specific and probability-corrected measures or agreements, a graphic method and the results of login models for oral agreements.