↓ Skip to main content

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Overview of attention for article published in BMC Bioinformatics, January 2017
Altmetric Badge

About this Attention Score

  • In the top 25% of all research outputs scored by Altmetric
  • High Attention Score compared to outputs of the same age (89th percentile)
  • High Attention Score compared to outputs of the same age and source (95th percentile)

Mentioned by

blogs
1 blog
twitter
3 X users
wikipedia
1 Wikipedia page

Citations

dimensions_citation
92 Dimensions

Readers on

mendeley
126 Mendeley
citeulike
1 CiteULike
You are seeing a free-to-access but limited selection of the activity Altmetric has collected about this research output. Click here to find out more.
Title
Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers
Published in
BMC Bioinformatics, January 2017
DOI 10.1186/s12859-017-1486-2
Pubmed ID
Authors

Rob Eisinga, Tom Heskes, Ben Pelzer, Manfred Te Grotenhuis

Abstract

The Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation. We propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets. We provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip .

X Demographics

X Demographics

The data shown below were collected from the profiles of 3 X users who shared this research output. Click here to find out more about how the information was compiled.
Mendeley readers

Mendeley readers

The data shown below were compiled from readership statistics for 126 Mendeley readers of this research output. Click here to see the associated Mendeley record.

Geographical breakdown

Country Count As %
Indonesia 1 <1%
United States 1 <1%
Portugal 1 <1%
Germany 1 <1%
Unknown 122 97%

Demographic breakdown

Readers by professional status Count As %
Researcher 16 13%
Student > Master 15 12%
Student > Ph. D. Student 14 11%
Student > Bachelor 11 9%
Student > Doctoral Student 7 6%
Other 31 25%
Unknown 32 25%
Readers by discipline Count As %
Engineering 17 13%
Computer Science 16 13%
Agricultural and Biological Sciences 12 10%
Medicine and Dentistry 6 5%
Biochemistry, Genetics and Molecular Biology 4 3%
Other 33 26%
Unknown 38 30%
Attention Score in Context

Attention Score in Context

This research output has an Altmetric Attention Score of 15. This is our high-level measure of the quality and quantity of online attention that it has received. This Attention Score, as well as the ranking and number of research outputs shown below, was calculated when the research output was last mentioned on 09 December 2018.
All research outputs
#2,014,945
of 22,950,943 outputs
Outputs from BMC Bioinformatics
#522
of 7,308 outputs
Outputs of similar age
#44,939
of 419,016 outputs
Outputs of similar age from BMC Bioinformatics
#7
of 144 outputs
Altmetric has tracked 22,950,943 research outputs across all sources so far. Compared to these this one has done particularly well and is in the 91st percentile: it's in the top 10% of all research outputs ever tracked by Altmetric.
So far Altmetric has tracked 7,308 research outputs from this source. They typically receive a little more attention than average, with a mean Attention Score of 5.4. This one has done particularly well, scoring higher than 92% of its peers.
Older research outputs will score higher simply because they've had more time to accumulate mentions. To account for age we can compare this Altmetric Attention Score to the 419,016 tracked outputs that were published within six weeks on either side of this one in any source. This one has done well, scoring higher than 89% of its contemporaries.
We're also able to compare this research output to 144 others from the same source and published within six weeks on either side of this one. This one has done particularly well, scoring higher than 95% of its contemporaries.