Comparing Algorithms w/ Small Samples

 

• do 5*2 cross validation
• each time train both algorithms and test them on each set - this gives 4 error estimates - ,
• use these measures in the augmented t test as follows
• is for algorithm and trained on and tested on
• subtract corresponding error estimates to get two estimated differences and likewise for
• the estimated variance is ( is 2 in this case)
• 
• do this for each cross validation so is the variance computed for the repetition
• 5x2cv statistic is:
• used 2-fold cv gives large test sets and disjoint training sets, but but training sets are only 1/2 the size which may cause problems
• 5 replications shown to produce lowest Type I error (incorrectly detecting a difference when no difference exists)