\name{resamples}
\alias{resamples.default}
\alias{resamples}
\alias{summary.resamples}
\alias{sort.resamples}
\alias{modelCor}
\title{
Collation and Visualization of Resampling Results
}
\description{
These functions provide methods for collection, analyzing and visualizing a set
of resampling results from a common data set.
}
\usage{
resamples(x, ...)

\method{resamples}{default}(x, modelNames = names(x), ...)

\method{summary}{resamples}(object, ...)

\method{sort}{resamples}(x, decreasing = FALSE, metric = x$metric[1], FUN = mean, ...)

modelCor(x, metric = x$metric[1], ...)
}
\arguments{
  \item{x}{a list of two or more objects of class \code{\link{train}}, \code{\link{sbf}} or \code{\link{rfe}}  with a common set of resampling indices in the \code{control} object. For \code{sort.resamples}, it is an object generated by \code{resamples}.}
  \item{modelNames}{an optional set of names to give to the resampling results}
  \item{object}{an object generated by \code{resamples}}
  \item{metric}{a character string for the performance measure used to sort or computing the between-model correlations}
  \item{decreasing}{logical. Should the sort be increasing or decreasing? }
  \item{FUN}{a function whose first argument is a vector and returns a scalar, to be applied to each model's performance measure.}
  \item{\dots}{only used for \code{sort} and \code{modelCor} and captures arguments to pass to \code{sort} or \code{FUN}.}
}
\details{
The ideas and methods here are based on Hothorn et al (2005) and Eugster et al (2008).

The results from  \code{\link{train}} can have more than one performance metric per resample. Each metric in the input object is saved.

\code{resamples} checks that the resampling results match; that is, the indices in the object \code{trainObject$control$index} are the same. Also, the argument \code{\link{trainControl}} \code{returnResamp} should have a value of \code{"final"} for each model.

The summary function computes summary statistics across each model/metric combination.
 
}
\value{
For \code{resamples}: an object with class \code{"resamples"} with elements
  \item{call }{the call}
  \item{values }{a data frame of results where rows correspond to resampled data sets and columns indicate the model and metric}
  \item{models }{a character string of model labels}
  \item{metrics }{a character string of performance metrics}
  \item{methods }{a character string of the \code{\link{train}} \code{method} argument values for each model }
For \code{sort.resamples} a character string in the sorted order is generated. \code{modelCor} returns a correlation matrix.
}
\author{
Max Kuhn
}

\seealso{
\code{\link{train}},  \code{\link{trainControl}}, \code{\link{diff.resamples}}, \code{\link{xyplot.resamples}}, \code{\link{densityplot.resamples}}, \code{\link{bwplot.resamples}}, \code{\link{splom.resamples}}}

\references{Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30}

\examples{

data(BloodBrain)
set.seed(1)

## tmp <- createDataPartition(logBBB,
##                            p = .8,
##                            times = 100)

## rpartFit <- train(bbbDescr, logBBB,
##                   "rpart", 
##                   tuneLength = 16,
##                   trControl = trainControl(
##                     method = "LGOCV", index = tmp))

## ctreeFit <- train(bbbDescr, logBBB,
##                   "ctree", 
##                   trControl = trainControl(
##                     method = "LGOCV", index = tmp))

## earthFit <- train(bbbDescr, logBBB,
##                   "earth",
##                   tuneLength = 20,
##                   trControl = trainControl(
##                     method = "LGOCV", index = tmp))

## or load pre-calculated results using:
## load(url("http://caret.r-forge.r-project.org/exampleModels.RData"))

## resamps <- resamples(list(CART = rpartFit,
##                           CondInfTree = ctreeFit,
##                           MARS = earthFit))

## resamps
## summary(resamps)
}

\keyword{ models }
