\name{calibration}
\alias{calibration}
\alias{calibration.formula}
\alias{calibration.default}
\alias{xyplot.calibration}
\alias{panel.calibration}
\title{
Probability Calibration Plot
}
\description{
For classification models, this function creates a 'calibration plot' that describes how consistent model probabilities are with observed event rates.
}
\usage{

calibration(x, ...)

\method{calibration}{formula}(x, data = NULL, 
            class = NULL, 
            cuts = 11, subset = TRUE, 
            lattice.options = NULL, ...)

\method{xyplot}{calibration}(x, data, ...)

panel.calibration(...)
}
\arguments{
  \item{x}{
a \code{lattice} formula (see \code{\link[lattice:xyplot]{xyplot}} for syntax) where the left-hand side of the formula is a factor class variable of the observed outcome and the right-hand side specifies one or model columns corresponding to a numeric ranking variable for a model (e.g. class probabilities). The classification variable should have two levels. 
}
  \item{data}{
For \code{calibration.formula}, a data frame (or more precisely, anything that is a valid \code{envir} argument in \code{eval}, e.g., a list or an environment) containing values for any variables in the formula, as well as \code{groups} and \code{subset} if applicable. If not found in \code{data}, or if \code{data} is unspecified, the variables are looked for in the environment of the formula. This argument is not used for \code{xyplot.calibration}.
}
  \item{class}{
a character string for the class of interest
}
  \item{cuts}{
the number of splits of the data are used to create the plot. By default, it uses as many cuts as there are rows in \code{data}
}
  \item{subset}{
An expression that evaluates to a logical or integer indexing vector. It is evaluated in \code{data}. Only the resulting rows of \code{data} are used for the plot. 
}
  \item{lattice.options}{
A list that could be supplied to \code{\link[lattice:lattice.options]{lattice.options}} 
}
  \item{\dots}{
options to pass through to \code{\link[lattice:xyplot]{xyplot}} or the panel function (not used in \code{calibration.formula}).
}
}
\details{
\code{calibration.formula} is used to process the data and \code{xyplot.calibration} is used to create the plot.

To construct the calibration plot, the following steps are used for each model:

\enumerate{
   \item The data are split into \code{cuts - 1} roughly equal groups by their class probabilities
   \item the number of samples with true results equal to \code{class} are determined
   \item the event rate is determined for each bin}
\code{xyplot.calibration} produces a plot of the observed event rate by the mid-point of the bins. 

This implementation uses the \pkg{lattice} function \code{\link[lattice:xyplot]{xyplot}}, so plot elements can be changed via panel functions, \code{\link[lattice:trellis.par.get]{trellis.par.set}} or other means. \code{calibration} uses the panel function \code{\link{panel.calibration}} by default, but it can be changed by passing that argument into \code{xyplot.calibration}.

The folowing elements are set by default in the plot but can be changed by passing new values into \code{xyplot.calibration}: \code{xlab = "Bin Midpoint"}, \code{ylab = "Observed Event Percentage"}, \code{type = "o"}, \code{ylim = extendrange(c(0, 100))},\code{xlim = extendrange(c(0, 100))} and \code{panel = panel.calibration}

}
\value{
\code{calibration.formula} returns a list with elements:
  \item{data}{the data used for plotting}
  \item{cuts}{the number of cuts}
  \item{class}{the event class}
  \item{probNames}{the names of the model probabilities}

\code{xyplot.calibration} returns a \pkg{lattice} object
}

\author{
Max Kuhn, some \pkg{lattice} code and documentation by Deepayan Sarkar
}

\seealso{
\code{\link[lattice:xyplot]{xyplot}}, \code{\link[lattice:trellis.par.get]{trellis.par.set}}} 

\examples{
\dontrun{
data(mdrr)
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .5)]


inTrain <- createDataPartition(mdrrClass)
trainX <- mdrrDescr[inTrain[[1]], ]
trainY <- mdrrClass[inTrain[[1]]]
testX <- mdrrDescr[-inTrain[[1]], ]
testY <- mdrrClass[-inTrain[[1]]]

library(MASS)

ldaFit <- lda(trainX, trainY)
qdaFit <- qda(trainX, trainY)

testProbs <- data.frame(obs = testY,
                        lda = predict(ldaFit, testX)$posterior[,1],
                        qda = predict(qdaFit, testX)$posterior[,1])

calibration(obs ~ lda + qda, data = testProbs)

calPlotData <- calibration(obs ~ lda + qda, data = testProbs)
calPlotData

xyplot(calPlotData, auto.key = list(columns = 2))
}
}
\keyword{hplot}