% \VignetteIndexEntry{caret Manual -- Model Building}
% \VignetteDepends{caret}
% \VignettePackage{caret}
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage[pdftex]{graphicx}
\usepackage{color}
\usepackage{xspace}
\usepackage{fancyvrb}
\usepackage{fancyhdr}
\usepackage{lastpage}
\usepackage{longtable} 
\usepackage{algorithm2e}
\usepackage[
         colorlinks=true,
         linkcolor=blue,
         citecolor=blue,
         urlcolor=blue]
         {hyperref}
         \usepackage{lscape}

\usepackage{Sweave}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% define new colors for use
\definecolor{darkgreen}{rgb}{0,0.6,0}
\definecolor{darkred}{rgb}{0.6,0.0,0}
\definecolor{lightbrown}{rgb}{1,0.9,0.8}
\definecolor{brown}{rgb}{0.6,0.3,0.3}
\definecolor{darkblue}{rgb}{0,0,0.8}
\definecolor{darkmagenta}{rgb}{0.5,0,0.5}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand{\bld}[1]{\mbox{\boldmath $#1$}}
\newcommand{\shell}[1]{\mbox{$#1$}}
\renewcommand{\vec}[1]{\mbox{\bf {#1}}}

\newcommand{\ReallySmallSpacing}{\renewcommand{\baselinestretch}{.6}\Large\normalsize}
\newcommand{\SmallSpacing}{\renewcommand{\baselinestretch}{1.1}\Large\normalsize}

\newcommand{\halfs}{\frac{1}{2}}

\setlength{\oddsidemargin}{-.25 truein}
\setlength{\evensidemargin}{0truein}
\setlength{\topmargin}{-0.2truein}
\setlength{\textwidth}{7 truein}
\setlength{\textheight}{8.5 truein}
\setlength{\parindent}{0.20truein}
\setlength{\parskip}{0.10truein}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\pagestyle{fancy}
\lhead{}
\chead{The {\tt caret} Package}
\rhead{}
\lfoot{}
\cfoot{}
\rfoot{\thepage\ of \pageref{LastPage}}
\renewcommand{\headrulewidth}{1pt}
\renewcommand{\footrulewidth}{1pt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{The {\tt caret} Package}
\author{Max Kuhn \\ max.kuhn@pfizer.com}


\begin{document}

\maketitle

\thispagestyle{empty}
	
<<loadLibs, results = hide, echo = FALSE>>=
library(caret)
library(kernlab)
library(gbm)
library(ipred)
library(grid)
library(randomForest)
data(BloodBrain)
data(mdrr)

@

\section{Model Training and Parameter Tuning}\label{S:train}

\texttt{caret} has several functions that attempt to streamline the model building and evaluation process. 

The \texttt{train} function can be used to
\begin{itemize}
   \item evaluate, using resampling, the effect of model tuning parameters on performance
   \item choose the ``optimal'' model across these parameters 
   \item estimate model performance from a training set
\end{itemize}

To optimize tuning parameters of models, \texttt{train} can be used to fit many predictive models over a grid of parameters and return the ``best'' model (based on resampling statistics). See Table \ref{T:methods} for the models currently available.

As an example, the multidrug resistance reversal (MDRR) agent data is used to determine a predictive model for the ``ability of a compound to reverse a leukemia cell's resistance to adriamycin'' (\href{http://pubs.acs.org/cgi-bin/abstract.cgi/jcisd8/2005/45/i03/abs/ci0500379.html}{Svetnik et al, 2003}). For each sample (i.e. compound), predictors are calculated that reflect characteristics of the molecular structure. These molecular descriptors are then used to predict assay results that reflect resistance. 

The data are accessed using \texttt{data(mdrr)}. This creates a data frame of predictors called \texttt{mdrrDescr} and a factor vector with the observed class called \texttt{mdrrClass}.

To start, we will:
 
\begin{itemize}
   \item use unsupervised filters to remove predictors with unattractive characteristics (e.g. spare distributions or high inter--predictor correlations)
   \item split the entire data set into a training and test set
   \item center and scale the training and test set using the predictor means and standard deviations from the training set
\end{itemize}

See the package vignette ``caret Manual -- Data and Functions'' for more details about these operations.

\begin{small}
<<preProc>>=
print(ncol(mdrrDescr))
nzv <- nearZeroVar(mdrrDescr)
filteredDescr <- mdrrDescr[, -nzv]

print(ncol(filteredDescr))

descrCor <- cor(filteredDescr)
highlyCorDescr <- findCorrelation(descrCor, cutoff = .75)
filteredDescr <- filteredDescr[,-highlyCorDescr]

print(ncol(filteredDescr))

set.seed(1)
inTrain <- sample(seq(along = mdrrClass), length(mdrrClass)/2)

trainDescr <- filteredDescr[inTrain,]
testDescr <- filteredDescr[-inTrain,]
trainMDRR <- mdrrClass[inTrain]
testMDRR <- mdrrClass[-inTrain]

print(length(trainMDRR))
print(length(testMDRR))

preProcValues <- preProcess(trainDescr)

trainDescr <- predict(preProcValues, trainDescr)
testDescr <-  predict(preProcValues, testDescr)
@
\end{small}

To estimate model performance across the tuning parameters ``leave group out cross--validation'' (\texttt{LGOCV}) can be used. This technique is repeated splitting of the data into training and test sets (without replacement). If the resampling method is not specified, simple bootstrapping is used. To train a support vector machine classification model (radial basis function kernel) on these multidrug resistance reversal agent data, we can first setup a control object\footnote{This is optional; to use the default specifications, the control object does not need to be specified} that specifies the type of resampling used, the number of data splits (30), the proportion of data in the sub--training sets (75$\%$) and whether the iterations should be printed as they occur. In this case, we need to specify the proportion of samples used in each resampled training set. We also set the seed.

\begin{small}
<<setup>>=
fitControl <- trainControl(
                           method = "LGOCV",
                           p = .75,
                           number = 30,
                           returnResamp = "all",
                           verboseIter = FALSE)
set.seed(2)
@
\end{small}


The first two arguments to \texttt{train} are the predictor and outcome data objects, respectively. The third argument, \texttt{method}, specifies the type of model. For this model, the tuning parameters are the cost value (the \texttt{C} argument in \texttt{kernlab}'s \texttt{ksvm} function) and the radius of the RBF (the \texttt{sigma} argument to the kernel function). The \texttt{tuneLength} argument sets the size of the grid used to search the tuning parameter space and \texttt{trControl} is the control parameter for the \texttt{train} function. 

\begin{small}
<<mdrrModel1>>=
svmFit <- train(trainDescr, trainMDRR, method = "svmRadial", tuneLength = 4, trControl = fitControl)
svmFit
@
\end{small}


There are two tuning parameters for this model: \texttt{sigma} is a parameter for the kernel function that can be used to expand/contract the distance function and \texttt{C} is the cost parameter that can be used as a regularization term that controls the complexity of the model. For this model, the function \texttt{sigest} in the \texttt{kernlab} package is used to provide a good estimate of the \texttt{sigma} parameter, so that only the cost parameter is tuned. This tuning scheme is the default, but can be modified (details are below). 

The column labeled ``\texttt{Accuracy}'' is the overall agreement rate averaged over cross--validation iterations. The agreement standard deviation is also calculated from the cross-validation results. The column ``\texttt{Kappa}'' is Cohen's (unweighted) Kappa statistic  averaged across the resampling results


For regression models (i.e. a numeric outcome), a similar table would be produced showing the average root mean squared error and average $R^2$ value statistic across tuning parameters, otherwise known as $Q^2$ (see the note below related to this calculation).

\texttt{caret} works with specific models (see Table \ref{T:methods}). For these models, \texttt{train} can automatically create a grid of tuning parameters. By default, if $p$ is the number of tuning parameters, the grid size is $3^p$. For example, regularized discriminant analysis (RDA) models have two  parameters (\texttt{gamma} and \texttt{lambda}), both of which lie on $[0, 1]$. The default training grid would produce nine combinations in this two--dimensional space.

Alternatively, the grid can be specified by the user. The argument \texttt{tuneGrid} can take a data frame with columns for each tuning parameter (see Table \ref{T:methods} for specific details). The column names should be the same as the fitting function's arguments with a period preceding the name. For our RDA example, the names would be \texttt{.gamma} and \texttt{.lambda}. \texttt{train} will tune the model over each combination of values in the rows.

For a gradient boosting machine (GBM) model, the amount of ``shrinkage'' in a gradient boosting model is fixed at 0.1 and the other meta--parameters can be manually specified:
<<setPageWidth, echo = FALSE, results = hide>>=
currentWidth <- options("width")$width
options(width = 100)
@
\begin{small}
\begin{landscape}

<<mdrrModel2>>=
gbmGrid <-  expand.grid(.interaction.depth = c(1, 3), .n.trees = c(100, 300, 500), .shrinkage = 0.1)
set.seed(3)
gbmFit <- train(trainDescr, trainMDRR, "gbm", tuneGrid = gbmGrid, trControl = fitControl, verbose = FALSE)
gbmFit
@

\end{landscape}
\end{small}

<<resetPageWidth, echo = FALSE, results = hide>>=
options(width = currentWidth)
@

Some notes about the use of \texttt{train}:
\begin{itemize}
  
  \item There is a formula interface (e.g. \texttt{train(y~., data = someData)} that can be used. One of the issues with a large number of predictors is that the objects related to the formula which are saved can get very large. In these cases, it is best to stick with the non--formula interface described above.
	\item The function determines the type of problem (classification or regression) from the type of the response given in the \texttt{y} argument. 
	
	\item The \texttt{$\ldots$} option can be used to pass parameters to the fitting function. For example, in random forest models, you can specify the number of trees to be used in the call to \texttt{train}. In the example above, the default trace for a \texttt{gbm} model was turned off using the \texttt{verbose} argument to \texttt{gbm}.
	
	\item For regression models, the classical $R^2$ statistic cannot be compared between models that contain an intercept and models that do not. Also, some models do not have an intercept only null model.	
          
          \item [] To approximate this statistic across different types of models,  the square of the correlation between the observed and predicted outcomes is used. This means that the $R^2$ values produced by \texttt{train} will not match the results of \texttt{lm} and other functions. 
            
            \item [] Also, the correlation estimate does not take into account the degrees of freedom in a model and thus does not penalize models with more parameters. For some models (e.g random forests or on--linear support vector machines) there is no clear sense of the degrees of freedom, so this information cannot be used in $R^2$ if we would like to compare different models.
          
	
	\item The nearest shrunken centroid model of \href{http://projecteuclid.org/handle/euclid.ss/1056397488}{Tibshirani et al (2003)} is specified using \texttt{method = "pam"}. For this model, there must be at least two samples in each class. \texttt{train} will ignore classes where there are less than two samples per class from every model fit during bootstrapping or cross--validation (this model only).
	
	\item For recursive partitioning models, an initial model is fit to all of the training data to obtain the possible values of the maximum depth of any node (\texttt{maxdepth}). The tuning grid is created based on these values. If \texttt{tuneLength} is larger than the number of possible \texttt{maxdepth} values determined by the initial model, the grid will be truncated to the  \texttt{maxdepth} list.
	
	\item [] The same is also true for nearest shrunken centroid models, where an initial model is fit to find the range of possible threshold values, and MARS models (see the details below).
	
  \item For multivariate adaptive regression splines (MARS), the \texttt{earth} package is used with a model type of \texttt{mars} or \texttt{earth} is requested. The tuning parameters used by \texttt{train} are \texttt{degree} and \texttt{nprune}. The parameter \texttt{nk} is not automatically specified and, if not specified, the default in the \texttt{earth} function is used. 

  \item [] For example, suppose a training set with 40 predictors is used with \texttt{degree = 1} and \texttt{nprune = 20}. An initial model with \texttt{nk = 41} is fit and is pruned down to 20 terms. This number includes the intercept and may include ``singleton'' terms instead of pairs. 

  \item [] Alternate model training schemes can be used by passing \texttt{nk} and/or \texttt{pmethod} to the \texttt{earth} function.

  \item [] Also, there may be cases where the message such as ``specified 'nprune' 29 is greater than the number of available model terms 24, forcing 'nprune' to 24'' show up after the model fit. This can occur since the \texttt{earth} function may not actually use the number of terms in the initial model as specified by \texttt{nk}. This may be because the \texttt{earth} function removes terms with linear dependencies and  the forward pass counts as if terms were added in pairs (although singleton terms may be used). By default, the \texttt{train} function fits and initial MARS model is used to determine the number of possible terms in the training set to create the tuning grid. Resampled data sets may produce slightly different models that do not have as many possible values of  \texttt{nprune}.
	
   \item For the \texttt{glmboost} and \texttt{gamboost} functions from the \texttt{mboost} package, an additional tuning parameter, \texttt{prune}, is used by train. If \texttt{prune = "yes"}, the number of trees is reduced based on the AIC statistic. If \texttt{"no"}, the number of trees is kept at the value specified by the \texttt{mstop} parameter. See the \texttt{mboost} package vignette for more details about AIC pruning.	

   \item For some models (\texttt{pls}, \texttt{plsda}, \texttt{earth}, \texttt{rpart}, \texttt{gbm}, \texttt{gamboost}, \texttt{glmboost}, \texttt{blackboost}, \texttt{ctree}, \texttt{pam}, \texttt{superpc}, \texttt{enet} and \texttt{lasso}), the \texttt{train} function will fit a model that can be used to derive predictions for some sub-models. For example, for MARS (via the \texttt{earth} function), for a fixed degree, a model with a maximum number of terms will be fit and the predictions of all of the requested models with the same degree and smaller number of terms will be computed using \texttt{update.earth} instead of fitting a new model. When the \texttt{verboseIter} option is used, a line is printed for the ``top--level'' model (instead of each model in the tuning grid).

	\item There are \texttt{print} and \texttt{plot} methods. See Figures \ref{f:plots1} and \ref{f:plots2} for examples. This is also a function, \texttt{resampleHist}, that will plot a histogram or density plot of the resampled performance estimates for the optimal model. Figure \ref{f:plots2} shows and example of this type of plot for the support vector machine example.
	
	\item Using the first set of tuning parameters that are optimal (in the sense of accuracy or mean squared error), \texttt{train} automatically fits a model with these parameters to the entire training data set. That model object is accessible in the \texttt{finalModel} object within \texttt{train}. For example, \verb+gbmFit$finalModel+  is the same object that would have been produced using a direct call to the \texttt{gbm} function. 
	
\end{itemize}

There is additional functionality in \texttt{train} that is described in the next section.




\begin{longtable}{lllll}
\caption{Models used in \texttt{train}} \\ \label{T:methods}
{\bf Model} & {\bf \texttt{method} Value} & {\bf Package} & {\bf Tuning Parameters}\\
\hline \\
\endhead
\\
\multicolumn{5}{l}{{{\small \em (continued on next page)}}} \\
\endfoot
\hline
\endlastfoot
\multicolumn{5}{c}{{{ \em ``Dual--Use Models''}}} \\ 

      Generalized linear model &
         \texttt{glm} & 
            \texttt{stats}       & 
            None \\  
       
      Recursive Partitioning &
         \texttt{rpart} & 
            \texttt{rpart}       & 
            \texttt{maxdepth} \\  
       &
         \texttt{ctree} & 
            \texttt{party}       & 
            \texttt{mincriterion} \\   
       &
         \texttt{ctree2} & 
            \texttt{party}       & 
            \texttt{maxdepth} \\   
            
      Boosted Trees &
         \texttt{gbm} & 
            \texttt{gbm}       & 
            \texttt{interaction.depth}, \\
      & & & \texttt{n.trees}, \texttt{shrinkage}  \\

       &
         \texttt{blackboost} & 
            \texttt{gbm}       & 
            \texttt{maxdepth}, \texttt{mstop}\\
            
       &
         \texttt{ada} & 
            \texttt{ada}       & 
            \texttt{maxdepth}, \texttt{iter}, \texttt{nu}\\            
     
      Other Boosted Models &
         \texttt{glmboost} & 
            \texttt{mboost}       &          
            \texttt{mstop}\\   
      &      
         \texttt{gamboost} & 
            \texttt{mboost}       &          
            \texttt{mstop}\\      
                               
      Random Forests & 
         \texttt{rf} & 
            \texttt{randomForest}       & 
            \texttt{mtry} \\
                        
       & 
         \texttt{cforest} & 
            \texttt{party}       & 
            \texttt{mtry} \\     
                        
      Bagged Trees &
         \texttt{treebag} & 
            \texttt{ipred}       & 
            None \\                         
   
      Multivariate Adaptive  &
         \texttt{earth}, \texttt{mars} & 
            \texttt{earth}       & 
            \texttt{degree}, \texttt{nprune} \\            
      \:\: Regression Splines & & &\\
            
      Bagged MARS &
         \texttt{bagEarth} & 
            \texttt{caret},  \texttt{earth}      & 
            \texttt{degree}, \texttt{nprune} \\ 
            
      Elastic Net (glm) &               
         \texttt{glmnet} & 
            \texttt{glmnet}       &       
            \texttt{alpha}, \texttt{lambda} \\ 
            
      Neural Networks &               
         \texttt{nnet} & 
            \texttt{nnet}       &       
            \texttt{decay}, \texttt{size} \\      

       &               
         \texttt{pcaNNet} & 
            \texttt{pcaNNet}       &       
            \texttt{decay}, \texttt{size} \\              
            
      Partial Least Squares &
         \texttt{pls} & 
            \texttt{pls}, \texttt{caret}       & 
            \texttt{ncomp} \\      

      Sparse Partial Least Squares &
         \texttt{spls} & 
            \texttt{spls}, \texttt{caret}      & 
            \texttt{K}, \texttt{eta}, \texttt{kappa} \\   
        
      Support Vector Machines  &
         \texttt{svmLinear} & 
            \texttt{kernlab}       & 
            none \\              
      
  &
         \texttt{svmRadial} & 
            \texttt{kernlab}       & 
            \texttt{sigma}, \texttt{C} \\             
      
  &
         \texttt{svmPoly} & 
            \texttt{kernlab}       & 
            \texttt{scale}, \texttt{degree}, \texttt{C} \\            
      
     Gaussian Processes  &
         \texttt{gaussprLinear} & 
            \texttt{kernlab}       & 
            none \\             
      
  &
         \texttt{gaussprRadial} & 
            \texttt{kernlab}       & 
            \texttt{sigma} \\              
      
  &
         \texttt{gaussprPoly} & 
            \texttt{kernlab}       & 
            \texttt{scale}, \texttt{degree} \\            

                      
      $k$ Nearest Neighbors &
         \texttt{knn} & 
            \texttt{caret}       & 
            \texttt{k} \\   
      
      \\
\multicolumn{5}{c}{{{ \em Regression Only Models}}} \\      
    
      Linear Least Squares &
         \texttt{lm} & 
            \texttt{stats}       & 
            None \\  

       Principal Component Regression &
         \texttt{pcr} & 
            \texttt{pls}       & 
            \texttt{ncomp}  \\  
           
      Robust Linear Regression &
         \texttt{rlm} & 
            \texttt{MASS}       & 
            None \\              
                   
      Rule--Based Models &
         \texttt{M5Rules} & 
            \texttt{RWeka}    & 
            \texttt{pruned}  \\             

     Least Angle   Regression&
         \texttt{lars} & 
            \texttt{lars}       & 
            \texttt{fraction} \\   
         &
         \texttt{lars2} & 
            \texttt{lars}       & 
            \texttt{step} \\            
                
      Elastic Net &
         \texttt{enet} & 
            \texttt{elasticnet}      & 
            \texttt{lambda}, \texttt{fraction} \\          

      The Lasso &
         \texttt{lasso} & 
            \texttt{elasticnet}      & 
            \texttt{fraction} \\       

      Projection Pursuit   &
         \texttt{ppr} & 
            \texttt{stats}       & 
            \texttt{nterms} \\            
      \:\: Regression & & &\\                   
            
      Penalized Linear Models  &
         \texttt{penalized} & 
            \texttt{penalized}       & 
            \texttt{lambda1}, \texttt{lambda2} \\            
      \:\: Regression Splines & & &\\
      
      Relevance Vector Machines  &
         \texttt{rvmLinear} & 
            \texttt{kernlab}       & 
            none \\            
      
  &
         \texttt{rvmRadial} & 
            \texttt{kernlab}       & 
            \texttt{sigma} \\              
      
  &
         \texttt{rvmPoly} & 
            \texttt{kernlab}       & 
            \texttt{scale}, \texttt{degree} \\                        
            
      Supervised Principal &
         \texttt{superpc}  &
         \texttt{superpc}  &
         \texttt{n.components}, 
         \texttt{threshold} \\
      \:\: Components  & & &\\
      \\       
             
\multicolumn{5}{c}{{{\em Classification Only Models}}} \\ 
      Linear Discriminant Analysis &
         \texttt{lda} & 
            \texttt{MASS}       &          
            None\\
            
      Quadratic Discriminant  &
         \texttt{qda} & 
            \texttt{MASS}       &          
            None\\  
      \:\:  Analysis & & &\\               
            
      Stabilized Linear &
         \texttt{slda} & 
            \texttt{ipred}       &          
            \texttt{diagonal}\\  
      \:\: Discriminant Analysis & & &\\            
            
      Shrinkage Linear &
         \texttt{sda} & 
            \texttt{sda}       &          
            \texttt{diagonal}\\  
      \:\: Discriminant Analysis & & &\\
      
      Sparse Linear &
         \texttt{sparseLDA} & 
            \texttt{sparseLDA}       &          
            \texttt{NumVars}, \texttt{lambda} \\  
      \:\: Discriminant Analysis & & &\\      
      
      Stepwise Diagonal &
        \texttt{sddaLDA}, &
        \texttt{SDDA} &
        None \\
      \:\: Discriminant Analysis & \texttt{sddaQDA} & &\\
            
      Regularized Discriminant  &
         \texttt{rda} & 
            \texttt{klaR}       & 
            \texttt{lambda}, \texttt{gamma} & \\
      \: \: Analysis \\
      
      Mixture Discriminant  &
         \texttt{mda} & 
            \texttt{mda}       & 
            \texttt{subclasses} & \\
      \: \: Analysis \\      

      Sparse Mixture &
         \texttt{smda} & 
            \texttt{sparseLDA}       &          
            \texttt{NumVars}, \texttt{R}, \texttt{lambda} \\  
      \:\: Discriminant Analysis & & &\\    
      
      Penalized Discriminant  &
         \texttt{pda} & 
            \texttt{mda}       & 
            \texttt{lambda} & \\
      
     \: \: Analysis &
         \texttt{pda2} & 
            \texttt{mda}       & 
            \texttt{df} & \\     
      
      Flexible Discriminant  &
         \texttt{fda} & 
            \texttt{mda}, \texttt{earth}       & 
            \texttt{degree}, \texttt{nprune} & \\
      \: \: Analysis (MARS basis)\\      
      
      Bagged FDA &
         \texttt{bagFDA} & 
            \texttt{caret},  \texttt{earth}       & 
            \texttt{degree}, \texttt{nprune} \\   
            
      Logistic/Multinomial  &
         \texttt{multinom} & 
            \texttt{nnet}       & 
            \texttt{decay}& \\   
      \: \: Regression \\ 
      
      LogitBoost &      
         \texttt{logitBoost} & 
            \texttt{caTools}       &          
            \texttt{nIter}\\              
            
      Logistic Model Trees &
         \texttt{LMT} & 
            \texttt{RWeka}    & 
            \texttt{iter}  \\  
            
      Rule--Based Models &
         \texttt{J48} & 
            \texttt{RWeka}    & 
            \texttt{C}  \\  
            
       &
         \texttt{OneR} & 
            \texttt{RWeka}    & 
            None  \\  
            
       &
         \texttt{PART} & 
            \texttt{RWeka}    & 
            \texttt{threshold}, \texttt{pruned}   \\  
            
        &
         \texttt{JRip} & 
            \texttt{RWeka}    & 
            \texttt{NumOpt}   \\
            
      Bayesian Multinomial  
      & \texttt{vbmpRadial} 
      & \texttt{vbmp} 
      & \texttt{estimateTheta} \\
      \:\: Probit Model \\
      
      Least Squares Support Vector  &
         \texttt{lssvmRadial} & 
            \texttt{kernlab}       & 
            \texttt{sigma} \\             
                  
      Nearest Shrunken Centroids &
         \texttt{pam} & 
            \texttt{pamr}       & 
            \texttt{threshold} \\  
                                                                   
      Naive Bayes &
         \texttt{nb} & 
            \texttt{klaR}       & 
            \texttt{usekernel} \\
        
      Generalized Partial &
         \texttt{gpls} & 
            \texttt{gpls}       & 
            \texttt{K.prov} \\
      \:\: Least Squares \\
      
      Learned Vector Quantization &
         \texttt{lvq} & 
            \texttt{class}       &          
            \texttt{k} \\            
\label{label-name}
\end{longtable}

\begin{figure}[p]
   \begin{center}		
      \includegraphics[clip, width = .45\textwidth]{svm1}
      \hspace*{.4 in}   
      \includegraphics[clip, width = .45\textwidth]{svm2}
      
      \vspace*{.5 in}   
      
      \includegraphics[clip, width = .45\textwidth]{gbm1}
      \hspace*{.4 in}
      \includegraphics[clip, width = .45\textwidth]{gbm2}    

      \caption{ Examples of output from \texttt{plot.tain}. {\bf top left} a plot produced using \texttt{plot(svmFit)} showing the relationship between SVM cost parameter and the resampled classification accuracy. Although this model has two tuning parameters, a constant value for the parameter \texttt{sigma} was used. {\bf top right} the same plot but the \texttt{xTrans} argument was used to log--transform the cost parameter. {\bf bottom left} a plot produced using \texttt{plot(gbmFit)} showing the relationship between the number of boosting iterations, the interaction depth and the resampled classification accuracy {\bf bottom right} the same plot, but the Kappa statistic is plotted using \texttt{plot(gbmFit metric = "Kappa")}}
      \label{f:plots1} 
    \end{center}
\end{figure} 
	   
\begin{figure}[p]
   \begin{center}	
      \includegraphics[clip, width = .45\textwidth]{gbm3}    
   
      \vspace*{1 in}      
      
      \includegraphics[clip, width = .7\textwidth]{resampHist}
   
 \caption{More examples. {\bf top:} A plot produced using \texttt{plot(gbmFit metric = "Kappa", plotType = "level")} showing the relationship (using a \texttt{levelplot}) between the number of boosting iterations, the interaction depth and the resampled estimate of the Kappa statistic. {\bf bottom:} A plot of the resampling estimates of performance from the optimal support vector machine model produced using \texttt{resampleHist(svmFit, type = "density", layout = c(2, 1), adjust = 1.5)}.}   
      \label{f:plots2}         
   \end{center}
\end{figure}   

\clearpage

\section{Customizing the Tuning Process}

There are a few ways to customize the process of selecting tuning/complexity parameters. First, as previously shown with the boosted tree code, you can choose specific values of the tuning parameter (instead of the defaults).

Secondly, the user can change the metric used to determine the best settings. By default, RMSE and $R^2$ are computed for regression while accuracy and Kappa are computed for classification. Also by default, the parameter values are chosen using RMSE and accuracy, respectively  for regression and classification. The \texttt{metric} argument of the \texttt{train} function allows the user to control which the optimality criterion is used. For example, in problems where there are a low percentage of samples in one class, using \texttt{metric = "Kappa"} can improve quality of the final model.

If none of these parameters are satisfactory, the user can also compute custom performance metrics. The \texttt{trainControl} function has a argument called \texttt{summaryFunction} that specifies a function for computing performance. The function should have these arguments:
\begin{itemize}
\item \texttt{data} is a reference for a data frame or matrix with columns called \texttt{obs} and \texttt{pred} for the observed and predicted outcome values (either numeric data for regression or character values for classification). Currently, class probabilities are not passed to the function. The values in data are the held--out predictions (and their associated reference values) for a single combination of tuning parameters.
\item \texttt{lev} is a character string that has the outcome factor levels taken from the training data. For regression, a value of \texttt{NULL} is passed into the function.
\item \texttt{model} is a character string for the model being used (i.e. the value passed to the \texttt{method} value of \texttt{train}).
\end{itemize}
The output to the function should be a vector of numeric summary metrics with non--null names. 

As an example, classification accuracy in two--class problems can be decomposed into sensitivity and specificity. We can use these values to tune the parameters using the following function:
<<summaryFunc>>=




newSummary <- function (data, lev, model)
  {
    out <- c(sensitivity(data[, "pred"], data[, "obs"], lev[1]),
      specificity(data[, "pred"], data[, "obs"], lev[2]))
    
    names(out) <- c("Sens", "Spec")
    out
  }
@ 

To rebuild the support vector machine model using this criterion, we can see the relationship between the tuning parameters and sensitivity/specificity via the following code:
<<reTune>>=
fitControl$summaryFunction <- newSummary
set.seed(2)
svmNew <- train(trainDescr, trainMDRR, method = "svmRadial", metric = "Spec", tuneLength = 4, trControl = fitControl)
svmNew
@ 
Based on this model and the original SVM model, 60$\%$ accuracy can be achieved  by being very biased towards sensitivity.

The third method for customizing the tuning process is to modify the algorithm that is used to select the ``best'' parameter values, given the performance numbers. By default, the \texttt{train} function chooses the model with the largest performance value (or smallest, for mean squared error in regression models). Other schemes for selecting model can be used.  Breiman et al (1984) suggested the ``one standard error rule'' for simple tree--based models. In this case, the model with the best performance value is identified and, using resampling, we can estimate the standard error of performance. The final model used was the simplest model within one standard error of the (empirically) best model. With simple trees this makes sense, since these models will start to overfit as they become more and more specific to the training data.

\texttt{train} allows the user to specify alternate rules for selecting the final model. The argument \texttt{selectionFunction}  can be used to supply a function to algorithmically determine the final model. There are three existing functions in the package: \texttt{best} is chooses the largest/smallest value, \texttt{oneSE} attempts to capture the spirit of Breiman et al (1984) and \texttt{tolerance} selects the least complex model within some percent tolerance of the best value. See \texttt{?best} for more details.

User--defined functions can be used, as long as they have the following arguments:
\begin{itemize}
  \item \texttt{x} is a data frame containing the tune parameters and their associated performance metrics. Each row corresponds to a different tuning parameter combination
    \item \texttt{metric} a character string indicating which performance metric should be optimized (this is passed in directly from the \texttt{metric} argument of \texttt{train}. 
      \item \texttt{maximize} is a single logical value indicating whether larger values of the performance metric are better (this is also directly passed from the call to \texttt{train}).
  \end{itemize}
  The function should output a single integer indicating which row in \texttt{x} is chosen.

As an example, if we chose the previous SVM model on the basis of specificity, we would choose a cost value of 100, the most complex model. Lower cost values would produce approximately the same performance with less complex models (with the exception of cost = 0.1). The tolerance function could be used to find a less complex model based on $(x-x_{best})/x_{best}\times 100$, which is the percent difference. For example, to select cost values based on $2\%$ and 6$\%$ losses of performance:
<<tolerance>>=
whichTwoPct <- tolerance(svmNew$results, "Spec", 2, TRUE)  
cat("best model within 2 pct of best:\n")
svmNew$results[whichTwoPct,]

whichSixPct <- tolerance(svmNew$results, "Spec", 6, TRUE)  
cat("\n\nbest model within 6 pct of best:\n")
svmNew$results[whichSixPct,]
@ 

  The main issue with these functions is related to ordering the models from simplest to complex. In some cases, this is easy (e.g. simple trees, partial least squares), but in most cases, the ordering of models is subjective. For example, is a boosted tree model using 100 iterations and a tree depth of 2 more complex than one with 50 iterations and a depth of 8? The package makes some choices regarding the orderings. In the case of boosted trees, the package assumes that increasing the number of iterations adds complexity at a faster rate than increasing the tree depth, so models are ordered on the number of iterations then ordered with depth. See \texttt{?best} for more examples for specific models.

  Finally, the function \texttt{trainControl}, generates parameters that further control how models are resampled with possible values:
  \begin{itemize}
  \item \texttt{method}: The resampling method: \texttt{boot}, \texttt{cv}, \texttt{LOOCV}, \texttt{LGOCV}  and \texttt{oob}. The last value, out--of--bag estimates, can only be used by random forest, bagged trees, bagged earth, bagged flexible discriminant analysis, or conditional tree forest models. GBM models are not included (the \texttt{gbm} package maintainer has indicated that it would not be a good idea to choose tuning parameter values based on the model OOB error estimates with boosted trees). Also, for leave--one--out cross--validation, no uncertainty estimates are given for the resampled performance measures.
  \item \texttt{number}: Either the number of folds or number of resampling iterations
  \item \texttt{verboseIter}: A logical for printing a training log.
  \item \texttt{returnData}: A logical for saving the data
  \item \texttt{p}: For leave-group out cross-validation: the training percentage
  \item \texttt{index}: a list with elements for each resampling iteration. Each list element is the sample rows used for training at that iteration. When these values are not specified, \texttt{caret} will generate them.
  \item \texttt{summaryFunction}: previously  mentioned
  \item \texttt{selectionFunction}: previously  mentioned    
  \item \texttt{returnResamp}: a character string containing one of the following values: \texttt{"all"}, \texttt{"final"} or \texttt{"none"}. This specifies how much of the resampled performance measures to save. 
  \end{itemize}


<<makePlots, results = hide, echo = FALSE>>=
pdf(paste(getwd(), "/svm1.pdf", sep = ""), width = 5, height = 5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plot(svmFit))
dev.off()
pdf(paste(getwd(), "/svm2.pdf", sep = ""), width = 5, height = 5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plot(svmFit, xTrans = function(u) log(u, base = 10)))
dev.off()
pdf(paste(getwd(), "/gbm1.pdf", sep = ""), width = 5, height = 5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plot(gbmFit))
dev.off()
pdf(paste(getwd(), "/gbm2.pdf", sep = ""), width = 5, height = 5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plot(gbmFit, metric = "Kappa"))
dev.off()
pdf(paste(getwd(), "/gbm3.pdf", sep = ""), width = 5, height = 5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plot(gbmFit, meric = "Kappa", plotType = "level"))
dev.off()
pdf(paste(getwd(), "/resampHist.pdf", sep = ""), width = 7, height = 3.5)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(resampleHist(svmFit, type = "density", layout = c(2, 1), adjust = 1.5))
dev.off()
@

\section{Extracting Predictions and Class Probabilities}

As previously mentioned, objects produced by the \texttt{train} function contain the ``optimized'' model in the \texttt{finalModel} sub--object. Predictions can be made from these objects as usual. In some cases, such as \texttt{pls} or \texttt{gbm} objects, additional  parameters from the optimized fit may need to be specified. In these cases, the \texttt{train} objects uses the results of the parameter optimization to predict new samples.

For example, we can load the Boston Housing data:

\begin{small}
<<bhExample>>=
library(mlbench)
data(BostonHousing)
# we could use the formula interface too
bhDesignMatrix <-  model.matrix(medv ~. - 1, BostonHousing)
@
\end{small}

\noindent split the data into random training/test groups:

\begin{small}
<<bhSplit>>=
set.seed(4)
inTrain <- createDataPartition(BostonHousing$medv, p = .8, list = FALSE, times = 1)
trainBH <- bhDesignMatrix[inTrain,]
testBH <- bhDesignMatrix[-inTrain,]

preProc <- preProcess(trainBH)
trainBH <- predict(preProc, trainBH)
testBH  <- predict(preProc,  testBH)

trainMedv <- BostonHousing$medv[inTrain]
testMedv <- BostonHousing$medv[-inTrain]
@
\end{small}

\noindent fit partial least squares and multivariate adaptive regression spline models:

\begin{small}
<<bhModels>>=
set.seed(5)
plsFit <- train(trainBH, trainMedv, "pls", tuneLength = 10, trControl = trainControl(verboseIter = FALSE))
set.seed(5)
marsFit <- train(trainBH, trainMedv, "earth", tuneLength = 10, trControl = trainControl(verboseIter = FALSE))

@
\end{small}
To obtain predictions  for the PLS model, \texttt{predict.mvr} can be used. In this case, the number of components must be manually specified or all of the sub-models are predicted:

\begin{small}
<<plsPrediction1>>=
plsPred1 <- predict(plsFit$finalModel, newdata = as.matrix(testBH))
dim(plsPred1)
@ 
\end{small}
Alternatively, \texttt{predict.train} can be used to get a vector of predictions for the optimal model only:
\begin{small}
<<plsPrediction1>>=
plsPred2 <- predict(plsFit, newdata = testBH)
length(plsPred2)
@
\end{small}
For multiple models, the objects can be grouped using a list and predicted simultaneously:
\begin{small}
<<bhPrediction1>>=
bhModels <- list(
                 pls = plsFit,
                 mars = marsFit)

bhPred1 <- predict(bhModels, newdata = testBH)
str(bhPred1)
@
\end{small}
In some cases,observed outcomes and their associated predictions may be needed for a set of models. In this case, \texttt{extractPrediction} can be used. This function takes a list of models and test and/or unknown samples as inputs and returns a data frame of predictions:
\begin{small}
<<bhPrediction1>>=
allPred <- extractPrediction(bhModels,
                             testX = testBH,
                             testY = testMedv)
testPred <- subset(allPred, dataType == "Test")
head(testPred)

by(
   testPred, 
   list(model = testPred$model), 
   function(x) postResample(x$pred, x$obs))
@
\end{small}
The output of \texttt{extractPrediction} is a data frame with columns:
   \begin{itemize}   
      \item \texttt{obs}, the observed data
      \item \texttt{pred}, the predicted values from each model
      \item \texttt{model}, a character string (``\texttt{rpart}'', ``\texttt{pls}'' etc.)
      \item \texttt{dataType}, a character string for the type of data:
      \begin{itemize}
         \item ``\texttt{Training}'' data are the predictions on the training data from
            the optimal model,
         \item ``\texttt{Test}'' denote the predictions on the test set (if one is specified),
         \item ``\texttt{Unknown}'' data are the predictions on the unknown samples (if specified). 
         Only the predictions are produced for these data. Also, if the quick prediction of the unknowns
         is the primary goal, the argument \texttt{unkOnly} can be used to only process the unknowns.
      \end{itemize}
   \end{itemize}      

Some classification models can produce probabilities for each class. The functions \texttt{predict.train} and \texttt{predict.list} can be used with the \texttt{type = "probs"} argument to produce data frames of class probabilities (with one column per class). Also, the function \texttt{extractProbs} can be used to get these probabilities from one or more models. The results are very similar to what is produced by \texttt{extractPrediction} but with columns for each class. The column \texttt{pred} is still the predicted class from the model. 


\section{Evaluating Models}

A function, \texttt{postResample}, can be used obtain the same performance measures as generated by \texttt{train}. 

\texttt{caret} also contains several functions that can be used to describe the performance of classification models. The functions \texttt{sensitivity}, \texttt{specificity}, \texttt{posPredValue} and \texttt{negPredValue} can be used to characterize performance where there are two classes. By default, the first level of the outcome factor is used to define the ``positive'' result (i.e. the event of interest), although this can be changed. 

The function \texttt{confusionMatrix} can also be used to summarize the results of a classification model:

\begin{small}
<<mbrConfusion>>=
mbrrPredictions <- extractPrediction(list(svmFit), testX = testDescr, testY = testMDRR)
mbrrPredictions <- mbrrPredictions[mbrrPredictions$dataType == "Test",]
sensitivity(mbrrPredictions$pred, mbrrPredictions$obs)
confusionMatrix(mbrrPredictions$pred, mbrrPredictions$obs)
@
\end{small}

The ``no--information rate'' is the largest proportion of the observed classes (there were more actives than inactives in this test set). A hypothesis test is also computed to evaluate whether the overall accuarcy rate is greater than the rate of the largest class. Also, the prevalence of the ``postivie event'' is computed from the data (unless passed in as an argument), the detection rate (the rate of true events also 
predicted to be events) and the detection prevalence (the prevalence of predicted events).

Suppose a $2 \times 2$ table with notation

\begin{tabular}{r|c|c|}
                    & \multicolumn{2}{c}{{\bf Reference}}          \\
\cline{2-3}
         {\bf Predicted}  & Event     & No Event \\
\cline{2-3}
         Event      & A         & B        \\
\cline{2-3}
         No Event   & C         & D      \\
\cline{2-3}  
\end{tabular}

The formulas used here are:
\begin{align}
Sensitivity &= \frac{A}{A+C}\notag \\
Specificity &= \frac{D}{B+D}\notag \\
Prevalence &= \frac{A+C}{A+B+C+D}\notag \\
PPV &= \frac{sensitivity \times prevalence}{((sensitivity \times prevalence) + ((1-specificity) \times(1-prevalence))}\notag \\
NPV &= \frac{specificity \times (1-prevalence)}{((1-sensitivity) \times prevalence) + ((specificity) \times(1-prevalence))}\notag \\
Detection\: Rate &=  \frac{A}{A+B+C+D}\notag \\
Detection\: Prevalence &=  \frac{A + B}{A+B+C+D}\notag 
\end{align}

When there are three or more classes, \texttt{confusionMatrix} will show the confusion matrix and a set of ``one--versus--all'' results. For example, in a three class problem, the sensitivity of the first class is calculated against all the samples in the second and third classes (and so on).


\subsection*{ROC Curves}

The function \texttt{roc}\footnote{I'm looking into using the \texttt{ROCR} package for ROC curves, so don't get too attached to these functions} can be used to calculate the sensitivity and specificity used in an ROC plot. For example, using the previous support vector machine fit to the MBRR data, the predicted class probabilities on the test set can used to create an ROC curve. The area under the ROC curve, via the trapezoidal rule, is calculated using the \texttt{aucRoc} function. 

\begin{small}
<<mbrrROC>>=
mbrrProbs <- extractProb(list(svmFit), testX = testDescr, testY = testMDRR)
mbrrProbs <- mbrrProbs[mbrrProbs$dataType == "Test",]
mbrrROC <- roc(mbrrProbs$Active, mbrrProbs$obs)
aucRoc(mbrrROC)
@
\end{small}

See Figure \ref{f:mbrrROC} for an example.

\subsection*{Plotting Predictions and Probabilities}

Two functions, \texttt{plotObsVsPred} and \texttt{plotClassProbs}, are interfaces to lattice to plot model results. For regression, \texttt{plotObsVsPred} plots the observed versus predicted values by model type and data (e.g. test). See Figures \ref{f:bhPredPlot} and \ref{f:mbrrROC}  for examples. For classification data, \texttt{plotObsVsPred} plots the accuracy rates for models/data in a dotplot. 


<<mbrrPlots, echo = FALSE, results = hide>>=
pdf(paste(getwd(), "/roc.pdf", sep = ""), width = 6.5, height = 7)
   plot(1 - mbrrROC[,"specificity"], mbrrROC[, "sensitivity"], type = "s", xlab = "1 - Specificity", ylab = "Sensitivity")
   abline(0,1, col = "grey", lty = 2)
dev.off()

pdf(paste(getwd(), "/svmProbs.pdf", sep = ""), width = 9, height = 7)
   trellis.par.set(caretTheme(), warn = FALSE)
   print(plotClassProbs(mbrrProbs))
dev.off()
@

\begin{figure}[ht]
   \begin{center}      
      \includegraphics[clip, width = .6\textwidth]{svmProbs}   
      \caption{The predicted class probabilities from a support vector machine fit for the MBRR test set. This plot was created using \texttt{plotClassProbs(mbrrProbs)}.}
      \label{f:mbrrProbs}         
      \vspace*{.5 in}      
       \includegraphics[clip, width = .5\textwidth]{roc}   
      \caption{An ROC curve from the predicted class probabilities from a support vector machine fit for the MBRR test set. }
      \label{f:mbrrROC}       
   \end{center}
\end{figure}  

\begin{figure}
   \begin{center}      
<<bhPredPlot, echo = FALSE, fig = TRUE>>=
trellis.par.set(caretTheme(), warn = FALSE)
print(plotObsVsPred(testPred))
@         
      \caption{The results of using \texttt{ plotObsVsPred } to show plots of the observed median home price against the predictions from two models. The plot shows the training and test sets in the same Lattice plot}
      \label{f:bhPredPlot}         
   \end{center}
\end{figure}


To plot class probabilities, \texttt{plotClassProbs} will display the results by model, data and true class (for example, Figure \ref{f:mbrrProbs}).     


\section{Session Information}

<<<session, echo=FALSE, results=tex>>=
toLatex(sessionInfo())
@ 


\section{References}

\begin{description}

   
   \item Breiman, Friedman, Olshen, and Stone. (1984) {\it Classification and Regression Trees}. Wadsworth.


   \item Svetnik, V., Wang, T., Tong, C., Liaw, A., Sheridan, R. P. and Song, Q. (2005), ``Boosting: An ensemble learning tool for compound classification and QSAR modeling,'' {\it Journal of Chemical Information and Modeling}, 45, 786 --799.
   
   \item Tibshirani, R., Hastie, T., Narasimhan, B., Chu, G. (2003), ``Class prediction by nearest shrunken centroids, with applications to DNA microarrays,'' {\it  Statistical Science}, 18, 104--117.


\end{description}

\end{document}
