%====================
%  Document Class
%====================
\documentclass{article}
%====================
%  Packages
%====================
\usepackage{mathpazo}
\usepackage{xfrac}
\usepackage[T1]{fontenc}
\usepackage{geometry}
\usepackage{url}
\usepackage[authoryear]{natbib}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{float}
\usepackage{animate}
\usepackage{setspace}
\onehalfspacing
\usepackage{booktabs}
\usepackage{bookmark}
\usepackage{caption}

\captionsetup{
  font=footnotesize,
  justification=raggedright,
  singlelinecheck=false
}
%====================
%  Custom Commands
%====================
\renewcommand{\sfdefault}{lmss}
\renewcommand{\ttdefault}{lmtt}
\newcommand\posscite[1]{\citeauthor{#1}'s (\citeyear{#1})} 
\newcommand\poscite[1]{\citeauthor{#1}' (\citeyear{#1})}

\newenvironment{sing_enum}{
\begin{enumerate}
  \setlength{\itemsep}{1pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
}{\end{enumerate}}

\newenvironment{sing_item}{
\begin{itemize}
  \setlength{\itemsep}{1pt}
  \setlength{\parskip}{0pt}
  \setlength{\parsep}{0pt}
}{\end{itemize}}
%====================
%  Formatting
%====================
%\geometry{verbose,tmargin=2.5cm,bmargin=2.5cm,lmargin=2.5cm,rmargin=2.5cm}

%====================
%  Other
%====================
\makeatletter
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
% \VignetteIndexEntry{Cleaning Text & Debugging}
% \VignetteEngine{knitr}

\makeatother
%===============================================================================
\begin{document}

\title{Cleaning Text \& Debugging}
\author{Tyler W. Rinker}
\date{\today}
\maketitle

\begin{figure}[h!]
  \centering
    \includegraphics[width=3.5in]{imgs/qdaplogo.png}
\end{figure}

The \textbf{qdap} package \citep{R-qdap} contains many functions that assume that the text strings supplied are cleaned and in the expected form.  Failing to prepare data may result in errors, warnings, and incorrect results.  This vignette will outline the checking and prepping of text as well as how to isolate and identify errors caused by unprepared text.

\newpage
\tableofcontents

\newpage
\section{qdap Text Assumptions}

\hspace{.4cm} Many of the analysis/scoring functions in \textbf{qdap} make the following assumptions about your data:

\begin{sing_enum}
  \item Each row contains a single sentence
  \item Each sentence contains a \textbf{qdap} end-mark ("?", ".", "!", "|")
  \item Each sentence contains only one punctuation end-mark
  \item Commas are followed by a space
  \item Numbers are unimportant (they are ignored unless converted to text equivalent)
  \item Symbols (non-text other than apostrophes and end-marks) are ignored
  \item Text elements contain alphabetic characters or are \texttt{NA}
  \item Text words are spelled correctly
  \item Text contains only ASCII characters
  \item Text contains no escape characters
\end{sing_enum}

\noindent If these assumptions are not met the integrity of \textbf{qdap}'s functions may be undermined.

\section{Cleaning and Debugging Procedures}

\hspace{.4cm} The follow procedure will help the user to avoid problems caused by poorly formatted text and cope with errors:

\begin{sing_enum}
  \item \emph{Check text} for potential problems with \texttt{check\_text} 
  \item Perform \texttt{check\_text}'s recommended \emph{cleaning procedures}
  \item \emph{Recheck text} for potential problems with \texttt{check\_text} 
  \item Run \emph{analysis}
  \item \emph{Debugging} (via halving) to isolate and identify errors when they occur
\end{sing_enum}



<<setup, include=FALSE, cache=FALSE>>=
library(knitr); library(qdap)
opts_chunk$set(fig.path='figure/minimal-', fig.align='center', fig.show='hold', 
    tidy=FALSE, cache=FALSE)
options(replace.assign=TRUE, width=90)
pdf.options(useDingbats = TRUE)
@

\newpage
\section{Checking Text}

\subsection{\texttt{check\_text} Introduced}

\hspace{.4cm} The \texttt{check\_text} function is designed to check text for the following potential sources of errors, warnings, and incorrect results:

\begin{sing_item}
  \item \textbf{non\_character} -- Text that is of \texttt{factor} class.
  \item \textbf{missing\_ending\_punctuation} -- Text with no end-mark at the end of the string.
  \item \textbf{empty} -- Text that contains an empty element (i.e., \texttt{""}).
  \item \textbf{double\_punctuation} - Text that contains two \textbf{qdap} punctuation marks in the same string.
  \item \textbf{non\_space\_after\_comma} -- Text that contains commas with no space after them.
  \item \textbf{no\_alpha} -- Text that contains string elements with no alphabetic characters.
  \item \textbf{non\_ascii} -- Text that contains non-ASCII characters.
  \item \textbf{missing\_value} -- Text that contains missing values (i.e., \texttt{NA}).
  \item \textbf{containing\_escaped} -- Text that contains escaped (see \texttt{?Quotes}).
  \item \textbf{containing\_digits} -- Text that contains digits.
  \item \textbf{indicating\_incomplete} -- Text that contains end-marks that indicate incomplete/trailing sentences.
  \item \textbf{potentially\_misspelled} -- Text that contains potentially misspelled words.
\end{sing_item}

The user simply supplies a text variable and \texttt{check\_text} will output a list.  The list prints to the console (or can be saved to an external file) as a prettified summary of potential text problems .  

\newpage
\subsection{\texttt{check\_text} Output Explained}

\hspace{.4cm} Here is a sample section for potential misspellings.  Notice that there are typically 4 elements within a section: (A) a section header, (B) the index within the vector for where the potential problems are located, (C) the actual text strings that raised the alert, and (D) a suggested fix for the problem.

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
======================
POTENTIALLY MISSPELLED
======================

The following observations were potentially misspelled:

2, 11, 13

The following text is potentially misspelled:

2: i want. <<thet>> them .
11: I like <<goud>> eggs!
13: <<tgreat>>

*Suggestion: Consider running `check_spelling_interactive`
\end{alltt}
\end{kframe}
\end{knitrout}


\newpage
\subsection{\texttt{check\_text} Example}

\hspace{.4cm} In this section you will see an instance of the use and output of \texttt{check\_text}.  Here is the data we'll use:


<<checking1>>=
x <- c("i like", "i want. thet them .", "I am ! that|", "", NA,
    "they,were there", ".", "   ", "?", "3;", "I like goud eggs!",
    "i 4like...", "\\tgreat",  "She said \"yes\"")
x
@

\noindent Output from \texttt{check\_text}:

<<checking2>>=
check_text(x)
@


\newpage
\section{Cleaning}

\hspace{.4cm} \textbf{qdap} has a number of functions that can be used to clean and format the text into the form expected by many \textbf{qdap} functions.  \texttt{check\_text} suggests the use of some of these cleaning functions.  The user should consider cleaning the text as recommended or have legitimate rationale for not doing so.  Failure to clean data as recomended may lead to errors, warnings, and incorrect results.

The following is a list of \textbf{qdap} cleaning \& formatting functions and their designed uses:

\begin{sing_item}
  \item \textbf{clean} -- Remove escaped characters
  \item \textbf{scrubber} -- General text cleaning function that removes extra white spaces other textual anomalies that may cause errors
  \item \textbf{strip} -- Strip text of unwanted characters
  \item \textbf{replace\_number} -- Replace numerically represented numbers with words
  \item \textbf{blank2NA} -- Replace blank (empty) cells 
  \item \textbf{comma\_spacer} -- Add a space after a comma
  \item \textbf{sentSplit} -- Split multi-sentence strings into individual sentences
  \item \textbf{incomplete\_replace} -- Replace incomplete sentence end marks (e.g., ``...'') with ``|''
  \item \textbf{check\_spelling\_interactive} -- Interactively check for misspelled words
  \item \textbf{Encoding} -- Set the declared encodings for a character vector
  \item \textbf{replace\_abbreviation} -- Replace abbreviations with long form
  \item \textbf{multigsub} -- Wrapper for gsub that takes a vector of search terms and a vector or single value of replacements
  \item \textbf{sub\_holder} --  Hold the place for particular character values, manipulate the string(s), and then revert place holders back
  \item \textbf{replace\_symbol} -- Replace select symbols with word equivalents  
  \item \textbf{replace\_contraction} -- Replace contractions with multi-word form 
  \item \textbf{Trim} --  Remove leading/trailing white spaces 
  \item \textbf{bracketX} --  Remove bracketed text 
  \item \textbf{genX} -- Remove text between markers
\end{sing_item}

\noindent While an exhaustive discussion of these functions is beyond the scope of this vignette the reader should be aware of these functions and their uses to clean and format text for analysis.

\newpage
\section{Debugging}

\hspace{.4cm} While the \texttt{check\_text} is useful for catching many text problems it can not detect all problems a user may encounter.  Some functions may provide useful error/warning message that help the user isolate the source of the problem.  At other times the source of the error is unknown.

When a user encounters such an error it is important to isolate and identify the source of the error.  Often text data sets (corpora) are larger and thus a reasonable strategy can be employed to isolate the source of the bug more quickly.  The use of \emph{halving} is one such approach.

In halving, after the user experiences an error, she recursively divides the data set into halves until the source of the error is found.  With each pass the non-offending portion is excluded while half of the offending portion of the data is used to test smaller and smaller chunks of the data for the location of the error.\footnote{Note that the halving technique can be used with code testing as well.}  

\subsection{Halving Example}

\hspace{.4cm} Here is an example of the entire halving process.  First we'll create an error generating function to use with the built in \textbf{qdap} data set \texttt{DATA}:

<<bug1>>=
fake_fun <- function(x) {
    stopifnot(!any(grepl("talk", x)))
    x
}
@

<<bug2, eval=FALSE>>=
with(DATA, fake_fun(state))
@

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
Error: !any(grepl("talk", x)) is not TRUE 
\end{alltt}
\end{kframe}
\end{knitrout}

\noindent An initial unknown error is raised.  Cut the data in half:

<<bug3>>=
with(DATA[1:5, ] , fake_fun(state))
@

\noindent No error is raised with this half.  Grab the other half of the data and confirm the error is in that portion of the data:


<<bug4, eval=FALSE>>=
with(DATA[5:11, ] , fake_fun(state))
@

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
Error: !any(grepl("talk", x)) is not TRUE 
\end{alltt}
\end{kframe}
\end{knitrout}

\noindent Confirmed!  So take half of the second half (the offending data):

<<bug5>>=
with(DATA[5:8, ] , fake_fun(state))
@

\noindent No error is raised with the first half of the original second half.  Grab the other half (last $\sfrac{1}{4}$ of the original data set) of the data and confirm the error is in that portion of the data:

<<bug6, eval=FALSE>>=
with(DATA[9:11, ] , fake_fun(state))
@

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
Error: !any(grepl("talk", x)) is not TRUE 
\end{alltt}
\end{kframe}
\end{knitrout}

\noindent Confirmed again!  At this point there are three rows left to test so we test them one at a time:


<<bug7, eval=FALSE>>=
with(DATA[9, ] , fake_fun(state))
@

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
Error: !any(grepl("talk", x)) is not TRUE 
\end{alltt}
\end{kframe}
\end{knitrout}

\noindent Source of the error identified!

Now that you have identified the location (row 9) identify the source.  Generally a visual inspection will turn up the source.  Other times it may elude you (for example an \texttt{Encoding} issue may be masked).  If you are eluded now you can begin the halving technique on the individual string.  Here we use \texttt{substring} to accomplish this task:

<<bug8, eval=FALSE>>=
with(DATA[9, ], fake_fun(substring(state, 1:20)))
@

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
Error: !any(grepl("talk", x)) is not TRUE 
\end{alltt}
\end{kframe}
\end{knitrout}

\noindent Found it!  Now let's test the other half:

<<bug9>>=
with(DATA[9, ], fake_fun(substring(state, 21)))
@

\noindent All clear!  We continue the halving technique until we've identified the source of the error, in this case the use of the word \emph{talk}.

At this point you may have either identified (a) improperly cleaned data or (b) a potential bug in \textbf{qdap}.  It is important that you can replicate the problem (either bad data or the bug).  If you have located (where) and identified the source of an error (what) you should be able to recreate the error with new data that contains the same type of error causing string(s).  If you cannot replicate the error you have not identified the source.

\section{Reporting a Potential Bug}

\hspace{.4cm} If you believe that there is a bug in \textbf{qdap} you can:


\begin{sing_enum}
  \item Fork the repository, correct the code, run a CRAN check, and send a pull request.
  \item Report the bug via \href{https://github.com/trinker/qdap/issues}{\textbf{qdap}'s GitHub issues page}.  Be sure that you:
    \begin{enumerate}
      \item Describe the bug (be specific; in other words, show work on your part to locate and identify the source of the error)
      \item Provide a \href{http://sscce.org/}{minimally reproducible example}, including a small data set and code to produce the error
    \end{enumerate}
\end{sing_enum}

\noindent The former is preferred if you are experienced with R programming, the latter if not.

\bibliographystyle{jss}
\bibliography{qdap}

\end{document}












