%\VignetteIndexEntry{deducorrect-imputation}
\documentclass[11pt, fleqn, a4paper]{article}
\usepackage[english]{babel}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows,decorations,calc}
\usetikzlibrary{decorations.pathreplacing}
\usepackage{amsmath, amssymb}
\usepackage{natbib}
\usepackage{algpseudocode}
\usepackage{algorithm}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
\usepackage{threeparttable}

\usepackage{mySweave}
% stimulate latex to put multiple floats on a page.
\hyphenation{time-stamp}

\newcommand{\true}{\textrm{\sc true} }
\newcommand{\false}{\textrm{\sc false} }

\usepackage{float}
 
\floatstyle{boxed}
\newfloat{Rcode}{t!}{rco}
\floatname{Rcode}{Figure}
\usepackage{ucap}



\title{Deductive imputation with the {\sf deducorrect} package}
\author{Mark van der Loo and Edwin de Jonge\\
{\small Package version \Sexpr{packageVersion("deducorrect")}}}
\begin{document}
\maketitle
\begin{abstract}
Numerical and categorical data used for statistical analyses is often plagued
with missing values and inconsistencies. In many cases, a number of missing
values may be derived, based on the consistency rules imposed on the data and
the observed values in a record. The methods used for such derivations are
called {\em deductive imputation}. In this paper, we describe the newly
developed deductive imputation functionality of {\sf R} package {\sf
deducorrect}. The package gained methods to deductively impute numerical as
well as categorical data. Methods for setting up a partial data editing system
are discussed as well.

{\em This vignette (at version 1.1-1) is a literal transcript of \cite{loo:2011c}.
Please use that paper when referencing imputation functionality of the package. This
vignette may be updated if the package is developed further.}


\end{abstract}

<<echo=FALSE,results=hide, keep.source=FALSE>>=
library(editrules)
library(deducorrect)
@
\maketitle

\newpage

\tableofcontents

\newpage
\section{Introduction}
The quality of raw survey data is only rarely sufficient to allow for immediate
statistical analysis. The presence of missing values (item nonresponse) and
inconsistencies impedes straightforward application of standard statistical
estimation methods, and statisticians often have to spend considerable effort
to counterbalance the effect of such errors. 

There are basically two ways to take the effect of data quality issues into
account.  The first is to adapt the statistical analysis such that the effects
of these issues are taken into account. One well-documented example is to use
weighting methods which take the effect of (selective) item nonresponse into
account \citep{kalton:1986,bethlehem:2011}.  The second way is to clean up the
dataset so that missing values are completed and inconsistencies have been
repaired. The latter method has the advantage that statistical analyses of the
data become to a degree independent of the models used in data cleaning.
Whichever way is chosen, in most cases additional assumptions are necessary to
clean data or interpret the results of data analyses. 

Recently, a number of near assumption-free data-cleaning methods have been
reported which rely almost purely on record consistency rules imposed {\em a
priori} on the data. Examples of such rules include account balances, positivity
demands on variables or forbidden value combinations in categorical data.  In a
previous paper \citep{loo:2011a} we reported on methods which use data
consistency rules and information in inconsistent records to track down and
repair typing errors, rounding errors and sign errors. The theory behind these
methods was first published by \citet{scholtus:2008, scholtus:2009} and the methods were
implemented by us in {\sf R} package {\sf deducorrect}. Since these so-called
deductive correction methods are based on adapting values, they are not suited
for completing missing values.

In this paper, we report on an extension of the {\sf deducorrect} package which
allows for deductive imputation of missing values in either numerical or
categorical data. The implemented methods were proposed by \citet{pannekoek:2006} and
\cite{waal:2011}. By deductive imputation we mean methods which use the
observed values in a record together with consistency rules imposed on the
record to uniquely derive values where possible.  The values may be missing
because of nonresponse, or they may be deemed missing by an error localization
algorithm such as implemented in the {\sf editrules} package
\citep{jonge:2011a, loo:2011d}. 

In section \ref{sdeduimpute}, we further introduce the concept of deductive
imputation and show the easiest way of imputing values with the deducorrect
package. In sections \ref{sdeductivenumerical} and \ref{sdeductivecategorical}
we expand a bit on the theory and demonstrate the use of lower-level
functionality of the package. Examples in {\sf R} code are given throughout to
help new users getting started.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Deductive imputation}
\label{sdeduimpute}

\subsection{Overview}
Deductive imputation relies on in-record consistency rules to derive the value
of variables which have not been completed from variables that have. These
methods therefore rely on the assumption that the values used in the derivation
have been completed correctly.  For example, suppose we have a numerical record
${\bf x}=(x_1,x_2,x_3)$, subject to the rules
\begin{eqnarray}
\label{rule1}
x_1 + x_2 &=& x_3\\
\label{rule2}
{\bf x} &\geq& \boldsymbol{0}. 
\end{eqnarray}
Suppose we are given two values of ${\bf x}$, for example $({\sf NA},x_2,x_3)$,
where {\sf NA} stands for Not Available. In principle, the unknown value is
easily derived from rule \eqref{rule1}, but one must take care not to violate
any other rules. For example, if $x_2<0$, the derived value for $x_1$ is most
likely not the true value, since at least one of the values used to derive
$x_1$ is invalid. Moreover, if $x_2>x_3$, the derived value for $x_1$ will be
negative, and therefore violate rule \eqref{rule2}. For categorical data,
analogous situations may arise.

The deductive imputation routines of the {\sf deducorrect} package offer two
mechanisms to avoid inconsistencies. The first is to explicitly check if
consistent deductive imputation is possible based on the observed values.  This
is switched on by default for the functions {\sf deduImpute}, {\sf
deductiveZeros}, the {\sf editmatrix} method of {\sf solSpace} and {\sf
deductiveLevels}. These functions will be discussed below. The second mechanism
is the ability to point out variables besides the missing ones, which should be
considered as if they were missing. A typical example would be to use the result
of an error localization algorithm which points out erroneous fields in a
record.


\begin{figure}[!t]
{
    \tikzstyle{block} = [rectangle, draw, fill=blue!20, 
        text width=5em, text centered, rounded corners, minimum height=3em]
    \tikzstyle{line} = [draw, -latex']
    \tikzstyle{cloud} = [draw, ellipse,fill=red!20, node distance=3cm,
        text width=9em, minimum height=2em, align=center]
\begin{tikzpicture}[node distance = 2cm, auto, decoration={brace,amplitude=10pt}]
    % Flow diagram blocks
    \node [block] (init0) {read data};
    \node [block, below of=init0]                       (init1)     {read rules};
    \node [block, below of=init1]                    (deducorr)  {deductive correction};
    \node [block, below of=deducorr]   (errloc)    {error localization};
 %   \node [block, below of=deducorr]                       (deduimp1)  {deductive imputation};
    \node [block, below of=errloc]                      (deduimp2)  {deductive imputation};
    \node [block, below of=deduimp2]                    (end)       {other imputation methods};
    % draw lines
    \path [line] (init0)   -- (init1);
    \path [line] (init1)   -- (deducorr);
    \path [line] (deducorr)-- (errloc);
%    \path [line] (deduimp1)-- (errloc);
    \path [line] (errloc)  -- (deduimp2);
    \path [line] (deduimp2)  -- (end);
    
    % functionality blocks
    \node [cloud, right of=init1, node distance=4cm] (editrules) {{\sf editarray}, {\sf editmatrix}};
%    \node [cloud, right of=deduimp1, node distance=4cm] (ddimp1) {{\sf deduImpute}};
    \node [cloud, right of=deducorr, node distance=4cm] (ddcor1) {{\sf correctTypos} {\sf correctSigns} {\sf correctRounding}};
    \node [cloud, right of=errloc, node distance=4cm] (ddcor2) {{\sf localizeErrors}};
    \node [cloud, right of=deduimp2, node distance=4cm] (ddimp2) {{\sf deduImpute}};
    % package annotation
    \draw [decorate, line width=1pt] ($(init1.north) + (7,0)$) -- ($(init1.south) + (7,0)$)
       node [midway, anchor=west, outer sep=3ex]{{\sf editrules}};
    \draw [decorate, line width=1pt] ($(deducorr.north) + (7,0)$) -- ($(deducorr.south) + (7,0)$)
       node [midway, anchor=west, outer sep=3ex]{{\sf deducorrect}};
    \draw [decorate, line width=1pt] ($(errloc.north) + (7,0)$) -- ($(errloc.south) + (7,0)$)
       node [midway, anchor=west, outer sep=3ex]{{\sf editrules}};
    \draw [decorate, line width=1pt] ($(deduimp2.north) + (7,0)$) -- ($(deduimp2.south) + (7,0)$)
       node [midway, anchor=west, outer sep=3ex]{{\sf deducorrect}};
\end{tikzpicture}
}
\ucaption{Flow diagram showing how functionality of the {\sf deducorrect} and
    {\sf editrules} can be combined to perform the deductive corrections, deductive
    imputations and error localization. All steps except deductive correction are
    available for numerical as well as categorical data. The ellipses indicate some
    of the {\sf R} functions from the packages noted on the right. Error localization
    is not strictly necessary to perform deductive imputation, but in this way
    the maximum number deductively imputable values will be derived.}
\label{flowchart}
\end{figure}

In the context of a complete automated data editing system, there are several
places where deductive imputation or correction can be applied. Typically, one
will apply such methods before the data is treated with more complicated
imputation models. Figure \ref{flowchart} shows a workflow for automatic
deductive data cleaning. It contains all (near) assumption-free corrections and
imputations of the {\sf deducorrect} package. If after these steps, missing
values or errors remain, one has to resort to other methods and accept extra
model assumptions. It should be noted that a common step such as detecting and
repairing unit measure errors is not included here.  However, such methods are
easily implemented in {\sf R}, and we refer to \cite{waal:2011} for an
overview. 

Deductive imputation appears after the error localization step in the process
flow chart of Figure \ref{flowchart}. At that point one is certain that the
missing variables together with the variables pointed out by the error
localization algorithm can be imputed consistent with the edit rules. Error
localization is not strictly necessary to perform deductive imputation with the
{\sf deducorrect} package since by default, the imputation routines check if
consistent imputation is possible.  However, the workflow in Figure
\ref{flowchart} guarantees that as many deductive imputations take place as
possible. For performance reasons, the user can choose to skip these checks
when the workflow of Figure \ref{flowchart} is followed.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Imputation with {\sf deduImpute}}
The simplest way to do deductive imputations with the {\sf deducorrect} package
is to use the {\sf deduImpute} function. It can be used for both numerical and
categorical data. The function accepts an {\sf editmatrix} or {\sf editarray}
containing the editrules and a {\sf data.frame} containing the records. The
return value is an object of class {\sf deducorrect}, similar to the values
returned by the {\sf correct-} functions of deducorrect [see
\citet{loo:2011a}].

For numerical data {\sf deduImpute} uses two methods (described in sections \ref{sssolspace} and
\ref{ssdeductivezeros}) to impute as many empty values as possible.  It uses
the functions {\sf solSpace} and {\sf deductiveZeros} iteratively for each
record until no deductive improvements can be made. Here, we will use the
example from \citet{waal:2011}, Chapter 9.2. This example uses the following
edits, based on a part of the Dutch Structural Business Survey balance account.
\begin{equation}
\begin{array}{rcl}
    x_1 + x_2      &=& x_3\\
    x_2           &=& x_4\\
    x_5 + x_6 + x_7 &=& x_8\\
    x_3 + x_8      &=& x_9\\
    x_9 - x_{10}     &=& x_{11}\\
    x_6 &\geq& 0\\
    x_7 &\geq& 0
\end{array}
\label{eqdeduImputeEdits}
\end{equation}
The rule $x_2=x_4$ may seem odd for readers not familiar with survey
statistics.  However, these rules correspond to cases where respondents have to
copy a figure from one page on a paper form to another\footnote{In spite of the
availability of web-based forms, many respondents still prefer paper forms.}.  
%
%
\begin{Rcode}
<<keep.source=TRUE>>=
E <- editmatrix(c(
         "x1 + x2      == x3",
         "x2           == x4",
         "x5 + x6 + x7 == x8",
         "x3 + x8      == x9",
         "x9 - x10     == x11",
         "x6 >= 0",
         "x7 >= 0"
))
dat <- data.frame(
    x1=c(145,145),
    x2=c(NA,NA),
    x3=c(155,155),
    x4=c(NA,NA),
    x5=c(NA, 86),
    x6=c(NA,NA),
    x7=c(NA,NA),
    x8=c(86,86),
    x9=c(NA,NA),
    x10=c(217,217),
    x11=c(NA,NA)
)
dat
d <- deduImpute(E,dat)
d$corrected
@
\ucaption{A simple example with {\sf deduImpute}. The return value is an object of class {\sf deducorrect}.}
\label{RdeduImpute}
\end{Rcode}
%
%
In Figure \ref{RdeduImpute} we give an example where the following record subject
to the edits in Eq.\ \eqref{eqdeduImputeEdits} is treated.
%
<<keep.source=FALSE,echo=FALSE>>=
dat[1,]
@
%
The record contains missing values. However, by assuming that all non-missing
values are correct, values can be derived for $x_2$, $x_4$, $x_9$ and $x_{11}$
just by considering the equality- and nonnegativity rules in the edit set.

The assumption that all missing values can be imputed consistently may not
alway be valid: the nonmissing values may have been filled in erroneously,
yielding faulty derived values to impute. The reason is that {\sf deduImpute}
does not take into account all edit rules: only nonnegativity rules and
equality rules are used to derive imputed values.

The {\sf deduImpute} function has two mechanisms to get around this. The first
is to set the option {\sf checkFeasibility=TRUE}. This causes solutions causing
new inconsistencies to be rejected. The second mechanism is to provide a
user-specified {\sf adapt} array to increase the number of variables which may
be imputed, missing or not. The {\sf adapt} array is a boolean array, stating
which variable may be changed in which record. A convenient example is to use
the {\sf adapt} array as generated by the {\sf localizeErrors} function from
the {\sf editrules} package. By specifying an {\sf adapt} array, {\sf
deduImpute} will try to fix records by imputing values which are either missing
or may be adapted according to {\sf adapt}.


For categorical data, {\sf deduImpute} uses the {\sf deductiveLevels} function,
discussed in section \ref{sdeductivecategorical}. The function accepts an 
{\sf editarray} holding the categorical edits and a {\sf data.frame} holding
records to be imputed.

Before introducing our example, we note that a categorical record is a member of
a discrete set, written as the cartesian product.
\begin{equation}
\label{eqdomain}
D = D_1\times D_2\times\ldots\times D_n,
\end{equation}
where each $D_k$ is the set of categories for a single variable.
An edit $e$ can be written as a subset of $D$:
\begin{equation}
e = A_1\times A_2\times\cdots\times A_n,
\end{equation}
where each $A_k\subset D_k$. The interpretation is that if a record ${\bf v}\in e$,
then that record is invalid.

Here, we reproduce example 9.3 of \cite{waal:2011} [first published by
\citep{kartika:2001}].  Consider four categorical variables with
domains $D_1=\{{\rm a},{\rm b},{\rm c},{\rm d}\}$, $D_2=D_3=\{{\rm a},{\rm
b},{\rm c}\}$ and $D_4=\{{\rm a},{\rm b}\}$. We define the edit rules
\begin{eqnarray}
    e_1&=& D_1\times \{ {\rm c}\}\times\{{\rm a},{\rm b}\}\times\{{\rm a}\} \\
    e_2&=& D_1\times\{{\rm b},{\rm c}\}\times D_3\times\{{\rm b}\} \\
    e_3&=& \{{\rm a},{\rm b},{\rm d}\}\times\{{\rm a},{\rm c}\}\times\{{\rm b},{\rm c}\}\times D_4\\
    e_4&=& \{{\rm c}\}\times D_2 \times \{{\rm b},{\rm c}\} \times \{{\rm a}\}.
\end{eqnarray}
Out of 72 possible records, only the following 20 are valid:
\begin{equation*}
    \begin{array}{cccc}
        ({\rm a,a,a,a})&({\rm b,a,a,a})&({\rm c,a,a,a})&({\rm d,a,a,a})\\
        ({\rm a,a,a,b})&({\rm b,a,a,b})&({\rm c,a,a,b})&({\rm d,a,a,b})\\
        ({\rm a,b,a,a})&({\rm b,b,a,a})&({\rm c,a,b,b})&({\rm d,b,a,a})\\
        ({\rm a,b,b,a})&({\rm b,b,b,a})&({\rm c,a,c,b})&({\rm d,b,b,a})\\
        ({\rm a,b,c,a})&({\rm b,b,c,a})&({\rm c,b,a,a})&({\rm d,b,c,a}).
    \end{array}
\end{equation*}
Figure \ref{RdeductiveLevels} shows how these rules can be defined in {\sf R}
using the {\sf editarray} function of the editrules package. Consider the
record ({\rm c},{\rm b},{\sf NA},{\sf NA}).  By simply considering the list of
valid records above it is clear that if $v_1$ and $v_2$ are assumed correct,
the only possible valid imputation is $v_3=v_4={\rm a}$.  Indeed this is
returned by {\sf deduImpute} in Figure \ref{RdeductiveLevels}. The record
$({\sf NA},{\sf NA},{\sf NA},{\rm b})$ cannot be imputed completely, since
there are six possible records with $v_4={\rm b}$. However, all of them have
$v_2={\rm a}$, so this may be imputed with certainly. Finally, the record
$({\rm b},{\rm c},{\rm a},{\sf NA})$ cannot be imputed since there is no valid
record with these values for $v_1$, $v_2$ and $v_3$. 

\begin{Rcode}
<<keep.source=TRUE>>=
M <- editarray(c(
"v1 %in% letters[1:4]",
"v2 %in% letters[1:3]",
"v3 %in% letters[1:3]",
"v4 %in% letters[1:2]",
"if (v2 == 'c'  & v3 != 'c' & v4 == 'a' ) FALSE",
"if (v2 != 'a'  & v4 == 'b') FALSE",
"if (v1 != 'c'  & v2 != 'b' & v3 != 'a') FALSE",
"if (v1 == 'c'  & v3 != 'a' & v4 == 'a' ) FALSE"
))
Mdat <- data.frame(
    v1 = c('c', NA,'b'),
    v2 = c('b', NA,'c'),
    v3 = c( NA, NA,'a'),
    v4 = c( NA,'b', NA),
    stringsAsFactors=FALSE
)
s <- deduImpute(M, Mdat)
s$corrected
s$status
s$corrections
@
\ucaption{Deductive imputations for categorical data using {\sf deduImpute}.}
\label{RdeductiveLevels}
\end{Rcode}


\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% DEDUCTIVE IMPUTATION OF NUMERICAL DATA
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Deductive imputation of numerical data}
\label{sdeductivenumerical}

The valid value combinations of numerical data records with $n$ variables are
usually limited to some subset of $\mathbb{R}^n$. Common cases include balance
accounts (linear restrictions) combined with linear inequality rules
(positivity rules for example). In such cases the set of valid records is a
convex polytope. In certain cases, when the values for a
number of variables have been fixed, the set of possible values for a number of
the remaining variables reduces to a point. In such cases deductive imputation
is possible.

\subsection{Imputation with {\sf solSpace} and {\sf imputess}}
\label{sssolspace}
\subsubsection{Area of application}
The combination of functions {\sf solSpace} and {\sf imputess} can be used
to impute numerical data under linear equality restrictions:
\begin{equation}
{\bf Ax} = {\bf b}\textrm{, with } {\bf A}\in\mathbb{R}^{m\times n},
\quad{\bf x}\in\mathbb{R}^n\textrm{ and }
{\bf b}\in\mathbb{R}^m.
\label{solspaceImputation}
\end{equation}
If {\bf x} has missing values, then {\sf solSpace} returns a representation
of the linear space of imputations valid under Eqn.\
\eqref{solspaceImputation}.  The function {\sf imputess} performs the actual
imputation. It is important to note that these functions do not take into
account the presence of any inequality restrictions.

\subsubsection{How it works}
\label{ssssolSpace}
Consider a numerical record $\bf x$ with $n_{miss}$ values missing.  
The values may be missing because of nonresponse, or they may be 
deemed missing by an error localization procedure (see the next subsection).
We will
write ${\bf x}=({\bf x}_{\rm obs},{\bf x}_{\rm miss})$, with ${\bf x}_{\rm
obs}$ the observed values and ${\bf x}_{\rm miss}$ the missing ones.  Supposing
further that $\bf x$ must obey a set of equality restrictions as in Eqn.
\eqref{solspaceImputation}, we may write
${\bf A}=[{\bf A}_{\rm obs},{\bf A}_{\rm miss}]$. Consequently we have \citep{waal:2011}
\begin{equation}
    {\bf A}_{\rm miss}{\bf x}_{\rm miss} = {\bf b} - {\bf A}_{\rm obs}{\bf x}_{\rm obs}.
\end{equation}
This gives
\begin{equation}
    {\bf x}_{\rm miss} = {\bf x}_0 + {\bf C}{\bf z},
\label{eqimpnum}
\end{equation}
with {\bf z} an arbitrary real vector of dimension $n_{\rm miss}$ and 
${\bf x}_0$ and {\bf C} constant.

The purpose of {\sf solSpace} is to compute ${\bf x}_0$ and ${\bf C}$.
Together they determine the vector space of values available for ${\bf x}_{\sf
miss}$.  Deductive imputation can be realized by observing that if any rows of
${\bf C}$ are filled with zeros, then the sole value for the corresponding
values of ${\bf x}_{\sf miss}$ are given the corresponding values in ${\bf
x}_0$. The values of ${\bf x}_0$ and ${\bf C}$ are given by
\begin{eqnarray}
    {\bf x}_0 &=& {\bf A}^+_{\rm miss}({\bf b}-{\bf A}_{\rm obs}{\bf x}_{\rm obs})\\
    {\bf C} &=& {\bf A}^+_{\rm miss}{\bf A}_{\rm miss} - \boldsymbol{1}.
\end{eqnarray}
Here, $\boldsymbol{1}$ is the identity matrix and ${\bf A}^+_{\rm miss}$ is the
pseudoinverse of {\bf A}, obeying
\begin{equation}
{\bf A}_{\rm miss}{\bf A}^+_{\rm miss}{\bf A}_{\rm miss}={\bf A}_{\rm miss}.
\end{equation}
See \citet{waal:2011} for details on the imputation method or \citet{greville:1959} for
an excellent discussion on the pseudoinverse.

\subsubsection{An example}
The {\sf solSpace} function returns the ${\bf x}_0$ and ${\bf C}$ as a
list. For example consider the first record from Figure \ref{RdeduImpute}:
<<echo=TRUE>>=
(x <- dat[1,])
@
Using the editmatrix defined in the same figure, we get:
<<keep.source=TRUE>>=
(s <- solSpace(E,x))
@
{\sf solSpace} has an extra argument {\sf adapt} which allows extra fields of
${\bf x}$ to be considered missing. An example of its use would be to determine
erroneous fields with {\sf errorLocalizer} (of the {\sf editrules} package) and
to determine the imputation space with {\sf solSpace}.

The top two and bottom two rows of {\bf C} in the example have zero
coefficients, yielding a unique solution for $x_2$, $x_4$, $x_9$ and $x_{11}$.
The unique values may be imputed with {\sf imputess}:
<<keep.source=TRUE>>=
imputess(x, s$x0, s$C)
@
If a ${\bf z}$-vector is provided as well [See Eq. \eqref{eqimpnum}], all values may be imputed. Here, we
choose ${\bf z}=\boldsymbol{0}$ (arbitrarily).
<<keep.source=TRUE>>=
( y <- imputess(x, s$x0, s$C, z=rep(0,ncol(s$C))) )
@
Using {\sf violatedEdits} from the editrules package, we may verify that this record satisfies every inequality
rule as well ({\sf E} as in figure \ref{RdeduImpute}).
<<keep.source=TRUE>>=
any(violatedEdits(E,y,tol=1e-8))
@

To demonstrate the use of the {\sf adapt} argument, consider the following case.
<<>>=
Ey <- editmatrix(c(
    "yt == y1 + y2 + y3",
    "y4 == 0"))
y <- c(yt=10, y1=NA, y2=3, y3=7,y4=12)
@
%Here, $y_4$ clearly violates the second rule. (A rule like this may arise when manipulating
%edit sets or pre-substituting a number of values). Since $y_4$ is not empty, {\sf solSpace}
%ignores it unless told otherwise.
<<>>=
(s <- solSpace(Ey,y))
@
However, using the {\sf adapt} argument, which is a logical indicator stating which
entries may be adapted, we get the following.
<<>>=
(s <- solSpace(Ey, y, adapt=c(FALSE,FALSE,FALSE,FALSE,TRUE)))
imputess(y,x0=s$x0,C=s$C)
@


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Imputation with {\sf deductiveZeros}}
\label{ssdeductivezeros}
\subsubsection{Area of application}
This method can be used to impute missing values in numerical records
subject to
\begin{eqnarray}
{\bf Ax} &=& {\bf b}\textrm{, with } {\bf A}\in\mathbb{R}^{m\times n},
\quad{\bf x}\in\mathbb{R}^n\textrm{ and }
{\bf b}\in\mathbb{R}^m
\\
{x}_j &\geq& 0 \textrm{ for at least one }j\in\{1,2,\ldots,n\}.
\label{deductiveZeroImputation}
\end{eqnarray}
Economic  survey data are often subject to account balances of the form $x_t = x_1 +
x_2+\cdots x_k$. For example, $x_t$ might be the total personnel cost and the
$x_i$ are costs related to permanent staff, temporary staff, externals, {\em
etc}. It is not uncommon for respondents to leave fields open which are not
relevant to them. For example, if a company has not hired any temporary staff,
the corresponding field might be left empty while a 0 would have been
appropriate.  

In such cases, missing values are bounded from above by the sum rules while
they are bounded from below by the nonnegativity constraint. If the missing
values are ignored, and the completed values add up to the required totals,
then missing values may be uniquely imputed with 0. The function {\sf
deductiveZeros} detects such cases.

\subsubsection{How it works}
Consider again the notation of Section \ref{ssssolSpace}. We write (following
notation of \cite{waal:2011}).
\begin{equation}
{\bf b}^* = {\bf b} - {\bf A}_{\rm obs}{\bf x}_{\rm obs}.
\end{equation}
If any $b^*_l=0$, this means that the sum rule ${\bf a}_l\cdot {\bf x}=b_l$ is
obeyed if missing values are ignored. For those cases, the following properties are
checked.
\begin{itemize}
\item Each $a_{{\rm miss},lj}$ has the same sign.
\item Each $a_{{\rm miss},lj}\not=0$ corresponds to a variable $x_j$ that is constrained to be nonnegative.
\end{itemize}
If these demands are obeyed, the corresponding value $x_{{\rm miss},j}$ may be imputed with 0.

\subsubsection{An example}
The function {\sf deductiveZeros} does not perform imputation itself but computes
an indicator stating which values may be imputed. As a first example consider
the following.
<<keep.source=TRUE>>=
Ey <- editmatrix(c(
    "yt == y1 + y2 + y3",
    "y1 >= 0", "y2 >= 0 ","y3 >= 0"))
y <- c(yt=10, y1=NA, y2=3, y3=7)
(I<-deductiveZeros(Ey,y))
@
The record $y$ can be imputed in one statement.
<<keep.source=TRUE>>=
y[I] <- 0
y
@


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% DEDUCTIVE IMPUTATION OF CATEGORICAL DATA
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Deductive Imputation of categorical data}
\label{sdeductivecategorical}

As shown in Eq. \eqref{eqdomain}, a categorical data record is a member of a
discrete set of value combinations $D$ (the domain). In practice, not every
record in $D$ may be acceptable. For example if 
\begin{equation}
D = \{\textrm{child},{\rm adult}\}\times \{{\rm married},{\rm unmarried}\},
\end{equation}
then the record $(\textrm{child},{\rm married})$ may be excluded from the set
of valid records. Therefore, if we have a record with $({\sf NA},{\rm
married})$, and assume that the marital status is correct, there is only one
possible value for the age class, namely ``adult''. So just like for numerical
data, if the known values limit the number of options for the unknowns to a
unique value, deductive imputation is possible.


\subsection{Imputation with {\sf deductiveLevels}}
\subsubsection{Area of application}
The function {\sf deductiveLevels} works on purely categorical data where the
number of categories for each variable is known and fixed, as in Eq.\
\eqref{eqdomain}.  It determines which missing values in a record are
determined uniquely by the known values, and these unique values are returned.

\subsubsection{How it works}
The algorithm behind {\sf deductiveLevels} has been described by
\citet{waal:2011} and is reproduced here in Algorithm \ref{algDeductiveLevels}.
The Algorithm is described in terms of the functions {\sf eliminate} and {\sf
substValue}, both of which are implemented in the {\sf editrules} package and
have been described extensively by \cite{loo:2011c}.  In short, {\sf
deductiveLevels} derives deductive imputations by first substituting all
observed values in the edit rules. Subsequently, all variables but one are
eliminated from the remaining edits. If only one possible value remains for the
remaining variable, it may be used as a deductive imputation and substituted in
the set of edits. This process is repeated until all missing values are
treated.

\begin{algorithm}[!t]
    \caption{{\sc deductiveLevels($E$,{\bf v})}}
    \label{algDeductiveLevels}
\begin{algorithmic}
\Require {An {\sf editarray} $E$, a partially complete record ${\bf v}$}
\State Determine the index $I\subset\{1,2,\ldots n\}$ in ${\bf v}$ of observed values.
\State $E\leftarrow\textrm{\sc substValue}(E,I,{\bf v}_{I})$
\If {$\lnot\textrm{\sc isFeasible}(E)$}
\State \Return $\varnothing$
\EndIf
\State $M\leftarrow \{1,2,\ldots,n\}\backslash I$ \Comment{Index of missing values in $\bf v$}
\State $T\leftarrow \varnothing$
\State $S\leftarrow \varnothing$ 
\While {$M\backslash T\not=\varnothing$}
    \State $m\leftarrow M_1$
    \State $F\leftarrow E $
    \For {$k \in M\backslash  m$} \Comment{Eliminate all but $k$ from $F$}
        \State $F\leftarrow\textrm{\sc eliminate}(F,k)$
        \EndFor 
    \If {There is one possible value $\tilde{v}$ for variable $m$ in $F$}
    \State $E\leftarrow\textrm{\sc substValue}(E,m,\tilde{v})$
        \State $M\leftarrow M\backslash m$
        \State $S\leftarrow S\cup(m,\tilde{v})$
    \Else
        \State $T\leftarrow T\cup m$
    \EndIf
\EndWhile
\Ensure Unique imputations $S$.
\end{algorithmic}
\end{algorithm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{An example}
Consider the variables $v_1=${\em gender}, $v_2=${\em pregnant} and $v_3=${\em chromosome}.
The value domain and edit rules are given by
\begin{eqnarray}
D_1 &=& \{ \textrm{male, female} \}\\
D_2 &=& \{\true ,\false \}\\
D_3 &=& \{ \textrm{xx, xy} \}\\
e_1 &=& \{ {\rm male} \}\times\{ \true \}\times D_3\\
e_2 &=& \{ {\rm male} \}\times D_2 \times \{ {\rm xx} \}.
\end{eqnarray}
The corresponding {\sf editarray} can be defined as follows.
<<keep.source=true>>=
E <- editarray(c(
    "gender     %in% c('male','female')",
    "pregnant   %in% c(TRUE,FALSE)",
    "chromosome %in% c('XX','XY')",
    "if (gender == 'male') !pregnant",
    "if (gender == 'male') chromosome == 'XY'"))
@
Now, consider the record $({\rm male},\false,{\sf NA})$. Using {\sf deductiveLevels} we find:
<<keep.source=true>>=
v <- c(gender='male',pregnant=FALSE,chromosome=NA)
(s <- deductiveLevels(E,v))
@
And imputation can be performed as follows:
<<keep.source=true>>=
v[names(s)] <- s
v
@
The {\sf deductiveLevels} function has an optional argument, allowing to switch
off the feasibility check. To illustrate this, consider the record $({\rm
male},\true,{\sf NA})$.  Clearly, there is no way to impute this record
consistently by just imputing the {\em chromosome} variable. If we choose
$v_3={\rm XX}$, this conflicts with the gender (male) if we choose ${\rm XY}$
this conflicts with the gender implied by $v_2$ (pregnant). In this case
{\sf deductiveLevels} returns {\sf NULL}.
<<keep.source=true>>=
v <- c(gender='male',pregnant=TRUE,chromosome=NA)
deductiveLevels(E,v)
@
The reason is that {\sf deductiveLevels} checks if feasible imputations are
possible after substituting all observed values into the edits. This
check can be time-consuming since it potentially involves many variable
elimination steps. It may be turned off by passing {\sf checkFeasibility=FALSE}:
<<keep.source=true>>=
deductiveLevels(E,v,checkFeasibility=FALSE)
@
However, one must be careful since, as shown above, the result may be an
inconsistent imputation. The reason to include this option is that users
may provide an additional parameter, called {\sf adapt}, allowing {\sf deductiveLevels}
to impute more variables. If the {\sf adapt} parameter is chosen such that 
missing values plus adaptable values can lead to consistent imputation, the 
consistency check may be turned off. For example, we may choose to adapt 
the pregnancy status.
<<keep.source=true>>=
adapt <- c(gender=FALSE,pregnant=TRUE,chromosome=TRUE)
(s <- deductiveLevels(E,v,adapt=adapt,checkFeasibility=FALSE))
@
So that the imputed value becomes
<<keep.source=true>>=
v[names(s)] <- s
v
@
which is indeed a valid record. In general, the {\sf adapt} parameter should be
derived via a consistent error localization mechanism, such as implemented in
the {\sf editrules} package. Only those cases it is safe to gain some performance
by switching the feasibility check off.

\section{Conclusions}
Missing values and inconsistencies in raw data often hinder statistical
analyses.  However, in many cases, correct values can be derived using only the
available values and consistency rules imposed on the data (deductive
imputation).  As of version {\sf 1.1}, the deducorrect package includes
functionality for deductive imputation of both numerical and categorical data.
The functionality of the {\sf deducorrect} package can be used as-is, or may be
integrated with the error localization functionality of the {\sf editrules}
package.

Future work on the package will include  several performance enhancements and
visualisation options.













\clearpage
\bibliographystyle{chicago}
\addcontentsline{toc}{section}{References}
\bibliography{deducorrect}
\end{document}
