<!--
%\VignetteEngine{knitr}
%\VignetteIndexEntry{An Introduction to qdap}
-->

```{r setup, include=FALSE}
# set global chunk options
library(knitr); library(tm); library(qdap); #library(reports)
opts_chunk$set(cache=FALSE, tidy=FALSE, warning=FALSE)

opts_knit$set(upload.fun = image_uri, self.contained=TRUE)
#library(knitcitations); library(reports); library(qdapTools)
#bib <- read.bibtex(dir()[tools::file_ext(dir()) == "bib"][1])

source("funs/utils_functions.txt")
source("funs/extra_functions.txt")    
    
#  BU <- "http://trinker.github.io/qdap/" 
BU <- "http://trinker.github.io/qdap_dev/" #switch before upload
LN <- function(fun, base=BU) paste0(BU, fun, ".html")
FUN <- function(fun, fun2 = fun, base=BU) {
    HR2(LN(fun2), paste0("<code>", fun,"</code>"), copy2clip = FALSE)
}
BU2 <- "http://trinker.github.io/qdapDictionaries/" #switch before upload
LN2 <- function(fun, base=BU2) paste0(BU2, fun, ".html")
FUN2 <- function(fun, base=BU2) {
    HR2(LN2(fun), paste0("<code>", fun,"</code>"), copy2clip = FALSE)
}
yt <- function(URL) {
  paste0("<a href=\"", URL, "\" target=\"_blank\" style=\"text-decoration: none\"><b><font size=\"5\" color=\"#B22222\">[YT]</font></b></a>\n")
}
#cite in text using `r citet(bib[1])`
uri_embed <- function(path, add="") {
    uri <- knitr::image_uri(path)
    cat(paste0("<img ", add, " src=\"", uri, "\" />"))
}
```

# qdap Package Vignette
# Tyler Rinker

qdap (<a href="http://github.com/trinker/qdap">Rinker, 2013</a>) is an R package designed to assist in quantitative discourse analysis. The package stands as a bridge between qualitative transcripts of dialogue and statistical analysis and visualization.  qdap was born out of a frustration with current discourse analysis programs. Packaged programs are a closed system, meaning the researcher using the method has little, if any, influence on the program applied to her data.

R already has thousands of excellent packages for statistics and visualization. qdap is designed to stand as a bridge between the qualitative discourse of a transcript and the computational power and freedom that R offers. As qdap returns the power to the researcher it will also allow the researcher to be more efficient and thus effective and productive in data analysis.  The qdap package provides researchers with the tools to analyze data and more importantly is a dynamic system governed by the data, shaped by theory, and continuously refined by the field.

...if you can dream up an analysis then qdap and R can help get you there.

```{r, echo=FALSE, results='asis'}
uri_embed("imgs/qdaplogo.png", 
    "width=\"350\", height=\"250\" style=\"display:block; margin-left:auto; margin-right:auto;\"")
```

The following vignette is a loose chronological road map for utilizing the tools provided by qdap.  

<hr>
<h3 id="toc">Select from sections below:</h3>

<div style="float: left; width: 50%;">
<ul>
<div>1.  `r HR("#project", "Starting a New Project")`    </div> 
<div>2.  `r HR("#import_export", "Import/Export Discourse Data")`    </div> 
<div>3.  `r HR("#viewing", "View the Data")`    </div> 
<div>4.  `r HR("#tools", "Generic qdap Tools")`    </div> 
<div>5.  `r HR("#cleaning", "Cleaning/Preparing the Data")`    </div> 
<div>6.  `r HR("#reshaping", "Reshaping the Data")`    </div> 
<div>7.  `r HR("#word", "Extract Words")`    </div> 
<div>8.  `r HR("#coding", "Qualitative Coding System")`    </div> 
<div>9.  `r HR("#counts", "Word Counts and Descriptive Statistics")`    </div> 
<div>10.  `r HR("#measures", "Word Measures and Scoring")`    </div> 
<div>11.  `r HR("#visualization", "Visualizing Discourse Data")`    </div> 
<div>12.  `r HR("#id", "ID Sentences")`    </div> 
<div>13.  `r HR("#tm", "tm Package Compatability")`    </div> 
<div>14.  `r HR("#data", "Data Sets")`    </div> 
<div>15.  `r HR("#dict", "Dictionaries and Word Lists")`    </div>   
<div>16.  `r HR("#install", "Installation Issues")`    </div>     
<div>17.  `r HR("#connect", "Recommended Packages (Extending qdap)")`    </div>   

</ul>
</div>
<div style="float: right; width: 50%;">
<ul>
<div><b>Symbol Conventions:</b></div>  
<div>`r FT(orange, 5, text="&diams;")` = Example (R code)    </div> 
<div><b>`r FT(firebrick, 5, text="[YT]")`</b> = Video Demo (click to watch)    </div> 
</ul>
</div>
<br style="clear:both;"/>

```{r, echo=FALSE, eval=FALSE}
## library(qdap)
## dat <- data.frame(
##     x = c("project", "import_export", "tools", "cleaning", "viewing", 
##         "reshaping", "word", "coding", "counts", "measures", "visualization", 
##         "id", "data", "dict", "install"),
##     
##     y = c("Starting a New Project", "Import/Export Discourse Data", 
##         "Generic qdap Tools", "Cleaning/Preparing the Data", "View the Data", 
##         "Reshaping the Data", "Extract/Analyze Words", "Qualitative Coding System", 
##         "Word Counts and Descriptive Statistics", "Word Measures and Scoring", 
##         "Visualizing Discourse Data", "ID Sentences", 
##         "Data Sets", "Dictionaries and Word Lists", "Installation Issues")
## )
## 
## FUN <- function(x, y) {
##     cat("\n\n")
##     m <- paste0("<div>", 1:length(x), ".  `r HR(\"#", x, "\", \"", y, "\")`    </div> ")
##     cat(paste(m, collapse="\n")); cat("\n")
##     cat("\n\n")
##     n <- paste0("<h3 id=\"", x, "\">", y, "</h3>")
##     cat(paste(n, collapse="\n")); cat("\n")
## }
## 
## FUN(dat[, 1], dat[, 2])
## 
## path <- "C:/Users/trinker/GitHub/trinker.github.com/qdap_dev"
## #  path <- "C:/Users/trinker/GitHub/trinker.github.com/qdap"
## URL <- "http://trinker.github.io/qdap_dev/"
## #  url <- "http://trinker.github.io/qdap"
## 
## inds <- readLines(file.path(path, "index.html"))
## h3s <- grep("<h3", inds)
## h2s <- grep("<h2", inds)
## 
## inds <- inds[head(h3s, 1):(tail(h2s, 1) - 1)]
## inds <- inds[12: tail(grep("</ul>", inds), 1)]
## h3s <- grep("<h3", inds)
## dat2 <- data.frame(start = h3s + 4, end = c(tail(h3s, -1) - 1, length(inds)))
## 
## inds <- substring(inds, 5)
## 
## 
## 
## invisible(lapply(1:nrow(dat2), function(i) {
##     rws <- inds[dat2[i, 1]:dat2[i, 2]]
##     
##     funs <- unlist(genXtract(rws, ".html\">", "</a>"))
##     descripts <- unlist(genXtract(rws, "<br />", "</li>"))
##     
##     rws <- rws[grepl("<code>", rws)]
##     rws <- paste0("<form action=\"", file.path(URL, paste0(funs, ".html")), " target=\"_blank\" \">
##     <input type=\"submit\" value=\"", funs, "\"> - ", descripts, "\n</form>", "\n") 
##     
##     
##     cat(paste0("============\nfun group", i, "\n============\n"))
##     cat(paste0("The following functions will be utilized in this section (click to view more):    \n\n"))
##     cat(paste(rws, collapse = "\n")); cat("\n")
## }))


```

<h3 id="project">Starting a New Project `r yt("http://youtu.be/u8AJiyMffmc")`</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/new_project.html" target="_blank">
    <input type="submit" value="new_project"> - Project Template
</form>
</div>


The function `r FUN("new_project")` is designed to generate project template of multiple nested directories that organize and guide the researcher through a qualitative study, from data collection to analysis and report/presentation generation.  This workflow framework will enable the researcher to be better organized and more efficient in all stages of the research process.  `r FUN("new_project")` utilizes the `r HR2("http://cran.r-project.org/web/packages/reports/reports.pdf", "reports package")` (<a href="http://github.com/trinker/reports">Rinker, 2013b</a>) 

Please see the following links for PDF descriptions of the contents of the `r FUN("new_project")` and the reports directory. `r VS(2)`

<div style="text-align: center;">
<table width="30%" style="text-align: center;margin: 0px auto;">
<colgroup>
<col width="110" />
<col width="110" />
</colgroup>
<tr>
<tr style="text-align: center;">
<td style="text-align: center;">Project<br> Workflow</td>
<td style="text-align: center;">Report<br> Workflow</td>
</tr>
<tr>
<td style="text-align: center; onClick="document.location.href='https://copy.com/4VekuLlUqix0CfSw/PROJECT_WORKFLOW_GUIDE.pdf?download=1';">
<a href="https://copy.com/4VekuLlUqix0CfSw/PROJECT_WORKFLOW_GUIDE.pdf?download=1';"><img src="http://drupal.org/files/project-images/Download%20Views%20PDF_2.png" width="50" height="75"><br></a>
<a href="https://copy.com/4VekuLlUqix0CfSw/PROJECT_WORKFLOW_GUIDE.pdf?download=1" target="_blank">click here</a>
<td style="text-align: center; onClick="https://copy.com/csVvdAm2vikGlkIU/REPORT_WORKFLOW_GUIDE.pdf?download=1';">
<p><a href="https://copy.com/csVvdAm2vikGlkIU/REPORT_WORKFLOW_GUIDE.pdf?download=1';"  target="_blank"><img src="http://drupal.org/files/project-images/Download%20Views%20PDF_2.png" width="50" height="75"><br></a>
<a href="https://copy.com/csVvdAm2vikGlkIU/REPORT_WORKFLOW_GUIDE.pdf?download=1" target="_blank">click here</a></p></td>
</tr>
</table>
</div>

<h4 id="extra">extra_functions `r yt("http://youtu.be/yuFyz7IW0Us")`</h4>    
The `r FUN("new_project")` template is designed to be utilized with `r HR2("http://www.rstudio.com/ide/download/", "RStudio")`.  Upon clicking the `xxx.Rproj` file the template will be loaded into RStudio.  The .Rprofile script will be sourced upon start up, allowing the user to automatically load packages, functions, etc. related to the project.  The file `extra_functions.R` is sourced, loading custom functions.  Already included are two functions, `email` and `todo`, used to generate project member emails and track project tasks.  This auto sourcing greatly enhances efficiency in workflow.


<h3 id="import_export">Import/Export Discourse Data</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/condense.html" target="_blank">
    <input type="submit" value="condense"> - `r HR("#mcsv", "Condense Dataframe Columns")`
</form>

<form action="http://trinker.github.io/qdap_dev/dir_map.html" target="_blank">
    <input type="submit" value="dir_map"> - `r HR("#readin", "Map Transcript Files from a Directory to a Script")`
</form>

<form action="http://trinker.github.io/qdap_dev/mcsv_r.html" target="_blank">
    <input type="submit" value="mcsv_r"><input type="submit" value="mcsv_w"> - `r HR("#mcsv", "Read/Write Multiple csv Files at a Time")`
</form>

<form action="http://trinker.github.io/qdap_dev/read.transcript.html" target="_blank">
    <input type="submit" value="read.transcript"> - `r HR("#readin", "Read Transcripts Into R")`
</form>
</div>

<h4 id="readin">Reading In Transcript Data `r yt("http://youtu.be/UxgOScggLBg")`</h4>    

This subsection covers how to read in transcript data.  Generally the researcher will have data stored as a .docx (Microsoft Word or Open/Libre Office) or .xlsx/.csv (spreadsheet format).  It is of great importance that the researcher manually writes/parses their transcripts to avoid potential analysis problems later.  All sentences should contain appropriate qdap punctuation (declarative = ., interrogative = ?, exclamatory = !, interrupted = | or `r FUN("imperative")` = *., *?, *!, *|).  Additionally, if a sentence contains an end mark/punctuation it should have accompanying text/dialogue.  Two functions are useful for reading in data, `r FUN("read.transcript")` and `r FUN("dir_map")`.  `r FUN("read.transcript")` detects file type (.docx/.csv/.xlsx) and reads in a single transcript whereas `r FUN("dir_map")` generates code that utilizes `r FUN("read.transcript")` for each of the multiple transcripts in a single directory.  Note that `r FUN("read.transcript")` expects a two column formatted transcript (usually with person on the left and dialogue on the right).

Five arguments are of particular importance to read.transcript: 

<table>
<tr><td><code>file</code></td>
<td><p>The name of the file which the data are to be
read from. Each row of the table appears as one line of
the file. If it does not contain an absolute path, the
file name is relative to the current working directory,
<code>getwd()</code>.</p></td></tr>
<tr><td><code>col.names</code></td>
<td>
<p>A character vector specifying the column
names of the transcript columns.</p>
</td></tr>
<tr><td><code>header</code></td>
<td>
<p>logical.  If <code>TRUE</code> the file contains
the names of the variables as its first line.</p>
</td></tr>
<tr><td><code>sep</code></td>
<td>
<p>The field separator character. Values on each
line of the file are separated by this character.  The
default of <code>NULL</code> instructs
<code><a href="read.transcript.html">read.transcript</a></code> to use a separator
suitable for the file type being read in.</p>
</td></tr>
<tr><td><code>skip</code></td>
<td>
<p>Integer; the number of lines of the data file
to skip before beginning to read data.</p>
</td></tr>
</table>

Often transcripts contain extraneous material at the top and the argument `r CN("skip = ?")` must be used to skip these extra lines.  Some sort of unique separator must also be used to separate the person column from the text column.  By default `r CN('sep = ":"')` is assumed.  If your transcripts do not contain a separator one must be inserted manually.  Also note that the researcher may want to prepare the transcripts with brackets to denote non spoken annotations as well dialogue that is read rather than spoken.  For more on bracket parsing see `r HR("#bracket", "Bracket/General Chunk Extraction")`.

<div class="middleDiv">
<b>`r FT(red, 4, text="Note: It is important that all sentences contain valid qdap punctuation (<font face=\"courier\">.</font>, <font face=\"courier\">?</font>, <font face=\"courier\">!</font>, <font face=\"courier\">|</font>) in your transcripts.  Many qdap functions are dependent upon this assumption.")`</b>
</div>

`r FT(orange, 5, text="&diams;")` **Reading In Data**- *read.transcript* `r FT(orange, 5, text="&diams;")`
<pre><code class="r">## Location of sample transcripts from the qdap package
(doc1 <- system.file("extdata/transcripts/trans1.docx", package = "qdap"))
(doc2 <- system.file("extdata/transcripts/trans2.docx", package = "qdap"))
(doc3 <- system.file("extdata/transcripts/trans3.docx", package = "qdap"))
(doc4 <- system.file("extdata/transcripts/trans4.xlsx", package = "qdap"))</code></pre>

<pre><code class="r">dat1 <- read.transcript(doc1)
truncdf(dat1, 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it's time to learn. [Student di
## 3 Multiple Students        Yes teacher we're ready to learn.
## 4     [Cross Talk 3                                      00]
## 5         Teacher 4 Let's read this terrific book together. </code></pre>


<pre><code class="r">dat2 <- read.transcript(doc1, col.names = c("person", "dialogue"))
truncdf(dat2, 40)</code></pre>

<pre><code>##              person                                 dialogue
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it's time to learn. [Student di
## 3 Multiple Students        Yes teacher we're ready to learn.
## 4     [Cross Talk 3                                      00]
## 5         Teacher 4 Let's read this terrific book together. </code></pre>

<pre><code class="r">dat2b <- rm_row(dat2, "person", "[C") #remove bracket row
truncdf(dat2b, 40)</code></pre>

<pre><code>##              person                                 dialogue
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it's time to learn. [Student di
## 3 Multiple Students        Yes teacher we're ready to learn.
## 4         Teacher 4 Let's read this terrific book together. </code></pre>


<pre><code class="r">## Be aware of the need to `skip` non transcript lines
## Incorrect read; Needed to use `skip`
read.transcript(doc2)</code></pre>

<pre><code>Error in data.frame(X1 = speaker, X2 = pvalues, stringsAsFactors = FALSE) : 
  arguments imply differing number of rows: 7, 8</code></pre>


<pre><code class="r">## Correct: Used `skip`
dat3 <- read.transcript(doc2, skip = 1)
truncdf(dat3, 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it's time to learn. [Student di
## 3 Multiple Students        Yes teacher we're ready to learn.
## 4     [Cross Talk 3                                      00]
## 5         Teacher 4 Let's read this terrific book together. 
</code></pre>

<pre><code class="r">## Be Aware of the `sep` Used
## Incorrect Read; Wrong `sep` Provided (used default `:`)
read.transcript(doc3, skip = 1)</code></pre>

<pre><code>##Dialogue and Person Columns Mixed Inappropriately
## X1
## 1 [Cross Talk 3
##                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              X2
## 1 Teacher 4-Students it's time to learn. [Student discussion; unintelligible] Multiple Students-Yes teacher we're ready to learn. 00] Teacher 4-Let's read this terrific book together. It's called Moo Baa La La La and what was I going to ... Oh yes The story is by Sandra Boynton. A cow says Moo. A Sheep says Baa. Three singing pigs say LA LA LA! "No, no!" you say, that isn't right. The pigs say oink all day and night. Rhinoceroses snort and snuff. And little dogs go ruff ruff ruff! Some other dogs go bow wow wow! And cats and kittens say Meow! Quack! Says the duck. A horse says neigh. It's quiet now. What do you say?
</code></pre>


<pre><code class="r">## Correct `sep` Used
dat4 <- read.transcript(doc3, sep = "-", skip = 1)
truncdf(dat4, 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1         Teacher 4 Students it's time to learn. [Student di
## 2 Multiple Students Yes teacher we're ready to learn. [Cross
## 3         Teacher 4 Let's read this terrific book together. </code></pre>


<pre><code class="r">## Read In .xlsx Data
dat5 <- read.transcript(doc4)
truncdf(dat5, 40)</code></pre>

<pre><code>##                   V1                                       V2
## 1      Researcher 2:                         October 7, 1892.
## 2         Teacher 4:             Students it's time to learn.
## 3               <NA>                                     <NA>
## 4 Multiple Students:        Yes teacher we're ready to learn.
## 5               <NA>                                     <NA>
## 6         Teacher 4: Let's read this terrific book together. 
</code></pre>

<pre><code class="r">## Reading In Text
trans <- "sam: Computer is fun. Not too fun.
greg: No it's not, it's dumb.
teacher: What should we do?
sam: You liar, it stinks!"

read.transcript(text=trans)</code></pre>

<pre><code>##        V1                            V2
## 1     sam Computer is fun. Not too fun.
## 2    greg         No its not, its dumb.
## 3 teacher            What should we do?
## 4     sam          You liar, it stinks!
</code></pre>

The `r FUN("dir_map")` function enables the researcher to produce multiple lines of code, one line with `r FUN("read.transcript")` for each file in a directory, which is then optionally copied to the clipboard for easy insertion into a script.  Note that setting the argument `r CN("use.path = FALSE")` may allow the code to be more portable in that a static path is not supplied to the `r FUN("read.transcript")` scripts.

`r FT(orange, 5, text="&diams;")` **Reading In Data**- *dir_map* `r FT(orange, 5, text="&diams;")`

<pre><code class="r">(DIR <- system.file("extdata/transcripts", package = "qdap"))
dir_map(DIR)</code></pre>

...will produce...

<pre><code>dat1 <- read.transcript('~/extdata/transcripts/trans1.docx', col.names = c('person', 'dialogue'), skip = 0)
dat2 <- read.transcript('~/extdata/transcripts/trans2.docx', col.names = c('person', 'dialogue'), skip = 0)
dat3 <- read.transcript('~/extdata/transcripts/trans3.docx', col.names = c('person', 'dialogue'), skip = 0)
dat4 <- read.transcript('~/extdata/transcripts/trans4.xlsx', col.names = c('person', 'dialogue'), skip = 0)</code></pre>


<h4 id="mcsv">Reading/Writing Multiple .csv Files `r yt("http://youtu.be/aeZKJGEfD7U")`</h4>    

The `r CN("mcsv_x")` family of functions are utilized to read (`r FUN("mcsv_r")`) and write (`r FUN("mcsv_w")`) multiple csv files at once.  `r FUN("mcsv_w")` takes an arbitrary number of dataframes and outputs them to the supplied directory( `r CN("dir = ?")`).  An attempt will be made to output the dataframes from qdap functions that output lists of dataframes.  Note that dataframes that contain columns that are lists must be condensed prior to writing with other R dataframe writing functions (e.g., `write.csv`) using the `r FUN("condense")` function.  By default `r FUN("mcsv_w")` attempts to utilize `r FUN("condense")`.

The `r FUN("mcsv_r")` function reads multiple files at once and then assigns then dataframes to identically named objects (minus the file extension) in the global environment.  Additionally, all of the dataframes that are read in are also assigned to an inclusive list (name `L1` by default).

`r FT(orange, 5, text="&diams;")` **Reading and Writing Multiple csvs** `r FT(orange, 5, text="&diams;")`

```{r, eval=FALSE}
## Make new minimal data sets
mtcarsb <- mtcars[1:5, ]; CO2b <- CO2[1:5, ]

## Write multiple csvs and assign the directory path to `a`
a <- mcsv_w(mtcarsb, CO2b, dir="foo")

## New data sets gone from .GlobalEnv
rm("mtcarsb", "CO2b")  

## View the files in `a` and assign to `nms`
(nms <- dir(a))

## Read in and notice the dataframes have been assigned in .GlobalEnv
mcsv_r(file.path(a, nms))
mtcarsb; CO2b
L1

## The dataframe names and list of dataframe can be altered
mcsv_r(file.path(a, nms), a.name = paste0("bot", 1:2), l.name = "bots_stink")
bot1; bot2
bots_stink

## Clean up
delete("foo")
```

`r FT(orange, 5, text="&diams;")` **Writing Lists of Dataframes to csvs** `r FT(orange, 5, text="&diams;")`
```{r, eval=FALSE}
## poldat and termco produce lists of dataframes
poldat <- with(DATA, polarity(state, person))
term <- c("the ", "she", " wh")
termdat <- with(raj.act.1,  termco(dialogue, person, term))

## View the lists of dataframes
str(poldat); str(termdat)

## Write the lists of dataframes to csv
mcsv_w(poldat, termdat, mtcars, CO2, dir="foo2")

## Clean up
delete("foo2")
```

<h3 id="viewing">View the Data</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/data_viewing.html" target="_blank">
    <input type="submit" value="truncdf"><input type="submit" value="htruncdf"><input type="submit" value="ltruncdf"><input type="submit" value="qview"> - `r HR("#trunc", "Truncated Dataframe Viewing")`
</form>

<form action="http://trinker.github.io/qdap_dev/data_viewing.html" target="_blank">
    <input type="submit" value="lview"> - `r HR("#unclass", "Unclass qdap Object to View List of Dataframes")`
</form>

<form action="http://trinker.github.io/qdap_dev/left_just.html" target="_blank">
    <input type="submit" value="left_just"><input type="submit" value="right_just"> - `r HR("#just", "Text Justification")`
</form>

<form action="http://trinker.github.io/qdap_dev/Search.html" target="_blank">
    <input type="submit" value="Search"> - `r HR("#search", "Search Columns of a Dataframe")`
</form>
</div>

The nature of dialogue data makes it large and cumbersome to view in R.  This section explores qdap tools designed for more comfortable viewing of R dialogue oriented text dataframes.  

<h4 id="trunc">Truncated Dataframe Viewing</h4> 

The <a href="http://trinker.github.io/qdap_dev/htruncdf.html" target="_blank"><code>_truncdf</code></a> family of functions (trunc + dataframe = `r FUN("truncdf", "data_viewing")`) are designed to truncate the width of columns and number of rows in dataframes and lists of dataframes.  The `r CN("l")` and `r CN("h")` in front of `r CN("trunc")` stands for <b><font color="blue">l</font>ist</b> and <b><font color="blue">h</font>ead</b> and are extensions of `r FUN("truncdf", "data_viewing")`.  `r FUN("qview", "data_viewing")` is a wrapper for `r FUN("htruncdf", "data_viewing")` that also displays number of rows, columns, and the dataframe name.


`r FT(orange, 5, text="&diams;")` **Truncated Data Viewing** `r FT(orange, 5, text="&diams;")`

```{r}
truncdf(raj[1:10, ])
truncdf(raj[1:10, ], 40)
htruncdf(raj)
htruncdf(raj, 20)
htruncdf(raj, ,20)
ltruncdf(rajPOS, width = 4)
```

<pre><code class="r">qview(raj)</code></pre>

<pre><code>## ========================================================================
## nrow =  840           ncol =  3             raj
## ========================================================================
##     person   dialogue act
## 1  Sampson Gregory, o   1
## 2  Gregory No, for th   1
## 3  Sampson I mean, an   1
## 4  Gregory Ay, while    1
## 5  Sampson I strike q   1
## 6  Gregory But thou a   1
## 7  Sampson A dog of t   1
## 8  Gregory To move is   1
## 9  Sampson A dog of t   1
## 10 Gregory That shows   1</code></pre>

<pre><code class="r">qview(CO2)</code></pre>

<pre><code>## ========================================================================
## nrow =  84           ncol =  5             CO2
## ========================================================================
##    Plant   Type  Treatment conc uptake
## 1    Qn1 Quebec nonchilled   95     16
## 2    Qn1 Quebec nonchilled  175   30.4
## 3    Qn1 Quebec nonchilled  250   34.8
## 4    Qn1 Quebec nonchilled  350   37.2
## 5    Qn1 Quebec nonchilled  500   35.3
## 6    Qn1 Quebec nonchilled  675   39.2
## 7    Qn1 Quebec nonchilled 1000   39.7
## 8    Qn2 Quebec nonchilled   95   13.6
## 9    Qn2 Quebec nonchilled  175   27.3
## 10   Qn2 Quebec nonchilled  250   37.1</code></pre>

<h4 id="unclass">Unclass qdap Object to View List of Dataframes</h4> 

Many qdap objects are lists that print as a single dataframe, though the rest of the objects in the list are available.  The `r FUN("lview", "data_viewing")` function unclasses the object and assigns "list".


```{r}
lview(question_type(DATA.SPLIT$state, DATA.SPLIT$person))
```

<h4 id="just">Text Justification</h4> 

By default text data (character vectors) are displayed as right justified in R.  This can be difficult and unnatural to read, particularly as the length of the sentences increase.  The `r FUN("left_just")` function creates a more natural left justification of text.  Note that `r FUN("left_just")` inserts spaces to achieve the justification. This could interfere with analysis and therefore the output from `r FUN("left_just")` should only be used for visualization purposes, not analysis.

`r FT(orange, 5, text="&diams;")` **Justified Data Viewing** `r FT(orange, 5, text="&diams;")`    

```{r}
## The unnatural state of R text data
DATA
## left just to the rescue
left_just(DATA)
## Left just select column(s)
left_just(DATA, c("sex", "state"))
left_just(CO2[1:15,])
right_just(left_just(CO2[1:15,]))
```

<h4 id="search">Search Columns of a Dataframe</h4> 

A task of many analyses is to search a dataframe for a particular phrase and return those rows/observations that contain that term.  The researcher may optionally choose to specify a particular column to search (`r CN("column.name")`) or search the entire dataframe.

`r FT(orange, 5, text="&diams;")` **Search Dataframes** `r FT(orange, 5, text="&diams;")`

```{r}
(SampDF <- data.frame("islands"=names(islands)[1:32],mtcars, row.names=NULL))
Search(SampDF, "Cuba", "islands")
Search(SampDF, "New", "islands")
Search(SampDF, "Ho")
Search(SampDF, "Ho", max.distance = 0)
Search(SampDF, "Axel Heiberg")
Search(SampDF, 19) #too much tolerance in max.distance
Search(SampDF, 19, max.distance = 0)
Search(SampDF, 19, "qsec", max.distance = 0)
```


<h3 id="tools">Generic qdap Tools</h3>

This manual arranges functions into categories in the order a researcher is likely to use them.  The Generic qdap Tools section does not fit this convention, however, because these tools may be used throughout all stages of analysis it is important that the reader is familiar with them.  It is important to note that after reading in transcript data the researcher will likely that the next step is the need to parse the dataframe utilizing the techniques found in the `r HR("#cleaning", "Cleaning/Preparing the Data")` section.

<div class="funs">
The following functions will be utilized in this section (click to view more):<br>    

<form class="form_left" action="http://trinker.github.io/qdap_dev/hms2sec.html" target="_blank">
    <input type="submit" value="hms2sec"> 
</form>

<form action="http://trinker.github.io/qdap_dev/sec2hms.html" target="_blank">
    <input type="submit" value="sec2hms"> - `r HR("#time", "Time Conversion")` 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/lookup.html" target="_blank">
    <input type="submit" value="lookup"><input type="submit" value="%l%"> 
</form>

<form action="http://trinker.github.io/qdap_dev/hash.html" target="_blank">
    <input type="submit" value="hash"><input type="submit" value="hash_look"><input type="submit" value="%hl%"> - `r HR("#hash", "Hash Table/Dictionary Lookup")`
</form>


<form action="http://trinker.github.io/qdap_dev/qcv.html" target="_blank">
    <input type="submit" value="qcv"> - `r HR("#qcv", "Quick Character Vector")`
</form>

<form action="http://trinker.github.io/qdap_dev/url_dl.html" target="_blank">
    <input type="submit" value="url_dl"> - `r HR("#urldl", "Download Instructional Documents")`
</form>
</div>

<h4 id="qcv">Quick Character Vector</h4> 

Often it can be tedious to supply quotes to character vectors when dealing with large vectors.  `r FUN("qcv")` replaces the typical `r  CN('c("A", "B", "C", "...")')` approach to creating character vectors.  Instead the user supplies `r CN("qcv(A, B, C, ...)")`.  This format assumes single words separated by commas.  If your data/string does not fit this approach the combined `terms` and `split` argument can be utilized.

`r FT(orange, 5, text="&diams;")` **Quick Character Vector** `r FT(orange, 5, text="&diams;")`

```{r}
qcv(I, like, dogs)
qcv(terms = "I like, big dogs", split = ",")
qcv(I, like, dogs, space.wrap = TRUE)
qcv(I, like, dogs, trailing = TRUE)
qcv(I, like, dogs, leading = TRUE)
qcv(terms = "mpg cyl  disp  hp drat    wt  qsec vs am gear carb")
```

<h4 id="hash">Dictionary/Lookup</h4>  

Often the researcher who deals with text data will have the need to lookup values quickly and return an accompanying value.  This is often called a dictionary, hash, or lookup.  This can be used to find corresponding values or recode variables etc.  The `r FUN("lookup")` & `r HR2("%l%")` functions provide a fast environment lookup for single usage. The `r FUN("hash")` & `r HR2("http://trinker.github.io/qdap_dev/hash.html", "hash_lookup")`/`r HR2("http://trinker.github.io/qdap_dev/hash.html", "%hl%")` functions provide a fast environment lookup for multiple uses of the same hash table.

`r FT(orange, 5, text="&diams;")` **`r FUN("lookup")`**- *Dictionary/Look Up Examples* `r FT(orange, 5, text="&diams;")`

```{r}
lookup(1:5, data.frame(1:4, 11:14))
lookup(LETTERS[1:5], data.frame(LETTERS[1:4], 11:14), missing = NULL)
lookup(LETTERS[1:5], data.frame(LETTERS[1:5], 100:104))
```

<pre><code class="r">## Fast with very large vectors
key <- data.frame(x=1:2, y=c("A", "B"))
set.seed(10)
big.vec <- sample(1:2, 3000000, T)
out <- lookup(big.vec, key)
out[1:20]</code></pre>


<pre><code>##  [1] "B" "A" "A" "B" "A" "A" "A" "A" "B" "A" "B" "B" "A"
## [14] "B" "A" "A" "A" "A" "A" "B"</code></pre>

```{r}
## Supply a named list of vectors to key.match

codes <- list(A=c(1, 2, 4),
    B = c(3, 5),
    C = 7,
    D = c(6, 8:10)
)

lookup(1:10, codes) #or
1:10 %l% codes
```

```{r}
## Supply a single vector to key.match and key.assign
lookup(mtcars$carb, sort(unique(mtcars$carb)),
    c('one', 'two', 'three', 'four', 'six', 'eight'))
lookup(mtcars$carb, sort(unique(mtcars$carb)),
    seq(10, 60, by=10))
```

`r FT(orange, 5, text="&diams;")` **`r FUN("hash")`/`r FUN("hash_look")`**- *Dictionary/Look Up Examples* `r FT(orange, 5, text="&diams;")`

```{r}
## Create a fake data set of hash values
(DF <- aggregate(mpg~as.character(carb), mtcars, mean))

## Use `hash` to create a lookup environment
hashTab <- hash(DF)  

## Create a vector to lookup
x <- sample(DF[, 1], 20, TRUE)

## Lookup x in the hash with `hash_look` or `%hl%`
hash_look(x, hashTab)
x %hl% hashTab
```

<h4 id="time">Time Conversion</h4>  

Researchers dealing with transcripts may have the need to convert between traditional Hours:Minutes:Seconds format and seconds.  The `r FUN("hms2sec")` and `r FUN("sec2hms")` functions offer this type of time conversion.


`r FT(orange, 5, text="&diams;")` **Time Conversion Examples** `r FT(orange, 5, text="&diams;")`

```{r}
hms2sec(c("02:00:03", "04:03:01"))
hms2sec(sec2hms(c(222, 1234, 55)))
sec2hms(c(256, 3456, 56565))
```

<h4 id="urldl">Download Documents</h4>  
 
`r FUN("url_dl")` is a function used to provide qdap users with examples taken from the Internet.  It is useful for most document downloads from the Internet.

`r FT(orange, 5, text="&diams;")` **url_dl Examples** `r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Example 1 (download from dropbox)
# download transcript of the debate to working directory
url_dl(pres.deb1.docx, pres.deb2.docx, pres.deb3.docx)

# load multiple files with read transcript and assign to working directory
dat1 <- read.transcript("pres.deb1.docx", c("person", "dialogue"))
dat2 <- read.transcript("pres.deb2.docx", c("person", "dialogue"))
dat3 <- read.transcript("pres.deb3.docx", c("person", "dialogue"))

docs <- qcv(pres.deb1.docx, pres.deb2.docx, pres.deb3.docx)
dir() %in% docs
delete(docs)    #remove the documents
dir() %in% docs

## Example 2 (quoted string urls)
url_dl("https://dl.dropboxusercontent.com/u/61803503/qdap.pdf",
   "http://www.cran.r-project.org/doc/manuals/R-intro.pdf")

## Clean up
delete(qcv(qdap.pdf, R-intro.pdf))</code></pre>


<h3 id="cleaning">Cleaning/Preparing the Data</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/bracketX.html" target="_blank">
    <input type="submit" value="bracketX"><input type="submit" value="bracketXtract"><input type="submit" value="genX"><input type="submit" value="genXtract"> - `r HR("#bracket", "Bracket/General Chunk Extraction")`     
</form>

<form action="http://trinker.github.io/qdap_dev/beg2char.html" target="_blank">
    <input type="submit" value="beg2char"><input type="submit" value="char2end"> - `r HR("#grab", "Grab Begin/End of String to Character")` 
</form>

<form action="http://trinker.github.io/qdap_dev/capitalizer.html" target="_blank">
    <input type="submit" value="capitalizer"> - `r HR("#caps", "Capitalize Select Words")` 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/clean.html" target="_blank">
    <input type="submit" value="clean">
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/scrubber.html" target="_blank">
    <input type="submit" value="scrubber">
</form>

<form action="http://trinker.github.io/qdap_dev/Trim.html" target="_blank">
    <input type="submit" value="Trim">- `r HR("#clean", "Clean Imported Text: Remove Escaped Characters & Leading/Trailing White Space")`
</form> 
 

<form action="http://trinker.github.io/qdap_dev/incomplete_replace.html" target="_blank">
    <input type="submit" value="incomplete_replace"><input type="submit" value="incomp"> - `r HR("#inc", "Denote Incomplete End Marks With \"|\"")`
</form>

<form action="http://trinker.github.io/qdap_dev/multigsub.html" target="_blank">
    <input type="submit" value="multigsub"><input type="submit" value="mgsub"> - `r HR("#mgsub", "Multiple gsub")`
</form>

<form action="http://trinker.github.io/qdap_dev/name2sex.html" target="_blank">
    <input type="submit" value="name2sex"> - `r HR("#nms", "Names to Gender Prediction")`
</form>

<form action="http://trinker.github.io/qdap_dev/potential_NA.html" target="_blank">
    <input type="submit" value="potential_NA"> - `r HR("#na", "Search for Potential Missing Values")`
</form>

<form action="http://trinker.github.io/qdap_dev/qprep.html" target="_blank">
    <input type="submit" value="qprep"> - `r HR("#replace", "Quick Preparation of Text")`
</form>

<form action="http://trinker.github.io/qdap_dev/replace_abbreviation.html" target="_blank">
    <input type="submit" value="replace_abbreviation"> - `r HR("#replace", "Replace Abbreviations")`
</form>

<form action="http://trinker.github.io/qdap_dev/replace_contraction.html" target="_blank">
    <input type="submit" value="replace_contraction"> - `r HR("#replace", "Replace Contractions")`
</form>

<form action="http://trinker.github.io/qdap_dev/replace_number.html" target="_blank">
    <input type="submit" value="replace_number"> - `r HR("#replace", "Replace Numbers With Text Representation")`
</form>

<form action="http://trinker.github.io/qdap_dev/replace_symbol.html" target="_blank">
    <input type="submit" value="replace_symbol"> - `r HR("#replace", "Replace Symbols With Word Equivalents")`
</form>

<form action="http://trinker.github.io/qdap_dev/rm_row.html" target="_blank">
    <input type="submit" value="rm_row"><input type="submit" value="rm_empty_row"> - `r HR("#mark", "Remove Rows That Contain Markers")`
</form>

<form action="http://trinker.github.io/qdap_dev/space_fill.html" target="_blank">
    <input type="submit" value="space_fill"> - `r HR("#fill", "Replace Spaces")`
</form>

<form action="http://trinker.github.io/qdap_dev/stemmer.html" target="_blank">
    <input type="submit" value="stemmer"><input type="submit" value="stem_words"><input type="submit" value="stem2df"> - `r HR("#stem", "Stem Text")`
</form>
 
</div>

<h4 id="bracket">Bracket/General Chunk Extraction `r yt("http://youtu.be/B4lvZGo_6bA")`</h4>   

After reading in the data the researcher may want to remove all non-dialogue text from the transcript dataframe such as transcriber annotations.  This can be accomplished with the `r FUN("bracketX")` family of functions, which removes text found between two brackets (`r CN("( )")`, `r CN("{ }")`, `r CN("[ ]")`, `r CN("< >")`) or more generally using `r FUN("genX")` and `r FUN("genXtract")` to remove text between two character reference points. 

If the bracketed text is useful to analysis it is recommended that the researcher assigns the un-bracketed text to a new column.


`r FT(orange, 5, text="&diams;")` **Extracting Chunks 1**- *bracketX/bracketXtract* `r FT(orange, 5, text="&diams;")`

```{r}
## A fake data set
examp <- structure(list(person = structure(c(1L, 2L, 1L, 3L),
    .Label = c("bob", "greg", "sue"), class = "factor"), text =
    c("I love chicken [unintelligible]!",
    "Me too! (laughter) It's so good.[interrupting]",
    "Yep it's awesome {reading}.", "Agreed. {is so much fun}")), .Names =
    c("person", "text"), row.names = c(NA, -4L), class = "data.frame")
examp
bracketX(examp$text, "square")
bracketX(examp$text, "curly")
bracketX(examp$text, c("square", "round"))
bracketX(examp$text)
bracketXtract(examp$text, "square")
bracketXtract(examp$text, "curly")
bracketXtract(examp$text, c("square", "round"))
bracketXtract(examp$text, c("square", "round"), merge = FALSE)
bracketXtract(examp$text)
bracketXtract(examp$text, with = TRUE)
```

Often a researcher will want to extract some text from the transcript and put it back together.  One example is the reconstructing of material read from a book, poem, play or other text.  This information is generally dispersed throughout the dialogue (within classroom/teaching procedures).   If this text is denoted with a particular identifying bracket such as curly braces this text can be extracted and then pasted back together.

`r FT(orange, 5, text="&diams;")` **Extracting Chunks 2**- *Recombining Chunks* `r FT(orange, 5, text="&diams;")`

```{r}
paste2(bracketXtract(examp$text, "curly"), " ")
```

The researcher may need a more general extraction method that allows for any left/right boundaries to be specified.  This is useful in that many qualitative transcription/coding programs have specific syntax for various dialogue markup for events that must be parsed from the data set.  The `r FUN("genX")` and `r FUN("genXtract")` functions have such capabilities.

`r FT(orange, 5, text="&diams;")` **Extracting Chunks 3**- *genX/genXtract* `r FT(orange, 5, text="&diams;")`
```{r}
DATA$state  
## Look at the difference in number 1 and 10 from above
genX(DATA$state, c("is", "we"), c("too", "on"))
## A fake data set
x <- c("Where is the /big dog#?",
    "I think he's @arunning@b with /little cat#.")
x
genXtract(x, c("/", "@a"), c("#", "@b"))
## A fake data set
x2 <- c("Where is the L1big dogL2?",
    "I think he's 98running99 with L1little catL2.")
x2
genXtract(x2, c("L1", 98), c("L2", 99))
```

<h4 id="na">Search for Potential Missing Values</h4>

After reading in data, removing non-dialogue (via `r FUN("bracketX")`), and viewing it the researcher will want to find text rows that do not contain proper punctuation and or that contain punctuation and no text.  This is accomplished with the <a href="http://trinker.github.io/qdap_dev/htruncdf.html" target="_blank"><code>_truncdf</code></a> family of functions and `r FUN("potential_NA")` functions as the researcher manually parses the original transcripts, makes alterations and re-reads the data back into qdap.  This important procedure is not an automatic process, requiring that the researcher give attention to detail in comparing the R dataframe with the original transcript.

`r FT(orange, 5, text="&diams;")` **Identifying and Coding Missing Values** `r FT(orange, 5, text="&diams;")`
```{r}
## Create A Data Set With Punctuation and No Text
(DATA$state[c(3, 7, 10)] <- c(".", ".", NA))
DATA
potential_NA(DATA$state, 20)
potential_NA(DATA$state)
## Use To Selctively Replace Cells With Missing Values
DATA$state[potential_NA(DATA$state, 20)$row[-c(3)]] <- NA
DATA
## Reset DATA
DATA <- qdap::DATA
```

<h4 id="mark">Remove Rows That Contain Markers</h4>

The researcher may wish to remove empty rows (using `r FUN("rm_empty_row")`) and/or rows that contain certain markers (using `r FUN("rm_row")`).  Sometimes empty rows are read into the dataframe from the transcript.  These rows should be completely removed from the data set rather than denoting with `NA`.  The `r FUN("rm_empty_row")` removes completely empty rows (those rows with only 1 or more blank spaces) from the dataframe.

`r FT(orange, 5, text="&diams;")` **Remove Empty Rows**`r FT(orange, 5, text="&diams;")`
```{r}
(dat <- rbind.data.frame(DATA[, c(1, 4)], matrix(rep(" ", 4),
   ncol =2, dimnames=list(12:13, colnames(DATA)[c(1, 4)]))))
rm_empty_row(dat)
```

Other times the researcher may wish to use `r FUN("rm_row")` to remove rows from the dataframe/analysis based on transcription conventions or to remove demographic characteristics.  For example, in the example below the transcript is read in with <b>[Cross Talk 3</b>.  This is a transcription convention and we would want to parse these rows from the transcript.  A second example shows the removal of people from the dataframe.

`r FT(orange, 5, text="&diams;")` **Remove Selected Rows**`r FT(orange, 5, text="&diams;")`


<pre><code class="r">## Read in transcript
dat2 <- read.transcript(system.file("extdata/transcripts/trans1.docx", 
    package = "qdap"))
truncdf(dat2, 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it&#39;s time to learn. [Student di
## 3 Multiple Students        Yes teacher we&#39;re ready to learn.
## 4     [Cross Talk 3                                      00]
## 5         Teacher 4 Let&#39;s read this terrific book together.
</code></pre>

<pre><code class="r">## Use column names to remove rows
truncdf(rm_row(dat2, "X1", "[C"), 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it&#39;s time to learn. [Student di
## 3 Multiple Students        Yes teacher we&#39;re ready to learn.
## 4         Teacher 4 Let&#39;s read this terrific book together.
</code></pre>

<pre><code class="r">## Use column numbers to remove rows
truncdf(rm_row(dat2, 2, "[C"), 40)</code></pre>

<pre><code>##                  X1                                       X2
## 1      Researcher 2                         October 7, 1892.
## 2         Teacher 4 Students it&#39;s time to learn. [Student di
## 3 Multiple Students        Yes teacher we&#39;re ready to learn.
## 4     [Cross Talk 3                                      00]
## 5         Teacher 4 Let&#39;s read this terrific book together.
</code></pre>

<pre><code class="r">## Also remove people etc. from the analysis
rm_row(DATA, 1, c("sam", "greg"))</code></pre>

<pre><code>##       person sex adult                         state code
## 1    teacher   m     1            What should we do?   K3
## 2      sally   f     0        How can we be certain?   K6
## 3      sally   f     0   What are you talking about?   K9
## 4 researcher   f     1 Shall we move on?  Good then.  K10
</code></pre>

<h4 id="clean">Remove Extra Spaces and Escaped Characters</h4> 

An important step in the cleaning process is the removal of extra white spaces (use `r FUN("Trim")`) and `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/Quotes.html", "escaped characters")` (use `r FUN("clean")`).  The `r FUN("scrubber")` function wraps both `r FUN("Trim")` and `r FUN("clean")` and adds in the functionality of some of the `r CN("replace_")` family of functions.

`r FT(orange, 5, text="&diams;")` **Remove Extra Spaces and Escaped Characters**`r FT(orange, 5, text="&diams;")`
```{r}
x1 <- "I go \r
    to the \tnext line"
x1
clean(x1)
x2 <- c("  talkstats.com ", "   really? ", " yeah")
x2
Trim(x2)
x3 <- c("I like 456 dogs\t  , don't you?\"")
x3
scrubber(x3)
scrubber(x3, TRUE)
```

<h4 id="replace">Replacement Functions</h4>

The replacement family of functions replace various text elements within the transcripts with alphabetic versions that are more suited to analysis.  These alterations may affect word counts and other alphabetic dependent forms of analysis.

The `r FUN("replace_abbreviation")` replaces standard abbreviations that utilize periods with forms that do not rely on periods.  This is necessary in that many sentence specific functions (e.g., `r FUN("sentSplit")` and `r FUN("word_stats")`) rely on period usage acting as sentence end marks.  The researcher may augment the standard `r FUN2("abbreviations")` dictionary from qdapDictionaries with field specific abbreviations.

`r FT(orange, 5, text="&diams;")` **Replace Abbreviations**`r FT(orange, 5, text="&diams;")`
```{r}
## Use the standard contractions dictionary
x <- c("Mr. Jones is here at 7:30 p.m.",
    "Check it out at www.github.com/trinker/qdap",
    "i.e. He's a sr. dr.; the best in 2012 A.D.",
    "the robot at t.s. is 10ft. 3in.")
x
replace_abbreviation(x)
## Augment the standard dictionary with replacement vectors
abv <- c("in.", "ft.", "t.s.")
repl <- c("inch", "feet", "talkstats")
replace_abbreviation(x, abv, repl)
## Augment the standard dictionary with a replacement dataframe
(KEY <- rbind(abbreviations, data.frame(abv = abv, rep = repl)))
replace_abbreviation(x, KEY)
```

The `r FUN("replace_contraction")` replaces contractions with equivalent multi-word forms.  This is useful for some word/sentence statistics.  The researcher may augment the `r FUN2("contractions")` dictionary supplied by qdapDictionaries, however, the word list is exhaustive.

`r FT(orange, 5, text="&diams;")` **Replace Contractions**`r FT(orange, 5, text="&diams;")`
```{r}
x <- c("Mr. Jones isn't going.",
    "Check it out what's going on.",
    "He's here but didn't go.",
    "the robot at t.s. wasn't nice",
    "he'd like it if i'd go away")
x
replace_contraction(x)
```

The `r FUN("replace_number")` function utilizes The work of John <a href="">Fox (2005)</a> to turn numeric representations of numbers into their textual equivalents.  This is useful for word statistics that require the text version of dialogue.

`r FT(orange, 5, text="&diams;")` **Replace Numbers**-*Numeral Representation*`r FT(orange, 5, text="&diams;")`
```{r}
x <- c("I like 346457 ice cream cones.", "They are 99 percent good")
replace_number(x)
## Replace numbers that contain commas as well
y <- c("I like 346,457 ice cream cones.", "They are 99 percent good")
replace_number(y)
## Combine numbers as one word/string
replace_number(x, FALSE)
```


The `r FUN("replace_symbol")` converts ($) to "dollar", (%) to "percent", (#) to "number", (@) to "at", (&) to "and", (w/) to "with".  Additional substitutions can be undertaken with the `r FUN("multigsub")` function. 

`r FT(orange, 5, text="&diams;")` **Replace Symbols**`r FT(orange, 5, text="&diams;")`
```{r}
x <- c("I am @ Jon's & Jim's w/ Marry",
    "I owe $41 for food",
    "two is 10% of a #")
x
replace_symbol(x)
replace_number(replace_symbol(x))
```

The `r FUN("qprep")` function is a wrapper for several other replacement family function that allows for more speedy cleaning of the text.  This approach, while speedy, reduces the flexibility and care that is undertaken by the researcher when the individual replacement functions are utilized.  The function is intended for analysis that requires less care.

`r FT(orange, 5, text="&diams;")` **General Replacement (Quick Preparation)**`r FT(orange, 5, text="&diams;")`
```{r}
x <- "I like 60 (laughter) #d-bot and $6 @ the store w/o 8p.m."
x
qprep(x)
```


<h4 id="fill">Replace Spaces</h4>

Many qdap functions break sentences up into words based on the spaces between words.  Often the researcher will want to keep a group of words as a single unit.  The `r FUN("space_fill")` allows the researcher to replace spaces between selected phrases with <b><font color="blue" face="courier">&#126;&#126;</font></b>.  By default <b><font color="blue" face="courier">&#126;&#126;</font></b> is recognized by many qdap functions as a space separator.

`r FT(orange, 5, text="&diams;")` **Space Fill Examples**`r FT(orange, 5, text="&diams;")`
```{r}
## Fake Data
x <- c("I want to hear the Dr. Martin Luther King Jr. speech.",
    "I also want to go to the white House to see President Obama speak.")
x
## Words to keep as a single unit
keeps <- c("Dr. Martin Luther King Jr.", "The White House", "President Obama")
text <- space_fill(x, keeps)
text
## strip Example
strip(text, lower=FALSE)
## bag_o_words Example
bag_o_words(text, lower=FALSE)
## wfm Example
wfm(text, c("greg", "bob"))
## trans_cloud Example
obs <- strip(space_fill(keeps, keeps), lower=FALSE)
trans_cloud(text, c("greg", "bob"), target.words=list(obs), caps.list=obs, 
    cloud.colors=qcv(red, gray65), expand.target = FALSE, title.padj = .7,
    legend = c("space_filled", "other"), title.cex = 2, title.color = "blue", 
    max.word.size = 3)
```

<h4 id="mgsub">Multiple gsub</h4>

The researcher may have the need to make multiple substitutions in a text.  An example of when this is needed is when a transcript is marked up with transcription coding convention specific to a particular transcription method.  These codes, while useful in some contexts, may lead to inaccurate word statistics.  The base R function `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/grep.html", "gsub")` makes a single replacement of these types of coding conventions. The `r FUN("multigsub")` (alias `r FUN("mgsub")`) takes a vector of patterns to search for as well as a vector of replacements.  Note that the replacements occur sequentially rather than all at once. This means a previous (first in pattern string) sub could alter or be altered by a later sub.  `r FUN("mgsub")` is useful throughout multiple stages of the research process.

`r FT(orange, 5, text="&diams;")` **Multiple Substitutions**`r FT(orange, 5, text="&diams;")`
```{r}
left_just(DATA[, c(1, 4)])
multigsub(c("it's", "I'm"), c("it is", "I am"), DATA$state)
mgsub(c("it's", "I'm"), c("it is", "I am"), DATA$state)
mgsub(c("it's", "I'm"), "SINGLE REPLACEMENT", DATA$state)
mgsub("[[:punct:]]", "PUNC", DATA$state, fixed = FALSE)
## Iterative "I'm" converts to "I am" which converts to "INTERATIVE"
mgsub(c("it's", "I'm", "I am"), c("it is", "I am", "ITERATIVE"), DATA$state)
```


<h4 id="nms">Names to Gender Prediction</h4>

A researcher may face a list of names and be uncertain about gender of the participants.  The `r FUN("name2sex")` function utilizes the [**gender**](http://cran.r-project.org/web/packages/gender/) package to predict names based on Social Security Administration data, defaulting to the period from 1932-2012.

`r FT(orange, 5, text="&diams;")` **Name to Gender Prediction**`r FT(orange, 5, text="&diams;")`
<pre><code class="r">name2sex(qcv(mary, jenn, linda, JAME, GABRIEL, OLIVA, tyler, jamie, JAMES, 
    tyrone, cheryl, drew))
</code></pre>

<pre><code>[1] F F F M M F M F M M F M
Levels: F M
</code></pre>

<h4 id="stem">Stem Text</h4>

During the initial cleaning stage of analysis the researcher may choose to create a stemmed version of the dialogue, that is words are reduced to their root words.  The `r FUN("stemmer")` family of functions allow the researcher to create stemmed text.  The `r FUN("stem2df")` function wraps `r FUN("stemmer")` to quickly create a dataframe with the stemmed column added.

`r FT(orange, 5, text="&diams;")` **Stemming**`r FT(orange, 5, text="&diams;")`
```{r}
## stem2df EXAMPLE:
(stemdat <- stem2df(DATA, "state", "new"))
with(stemdat, trans_cloud(new, sex, title.cex = 2.5, 
    title.color = "blue", max.word.size = 5, title.padj = .7))
## stemmer EXAMPLE:
stemmer(DATA$state)
## stem_words EXAMPLE:
stem_words(doggies, jumping, swims)
```


<h4 id="grab">Grab Begin/End of String to Character</h4>

At times it is handy to be able to grab from the beginning or end of a string to a specific character.  The `r FUN("beg2char")` function allows you to grab from the beginning of a string to the n<sup>th</sup> occurrence of a character.  The counterpart function, `r FUN("char2end")`, grab from the n<sup>th</sup> occurrence of a character to the end of a string to. This behavior is useful if the transcript contains annotations at the beginning or end of a line that should be eliminated.

`r FT(orange, 5, text="&diams;")` **Grab From Character to Beginning/End of String**`r FT(orange, 5, text="&diams;")`
```{r}
x <- c("a_b_c_d", "1_2_3_4", "<_?_._:")
beg2char(x, "_")
beg2char(x, "_", 4)
char2end(x, "_")
char2end(x, "_", 2)
char2end(x, "_", 3, include=TRUE)
(x2 <- gsub("_", " ", x))
beg2char(x2, " ", 2)
(x3 <- gsub("_", "\\^", x))
char2end(x3, "^", 2)
```

<h4 id="inc">Denote Incomplete End Marks With "|"</h4> 

Often incomplete sentences have a different function than complete sentences.  The researcher may want to denote incomplete sentences for consideration in later analysis.  Traditionally, incomplete sentence are denoted with the following end marks (.., ..., .?, ..?, en & em).  The `r FUN("incomplete_replace")` can identify and replace the traditional end marks with a standard form `r FT(blue, text="\"|\"")`.

`r FT(orange, 5, text="&diams;")` **Incomplete Sentence Identification**`r FT(orange, 5, text="&diams;")`
```{r}
x <- c("the...",  "I.?", "you.", "threw..", "we?")
incomplete_replace(x)
incomp(x)
incomp(x, scan.mode = TRUE)
```


<h4 id="caps">Capitalize Select Words</h4>

The `r FUN("capitalizer")` functions allows the researcher to specify words within a vector to be capitalized.  By default `r FT(blue, text="I")`, and contractions containing `r FT(blue, text="I")`, are capitalized.  Additional words can be specified through the `r CN("caps.list")` argument.  To capitalize words within strings the `r FUN("mgsub")` can be used.

`r FT(orange, 5, text="&diams;")` **Word Capitalization**`r FT(orange, 5, text="&diams;")`
```{r}
capitalizer(bag_o_words("i like it but i'm not certain"), "like")
capitalizer(bag_o_words("i like it but i'm not certain"), "like", FALSE)
```

<h3 id="reshaping">Reshaping the Data</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/adjacency_matrix.html" target="_blank">
    <input type="submit" value="adjacency_matrix"><input type="submit" value="adjmat"> - `r HR("#adj", "Create Adjacency Matrix")`
</form>


<form class="form_left" action="http://trinker.github.io/qdap_dev/gantt.html" target="_blank">
    <input type="submit" value="gantt">
</form>
<form action="http://trinker.github.io/qdap_dev/gantt_rep.html" target="_blank">    
    <input type="submit" value="gantt_rep">- `r HR("#ganttspan", "Generate Unit Spans")`
</form>


<form action="http://trinker.github.io/qdap_dev/key_merge.html" target="_blank">
    <input type="submit" value="key_merge"> - `r HR("#merge", "Merge Demographic Information with Person/Text Transcript")`
</form>


<form class="form_left" action="http://trinker.github.io/qdap_dev/paste2.html" target="_blank">
    <input type="submit" value="paste2"/>
</form>
<form class="form_left" action="http://trinker.github.io/qdap_dev/paste2.html" target="_blank">
    <input type="submit" value="colpaste2df"/>
</form>
<form class="form_left" action="http://trinker.github.io/qdap_dev/colSplit.html" target="_blank">
    <input type="submit" value="colSplit"/>
</form>
<form class="form_left" action="http://trinker.github.io/qdap_dev/colsplit2df.html" target="_blank">
    <input type="submit" value="colsplit2df"/>
</form>
<form action="http://trinker.github.io/qdap_dev/colsplit2df.html" target="_blank">
    <input type="submit" value="lcolsplit2df"/>- `r HR("#paste2", "Paste and Separate Columns")`
</form>


<form class="form_left" action="http://trinker.github.io/qdap_dev/sentSplit.html" target="_blank">
    <input type="submit" value="sentSplit"><input type="submit" value="sentCombine"><input type="submit" value="TOT">
</form>

<form action="http://trinker.github.io/qdap_dev/speakerSplit.html" target="_blank">
    <input type="submit" value="speakerSplit"> - `r HR("#sentsplit", "Sentence Splitting/Combining")`
</form>

</div>

<h4 id="sentsplit">Sentence Splitting/Combining</h4>

Many functions in the qdap package require that the dialogue is broken apart into individual sentences, failure to do so may invalidate many of the outputs from the analysis and will lead to lead to warnings.  After reading in and cleaning the data the next step should be to split the text variable into individual sentences.  The `r FUN("sentSplit")` function outputs a dataframe with the text variable split into individual sentences and repeats the demographic variables as necessary.  Additionally, a turn of talk (`r FT(red, text="tot column")`) variable is added that keeps track of the original turn of talk (row number) and the sentence number per turn of talk.  The researcher may also want to create a second text column that has been stemmed for future analysis by setting `r CN("stem.col = TRUE")`, though this is more time intensive.

`r FT(orange, 5, text="&diams;")` **`r FUN("sentSplit")` Example**`r FT(orange, 5, text="&diams;")`

```{r}
sentSplit(DATA, "state")
sentSplit(DATA, "state", stem.col = TRUE)
sentSplit(raj, "dialogue")[1:11, ]
```

`r FT(orange, 5, text="&diams;")` **`r FUN("sentSplit")`** - *plot Method*`r FT(orange, 5, text="&diams;")`

```{r}
plot(sentSplit(DATA, "state"), grouping.var = "person")
plot(sentSplit(DATA, "state"), grouping.var = "sex")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("TOT", "sentSplit")` Example** `r FT(orange, 5, text="&diams;")`

```{r}
## Convert tot column with sub sentences to turns of talk
dat <- sentSplit(DATA, "state")
TOT(dat$tot)
```

Within dialogue (particularly classroom dialogue) several speakers may say the same speech at the same.  The transcripts may lump this speech together in the form of: 

<TABLE>
    <TR> <TD align="right"><b>Person</b></TD> <TD align="right"><b>Dialogue</b></TD> </TR>
    <TR> <TD align="right"> John, Josh & Imani `r HS(8)`</TD> <TD align="right"> Yes Mrs. Smith. `r HS(8)`</TD> </TR>
</TABLE>

The `r FUN("speakerSplit")` function attributes this text to each of the people as separate entries.  The default behavior is the search for the person separators of <font face="courier">sep = c("and", "&", ",")</font>, though other separators may be specified.

`r FT(orange, 5, text="&diams;")` **Break and Stretch if Multiple Persons per Cell**`r FT(orange, 5, text="&diams;")`

```{r}
## Create data set with multiple speakers per turn of talk
DATA$person <- as.character(DATA$person)
DATA$person[c(1, 4, 6)] <- c("greg, sally, & sam",
    "greg, sally", "sam and sally")
speakerSplit(DATA)
## Change the separator
DATA$person[c(1, 4, 6)] <- c("greg_sally_sam",
    "greg.sally", "sam; sally")
speakerSplit(DATA, sep = c(".", "_", ";"))
## Reset DATA
DATA <- qdap::DATA  
```

The `r FUN("sentCombine")` function is the opposite of the `r FUN("sentSplit")`, combining sentences into a single turn of talk per grouping variable.

`r FT(orange, 5, text="&diams;")` **Sentence Combining**`r FT(orange, 5, text="&diams;")`

```{r}
dat <- sentSplit(DATA, "state")
## Combine by person
sentCombine(dat$state, dat$person)
## Combine by sex
truncdf(sentCombine(dat$state, dat$sex), 65)
```

<h4 id="merge">Merge Demographic Information with Person/Text Transcript</h4>

It is more efficient to maintain a dialogue dataframe (consisting of a column for people and a column for dialogue) and a separate demographics dataframe (a person column and demographic column(s)) and then merge the two during analysis.  The `r FUN("key_merge")` function is a wrapper for the `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/merge.html", "merge")` function from R's base install that merges the dialogue and demographics dataframe. `r FUN("key_merge")` attempts to guess the person column and outputs a qdap friendly dataframe.

`r FT(orange, 5, text="&diams;")` **Merging Demographic Information**`r FT(orange, 5, text="&diams;")`

```{r}
## A dialogue dataframe and a demographics dataframe
ltruncdf(list(dialogue=raj, demographics=raj.demographics), 10, 50)
## Merge the two
merged.raj <- key_merge(raj, raj.demographics)
htruncdf(merged.raj, 10, 40)
```

<h4 id="paste2">Paste and Split Columns</h4>

Many functions in qdap utilize the `r FUN("paste2")` function, which pastes multiple columns/lists of vectors.  `r FUN("paste2")` differs from base R's `r HR2("http://127.0.0.1:16084/library/base/html/paste.html", "paste")` function in that `r FUN("paste2")` can paste unspecified columns or a list of vectors together.  The `r FUN("colpaste2df", "paste2")` function, a wrapper for `r FUN("paste2")`, pastes multiple columns together and outputs an appropriately named dataframe.  The `r FUN("colsplit2df")` and `r FUN("lcolsplit2df", "colsplit2df")` are useful because they can split the output from qdap functions that contain dataframes with pasted columns.

`r FT(orange, 5, text="&diams;")` **Using `r FUN("paste2")` and `r FUN("colSplit")`**: *Pasting & Splitting Vectors and Dataframes*`r FT(orange, 5, text="&diams;")`

```{r}
## Pasting a list of vectors
paste2(rep(list(state.abb[1:8],  month.abb[1:8]) , 2), sep = "|_|")
## Pasting a dataframe
foo1 <- paste2(CO2[, 1:3])
head(foo1, 12)
## Splitting a pasted column
bar1 <- colSplit(foo1)
head(bar1, 10)
```

`r FT(orange, 5, text="&diams;")` **`r FUN("colpaste2df")` & `r FUN("colsplit2df")`**: *Splitting Columns in Dataframes*`r FT(orange, 5, text="&diams;")`

```{r}
## Create a dataset with a pasted column
(dat <- colpaste2df(head(CO2), 1:3, keep.orig = FALSE)[, c(3, 1:2)])
## Split column
colsplit2df(dat)
## Specify names
colsplit2df(dat, new.names = qcv(A, B, C))
## Keep the original pasted column
colsplit2df(dat, new.names = qcv(A, B, C), keep.orig = TRUE)
## Pasting columns and output a dataframe
colpaste2df(head(mtcars)[, 1:5], qcv(mpg, cyl, disp), sep ="_", name.sep = "|")
colpaste2df(head(CO2)[, -3], list(1:2, qcv("conc", "uptake")))
```

`r FT(orange, 5, text="&diams;")` **`r FUN("lcolsplit2df")`**: *Splitting Columns in Lists of Dataframes*`r FT(orange, 5, text="&diams;")`

```{r}
## A list with dataframes that contain pasted columns
x <- question_type(DATA.SPLIT$state, list(DATA.SPLIT$sex, DATA.SPLIT$adult))
ltruncdf(x[1:4])
z <- lcolsplit2df(x)
ltruncdf(z[1:4])
```

<h4 id="ganttspan">Generate Unit Spans</h4>

Often a researcher will want to view the patterns of the discourse by grouping variables over time.  This requires the data to have start and end times based on units (sentence, turn of talk, or word).  The `r FUN("gantt")` function provides the user with unit spans (start and end times) with the `r FUN("gantt_rep")` extending this capability to repeated measures.  The `r FUN("gantt")` function has a basic plotting method to allow visualization of the unit span data, however, the `r FUN("gantt_wrap")` function extends the `r FUN("gantt")` and `r FUN("gantt_rep")` functions to plot precise depictions (Gantt plots) of the unit span data.  Note that if the researcher is only interested in the plotting the data as a Gantt plot, the `r FUN("gantt_plot")` function combines the `r FUN("gantt")`/`r FUN("gantt_rep")` functions with the `r FUN("gantt")` function  

`r FT(orange, 5, text="&diams;")` **Unit Spans**`r FT(orange, 5, text="&diams;")`

```{r}
## Unit Span Dataframe
dat <- gantt(mraja1$dialogue, mraja1$person) 
head(dat, 12)
plot(dat)
plot(dat, base = TRUE)
```

`r FT(orange, 5, text="&diams;")` **Repeated Measures Unit Spans**`r FT(orange, 5, text="&diams;")`

```{r}
## Repeated Measures Unit Span Dataframe
dat2 <- with(rajSPLIT, gantt_rep(act, dialogue, list(fam.aff, sex)))
head(dat2, 12)
## Plotting Repeated Measures Unit Span Dataframe
plot(dat2)
gantt_wrap(dat2, "fam.aff_sex", facet.vars = "act",
    title = "Repeated Measures Gantt Plot")
```

<h4 id="adj">Create Adjacency Matrix</h4>

It is useful to convert data to an adjacency matrix for examining relationships between grouping variables in word usage.  The `r FUN("adjaceny_matrix")` (aka: `r FUN("adjmat")`) provide this capability, interacting with a `r FUN("termco")` or `r FUN("wfm", "Word_Frequency_Matrix")` object.  In the first example below Sam and Greg share 4 words in common, whereas, the Teacher and Greg share no words.  The adjacency matrix can be passed to a network graphing package such as the `r HR2("http://igraph.sourceforge.net/", "igraph")` package for visualization of the data structure as seen in Example 3.


`r FT(orange, 5, text="&diams;")` **Adjacency Matrix**: *Example 1*`r FT(orange, 5, text="&diams;")`

```{r, eval=FALSE}
adjacency_matrix(wfm(DATA$state, DATA$person))
```

<pre><code>## Adjacency Matrix:
## 
##            greg researcher sally sam
## researcher    0                     
## sally         1          1          
## sam           4          0     1    
## teacher       0          1     2   0
## 
## 
## Summed occurrences:
## 
##       greg researcher      sally        sam    teacher 
##         18          6         10         11          4 
</code></pre>

`r FT(orange, 5, text="&diams;")` **Adjacency Matrix**: *Example 2*`r FT(orange, 5, text="&diams;")`

```{r, eval=FALSE}
words <- c(" education", " war ", " econom", " job", "governor ")
(terms <- with(pres_debates2012, termco(dialogue, person, words)))
adjmat(terms)
```

<pre><code>## Adjacency Matrix:
## 
##           OBAMA ROMNEY CROWLEY LEHRER QUESTION
## ROMNEY        5                               
## CROWLEY       2      2                        
## LEHRER        4      4       2                
## QUESTION      4      4       2      4         
## SCHIEFFER     2      2       1      1        1
## 
## 
## Summed occurrences:
## 
##     OBAMA    ROMNEY   CROWLEY    LEHRER  QUESTION SCHIEFFER 
##         5         5         2         4         4         2 
</code></pre>

It is often useful to plot the adjacency matrix as a network.  The `r HR("http://cran.r-project.org/web/packages/igraph/index.html", "igraph package")` provides this functionality.

<p id="plotadj">`r FT(orange, 5, text="&diams;")` <b>Plotting an Adjacency Matrix</b>: <em>Example 1</em>`r FT(orange, 5, text="&diams;")`</p>


```{r, echo=-7}
library(igraph)
dat <- adjacency_matrix(wfm(DATA$state, DATA$person, stopword = Top25Words))
g <- graph.adjacency(dat$adjacency, weighted=TRUE, mode ="undirected")
g <- simplify(g)
V(g)$label <- V(g)$name
V(g)$degree <- igraph::degree(g)
set.seed(14)
plot(g, layout=layout.auto(g))
```

The following example will visualize the presidential debates data as a network plot.

<p id="plotadj2">`r FT(orange, 5, text="&diams;")` <b>Plotting an Adjacency Matrix</b>: <em>Example 2</em>`r FT(orange, 5, text="&diams;")`</p>

```{r warning=FALSE}
library(igraph)

## Subset the presidential debates data set
subpres <- pres_debates2012[pres_debates2012$person %in% qcv(ROMNEY, OBAMA), ]

## Create a word frequency matrix
dat <- with(subpres, wfm(dialogue, list(person, time), stopword = Top200Words))

## Generate an adjacency matrix
adjdat <- adjacency_matrix(dat)
X <- adjdat$adjacency

g <- graph.adjacency(X, weighted=TRUE, mode ="undirected")
g <- simplify(g)
V(g)$label <- V(g)$name
V(g)$degree <- igraph::degree(g)
plot(g, layout=layout.auto(g))
```

We can easily add information to the network plot utilizing the `r FUN("Dissimilarity")` function to obtain weights and distance measures for use with the plot.

`r FT(orange, 5, text="&diams;")` **Plotting an Adjacency Matrix**: *Example 2b*`r FT(orange, 5, text="&diams;")`

```{r}
edge.weight <- 15  #a maximizing thickness constant
d <- as.matrix(Dissimilarity(dat))
d2 <- d[lower.tri(d)]
z1 <- edge.weight*d2^2/max(d2)
z2 <- c(round(d2, 3))
E(g)$width <- c(z1)[c(z1) != 0] 
E(g)$label <- c(z2)[c(z2) != 0]
plot(g, layout=layout.auto(g))
plot(g, layout=layout.auto(g), edge.curved =TRUE)
```

`r FT(orange, 5, text="&diams;")` **Plotting an Adjacency Matrix**: *Try the plot interactively!*`r FT(orange, 5, text="&diams;")`

                     
<pre><code class="r">tkplot(g)
</code></pre>



<h3 id="word">Extract Words</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/all_words.html" target="_blank">
    <input type="submit" value="all_words"> - `r HR("#all_words", "Searches Text Column for Words")`
</form>

<form action="http://trinker.github.io/qdap_dev/bag_o_words.html" target="_blank">
    <input type="submit" value="bag_o_words"><input type="submit" value="breaker"><input type="submit" value="word.split"> - `r HR("#bag", "Bag of Words")`
</form>

<form action="http://trinker.github.io/qdap_dev/common.html" target="_blank">
    <input type="submit" value="common"> - `r HR("#common", "Find Common Words Between Groups")`
</form>

<form action="http://trinker.github.io/qdap_dev/exclude.html" target="_blank">
    <input type="submit" value="exclude"> - `r HR("#exclude", "Exclude Elements From a Vector")`
</form>

<form action="http://trinker.github.io/qdap_dev/ngrams.html" target="_blank">
    <input type="submit" value="ngrams"> - `r HR("#ngram", "Generate ngrams")`
</form>

<form action="http://trinker.github.io/qdap_dev/rm_stopwords.html" target="_blank">
    <input type="submit" value="rm_stopwords"> - `r HR("#stopwords", "Remove Stopwords")`
</form>

<form action="http://trinker.github.io/qdap_dev/strip.html" target="_blank">
    <input type="submit" value="strip"> - `r HR("#strip", "Strip Text of Unwanted Characters/Capitalization")`
</form>

<form action="http://trinker.github.io/qdap_dev/synonyms.html" target="_blank">
    <input type="submit" value="synonyms"><input type="submit" value="syn"> - `r HR("#syn", "Search For Synonyms")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_associate.html" target="_blank">
    <input type="submit" value="word_associate"> - `r HR("#assoc", "Find Associated Words")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_diff_list.html" target="_blank">
    <input type="submit" value="word_diff_list"> - `r HR("#diffs", "Differences In Word Use Between Groups")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_list.html" target="_blank">
    <input type="submit" value="word_list"> - `r HR("#word_list", "Raw Word Lists/Frequency Counts")`
</form>
</div>

This section overviews functions that can extract words and word lists from dialogue text.  The subsections describing function use are in alphabetical order as there is no set chronology for use.

<h4 id="all_words">Searches Text Column for Words</h4>

The `r FUN("all_words")` breaks the dialogue into a bag of words and searches based on the criteria arguments `r CN("begins.with")` and `r CN("contains")`.  The resulting word list can be useful for analysis or to pass to qdap functions that deal with `r HR("#counts", "Word Counts and Descriptive Statistics")`.

`r FT(orange, 5, text="&diams;")` **`r FUN("all_words")`**`r FT(orange, 5, text="&diams;")`


```{r}
## Words starting with `re`
x1 <- all_words(raj$dialogue, begins.with="re")
head(x1, 10)
## Words containing with `conc`
all_words(raj$dialogue, contains = "conc")
## All words ordered by frequency
x2 <- all_words(raj$dialogue, alphabetical = FALSE)
head(x2, 10)
```

<h4 id="bag">Word Splitting (Bag of Words)</h4>

The qdap package utilizes the following functions to turn text into a bag of words (word order is preserved):


<TABLE>
    <TR> <TD align="right"><font face="courier"><b>`r HR("http://trinker.github.io/qdap_dev/bag_o_words.html", "bag_o_words")` </b></font></TD> <TD align="right">Reduces a text column to a <b>single</b> vector bag of words.</TD> </TR>
    <TR> <TD align="right"><font face="courier"><b>`r HR("http://trinker.github.io/qdap_dev/bag_o_words.html", "breaker")`</b></font></TD> <TD align="right"> Reduces a text column to a <b>single</b> vector bag of words and qdap recognized end marks.</TD> </TR>
    <TR> <TD align="right"><font face="courier"><b>`r HR("http://trinker.github.io/qdap_dev/bag_o_words.html", "word.split")`</b></font></TD> <TD align="right"> Reduces a text column to a <b>list</b> of vectors of bag of words and qdap recognized end marks (i.e., ".", "!", "?", "*", "-").</TD> </TR>
</TABLE>

Bag of words can be useful for any number of reasons within the scope of analyzing discourse.  Many other qdap functions employ or mention these three functions as seen in the following counts for the three word splitting functions.

```{r, eval=FALSE, echo=FALSE, include = FALSE}
library(acc.roxygen2)
x <- search_repo(bag_o_words, breaker, word.split)
print(xtable(x), type="html")
```


<TABLE border=1>
 <TR> <TD align="right">  </TD> <TD><b>Function</b> </TD> <TD> <b>bag_o_words</b> </TD> <TD> <b>breaker</b></TD> <TD> <b>word.split</b></TD> </TR>
 
 
  <TR> <TD align="right"> 1 </TD> <TD> all_words.R                   </TD> <TD> 1 </TD> <TD> - </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 2 </TD> <TD> automated_readability_index.R </TD> <TD> - </TD> <TD> - </TD> <TD> 2 </TD> </TR>
  <TR> <TD align="right"> 3 </TD> <TD> bag_o_words.R                 </TD> <TD> 10 </TD> <TD> 6 </TD> <TD> 3 </TD> </TR>
  <TR> <TD align="right"> 4 </TD> <TD> capitalizer.R                 </TD> <TD> 3 </TD> <TD> 1 </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 5 </TD> <TD> imperative.R                  </TD> <TD> - </TD> <TD> 3 </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 6 </TD> <TD> ngrams.R                      </TD> <TD> 1 </TD> <TD> - </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 7 </TD> <TD> polarity.R                    </TD> <TD> 2 </TD> <TD> - </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 8 </TD> <TD> rm_stopwords.R                   </TD> <TD> 1 </TD> <TD> 3 </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 9 </TD> <TD> textLISTER.R                  </TD> <TD> - </TD> <TD> - </TD> <TD> 2 </TD> </TR>
  <TR> <TD align="right"> 10 </TD> <TD> trans_cloud.R                 </TD> <TD> 1 </TD> <TD> 1 </TD> <TD> - </TD> </TR>
  <TR> <TD align="right"> 11 </TD> <TD> wfm.R                         </TD> <TD> 1 </TD> <TD> - </TD> <TD> - </TD> </TR>
   </TABLE>
<br>

`r FT(orange, 5, text="&diams;")` **Word Splitting Examples**`r FT(orange, 5, text="&diams;")`

```{r}
bag_o_words("I'm going home!")
bag_o_words("I'm going home!", apostrophe.remove = TRUE)
bag_o_words(DATA$state)
by(DATA$state, DATA$person, bag_o_words)
lapply(DATA$state,  bag_o_words)
breaker(DATA$state)
by(DATA$state, DATA$person, breaker)
lapply(DATA$state,  breaker)
word_split(c(NA, DATA$state))
```


<h4 id="common">Find Common Words Between Groups</h4>

The `r FUN("common")` function finds items that are common between n vectors 
(i.e., subjects or grouping variables).  This is useful for determining common language choices shared across participants in a conversation.

`r FT(orange, 5, text="&diams;")` **Words in Common Examples**`r FT(orange, 5, text="&diams;")`

```{r}
## Create vectors of words
a <- c("a", "cat", "dog", "the", "the")
b <- c("corn", "a", "chicken", "the")
d <- c("house", "feed", "a", "the", "chicken")

## Supply individual vectors
common(a, b, d, overlap=2)
common(a, b, d, overlap=3)
## Supply a list of vectors
common(list(a, b, d))
## Using to find common words between subjects
common(word_list(DATA$state, DATA$person)$cwl, overlap = 2)
```


<h4 id="exclude">Exclude Elements From a Vector</h4>

It is often useful and more efficient to start with a preset vector of words and eliminate or `r FUN("exclude")` the words you do not wish to include.  Examples could range from excluding an individual(s) from a column of participant names or excluding a few select word(s) from a pre-defined qdap word list.  This is particularly useful for passing terms or stopwords to word counting functions like `r FUN("termco")` or `r FUN("trans_cloud")`.

`r FT(orange, 5, text="&diams;")` **`r FUN("exclude")` Examples**`r FT(orange, 5, text="&diams;")`
```{r}
exclude(1:10, 3, 4)
exclude(Top25Words, qcv(the, of, and))
exclude(Top25Words, "the", "of", "an")
#Using with `term_match` and `termco`
MTCH.LST <- exclude(term_match(DATA$state, qcv(th, i)), qcv(truth, stinks))
termco(DATA$state, DATA$person, MTCH.LST)
```

<h4 id="ngramn">Generate ngrams</h4>

Utilizing `r HR2("http://en.wikipedia.org/wiki/N-gram", "ngrams")` can be useful for gaining a sense of what terms are used in conjunction with other terms.  This is particularly useful in the analysis of dialogue when the combination of a particular vocabulary is meaningful.  The `r FUN("ngrams")` function provides a list of ngram related output that can be utilize in various analyses.

`r FT(orange, 5, text="&diams;")` **`r FUN("ngrams")` Example** *note that the output is only partial*`r FT(orange, 5, text="&diams;")`

```{r}
out <- ngrams(DATA$state, DATA$person, 2)
lapply(out[["all_n"]], function(x) sapply(x, paste, collapse = " "))
```

<h4 id="stopwords">Remove Stopwords</h4>

In analyzing discourse it may be helpful to remove certain words from the analysis as the words may not be meaningful or may overshadow the impact of other words.  The `r FUN("rm_stopwords")` function can be utilized to remove `r HR2("http://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html", "rm_stopwords")` from the dialogue before passing to further analysis.  It should be noted that many functions have a stopwords argument that allows for the removal of the stopwords within the function environment rather than altering the text in the primary discourse dataframe.  Careful researcher consideration must be given as to the functional impact of removing words from an analysis.

`r FT(orange, 5, text="&diams;")` **Stopword Removal Examples**`r FT(orange, 5, text="&diams;")`

```{r}
## The data
DATA$state
rm_stopwords(DATA$state, Top200Words)
rm_stopwords(DATA$state, Top200Words, strip = TRUE)
rm_stopwords(DATA$state, Top200Words, separate = FALSE)
rm_stopwords(DATA$state, Top200Words, unlist = TRUE, unique = TRUE)
```

<h4 id="strip">Strip Text of Unwanted Characters/Capitalization</h4>

It is often useful to remove capitalization and punctuation from the dialogue in order to standardize the text.  R is case sensitive.  By removing capital letters and extra punctuation with the `r FUN("strip")` function the text is more comparable.  In the following output we can see, through the `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/Comparison.html", "==")` comparison operator and `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/Comparison.html", "outer")` function that the use of `r FUN("strip")` makes the different forms of `r FT(blue, text="Dan")` comparable.

```{r}
x <- c("Dan", "dan", "dan.", "DAN")
y <- outer(x, x, "==")
dimnames(y) <- list(x, x); y
x <- strip(c("Dan", "dan", "dan.", "DAN"))
y <- outer(x, x, "==")
dimnames(y) <- list(x, x); y
```

As seen in the examples below, `r FUN("strip")` comes with multiple arguments to adjust the flexibility of the degree of text standardization.

`r FT(orange, 5, text="&diams;")` **`r FUN("strip")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
## Demonstrating the standardization of 
## The data
DATA$state
strip(DATA$state)
strip(DATA$state, apostrophe.remove=FALSE)
strip(DATA$state, char.keep = c("?", "."))
```

<h4 id="syn">Search For Synonyms</h4>

It is useful in discourse analysis to analyze vocabulary use.  This may mean searching for words similar to your initial word list.  The `r FUN("synonyms")` (aka `r FUN("syn")`) function generates synonyms from the `r HR2("http://trinker.github.io/qdapDictionaries/", "qdapDictionaries'")` `r HR2("http://trinker.github.io/qdapDictionaries/SYNONYM.html", "SYNONYM")` dictionary.  These synonyms can be returned as a list or a vector that can then be passed to other qdap functions.

`r FT(orange, 5, text="&diams;")` **Synonyms Examples**`r FT(orange, 5, text="&diams;")`

```{r}
synonyms(c("the", "cat", "teach"))
syn(c("the", "cat", "teach"), return.list = FALSE)
syn(c("the", "cat", "teach"), multiwords = FALSE)
```

<h4 id="assoc">Find Associated Words</h4>

`r FT(orange, 5, text="&diams;")` **Word Association Examples**`r FT(orange, 5, text="&diams;")`

```{r}
ms <- c(" I ", "you")
et <- c(" it", " tell", "tru")
word_associate(DATA2$state, DATA2$person, match.string = ms,
    wordcloud = TRUE,  proportional = TRUE,
    network.plot = TRUE,  nw.label.proportional = TRUE, extra.terms = et,
    cloud.legend =c("A", "B", "C"),
    title.color = "blue", cloud.colors = c("red", "purple", "gray70"))

```

<h4 id="diffs">Differences In Word Use Between Groups</h4>

`r FT(orange, 5, text="&diams;")` **Word Difference Examples**`r FT(orange, 5, text="&diams;")`

```{r}
out <- with(DATA, word_diff_list(text.var = state,
    grouping.var = list(sex, adult)))

ltruncdf(unlist(out, recursive = FALSE), n=4)
```

<h4 id="word_list">Raw Word Lists/Frequency Counts</h4>

`r FT(orange, 5, text="&diams;")` **`r FUN("word_list")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
with(DATA, word_list(state, person))
with(DATA, word_list(state, person, stopwords = Top25Words))
with(DATA, word_list(state, person, cap = FALSE, cap.list=c("do", "we")))
```
   
<h3 id="coding">Qualitative Coding System</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more): <br>   

<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_code.blank.html" target="_blank">
    <input type="submit" value="cm_code.blank"> 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_code.combine.html" target="_blank">
    <input type="submit" value="cm_code.combine"> 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_code.exclude.html" target="_blank">
    <input type="submit" value="cm_code.exclude"> 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_code.overlap.html" target="_blank">
    <input type="submit" value="cm_code.overlap"> 
</form>

<form action="http://trinker.github.io/qdap_dev/cm_code.transform.html" target="_blank">
    <input type="submit" value="cm_code.transform"> - `r HR("#reshape", "Combine, Exclude, and Overlap Codes")`
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_df.temp.html" target="_blank">
    <input type="submit" value="cm_df.temp"> 
</form>

<form action="http://trinker.github.io/qdap_dev/cm_2long.html" target="_blank">
    <input type="submit" value="cm_long"> - `r HR("#wordcsv", "Coding Words:  .csv Approach")`
</form>



<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_range.temp.html" target="_blank">
    <input type="submit" value="cm_range.temp">
</form>
<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_df.transcript.html" target="_blank">
    <input type="submit" value="cm_df.transcript"> 
</form>
<form action="http://trinker.github.io/qdap_dev/cm_2long.html" target="_blank">
    <input type="submit" value="cm_2long">  - `r HR("#wordtrans", "Coding Words: Transcript & List Approach")`
</form>



<form class="form_left" action="http://trinker.github.io/qdap_dev/cm_time.temp.html" target="_blank">
    <input type="submit" value="cm_time.temp">
</form>

<form action="http://trinker.github.io/qdap_dev/cm_2long.html" target="_blank">
    <input type="submit" value="cm_2long"> - `r HR("#timespan", "Coding Words: Time Spans Approach")` 
</form>


<form action="http://trinker.github.io/qdap_dev/cm_distance.html" target="_blank">
    <input type="submit" value="cm_distance"> - `r HR("#cmdist", "Distance Matrix Between Codes")`
</form>


</div>

A major task in qualitative work is coding either time or words with selected coding structures.  For example a researcher may code the teacher's dialogue as related to the resulting behavior of a student in a classroom as "high", "medium" or "low" engagement. The researcher may choose to apply the coding to:

- The dialogue
- The time spans

The coding process in qdap starts with the decision of whether to code the dialogue and/or the time spans.  After that the researcher may follow the sequential subsections in the `r HR("#coding", "Qualitative Coding System")` section outlined in these steps:

1. Making a template for coding dialogue/time spans
2. The actual coding  dialogue/time spans
3. Reading in the dialogue/time spans
4. Transforming codes (finding overlap and/or differences between word span/time span of codes)
5. Initial analysis

If you choose the route of coding words qdap gives two approaches.  Each has distinct benefits and disadvantages dependent upon the situation.  If you chose the coding of time spans qdap provides one option. 

If you chose the coding of words you may choose to code a csv file or to code the transcript directly (perhaps with markers or other forms of markup), record the ranges in a text list and then read in the data.  Both approaches can result in the same data being read back into qdap.  The csv approach may allow for extended capabilities (beyond the scope of this vignette) while the transcript/list approach is generally more efficient and takes the approach many qualitative researchers typically utilize in qualitative coding (it also has the added benefit of producing a hard copy).

The next three subsections will walk the reader through how to make a template, code in the template, and read the data back into R/qdap.  Subsections 4-5 will cover reshaping and initial analysis after the data has been read in (this approach is generally the same for all three coded data types).

1. `r HR("#wordcsv", "Coding Words - The .csv Approach")` - How to template, code, read in and reshape the data
2. `r HR("#wordtrans", "Coding Words - The Transcript/List Approach")` - How to template, code, read in  and reshape the data
3. `r HR("#timespan", "Coding Time Spans")` - How to template, code, read in and reshape the data
4. `r HR("#reshape", "Transforming Codes")`
5. `r HR("#analysis", "Initial Coding Analysis")`

Before getting started with subsections 1-3 the reader will want to know the naming scheme of the code matrix (`r FT(red, text="cm&#95;")`) functions used.  The initial `r FT(red, text="cm&#95;")` is utilized for any code matrix family of functions.  The functions containing `r FT(red, text="cm&#95;temp")` are template functions.  The `r FT(red, text="df")`, `r FT(red, text="range")`, or `r FT(red, text="time")` determine whether the csv (`r FT(red, text="df")`), Transcript/List (`r FT(red, text="range")`), or Time Span (`r FT(red, text="time")`) approach is being utilized.  `r FT(red, text="cm&#95;")` functions that bear `r FT(red, text="2long")` transform a read in list to a usable long format.

<h4 id="wordcsv">Coding Words - The .csv Approach `r yt("http://www.youtube.com/watch?v=tH242SIESIs")`</h4>

The csv approach utilizes `r FUN("cm_df.temp")` and `r FUN("cm_2long")` functions.  To utilize the csv template approach simply supply the dataframe, specify the text variable and provide a list of anticipated codes.  

`r FT(orange, 5, text="&diams;")` **Coding Words (csv approach)**: The Template `r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Codes
codes <- qcv(dc, sf, wes, pol, rejk, lk, azx, mmm)

## The csv template
X <- cm_df.temp(DATA, text.var = "state", codes = codes, file = "DATA.csv")
qview(X)
</code></pre>

<pre><code>========================================================================
nrow =  56           ncol =  14             X
========================================================================
   person sex adult code     text word.num dc sf wes pol rejk lk azx mmm
1     sam   m     0   K1 Computer        1  0  0   0   0    0  0   0   0
2     sam   m     0   K1       is        2  0  0   0   0    0  0   0   0
3     sam   m     0   K1     fun.        3  0  0   0   0    0  0   0   0
4     sam   m     0   K1      Not        4  0  0   0   0    0  0   0   0
5     sam   m     0   K1      too        5  0  0   0   0    0  0   0   0
6     sam   m     0   K1     fun.        6  0  0   0   0    0  0   0   0
7    greg   m     0   K2       No        7  0  0   0   0    0  0   0   0
8    greg   m     0   K2     it's        8  0  0   0   0    0  0   0   0
9    greg   m     0   K2     not,        9  0  0   0   0    0  0   0   0
10   greg   m     0   K2     it's       10  0  0   0   0    0  0   0   0
</code></pre>

After coding the data (see the `r HR2("http://www.youtube.com/watch?v=tH242SIESIs", "YouTube video")`) the data can be read back in with `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html", "read.csv")`.


`r FT(orange, 5, text="&diams;")` **Coding Words (csv approach)**: Read In and Reshape `r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Read in the data
dat <- read.csv("DATA.csv")

## Reshape to long format with word durations
cm_2long(dat)
</code></pre>

<pre><code>    code     person sex adult code.1     text word.num start end variable
1     dc        sam   m     0     K1 Computer        1     0   1      dat
2    wes        sam   m     0     K1 Computer        1     0   1      dat
3   rejk        sam   m     0     K1 Computer        1     0   1      dat
4    mmm        sam   m     0     K1 Computer        1     0   1      dat
5     lk        sam   m     0     K1       is        2     1   2      dat
6    azx        sam   m     0     K1       is        2     1   2      dat
.
.
.
198  wes       greg   m     0    K11 already?       56    55  56      dat
199 rejk       greg   m     0    K11 already?       56    55  56      dat
200   lk       greg   m     0    K11 already?       56    55  56      dat
201  azx       greg   m     0    K11 already?       56    55  56      dat
202  mmm       greg   m     0    K11 already?       56    55  56      dat
</code></pre>


<h4 id="wordtrans">Coding Words - The Transcript/List Approach `r yt("http://www.youtube.com/watch?v=cxcD-j0iI2U")`</h4>

The Transcript/List approach utilizes `r FUN("cm_df.transcript")`,  `r FUN("cm_range.temp")` and `r FUN("cm_2long")` functions.  To use the transcript template simply supply the dataframe, specify the text variable and provide a list of anticipated codes.  

`r FT(orange, 5, text="&diams;")` **Coding Words (Transcript/List approach)**: Transcript Template `r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Codes
codes <- qcv(AA, BB, CC)

## Transcript template
X <- cm_df.transcript(DATA$state, DATA$person, file="DATA.txt")
</code></pre>

<pre><code>sam:

                                  
     1        2  3    4   5   6   
     Computer is fun. Not too fun.

greg:

                            
     7  8    9    10   11   
     No it's not, it's dumb.

teacher:

                       
     12   13     14 15 
     What should we do?

sam:

                         
     16  17    18 19     
     You liar, it stinks!
</code></pre>

`r FT(orange, 5, text="&diams;")` **Coding Words (Transcript/List approach)**: List Template 1`r FT(orange, 5, text="&diams;")`

<pre><code class="r">### List template
cm_range.temp(codes, file = "foo1.txt")
</code></pre>

<pre><code>list(
    AA = qcv(terms=''),
    BB = qcv(terms=''),
    CC = qcv(terms='')
)
</code></pre>

This list below contains demographic variables.  If the researcher has demographic variables it is recommended to supply them at this point.  The demographic variables will be generated with durations automatically.

`r FT(orange, 5, text="&diams;")` **Coding Words (Transcript/List approach)**: List Template 2`r FT(orange, 5, text="&diams;")`

<pre><code class="r">### List template with demographic variables
with(DATA, cm_range.temp(codes = codes, text.var = state, 
    grouping.var = list(person, adult), file = "foo2.txt"))
</code></pre>

<pre><code>list(
    person_greg = qcv(terms='7:11, 20:24, 30:33, 49:56'),
    person_researcher = qcv(terms='42:48'),
    person_sally = qcv(terms='25:29, 37:41'),
    person_sam = qcv(terms='1:6, 16:19, 34:36'),
    person_teacher = qcv(terms='12:15'),
    adult_0 = qcv(terms='1:11, 16:41, 49:56'),
    adult_1 = qcv(terms='12:15, 42:48'),
    AA = qcv(terms=''),
    BB = qcv(terms=''),
    CC = qcv(terms='')
)
</code></pre>

After coding the data (see the `r HR2("http://www.youtube.com/watch?v=cxcD-j0iI2U", "YouTube video")`) the data can be read back in with `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/source.html", "source")`.  Be sure to assign list to an object (e.g., `dat <- list()`).

`r FT(orange, 5, text="&diams;")` **Coding Words (Transcript/List approach)**: Read in the data`r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Read it in
source("foo1.txt")

### View it
Time1
</code></pre>

<pre><code>$AA
[1] "1"

$BB
[1] "1:2,"  "3:10," "19"   

$CC
[1] "1:9,"    "100:150"
</code></pre>

This format is not particularly useful.  The data can be reshaped to long format with durations via `r FUN("cm_2long")`:

`r FT(orange, 5, text="&diams;")` **Coding Words (Transcript/List approach)**: Long format`r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Long format with durations
datL <- cm_2long(Time1)
datL
</code></pre>

<pre><code>  code start end variable
1   AA     0   1    Time1
2   BB     0   2    Time1
3   BB     2  10    Time1
4   BB    18  19    Time1
5   CC     0   9    Time1
6   CC    99 150    Time1
</code></pre>

<h4 id="timespan">Coding Time Spans `r yt("http://youtu.be/XC-RXeY63bM")`</h4>

The Time Span approach utilizes the `r FUN("cm_time.temp")` and `r FUN("cm_2long")` functions.  To generate the timespan template approach simply supply the list of anticipated codes and a start/end time.  


`r FT(orange, 5, text="&diams;")` **Coding Times Spans**: Time Span Template `r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Codes
## Time span template
X <- cm_time.temp(start = ":14", end = "7:40", file="timespans.txt")
X <- cm_time.temp(start = ":14", end = "7:40", file="timespans.doc")
</code></pre>


<pre><code>[0]                                14 15 16 ... 51 52 53 54 55 56 57 58 59
[1]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[2]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[3]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[4]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[5]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[6]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53 54 55 56 57 58 59
[7]0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ... 51 52 53                                                
</code></pre>


`r FT(orange, 5, text="&diams;")` **Coding Times Spans**: List Template 1`r FT(orange, 5, text="&diams;")`

<pre><code class="r">### List template
codes <- qcv(AA, BB, CC)
cm_time.temp(codes, file = "codelist.txt")
</code></pre>

<pre><code> list(                                                 
     transcript_time_span = qcv(terms="00:00 - 00:00"),
     AA = qcv(terms=""),                               
     BB = qcv(terms=""),                               
     CC = qcv(terms="")                                
 )  
</code></pre>

This list below contains demographic variables.  If the researcher has demographic variables it is recommended to supply them at this point.  

`r FT(orange, 5, text="&diams;")` **Coding Times Spans**: List Template 2`r FT(orange, 5, text="&diams;")`

<pre><code class="r">### List template with demographic variables
with(DATA, cm_time.temp(codes, list(person, adult), file = "codelist.txt"))
</code></pre>

<pre><code>list(
    transcript_time_span = qcv(terms="00:00 - 00:00"),
    person_sam = qcv(terms=""),
    person_greg = qcv(terms=""),
    person_teacher = qcv(terms=""),
    person_sally = qcv(terms=""),
    person_researcher = qcv(terms=""),
    adult_0 = qcv(terms=""),
    adult_1 = qcv(terms=""),
    AA = qcv(terms=""),
    BB = qcv(terms=""),
    CC = qcv(terms="")
)
</code></pre>

After coding the data (see the `r HR2("http://www.youtube.com/watch?v=XC-RXeY63bM&feature=youtu.be", "YouTube video")`) the data can be read back in with `r HR2("http://stat.ethz.ch/R-manual/R-devel/library/base/html/source.html", "source")`.  Be sure to assign list to an object (e.g., `dat <- list()`).  

`r FT(orange, 5, text="&diams;")` **Coding Times Spans**: Read in the data`r FT(orange, 5, text="&diams;")`


<pre><code class="r">## Read it in
source("codelist.txt")

### View it
Time1
</code></pre>

<pre><code>$transcript_time_span
[1] "00:00"   "-"       "1:12:00"

$A
[1] "2.40:3.00," "5.01,"      "6.52:7.00," "9.00"      

$B
[1] "2.40,"      "3.01:3.40," "5.01,"      "6.52:7.00," "9.00"      

$C
[1] "2.40:4.00,"  "5.01,"       "6.52:7.00,"  "9.00,"       "13.00:17.01"
</code></pre>

This format is not particularly useful.  The data can be reshaped to long format with durations via `r FUN("cm_2long")`:

`r FT(orange, 5, text="&diams;")` **Coding Times Spans**: Long format`r FT(orange, 5, text="&diams;")`

<pre><code class="r">## Long format with durations
datL <- cm_2long(Time1, v.name = "time")
datL
</code></pre>

<pre><code>   code start  end    Start      End variable
1     A   159  180 00:02:39 00:03:00    Time1
2     A   300  301 00:05:00 00:05:01    Time1
3     A   411  420 00:06:51 00:07:00    Time1
4     A   539  540 00:08:59 00:09:00    Time1
5     B   159  160 00:02:39 00:02:40    Time1
6     B   180  220 00:03:00 00:03:40    Time1
7     B   300  301 00:05:00 00:05:01    Time1
8     B   411  420 00:06:51 00:07:00    Time1
9     B   539  540 00:08:59 00:09:00    Time1
10    C   159  240 00:02:39 00:04:00    Time1
11    C   300  301 00:05:00 00:05:01    Time1
12    C   411  420 00:06:51 00:07:00    Time1
13    C   539  540 00:08:59 00:09:00    Time1
14    C   779 1021 00:12:59 00:17:01    Time1
</code></pre>


<h4 id="reshape">Transforming Codes</h4>

The researcher may want to determine where codes do and do not overlap with one other.  The `r FT(red, text="cm_")` family of functions bearing (`r FT(red, text="cm_code.")`) perform various transformative functions (Boolean search).  `r FUN("cm_code.combine")` will merge the spans (time or word) for given codes.  `r FUN("cm_code.exclude")` will give provide spans that exclude given codes.  `r FUN("cm_code.overlap")` will yield the spans where all of the given codes co-occur.  `r FUN("cm_code.transform")` is a wrapper for the previous three functions that produces one dataframe in a single call.  Lastly, `r FUN("cm_code.blank")` provides a more flexible framework that allows for the introduction of multiple logical operators between codes.  Most tasks can be handled with the `r FUN("cm_code.transform")` function.

For Examples of each click the links below:    
1. `r HR("#cm_code.combine", "cm_code.combine Examples")`     
2. `r HR("#cm_code.exclude", "cm_code.exclude Examples")`  
3. `r HR("#cm_code.overlap", "cm_code.overlap Examples")`       
4. `r HR("#cm_code.transform", "cm_code.transform Examples")`    
5. `r HR("#cm_code.blank", "cm_code.blank Examples")`    

For the sake of simplicity the uses of these functions will be demonstrated via a gantt plot for a visual comparison of the data sets.

The reader should note that all of the above functions utilize two helper functions (`r FUN("cm_long2dummy")` and `r FUN("cm_dummy2long")`) to stretch the spans into single units of measure (word or second) perform a calculation and then condense back to spans.  More advanced needs may require the explicit use of these functions, though they are beyond the scope of this vignette.  

The following data sets will be utilized throughout the demonstrations of the `r FT(red, text="cm_code.")` family of functions:

`r FT(orange, 5, text="&diams;")` **Common Data Sets** - Word Approach`r FT(orange, 5, text="&diams;")`

```{r}
foo <- list(
    AA = qcv(terms="1:10"),
    BB = qcv(terms="1:2, 3:10, 19"),
    CC = qcv(terms="1:3, 5:6")
)

foo2 <- list(
    AA = qcv(terms="4:8"),
    BB = qcv(terms="1:4, 10:12"),
    CC = qcv(terms="1, 11, 15:20"),
    DD = qcv(terms="")
)
```

```{r}
## Single time, long word approach
(x <- cm_2long(foo))
```

```{r echo=FALSE, fig.height = 2.5}
gantt_wrap(x, "code")
```

```{r}
## Repeated measures, long word approach
(z <- cm_2long(foo, foo2, v.name="time"))
```

```{r echo=FALSE, fig.height = 5}
gantt_wrap(z, "code", "time")
```


`r FT(orange, 5, text="&diams;")` **Common Data Sets** - Time Span Approach`r FT(orange, 5, text="&diams;")`

```{r}
bar1 <- list(
    transcript_time_span = qcv(00:00 - 1:12:00),
    A = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00"),
    B = qcv(terms = "2.40, 3.01:3.02, 5.01, 6.02:7.00, 9.00,
        1.12.00:1.19.01"),
    C = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00, 16.25:17.01")
)

bar2 <- list(
    transcript_time_span = qcv(00:00 - 1:12:00),
    A = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00"),
    B = qcv(terms = "2.40, 3.01:3.02, 5.01, 6.02:7.00, 9.00,
        1.12.00:1.19.01"),
    C = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00, 17.01")
)
```

```{r}
## Single time, long time approach
(dat <- cm_2long(bar1))
```

```{r echo=FALSE, fig.height = 2.5}
gantt_wrap(dat, "code")
```

```{r}
## Repeated measures, long time approach
(dats <- cm_2long(bar1, bar2, v.name = "time"))
```

```{r echo=FALSE, fig.height = 5}
gantt_wrap(dats, "code", "time")
```

<h5 id="cm_code.combine"><font color="green">cm_code.combine Examples</font></h5>

`r FUN("cm_code.combine")` provides all the spans (time/words) that are occupied by one or more of the combined codes.  For example, if we utilized `r FUN("cm_code.combine")` on code list X and Y the result would be any span where X or Y is located. This is the OR of the Boolean search.  Note that `combine.code.list` must be supplied as a list of named character vectors.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.combine")` Single Time** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
(cc1 <- cm_code.combine(x, list(ALL=qcv(AA, BB, CC))))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(cc1, "code")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.combine")` Repeated Measures** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
combines <- list(AB=qcv(AA, BB), ABC=qcv(AA, BB, CC))
(cc2 <- cm_code.combine(z, combines, rm.var = "time"))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cc2, "code", "time")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.combine")` Single Time** *Time Span Example*`r FT(orange, 5, text="&diams;")`

```{r}
combines2 <- list(AB=qcv(A, B), BC=qcv(B, C), ABC=qcv(A, B, C))
(cc3 <- cm_code.combine(dat, combines2))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(cc3, "code")
```

<h5 id="cm_code.exclude"><font color="green">cm_code.exclude Examples</font></h5>

`r FUN("cm_code.exclude")` provides all the spans (time/words) that are occupied by one or more of the combined codes with the exclusion of another code.  For example, if we utilized `r FUN("cm_code.combine")` on code list X and Y the result would be any span where X is located but Y is not. This is the NOT of the Boolean search.  The last term supplied to exclude.code.list is the excluded term.  All other terms are combined and the final code term is partitioned out.  Note that `exclude.code.list` must be supplied as a list of named character vectors.


`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.exclude")` Single Time** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
(ce1 <- cm_code.exclude(x, list(BnoC=qcv(BB, CC))))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ce1, "code")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.exclude")` Repeated Measures** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
exlist <- list(AnoB=qcv(AA, BB), ABnoC=qcv(AA, BB, CC))
(ce2 <- cm_code.exclude(z, exlist, rm.var = "time"))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(ce2, "code", "time")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.exclude")` Repeated Measures** *Time Span Example*`r FT(orange, 5, text="&diams;")`

```{r}
exlist2 <- list(AnoB=qcv(A, B), BnoC=qcv(B, C), ABnoC=qcv(A, B, C))
(ce3 <- cm_code.exclude(dats, exlist2, "time"))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ce3, "code")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.exclude")` Single Time** *Time Span Combined Exclude Example*`r FT(orange, 5, text="&diams;")`

```{r}
(ce4.1 <- cm_code.combine(dat, list(AB = qcv(A, B))))
(ce4.2 <- cm_code.exclude(ce4.1, list(CnoAB = qcv(C, AB))))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ce4.2, "code")
```

<h5 id="cm_code.overlap"><font color="green">cm_code.overlap Examples</font></h5>

`r FUN("cm_code.overlap")` provides all the spans (time/words) that are occupied by all of the given codes.  For example, if we utilized `r FUN("cm_code.overlap")` on code list X and Y the result would be any span where X and Y are both located. This is the AND of the Boolean search.  Note that `overlap.code.list` must be supplied as a list of named character vectors.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.overlap")` Single Time** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
(co1 <- cm_code.overlap(x, list(BC=qcv(BB, CC))))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(co1, "code")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.overlap")` Repeated Measures** *Word Example*`r FT(orange, 5, text="&diams;")`

```{r}
overlist <- list(AB=qcv(AA, BB), ABC=qcv(AA, BB, CC))
(co2 <- cm_code.overlap(z, overlist, rm.var = "time"))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(co2, "code", "time")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.overlap")` Repeated Measures** *Time Span Example*`r FT(orange, 5, text="&diams;")`

```{r}
overlist2 <- list(AB=qcv(A, B), BC=qcv(B, C), ABC=qcv(A, B, C))
(co3 <- cm_code.overlap(dats, overlist2, "time"))
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(co3, "code")
```

<h5 id="cm_code.transform"><font color="green">`r FUN("cm_code.transform")` Examples</font></h5>

`r FUN("cm_code.transform")` is merely a wrapper for `r FUN("cm_code.combine")`, `r FUN("cm_code.exclude")`, and `r FUN("cm_code.overlap")`.


`r FT(orange, 5, text="&diams;")` <b>`r FUN("cm_code.transform")`</b> - Example 1`r FT(orange, 5, text="&diams;")`

```{r}
ct1 <- cm_code.transform(x, 
    overlap.code.list = list(oABC=qcv(AA, BB, CC)),
    combine.code.list = list(ABC=qcv(AA, BB, CC)), 
    exclude.code.list = list(ABnoC=qcv(AA, BB, CC))
)
ct1
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ct1, "code")
```

`r FT(orange, 5, text="&diams;")` <b>`r FUN("cm_code.transform")`</b> - Example 2`r FT(orange, 5, text="&diams;")`

```{r}
ct2 <-cm_code.transform(z, 
    overlap.code.list = list(oABC=qcv(AA, BB, CC)),
    combine.code.list = list(ABC=qcv(AA, BB, CC)), 
    exclude.code.list = list(ABnoC=qcv(AA, BB, CC)), "time"
)
ct2
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ct2, "code")
```

`r FT(orange, 5, text="&diams;")` <b>`r FUN("cm_code.transform")`</b> - Example 3`r FT(orange, 5, text="&diams;")`

```{r}
ct3 <-cm_code.transform(dat, 
    overlap.code.list = list(oABC=qcv(A, B, C)),
    combine.code.list = list(ABC=qcv(A, B, C)), 
    exclude.code.list = list(ABnoC=qcv(A, B, C))
)
ct3
```

```{r, echo = FALSE, fig.height = 2.5}
gantt_wrap(ct3, "code")
```


<h5 id="cm_code.blank"><font color="green">cm_code.blank Examples</font></h5>

`r FUN("cm_code.blank")` provides flexible Boolean comparisons between word.time spans.  The `overlap` argument takes a logical value, an integer or a character string of binary operator couple with an integer.  It is important to understand how the function operates.  This initial step calls `r FUN("cm_long2dummy")` as seen below (stretching the spans to dummy coded columns), the comparison is conduted between columns, and then the columns are reverted back to spans via the `r FUN("cm)dummy2long")`.  This first example illustrates the stretching to dummy and reverting back to spans.

`r FT(orange, 5, text="&diams;")` **Long to dummy and dummy to long** `r FT(orange, 5, text="&diams;")`

```{r comment=NA}
long2dummy <- cm_long2dummy(x, "variable")
list(original =x,
    long_2_dummy_format = long2dummy[[1]],
    dummy_back_2_long = cm_dummy2long(long2dummy, "variable")
)
```

Now let's examine a few uses of `r FUN("cm_code.blank")`.  The first is to set `overlap = TRUE` (the default behavior).  This default behavior is identical to `r FUN("cm_code.overlap")` as seen below.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.blank")`** - `overlap = TRUE` `r FT(orange, 5, text="&diams;")`

```{r}
(cb1 <- cm_code.blank(x, list(ABC=qcv(AA, BB, CC))))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cb1, "code")
```

Next we'll set `overlap = FALSE` and see that it is identical to `r FUN("cm_code.combine")`.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.blank")`** - `overlap = FALSE` `r FT(orange, 5, text="&diams;")`

```{r}
(cb2 <- cm_code.blank(x, list(ABC=qcv(AA, BB, CC)), overlap = FALSE))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cb2, "code")
```


By first combining all codes (see `cb2` above) and then excluding the final code by setting
`overlap = 1` the behavior of `r FUN("cm_code.exclude")` can be mimicked. 

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.blank")`** - *mimicking `r FUN("cm_code.exclude")`* `r FT(orange, 5, text="&diams;")`

```{r}
## Using the output from `cb2` above.
(cb3 <- cm_code.blank(cb2, list(ABnoC=qcv(ABC, CC)), overlap = 1))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cb3, "code")
```

Next we shall find when at least two codes overlap by setting `overlap = ">1"`.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.blank")`** - *At least 2 codes overlap* `r FT(orange, 5, text="&diams;")`


```{r}
blanklist <- list(AB=qcv(AA, BB), ABC=qcv(AA, BB, CC))
(cb4 <- cm_code.blank(z, blanklist, rm.var = "time", overlap = ">1"))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cb4, "code", "time")
```

Last, we will find spans where not one of the codes occurred by setting `overlap = "==0"`.

`r FT(orange, 5, text="&diams;")` **`r FUN("cm_code.blank")`** - *Spans where no code occurs* `r FT(orange, 5, text="&diams;")`

```{r}
blanklist2 <- list(noAB=qcv(AA, BB), noABC=qcv(AA, BB, CC))
(cb5 <- cm_code.blank(z, blanklist2, rm.var = "time", overlap = "==0"))
```

```{r, echo = FALSE, fig.height = 5}
gantt_wrap(cb5, "code", "time")
```

<h4 id="analysis">Initial Coding Analysis</h4>

The `r FT(red, text="cm_")` family of functions has three approaches to initial analysis of codes.  The researcher may want to summarize, visualize or determine the proximity of codes to one another.  The following functions accomplish these tasks:

1. `r HR("#cmsum", "Summary")`    
2. `r HR("#cmplot", "Plotting")`    
2. `r HR("#cmdist", "Distance Measures")`    

<h5 id="cmsum"><font color="green">Summary</font></h5>

Most of the `r FT(red, text="cm_")` family of functions have a `r FUN("summary", "summary.cmspans")` method to allows for summaries of codes by group.  Note that these summaries can be wrapped with `r FUN("plot", "plot.sum_cmspans")` to print a heat map of the table of summaries.

`r FT(orange, 5, text="&diams;")` **Example 1: Summarizing Transcript/List Approach** `r FT(orange, 5, text="&diams;")`

```{r, message=FALSE}
## Two transcript lists
A <- list(
    person_greg = qcv(terms='7:11, 20:24, 30:33, 49:56'),
    person_researcher = qcv(terms='42:48'),
    person_sally = qcv(terms='25:29, 37:41'),
    person_sam = qcv(terms='1:6, 16:19, 34:36'),
    person_teacher = qcv(terms='12:15'),
    adult_0 = qcv(terms='1:11, 16:41, 49:56'),
    adult_1 = qcv(terms='12:15, 42:48'),
    AA = qcv(terms="1"),
    BB = qcv(terms="1:2, 3:10, 19"),
    CC = qcv(terms="1:9, 100:150")
)

B  <- list(
    person_greg = qcv(terms='7:11, 20:24, 30:33, 49:56'),
    person_researcher = qcv(terms='42:48'),
    person_sally = qcv(terms='25:29, 37:41'),
    person_sam = qcv(terms='1:6, 16:19, 34:36'),
    person_teacher = qcv(terms='12:15'),
    adult_0 = qcv(terms='1:11, 16:41, 49:56'),
    adult_1 = qcv(terms='12:15, 42:48'),
    AA = qcv(terms="40"),
    BB = qcv(terms="50:90"),
    CC = qcv(terms="60:90, 100:120, 150"),
    DD = qcv(terms="")
)

## Long format for transcript/list approach
v <- cm_2long(A, B, v.name = "time")
head(v)
```

```{r eval = FALSE}
## Summary of the data and plotting the summary
summary(v)
```

<pre><code>time              code total percent_total n percent_n  ave min max   mean(sd)
1  a       person_greg    22         12.0% 4     18.2%  5.5   4   8   5.5(1.7)
2  a person_researcher     7          3.8% 1      4.5%  7.0   7   7     7.0(0)
3  a      person_sally    10          5.4% 2      9.1%  5.0   5   5     5.0(0)
4  a        person_sam    13          7.1% 3     13.6%  4.3   3   6   4.3(1.5)
5  a    person_teacher     4          2.2% 1      4.5%  4.0   4   4     4.0(0)
6  a           adult_0    45         24.5% 3     13.6% 15.0   8  26  15.0(9.6)
7  a           adult_1    11          6.0% 2      9.1%  5.5   4   7   5.5(2.1)
8  a                AA     1           .5% 1      4.5%  1.0   1   1     1.0(0)
9  a                BB    11          6.0% 3     13.6%  3.7   1   8   3.7(3.8)
10 a                CC    60         32.6% 2      9.1% 30.0   9  51 30.0(29.7)
11 b       person_greg    22         10.6% 4     19.0%  5.5   4   8   5.5(1.7)
12 b person_researcher     7          3.4% 1      4.8%  7.0   7   7     7.0(0)
13 b      person_sally    10          4.8% 2      9.5%  5.0   5   5     5.0(0)
14 b        person_sam    13          6.3% 3     14.3%  4.3   3   6   4.3(1.5)
15 b    person_teacher     4          1.9% 1      4.8%  4.0   4   4     4.0(0)
16 b           adult_0    45         21.7% 3     14.3% 15.0   8  26  15.0(9.6)
17 b           adult_1    11          5.3% 2      9.5%  5.5   4   7   5.5(2.1)
18 b                AA     1           .5% 1      4.8%  1.0   1   1     1.0(0)
19 b                BB    41         19.8% 1      4.8% 41.0  41  41    41.0(0)
20 b                CC    53         25.6% 3     14.3% 17.7   1  31 17.7(15.3)
============================
Unit of measure: words
</code></pre>


```{r}
plot(summary(v))
plot(summary(v), facet.vars = "time")
```


`r FT(orange, 5, text="&diams;")` **Example 2: Summarizing Time Spans Approach** `r FT(orange, 5, text="&diams;")`

```{r, message=FALSE}
## Single time list
x <- list(
    transcript_time_span = qcv(00:00 - 1:12:00),
    A = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00"),
    B = qcv(terms = "2.40, 3.01:3.02, 5.01, 6.02:7.00,
        9.00, 1.12.00:1.19.01"),
    C = qcv(terms = "2.40:3.00, 5.01, 6.02:7.00, 9.00, 17.01")
)

## Long format for time span approach
z <-cm_2long(x)
head(z)
```

```{r eval =  FALSE}
## Summary of the data and plotting the summary
summary(z)
```

<pre><code>  code total percent_total n percent_n  ave min max    mean(sd)
1    A 01:22         12.6% 4     26.7% 20.5   1  59  20.5(27.3)
2    B 08:06         74.7% 6     40.0% 81.0   1 422 81.0(168.6)
3    C 01:23         12.7% 5     33.3% 16.6   1  59  16.6(25.2)
============================
Unit of measure: time
Columns measured in seconds unless in the form hh:mm:ss
</code></pre>

```{r}
plot(summary(z))
```

`r FT(orange, 5, text="&diams;")` **Trouble Shooting Summary: Suppress Measurement Units** `r FT(orange, 5, text="&diams;")`

```{r, eval = FALSE}
## suppress printing measurement units
suppressMessages(print(summary(z)))
```

<pre><code>  code total percent_total n percent_n  ave min max    mean(sd)
1    A 01:22         12.6% 4     26.7% 20.5   1  59  20.5(27.3)
2    B 08:06         74.7% 6     40.0% 81.0   1 422 81.0(168.6)
3    C 01:23         12.7% 5     33.3% 16.6   1  59  16.6(25.2)
</code></pre>


`r FT(orange, 5, text="&diams;")` **Trouble Shooting Summary: Print as Dataframe** `r FT(orange, 5, text="&diams;")`

```{r}
## remove print method
class(z) <- "data.frame"
z
```


<h5 id="cmplot"><font color="green">Plotting</font></h5>

Like `r FUN("summary", "summary.cmspans")`, most of the `r FT(red, text="cm_")` family of functions have a `r FUN("plot", "plot.cmspans")` method as well that allows a Gantt plot visualization of codes by group.

`r FT(orange, 5, text="&diams;")` **Gantt Plot of Transcript/List or Time Spans Data** `r FT(orange, 5, text="&diams;")`


```{r, message=FALSE}
## Two transcript lists
A <- list(
    person_greg = qcv(terms='7:11, 20:24, 30:33, 49:56'),
    person_researcher = qcv(terms='42:48'),
    person_sally = qcv(terms='25:29, 37:41'),
    person_sam = qcv(terms='1:6, 16:19, 34:36'),
    person_teacher = qcv(terms='12:15'),
    adult_0 = qcv(terms='1:11, 16:41, 49:56'),
    adult_1 = qcv(terms='12:15, 42:48'),
    AA = qcv(terms="1"),
    BB = qcv(terms="1:2, 3:10, 19"),
    CC = qcv(terms="1:9, 100:150")
)

B  <- list(
    person_greg = qcv(terms='7:11, 20:24, 30:33, 49:56'),
    person_researcher = qcv(terms='42:48'),
    person_sally = qcv(terms='25:29, 37:41'),
    person_sam = qcv(terms='1:6, 16:19, 34:36'),
    person_teacher = qcv(terms='12:15'),
    adult_0 = qcv(terms='1:11, 16:41, 49:56'),
    adult_1 = qcv(terms='12:15, 42:48'),
    AA = qcv(terms="40"),
    BB = qcv(terms="50:90"),
    CC = qcv(terms="60:90, 100:120, 150"),
    DD = qcv(terms="")
)

## Long format
x <- cm_2long(A, v.name = "time")
y <- cm_2long(A, B, v.name = "time")

## cm_code family
combs <- list(sam_n_sally = qcv(person_sam, person_sally))
z <- cm_code.combine(v, combs, "time")
```

```{r, fig.height = 4}
plot(x, title = "Single")
```

```{r}
plot(y, title = "Repeated Measure")
plot(z, title = "Combined Codes")
```

<h5 id="cmdist"><font color="green">Distance Measures</font></h5>

Often a research will want to know which codes are clustering closer to other codes (regardless of whether the codes represent word or time spans).  `r FUN("cm_distance")` allows the research to find the distances between codes and standardize the mean of the differences to allow for comparisons similar to a correlation.  The matrix output from `r FUN("cm_distance")` is arrived at by taking the means and standard deviations of the differences between codes and scaling them (without centering) and then multiplying the two together.  This results in a standardized distance measure that is non-negative, with values closer to zero indicating codes that are found in closer proximity.  

The researcher may also access the means, standard deviations and number of codes by indexing the list output for each transcript.  This distance measure compliments the Gantt plot.  

Note that the argument <b><font color="green" face="courier new">causal = FALSE</font></b> (the default) does not assume Code A comes before Code B whereas <b><font color="green" face="courier new">causal = TRUE</font></b> assumes the first code precedes the second code.  Generally, setting <b><font color="green" face="courier new">causal = FALSE</font></b> will result in larger mean of differences and accompanying standardized values.  Also note that row names are the first code and column names are the second comparison code.  The values for Code A compared to Code B will not be the same as Code B compared to Code A.  This is because, unlike a true distance measure, `r FUN("cm_distance")`'s matrix is asymmetrical.  `r FUN("cm_distance")`computes the distance by taking each span (start and end) for Code A and comparing it to the nearest start or end for Code B.  So for example there may be 6 Code A spans and thus six differences between A and B, whereas Code B may only have 3 spans and thus three differences between B and A.  This fact alone will lead to differences in A compared to B versus B compared to A.  


`r FT(orange, 5, text="&diams;")` `r FUN("cm_distance")` - *Initial Data Setup* `r FT(orange, 5, text="&diams;")`


```{r, message=FALSE}
x <- list(
    transcript_time_span = qcv(00:00 - 1:12:00),
    A = qcv(terms = "2.40:3.00, 6.32:7.00, 9.00,
        10.00:11.00, 33.23:40.00, 59.56"),
    B = qcv(terms = "3.01:3.02, 5.01,  19.00, 1.12.00:1.19.01"),
    C = qcv(terms = "2.40:3.00, 5.01, 6.32:7.00, 9.00, 17.01, 38.09:40.00")
)
y <- list(
    transcript_time_span = qcv(00:00 - 1:12:00),
    A = qcv(terms = "2.40:3.00, 6.32:7.00, 9.00,
        10.00:11.00, 23.44:25.00, 59.56"),
    B = qcv(terms = "3.01:3.02, 5.01, 7.05:8.00 19.30, 1.12.00:1.19.01"),
    C = qcv(terms = "2.40:3.00, 5.01, 6.32:7.30, 9.00, 17.01, 25.09:27.00")
)

## Long format
dat <- cm_2long(x, y)
```

```{r, echo=FALSE, fig.height=6}
plot(dat, title="Plot of the Codes")
```

`r FT(orange, 5, text="&diams;")` `r FUN("cm_distance")` - *Non-Causal Distance* `r FT(orange, 5, text="&diams;")`

```{r, eval = FALSE}
## a cm_distance output
(out1 <- cm_distance(dat, time.var = "variable"))
```

<pre><code>x

standardized:
     A    B    C
A 0.00 1.04 0.82
B 0.88 0.00 3.89
C 0.09 0.95 0.00


y

standardized:
     A    B    C
A 0.00 0.38 1.97
B 0.47 0.00 4.94
C 0.08 0.09 0.00
</code></pre>

```{r, eval = FALSE}
## The elements available from the output
names(out1)
```

<pre><code>[1] "x" "y"
</code></pre>

```{r, eval = FALSE}
## A list containing means, standard deviations and other 
## descriptive statistics for the differences between codes
out1$x
```

<pre><code>$mean
       A      B      C
A   0.00 367.67 208.67
B 322.50   0.00 509.00
C  74.67 265.00   0.00

$sd
       A      B      C
A   0.00 347.51 483.27
B 337.47   0.00 940.94
C 143.77 440.92   0.00

$n
  A B C
A 6 6 6
B 4 4 4
C 6 6 6

$combined
  A                B                 C                
A n=6              367.67(347.51)n=6 208.67(483.27)n=6
B 322.5(337.47)n=4 n=4               509(940.94)n=4   
C 74.67(143.77)n=6 265(440.92)n=6    n=6              

$standardized
     A    B    C
A 0.00 1.04 0.82
B 0.88 0.00 3.89
C 0.09 0.95 0.00
</code></pre>

`r FT(orange, 5, text="&diams;")` `r FUN("cm_distance")` - *Causal Distance* `r FT(orange, 5, text="&diams;")`

```{r, eval = FALSE}
## a cm_distance output `causal = TRUE`
cm_distance(dat, time.var = "variable", causal = TRUE)
```

<pre><code>x

standardized:
     A    B    C
A 0.66 0.84 0.08
B 0.29 3.96 0.49
C 0.40 0.86 0.37


y

standardized:
     A    B    C
A 1.11 1.63 0.08
B 0.03 2.95 0.04
C 0.70 1.27 0.11
</code></pre>

<h3 id="counts">Word Counts and Descriptive Statistics</h3>


<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/dist_tab.html" target="_blank">
    <input type="submit" value="dist_tab"> - `r HR("#freqtab", "SPSS Style Frequency Tables")`
</form>

<form action="http://trinker.github.io/qdap_dev/pos.html" target="_blank">
    <input type="submit" value="pos"><input type="submit" value="pos_by"><input type="submit" value="pos_tags"> - `r HR("#pos", "Parts of Speech Tagging & Counts")`
</form>

<form action="http://trinker.github.io/qdap_dev/question_type.html" target="_blank">
    <input type="submit" value="question_type"> - `r HR("#quest", "Question Type Counts")`
</form>

<form action="http://trinker.github.io/qdap_dev/syllable_sum.html" target="_blank">
    <input type="submit" value="syllable_sum"><input type="submit" value="combo_syllable_sum"><input type="submit" value="polysyllable_sum"><input type="submit" value="syllable_count"> - `r HR("#syll", "Syllabication and Counts")`
</form>

<form action="http://trinker.github.io/qdap_dev/as.tdm.html" target="_blank">
    <input type="submit" value="as.tdm"><input type="submit" value="as.dtm"> - `r HR("#as.tdm", "Convert/Generate Term Document Matrix or Document Term Matrix")`
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/termco.html" target="_blank">
    <input type="submit" value="termco"><input type="submit" value="term_match"><input type="submit" value="termco_d"><input type="submit" value="termco2mat">
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/termco_c.html" target="_blank">
    <input type="submit" value="termco_c"> 
</form>

<form action="http://trinker.github.io/qdap_dev/spaste.html" target="_blank">
    <input type="submit" value="spaste"> - `r HR("#termco", "Search For and Count Terms")`
</form>

<form action="http://trinker.github.io/qdap_dev/Word_Frequency_Matrix.html" target="_blank">
    <input type="submit" value="wfm"><input type="submit" value="wfdf"><input type="submit" value="wf_combine"><input type="submit" value="wfm_expanded"> - `r HR("#wfm", "Word Frequency Matrix")`
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/word_count.html" target="_blank">
    <input type="submit" value="word_count"> 
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/word_count.html" target="_blank">
    <input type="submit" value="character_count"><input type="submit" value="character_table"> 
</form>

<form action="http://trinker.github.io/qdap_dev/word_list.html" target="_blank">
    <input type="submit" value="word_list">  - `r HR("#wordcount", "Word & Character Counts")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_stats.html" target="_blank">
    <input type="submit" value="word_stats"> - `r HR("#wordstats", "Descriptive Word Statistics")`
</form>

</div>

A researcher often needs to quickly gather frequency counts for various words/word types.  qdap offers multiple functions designed to efficiently generate descriptive word statistics by any combination of grouping variables.  Many of the functions also offer proportional usage to more fairly compare between groups.  Additionally, many functions also have plotting methods to better visualize the data that is transformed.

<h4 id="wordstats">Descriptive Word Statistics</h4>

Often a researcher may want to get a general sense of how words are functioning for different grouping variables.  The `r FUN("word_stats")` function enables a quick picture of what is occurring within the data.  The displayed (printed) output is a dataframe, however, the output from word_stats is actually a list.  Use `r FUN("?word_stats", "word_stats")` to learn more.

The displayed output is a wide dataframe, hence the abbreviated column names.  The following column names and meanings will provide guidance in understanding the output:

<h5 id="wordstats">`r FUN("word_stats")` Column Names</h5>

- n.tot - number of turns of talk   
- n.sent - number of sentences   
- n.words - number of words   
- n.char - number of characters   
- n.syl - number of syllables   
- n.poly - number of polysyllables   
- sptot - syllables per turn of talk   
- wptot - words per turn of talk   
- wps - words per sentence   
- cps - characters per sentence   
- sps - syllables per sentence   
- psps - poly-syllables per sentence   
- cpw - characters per word   
- spw - syllables per word   
- n.state - number of statements   
- n.quest - number of questions   
- n.exclm - number of exclamations   
- n.incom - number of incomplete statements   
- p.state - proportion of statements   
- p.quest - proportion of questions   
- p.exclm - proportion of exclamations   
- p.incom - proportion of incomplete statements   
- n.hapax - number of hapax legomenon   
- n.dis - number of dis legomenon   
- grow.rate - proportion of hapax legomenon to words   
- prop.dis - proportion of dis legomenon to words

<div class="middleDiv">
<b>`r FT(red, 4, text="It is assumed you have run <font face=\"courier\">sentSplit</font> on the data.<br>If this is not the case the counts will not be accurate.")`</b>
</div>


`r FT(orange, 5, text="&diams;")` **`r FUN("word_stats")` Example** `r FT(orange, 5, text="&diams;")`

Note that the initial output is broken into three dataframe outputs because of the width of printed output from `r FUN("word_stats")` being so large.  The user will see that these three dataframes are actually one wide dataframe in the R output.

```{r, results = "hide"}
(desc_wrds <- with(mraja1spl, word_stats(dialogue, person, tot = tot)))
```

```{r, include = FALSE}
desc_wrds2 <- with(mraja1spl, word_stats(desc_wrds, person, tot = tot, digits = 1))
```

```{r, echo = FALSE, comment = NULL}
desc_wrds2$gts[, c(1, 2:9)]
```

```{r, echo = FALSE, comment = NULL}
desc_wrds2$gts[, c(1, 10:19)]
```

```{r, echo = FALSE, comment = NULL}
desc_wrds2$gts[, c(1, 20:26)]
```

```{r}
## The following shows all the available elements in the `word_stats` output
names(desc_wrds)
```

`r FUN("word_stats")` has a plot method that plots the output as a heat map.  This can be useful for finding high/low elements in the data set.

`r FT(orange, 5, text="&diams;")` **`r FUN("word_stats")` Plot** `r FT(orange, 5, text="&diams;")`


```{r}
plot(desc_wrds)
```

```{r, fig.width = 9}
plot(desc_wrds, label=TRUE, lab.digits = 1)
```

It takes considerable time to run `r FUN("word_stats")` because it is calculating syllable counts.  The user may re-use the object output from one run and bass this as the text variable (`text.var`) in a subsequent run with different grouping variables (`grouping.vars`) as long as the text variable has not changed.  The example below demonstrates how to re-use the output from one `r FUN("word_stats")` run in another run.

`r FT(orange, 5, text="&diams;")` **`r FUN("word_stats")` Re-use** `r FT(orange, 5, text="&diams;")`


```{r, eval = FALSE}
with(mraja1spl, word_stats(desc_wrds, list(sex, fam.aff, died), tot = tot))
```


<h4 id="wfm">Word Frequency Matrix</h4>

Many analyses with words involve a matrix based on the words.  qdap uses a *word frequency matrix* (`r FUN("wfm", "Word_Frequency_Matrix")`) or the less malleable dataframe version, *word frequency dataframe* (`r FUN("wfdf", "Word_Frequency_Matrix")`).  The `r FUN("wfm", "Word_Frequency_Matrix")` is a count of word usages per grouping variable(s).  This is a similar concept to the `r HR("http://cran.r-project.org/web/packages/tm/index.html", "tm package's")` Term Document Matrix, though instead of documents we are interested in the grouping variable's usage of terms.  `r FUN("wfm", "Word_Frequency_Matrix")` is the general function that should be used, however, the `r FUN("wfdf", "Word_Frequency_Matrix")` function does provide options for margin sums (row and column).  Also note that the `r FUN("wfm_expanded", "Word_Frequency_Matrix")` and `r FUN("wfm_combine", "Word_Frequency_Matrix")` can expand or combine terms within a word frequency matrix.

`r FT(orange, 5, text="&diams;")` **`r FUN("wfm", "Word_Frequency_Matrix")` Examples** `r FT(orange, 5, text="&diams;")`

```{r}
## By a single grouping variable
with(DATA, wfm(state, person))[1:15, ]
## By two grouping variables
with(DATA, wfm(state, list(sex, adult)))[1:15, ]
```

`r FT(orange, 5, text="&diams;")` **`r FUN("wfm", "Word_Frequency_Matrix")`: Keep Two Word Phrase as a Single Term** `r FT(orange, 5, text="&diams;")`

```{r}
## insert double tilde ("~~") to keep phrases(e. g., first last name)
space_keeps <- c(" fun", "I ")
state2 <- space_fill(DATA$state, space_keeps, rm.extra = FALSE)
with(DATA, wfm(state2, list(sex, adult)))[1:18, ]
```

At times it may be useful to view the correlation between word occurrences between turns of talk or other useful groupings.  The user can utilize the output from `r FUN("wfm", "Word_Frequency_Matrix")` to accomplish this.

<p id="corr">`r FT(orange, 5, text="&diams;")` **`r FUN("wfm", "Word_Frequency_Matrix")`: Word Correlations** `r FT(orange, 5, text="&diams;")`</p>

```{r, include = FALSE}
dat <- readRDS("data/wfmcor.rds")
```

<pre><code class="r">library(reports)
x <- factor(with(rajSPLIT, paste(act, pad(TOT(tot)), sep = "|")))
dat <- wfm(rajSPLIT$dialogue, x)
</code></pre>

```{r}
cor(t(dat)[, c("romeo", "juliet")])
cor(t(dat)[, c("romeo", "banished")])
cor(t(dat)[, c("romeo", "juliet", "hate", "love")])
```

```{r, fig.width = 8}
dat2 <- wfm(DATA$state, id(DATA))
qheat(cor(t(dat2)), low = "yellow", high = "red", 
    grid = "grey90", diag.na = TRUE, by.column = NULL) 
```


`r FT(orange, 5, text="&diams;")` **`r FUN("wfdf", "Word_Frequency_Matrix")` Examples**: *Add Margins* `r FT(orange, 5, text="&diams;")`

```{r}
with(DATA, wfdf(state, person, margins = TRUE))[c(1:15, 41:42), ]
with(DATA, wfdf(state, list(sex, adult), margins = TRUE))[c(1:15, 41:42), ]
```

`r FT(orange, 5, text="&diams;")` **`r FUN("wfm_expanded", "Word_Frequency_Matrix")`: Expand the wfm** `r FT(orange, 5, text="&diams;")`

```{r}
## Start with a word frequency matrix
z <- wfm(DATA$state, DATA$person)

## Note a single `you`
z[30:41, ]
## Note that there are two `you`s in the expanded version
wfm_expanded(z)[33:45, ] 
```

`r FT(orange, 5, text="&diams;")` **`r FUN("wfm_combine", "Word_Frequency_Matrix")`: Combine Terms in the wfm** `r FT(orange, 5, text="&diams;")`

```{r}
## Start with a word frequency matrix
x <- wfm(DATA$state, DATA$person)

## The terms to exclude
WL <- list(
    random = c("the", "fun", "i"), 
    yous = c("you", "your", "you're")
)

## Combine the terms
(out <- wfm_combine(x, WL))
## Pass the combined version to Chi Squared Test
chisq.test(out)
```


`r FT(orange, 5, text="&diams;")` **`r FUN("wfm", "Word_Frequency_Matrix")`: Correspondence Analysis Example** `r FT(orange, 5, text="&diams;")`

<pre><code class="r">library(ca)

## Grab Just the Candidates
dat <- pres_debates2012
dat <- dat[dat$person %in% qcv(ROMNEY, OBAMA), ]

## Stem the text
speech <- stemmer(dat$dialogue)

## With 25 words removed
mytable1 <- with(dat, wfm(speech, list(person, time), stopwords = Top25Words))

## CA
fit <- ca(mytable)
summary(fit)
plot(fit)
plot3d.ca(fit, labels=1)

## With 200 words removed
mytable2 <- with(dat, wfm(speech, list(person, time), stopwords = Top200Words))

## CA
fit2 <- ca(mytable2)
summary(fit2)
plot(fit2)
plot3d.ca(fit2, labels=1)
</code></pre>

<h5 id="as.tdm"><font color="green">Convert/Generate Term Document Matrix or Document Term Matrix</font></h5>

Some packages that could further the analysis of qdap expect a Document Term or Term Document Matrix.  qdap's `r FUN("wfm", "Word_Frequency_Matrix")` is similar to the `r HR("http://cran.r-project.org/web/packages/tm/index.html", "tm package's")` `r HR("http://www.inside-r.org/packages/cran/tm/docs/DocumentTermMatrix", "TermDocumentMatrix")`     and `r HR("http://www.inside-r.org/packages/cran/tm/docs/DocumentTermMatrix", "DocumentTermMatrix")`.  qdap does not try to replicate the extensive work of the`r HR("http://cran.r-project.org/web/packages/tm/index.html", "tm")` package, however, the `r FUN("as.tdm")` and `r FUN("as.dtm", "as.tdm")` do attempt to extend the work the researcher conducts in qdap to be utilized in other R packages.  For a vignette describing qdap-tm compatability use `browseVignettes(package = "qdap")` or `\r HR2("http://cran.r-project.org/web/packages/qdap/vignettes/tm_package_compatibility.pdf, "Click Here")`.

`r FT(orange, 5, text="&diams;")` **`r FUN("as.tdm")` Use** `r FT(orange, 5, text="&diams;")`


```{r}
x <- wfm(DATA$state, DATA$person)
## Term Document Matrix
as.tdm(x)
## Document Term Matrix
as.dtm(x)
```

```{r rval=FALSE, echoh=FALSE, include=FALSE}
## ```{r eval =  FALSE}
## ## Run Latent Semantic Analysis
## library(lsa)
## lsa(as.tdm(x), dims=dimcalc_share())
## ```
## 
## 
## <pre><code>$tk
##                  [,1]         [,2]
## about    -0.021153126  0.072269368
## already  -0.169239530 -0.124825133
## am       -0.169239530 -0.124825133
## are      -0.021153126  0.072269368
## be       -0.021153126  0.072269368
## can      -0.021153126  0.072269368
## certain  -0.021153126  0.072269368
## computer -0.090637878  0.215786300
## distrust -0.090637878  0.215786300
## do       -0.001903917  0.014326564
## dumb     -0.169239530 -0.124825133
## eat      -0.169239530 -0.124825133
## fun      -0.181275756  0.431572601
## good     -0.001108363  0.009865681
## how      -0.021153126  0.072269368
## hungry   -0.169239530 -0.124825133
## i        -0.259877408  0.090961168
## i'm      -0.169239530 -0.124825133
## is       -0.259877408  0.090961168
## it       -0.090637878  0.215786300
## it's     -0.338479060 -0.249650265
## let's    -0.169239530 -0.124825133
## liar     -0.090637878  0.215786300
## move     -0.001108363  0.009865681
## no       -0.338479060 -0.249650265
## not      -0.259877408  0.090961168
## on       -0.001108363  0.009865681
## shall    -0.001108363  0.009865681
## should   -0.001903917  0.014326564
## stinks   -0.090637878  0.215786300
## talking  -0.021153126  0.072269368
## telling  -0.169239530 -0.124825133
## the      -0.169239530 -0.124825133
## then     -0.001108363  0.009865681
## there    -0.169239530 -0.124825133
## too      -0.090637878  0.215786300
## truth    -0.169239530 -0.124825133
## way      -0.169239530 -0.124825133
## we       -0.024165406  0.096461613
## what     -0.023057043  0.086595932
## you      -0.371668412  0.379016836
## 
## $dk
##                    [,1]        [,2]
## greg       -0.876176894 -0.47984657
## researcher -0.005738152  0.03792516
## sally      -0.109512712  0.27781431
## sam        -0.469245067  0.82951496
## teacher    -0.009856846  0.05507346
## 
## $sk
## [1] 5.177141 3.844150
## 
## attr(,"class")
## [1] "LSAspace"
## </code></pre>
```


<h4 id="termco">Search For and Count Terms</h4>

The `r FUN("termco")` family of functions are some of the most useful qdap functions for quantitative discourse analysis.  `r FUN("termco")` searches for (an optionally groups) terms and outputs a raw count, percent, and combined (raw/percent) matrix of term counts by grouping variable.  The `r FUN("term_match", "termco")` `r FUN("all_words")` `r FUN("syn", "synonyms")`, `r FUN("exclude")`, and `r FUN("spaste")` are complementary  functions that are useful in developing word lists to provide to the <b><font color="green" face="courier new">match.list</font></b>.  

The <b><font color="green" face="courier new">match.list</font></b> acts to search for similarly grouped <em>themes</em>.  For example <font color="green" face="courier new">c(" read ", " reads", " reading", " reader")</font> may be a search for words associated with reading.  It is good practice to name the vectors of words that are stored in the <b><font color="green" face="courier new">match.list</font></b> .  This is the general form for how to set up a <b><font color="green" face="courier new">match.list</font></b>:

```{r eval = FALSE}
themes <- list(
    theme_1 = c(),
    theme_2 = c(),
    theme_n = c()
)
```

<p id="match">It is important to understand how the <b><font color="green" face="courier new">match.list</font></b> is handled by `r FUN("termco")`.  The <b><font color="green" face="courier new">match.list</font></b> is (optionally) case and character sensitive. Spacing is an important way to grab specific words and requires careful thought. For example using <font color="green">"read"</font> will find the words <font color="green">"bread"</font>, <font color="green">"read"</font>, <font color="green">"reading"</font>, and <font color="green">"ready"</font>. If you want to search for just the word <font color="purple">"read"</font> supply a vector of <font color="green" face="courier new">c(" read ", " reads", " reading", " reader")</font>. Notice the leading and trailing spaces.  A space acts as a boundary whereas starting/ending with a nonspace allows for greedy matching that will find words that contain this term.  A leading, trailing or both may be used to control how `r FUN("termco")` searches for the supplied terms.  So the reader may ask why not supply one string spaced as <font color="green">" read"</font>?  Keep in mind that `r FUN("termco")` would also find the word <font color="purple">"ready"</font></p>

This section's examples will first view the complementary  functions that augment the *themes* supplied to <b><font color="green" face="courier new">match.list</font></b> and then main `r FUN("termco")` function will be explored.


`r FUN("term_match", "termco")` looks through a text variable (usually the text found in the transcript) and finds/returns a vector of words containing a term(s).

`r FT(orange, 5, text="&diams;")` **`r FUN("term_match", "termco")` and  `r FUN("exclude")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
term_match(text.var = DATA$state, terms = qcv(the, trust), return.list = FALSE)
term_match(DATA$state, "i", FALSE)
exclude(term_match(DATA$state, "i", FALSE), talking, telling)
```

`r FUN("all_words")` is similar to `r FUN("term_match", "termco")`, however, the function looks at all the words found in a text variable (usually the transcript text) and returns words that begin with or contain the term(s).  The output can be arrange alphabetically or by frequency.  The output is a dataframe which helps the researcher to make decisions with regard to frequency of word use. 

`r FT(orange, 5, text="&diams;")` **`r FUN("all_words")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
x1 <- all_words(raj$dialogue, begins.with="re")
head(x1, 10)
all_words(raj$dialogue, begins.with="q")
all_words(raj$dialogue, contains="conc")
x2 <- all_words(raj$dialogue)
head(x2, 10)
x3 <- all_words(raj$dialogue, alphabetical = FALSE)
head(x3, 10)
```

The `r FUN("synonyms")` (short hand: `r FUN("syn", "synonyms")`) function finds words that are synonyms of a given set of terms and returns either a list of vector that can be passed to `r FUN("termco")`'s <b><font color="green" face="courier new">match.list</font></b>.

`r FT(orange, 5, text="&diams;")` **`r FUN("synonyms")` Examples**`r FT(orange, 5, text="&diams;")`

```{r, message=FALSE}
synonyms(c("the", "cat", "job", "environment", "read", "teach"))
head(syn(c("the", "cat", "job", "environment", "read", "teach"),
    return.list = FALSE), 30)
syn(c("the", "cat", "job", "environment", "read", "teach"), multiwords = FALSE)
```

`r FT(orange, 5, text="&diams;")` **`r FUN("termco")` - Simple Example**`r FT(orange, 5, text="&diams;")`

```{r}
## Make a small dialogue data set
(dat2 <- data.frame(dialogue=c("@bryan is bryan good @br",
    "indeed", "@ brian"), person=qcv(A, B, A)))
## The word list to search for
ml <- list(
    wrds=c("bryan", "indeed"), 
    "@", 
    bryan=c("bryan", "@ br", "@br")
)

## Search by person
with(dat2, termco(dialogue, person, match.list=ml))
## Search by person proportion output
with(dat2, termco(dialogue, person, match.list=ml, percent = FALSE))
```

<p id = "rajex">`r FT(orange, 5, text="&diams;")` <b>`r FUN("termco")` - Romeo and Juliet Act 1 Example</b>`r FT(orange, 5, text="&diams;")`</p>

```{r}
## Word list to search for
## Note: In the last vector using "the" will actually 
## include the other 3 versions
ml2 <- list(
    theme_1 = c(" the ", " a ", " an "),
    theme_2 = c(" I'" ),
    "good",
    the_words = c("the", " the ", " the", "the ")
)

(out <- with(raj.act.1,  termco(dialogue, person, ml2)))
## Available elements in the termco output (use dat$...)
names(out)
## Raw and proportion - useful for presenting in tables
out$rnp  
## Raw - useful for performing calculations
out$raw 
## Proportion - useful for performing calculations
out$prop
```

`r FT(orange, 5, text="&diams;")` **Using `r FUN("termco")` with `r FUN("term_match", "termco")` and `r FUN("exclude")`**`r FT(orange, 5, text="&diams;")`

```{r}
## Example 1
termco(DATA$state, DATA$person, exclude(term_match(DATA$state, qcv(th),
    FALSE), "truth"))
## Example 2
MTCH.LST <- exclude(term_match(DATA$state, qcv(th, i)), qcv(truth, stinks))
termco(DATA$state, DATA$person, MTCH.LST)
```

`r FT(orange, 5, text="&diams;")` **Using `r FUN("termco")` with `r FUN("syn")`**`r FT(orange, 5, text="&diams;")`

```{r}
syns <- synonyms("doubt")
syns[1]
```

```{r eval = FALSE}
termco(DATA$state, DATA$person, unlist(syns[1]))
```

```{r echo = FALSE}
termco(DATA$state, DATA$person, unlist(syns[1]))$rnp[, c(1:5, 9:10)]
```

```{r}
synonyms("doubt", FALSE)
termco(DATA$state, DATA$person, list(doubt = synonyms("doubt", FALSE)))
```

```{r eval = FALSE}
termco(DATA$state, DATA$person, syns)
```

```{r echo = FALSE}
termco(DATA[["state"]], DATA[["person"]], syns)$rnp[, c(1:4, 7:8)]
```

`r FUN("termco")` also has a plot method that plots a heat map of the `r FUN("termco")` output based on the percent usage by grouping variable.  This allows for rapid visualizations of patterns and enables fast spotting of extreme values.  Here are some plots from the `r HR("#rajex", "Romeo and Juliet Act 1 Example")` above.

`r FT(orange, 5, text="&diams;")` **Using `r FUN("termco")` Plotting**`r FT(orange, 5, text="&diams;")`

```{r, fig.height = 3.5}
plot(out)
```

```{r fig.width = 16, fig.height = 5}
plot(out, label = TRUE)
```

<h4 id="quest">Question Type Counts</h4>

A researcher may be interested in classifying and investigating the types of questions used within dialogue.  
`r FUN("question_type")` provides question classification.  The algorithm searches for the following interrogative words (and optionally, their negative contraction form as well):

```{r echo = FALSE, comment=NULL}
wrds <- c("are*", "can*", "correct", "could", "did*", "do*",  
    "does*", "had*", "has", "have*", "how", "is", "may", "might*", 
    "must*", "ok", "right", "shall", "should", "was*", "were*", "what", 
    "when", "where", "which", "who", "whom", "whose", "why", "will*", 
    "would*", "implied do/does/did")

wrds2 <- data.frame(matrix(c(wrds, rep("", 3)), ncol = 5))

padding <- max(nchar(wrds)) + 3
padding2 <- rev(sort(nchar(wrds)))[2] + 8
out <- apply(wrds2[, 1:4], 2, function(x) sprintf(paste0("%-", padding2, "s"), x))
out2 <- apply(wrds2[, 5, drop=FALSE], 2, function(x) sprintf(paste0("%-", padding, "s"), x))
cat(paste(paste2(cbind(out, out2), sep ="", trim=FALSE), collapse = "\n"))
```


The interrogative word that is found first (with the exception of "ok", "right" and "correct") in the question determines the sentence type. "ok", "right" and "correct" sentence types are determined if the sentence is a question with no other interrogative words found and "ok", "right" or "correct" is the last word of the sentence. Those interrogative sentences beginning with the word "you", "wanna", or "want" are categorized as implying do/does/did question type, though the use of do/does is not explicit. Those sentence beginning with "you" followed by a select interrogative word (and or their negative counter parts) above (marked with *) or 1-2 amplifier(s) followed by the select interrogative word are categorized by the select word rather than an implied do/does/did question type.  A sentence that is marked "ok" over rides an implied do/does/did label.  Those with undetermined sentence type are labeled unknown.

`r FT(orange, 5, text="&diams;")` **`r FUN("question_type")` - Basic Example**`r FT(orange, 5, text="&diams;")`

```{r}
## Basic Example
(x <- question_type(DATA.SPLIT$state, DATA.SPLIT$person))
## Available elements from output
names(x)
## Table of counts useful for additional analysis
x$count
## The raw output with question types
truncdf(x$raw, 15)
```

`r FUN("question_type")` also has a plot method that plots a heat map of the output.  This allows for rapid visualizations of patterns and enables fast spotting of extreme values.

`r FT(orange, 5, text="&diams;")` **`r FUN("question_type")` - Plotting Method**`r FT(orange, 5, text="&diams;")`

```{r fig.width = 5.5, fig.height = 4}
plot(x)
```

```{r fig.width = 5.5, fig.height = 4}
plot(x, label = TRUE, high = "red", low = "yellow", grid = NULL)
```

Negative forms of questions such as `r FT(green, text = "Don't you want the robots to leave?")` are, by default, grouped with their equivalent positive `r FT(green, text = "Do")` forms, such as `r FT(green, text = "Do you want the robots to leave?")`.  The researcher may choose to keep the two forms separate using the argument <b><font color="green">neg.cont = TRUE</font>
</b>

`r FT(orange, 5, text="&diams;")` **`r FUN("question_type")` - Include Negative Questions**`r FT(orange, 5, text="&diams;")`

```{r}
## Create a Dataframe with Do and Don't
(DATA.SPLIT2 <- rbind(DATA.SPLIT,
    c("sam", "1.1", "1", "m", "0", "K1", "Don't you think so?", "x"),
    c("sam", "1.1", "1", "m", "0", "K1", "Do you think so?", "x")
))[, c(1, 7)]
## Do and Don't Grouped Together
question_type(DATA.SPLIT2$state, DATA.SPLIT2$person)
```

```{r eval = FALSE}
## Do and Don't Grouped Separately
question_type(DATA.SPLIT2$state, DATA.SPLIT2$person, neg.cont = TRUE)
```

<pre><code>      person tot.quest    what  don't     do    how   shall implied_do/does/did
1       greg         1       0      0      0      0       0             1(100%)
2 researcher         1       0      0      0      0 1(100%)                   0
3      sally         2  1(50%)      0      0 1(50%)       0                   0
4        sam         2       0 1(50%) 1(50%)      0       0                   0
5    teacher         1 1(100%)      0      0      0       0                   0
</code></pre>

It may be helpful to access the indices of the question types in the **x[["inds"]]** output or access **x[["raw"]][, "n.row"]** for use with the **`r FUN("trans_context")`** function as seen below.

`r FT(orange, 5, text="&diams;")` **`r FUN("question_type")` - Passing to `r FUN("trans_context")`**`r FT(orange, 5, text="&diams;")`

```{r}
## The indices of all questions
x <- question_type(DATA.SPLIT$state, DATA.SPLIT$person)
(inds1 <- x[["inds"]])
```

```{r, eval = FALSE}
with(DATA.SPLIT, trans_context(state, person, inds = inds1, n.before = 2))
```

<pre><code>===================================
Event 1: [lines 2-6]

    sam:        Computer is fun. Not too fun.

    greg:       No it's not, it's dumb.

 ** teacher:    What should we do?

    sam:        You liar, it stinks!

    greg:       I am telling the truth! 

===================================
Event 2: [lines 5-9]

    sam:        You liar, it stinks!

    greg:       I am telling the truth!

 ** sally:      How can we be certain?

    greg:       There is no way.

    sam:        I distrust you. 

===================================
Event 3: [lines 8-12]

    greg:       There is no way.

    sam:        I distrust you.

 ** sally:      What are you talking about?

    researcher: Shall we move on? Good then.

    greg:       I'm hungry. Let's eat. You already? 

===================================
Event 4: [lines 9-13]

    sam:        I distrust you.

    sally:      What are you talking about?

 ** researcher: Shall we move on? Good then.

    greg:       I'm hungry. Let's eat. You already? 

===================================
Event 5: [lines 13-15]

    sally:      What are you talking about?

    researcher: Shall we move on? Good then.

 ** greg:       I'm hungry. Let's eat. You already? </code></pre>
 
```{r, eval = FALSE}
## Find what and how questions
inds2 <- x[["raw"]][x[["raw"]]$q.type %in% c("what", "how"), "n.row"]
with(DATA.SPLIT, trans_context(state, person, inds = inds2, n.before = 2))
```

<pre><code>===================================
Event 1: [lines 2-6]

    sam:        Computer is fun. Not too fun.

    greg:       No it's not, it's dumb.

 ** teacher:    What should we do?

    sam:        You liar, it stinks!

    greg:       I am telling the truth! 

===================================
Event 2: [lines 5-9]

    sam:        You liar, it stinks!

    greg:       I am telling the truth!

 ** sally:      How can we be certain?

    greg:       There is no way.

    sam:        I distrust you. 

===================================
Event 3: [lines 8-12]

    greg:       There is no way.

    sam:        I distrust you.

 ** sally:      What are you talking about?

    researcher: Shall we move on? Good then.

    greg:       I'm hungry. Let's eat. You already? </code></pre>


<h4 id="wordcount">Word & Character Counts</h4>

A research may have the need to view simple word or character counts for the sake of comparisons between grouping variables.  `r FUN("word_count")` (`r FUN("wc", "word_count")`), `r FUN("word_list")`, `r FUN("character_count", "word_count")`, `r FUN("character_table", "word_count")` (`r FUN("char_table", "word_count")`) serve the purposes of counting words and characters with `r FUN("word_list")` producing a lists of words usage by grouping variable and `r FUN("character_table", "word_count")` producing a count table of characters.  The following examples demonstrate the uses of these functions.  

`r FT(orange, 5, text="&diams;")` **`r FUN("word_count")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
word_count(DATA$state)
## `wc a shortened version of `word_count`
wc(DATA$state)
## Retain the text
wc(DATA$state, names = TRUE)
## Setting `byrow=FALSE` gives a total for the text variable
word_count(DATA$state, byrow=FALSE, names = TRUE)
## identical to `byrow=FALSE` above
sum(word_count(DATA$state))
## By grouping variable
tapply(DATA$state, DATA$person, wc, byrow=FALSE)
```

`r FT(orange, 5, text="&diams;")` **`r FUN("word_count")` Plotting Centered Word Counts**`r FT(orange, 5, text="&diams;")`

```{r, fig.height=13, fig.width=14.5, warning=FALSE}
## Scale variable 
raj2 <- raj
raj2$scaled <- unlist(tapply(wc(raj$dialogue), raj2$act, scale))
raj2$scaled2 <- unlist(tapply(wc(raj$dialogue), raj2$act, scale, scale = FALSE))
raj2$ID <- factor(unlist(tapply(raj2$act, raj2$act, seq_along)))

## Plot with ggplot2
library(ggplot2); library(grid)

ggplot(raj2, aes(x = ID, y = scaled, fill =person)) +
    geom_bar(stat="identity", position="identity") +
    facet_grid(act~.) + 
    ylab("Standard Deviations") + xlab("Turns of Talk") +
    guides(fill = guide_legend(nrow = 5, byrow = TRUE)) +
    theme(legend.position="bottom", legend.key.size = unit(.35, "cm"), 
        axis.text.x = element_blank(), axis.ticks.x = element_blank()) +
    ggtitle("Standardized Word Counts\nPer Turn of Talk")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("character_count", "word_count")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
character_count(DATA$state)
## Setting `byrow=FALSE` gives a total for the text variable
character_count(DATA$state, byrow=FALSE)
## identical to `byrow=FALSE` above
sum(character_count(DATA$state))
## By grouping variable
tapply(DATA$state, DATA$person, character_count, byrow=FALSE)
```

`r FT(orange, 5, text="&diams;")` **`r FUN("character_table", "word_count")` Example**`r FT(orange, 5, text="&diams;")`

```{r}
x <- character_table(DATA$state, DATA$person)
names(x)
counts(x)
proportions(x)[, 1:10]
scores(x)[, 1:7]
## Combine Columns
vowels <- c("a", "e", "i", "o", "u")
cons <- letters[!letters %in% c(vowels, qcv(j, q, x, z))]
colcomb2class(x, list(vowels = vowels, consonants = cons, other = 2:7))
```

`r FT(orange, 5, text="&diams;")` **`r FUN("character_table", "word_count")` Plot Method**`r FT(orange, 5, text="&diams;")`

```{r}
plot(x)
```

```{r}
plot(x, label = TRUE, high = "red", lab.digits = 1, zero.replace = "")
```

`r FT(orange, 5, text="&diams;")` **`r FUN("character_table", "word_count")` Additional Plotting**`r FT(orange, 5, text="&diams;")`

```{r message = FALSE}
library(ggplot2);library(reshape2)
dat <- char_table(DATA$state, list(DATA$sex, DATA$adult))
dat2 <- colsplit2df(melt(dat$raw), keep.orig = TRUE)
dat2$adult2 <- lookup(as.numeric(as.character(dat2$adult)), 
    c(0, 1), c("child", "adult"))
head(dat2, 15)
```

```{r message = FALSE, fig.width = 12}
ggplot(data = dat2, aes(y = variable, x = value, colour=sex)) +
    facet_grid(adult2~.) +
    geom_line(size=1, aes(group =variable), colour = "black") +
    geom_point()
```

```{r, fig.width = 12}
ggplot(data = dat2, aes(x = variable, y = value)) +
    geom_bar(aes(fill = variable), stat = "identity") +
    facet_grid(sex ~ adult2, margins = TRUE) +
    theme(legend.position="none")
```

<h4 id="freqtab">SPSS Style Frequency Tables</h4>

It is helpful to view the frequency distributions for a vector, matrix or dataframe.   The `r FUN("dist_tab")` function allows the researcher to quickly generate frequency distributions.


`r FT(orange, 5, text="&diams;")` **`r FUN("dist_tab")` Examples**`r FT(orange, 5, text="&diams;")`

```{r}
dist_tab(rnorm(10000), 10)
dist_tab(sample(c("red", "blue", "gray"), 100, T), right = FALSE)
dist_tab(CO2, 4)
```

```{r, eval = FALSE}
wdst <- with(mraja1spl, word_stats(dialogue, list(sex, fam.aff, died)))
dist_tab(wdst$gts[1:4], 5)
```

<pre><code>$`sex&fam.aff&died`
          interval Freq cum.Freq percent cum.percent
1      f.cap.FALSE    1        1    9.09        9.09
2       f.cap.TRUE    1        2    9.09       18.18
3      f.mont.TRUE    1        3    9.09       27.27
4      m.cap.FALSE    1        4    9.09       36.36
5       m.cap.TRUE    1        5    9.09       45.45
6    m.escal.FALSE    1        6    9.09       54.55
7     m.escal.TRUE    1        7    9.09       63.64
8     m.mont.FALSE    1        8    9.09       72.73
9      m.mont.TRUE    1        9    9.09       81.82
10    m.none.FALSE    1       10    9.09       90.91
11 none.none.FALSE    1       11    9.09      100.00

$n.sent
     interval Freq cum.Freq percent cum.percent
1 (3.85,34.7]    7        7   63.64       63.64
2 (34.7,65.6]    0        7    0.00       63.64
3 (65.6,96.4]    2        9   18.18       81.82
4  (96.4,127]    1       10    9.09       90.91
5   (127,158]    1       11    9.09      100.00

$n.words
            interval Freq cum.Freq percent cum.percent
1         (14.4,336]    6        6   54.55       54.55
2          (336,658]    2        8   18.18       72.73
3          (658,981]    1        9    9.09       81.82
4      (981,1.3e+03]    1       10    9.09       90.91
5 (1.3e+03,1.62e+03]    1       11    9.09      100.00

$n.char
             interval Freq cum.Freq percent cum.percent
1     (72.7,1.34e+03]    6        6   54.55       54.55
2  (1.34e+03,2.6e+03]    2        8   18.18       72.73
3  (2.6e+03,3.86e+03]    1        9    9.09       81.82
4 (3.86e+03,5.12e+03]    1       10    9.09       90.91
5 (5.12e+03,6.39e+03]    1       11    9.09      100.00
</code></pre>


<h4 id="pos">Parts of Speech Tagging & Counts</h4>

In some analysis of text the research may wish to gather information about parts of speech (POS).  The function `r FUN("pos")` and it's grouping variable counterpart, `r FUN("pos_by", "pos")`, can provide this functionality.  The `r FUN("pos")` functions are wrappers for POS related functions from the `r HR("http://cran.r-project.org/web/packages/openNLP/index.html", "openNLP")` package.  The `r FUN("pos_tags", "pos")` function provides a quick reference to what the POS tags utilized by `r HR("http://cran.r-project.org/web/packages/openNLP/index.html", "openNLP")` mean.  For more information on the POS tags see the `r HR("http://www.cis.upenn.edu/~treebank/", "Penn Treebank Project")`.

The following examples utilize the `r FUN("pos_by", "pos")` function as the `r FUN("pos")` function is used identically, except without specifying a `grouping.var`.  It is important to realize that POS tagging is a very slow process.  The speed can be increased by using the <b><font  color="green" face="courier">parallel = TRUE</font></b> argument. Additionally, the user can recycle the output from one run of `r FUN("pos")`, `r FUN("pos_by", "pos")` or `r FUN("formality")` and use it interchangeably between the `r FUN("pos_by", "pos")` and `r FUN("formality")` functions.  This reuses the POS tagging which is the time intensive part (and can be extracted via <b><font  color="green" face="courier">YOUR_OUTPUT_HERE[[&quot;POStagged&quot;]]</font></b> from any of the above objects).

`r FT(orange, 5, text="&diams;")` **`r FUN("pos_tags", "pos")`** - *Interpreting POS Tags*`r FT(orange, 5, text="&diams;")`

```{r}
pos_tags()
```

```{r, include = FALSE}
posbydat <- readRDS("data/posbydat.rds")
```


`r FT(orange, 5, text="&diams;")` **`r FUN("pos_by", "pos")`** - *POS by Group(s)*`r FT(orange, 5, text="&diams;")`

```{r, comment=NA}
posbydat <- with(DATA, pos_by(state, list(adult, sex)))
## Available elements
names(posbydat)
```


```{r, eval = FALSE}
## Inspecting the truncated output
lview(posbydat)
```

<pre><code>$text
[1] "computer is fun not too fun"     "no its not its dumb"           
[3] "what should we do"               "you liar it stinks"            
[5] "i am telling the truth"          "how can we be certain"         
[7] "there is no way"                 "i distrust you"                
[9] "what are you talking about"      "shall we move on good then"    
[11] "im hungry lets eat you already"


$POStagged
                               POStagged                   POStags word.count
1 computer/NN is/VBZ fun/NN ...RB fun/NN   NN, VBZ, NN, RB, RB, NN          6
2 no/DT its/PRP$ not/RB its/PRP$ dumb/JJ    DT, PRP$, RB, PRP$, JJ          5
3         what/WP should/MD we/PRP do/VB           WP, MD, PRP, VB          4
4      you/PRP liar/VBP it/PRP stinks/VB         PRP, VBP, PRP, VB          4
5 i/PRP am/VBP telling/VBG...DT truth/NN     PRP, VBP, VBG, DT, NN          5
6 how/WRB can/MD we/PRP ...VB certain/JJ      WRB, MD, PRP, VB, JJ          5
7           there/EX is/VBZ no/DT way/NN           EX, VBZ, DT, NN          4
8               i/FW distrust/NN you/PRP               FW, NN, PRP          3
9  what/WP are/VBP you/PR...VBG about/IN     WP, VBP, PRP, VBG, IN          5
10 shall/MD we/PRP move/VB ...JJ then/RB   MD, PRP, VB, IN, JJ, RB          6
11 im/PRP hungry/JJ let...PRP already/RB PRP, JJ, VBZ, VB, PRP, RB          6


$POSprop
   wrd.cnt propDT propEX   propFW   propIN   propJJ   propMD ... propWRB
1        6      0      0  0.00000  0.00000  0.00000  0.00000           0
2        5     20      0  0.00000  0.00000 20.00000  0.00000           0
3        4      0      0  0.00000  0.00000  0.00000 25.00000           0
4        4      0      0  0.00000  0.00000  0.00000  0.00000           0
5        5     20      0  0.00000  0.00000  0.00000  0.00000           0
6        5      0      0  0.00000  0.00000 20.00000 20.00000          20
7        4     25     25  0.00000  0.00000  0.00000  0.00000           0
8        3      0      0 33.33333  0.00000  0.00000  0.00000           0
9        5      0      0  0.00000 20.00000  0.00000  0.00000           0
10       6      0      0  0.00000 16.66667 16.66667 16.66667           0
11       6      0      0  0.00000  0.00000 16.66667  0.00000           0

$POSfreq
   wrd.cnt DT EX FW IN JJ MD NN PRP PRP$ RB VB VBG VBP VBZ WP WRB
1        6  0  0  0  0  0  0  3   0    0  2  0   0   0   1  0   0
2        5  1  0  0  0  1  0  0   0    2  1  0   0   0   0  0   0
3        4  0  0  0  0  0  1  0   1    0  0  1   0   0   0  1   0
4        4  0  0  0  0  0  0  0   2    0  0  1   0   1   0  0   0
5        5  1  0  0  0  0  0  1   1    0  0  0   1   1   0  0   0
6        5  0  0  0  0  1  1  0   1    0  0  1   0   0   0  0   1
7        4  1  1  0  0  0  0  1   0    0  0  0   0   0   1  0   0
8        3  0  0  1  0  0  0  1   1    0  0  0   0   0   0  0   0
9        5  0  0  0  1  0  0  0   1    0  0  0   1   1   0  1   0
10       6  0  0  0  1  1  1  0   1    0  1  1   0   0   0  0   0
11       6  0  0  0  0  1  0  0   2    0  1  1   0   0   1  0   0

$POSrnp
   wrd.cnt       DT       EX       FW       IN       JJ ...      WRB
1        6        0        0        0        0        0            0
2        5 1(20.0%)        0        0        0 1(20.0%)            0
3        4        0        0        0        0        0            0
4        4        0        0        0        0        0            0
5        5 1(20.0%)        0        0        0        0            0
6        5        0        0        0        0 1(20.0%)     1(20.0%)
7        4 1(25.0%) 1(25.0%)        0        0        0            0
8        3        0        0 1(33.3%)        0        0            0
9        5        0        0        0 1(20.0%)        0            0
10       6        0        0        0 1(16.7%) 1(16.7%)            0
11       6        0        0        0        0 1(16.7%)            0

$percent
[1] TRUE

$zero.replace
[1] 0

$pos.by.freq
  adult&sex wrd.cnt DT EX FW IN JJ MD NN PRP PRP$ RB VB VBG VBP VBZ WP WRB
1       0.f      10  0  0  0  1  1  1  0   2    0  0  1   1   1   0  1   1
2       0.m      33  3  1  1  0  2  0  6   6    2  4  2   1   2   3  0   0
3       1.f       6  0  0  0  1  1  1  0   1    0  1  1   0   0   0  0   0
4       1.m       4  0  0  0  0  0  1  0   1    0  0  1   0   0   0  1   0


$pos.by.prop
  adult&sex wrd.cnt       DT       EX       FW       IN        JJ ... WP
1       0.f      10 0.000000 0.000000 0.000000 10.00000 10.000000     10
2       0.m      33 9.090909 3.030303 3.030303  0.00000  6.060606      0
3       1.f       6 0.000000 0.000000 0.000000 16.66667 16.666667      0
4       1.m       4 0.000000 0.000000 0.000000  0.00000  0.000000     25


$pos.by.rnp
  adult&sex wrd.cnt      DT      EX      FW       IN       JJ ...       WP
1       0.f      10       0       0       0 1(10.0%) 1(10.0%)     1(10.0%)
2       0.m      33 3(9.1%) 1(3.0%) 1(3.0%)        0  2(6.1%)            0
3       1.f       6       0       0       0 1(16.7%) 1(16.7%)            0
4       1.m       4       0       0       0        0        0     1(25.0%)
</code></pre>


`r FT(orange, 5, text="&diams;")` **Plot Method**`r FT(orange, 5, text="&diams;")`

```{r}
plot(posbydat, values = TRUE, digits = 2)
```


`r FT(orange, 5, text="&diams;")` **`r FUN("pos_by", "pos")`** - *Recycling Saves Time*`r FT(orange, 5, text="&diams;")`

<pre><code class="r">posbydat2 <- with(DATA, pos_by(posbydat, list(person, sex)))
system.time(with(DATA, pos_by(posbydat, list(person, sex))))
</code></pre>


<pre><code>    user  system elapsed 
    0.07    0.00    0.07
</code></pre>

```{r}
## `pos_by` output Recycled for `formality`
with(DATA, formality(posbydat, list(person, sex)))
```


<h4 id="syll">Syllabication and Counts</h4>

Examining syllable counts can be a useful source of information in associating with education level, age, SES, gender, etc.  Several readability scores rely on syllable and polysyllable word counts.  qdap defines a *polysyllable* word as a word with 3 or more syllables, though some in the linguistics/literacy fields may include two syllable words.  `r FUN("syllable_count", "syllabication")` is the base function for `r FUN("syllable_sum", "syllabication")`, `r FUN("polysyllable_sum", "syllabication")`, and `r FUN("combo_syllable_sum", "syllabication")`, though is generally not of direct use to the researcher conducting discourse analysis.  `r FUN("syllable_count", "syllabication")` uses a dictionary lookup method augmented with a syllable algorithm for words not found in the dictionary.  Words not found in the dictionary are denoted with a <b>NF</b> in the  <b>in.dictionary</b> column of the output.


Here is a list of qdap `r FUN("syllabication", "syllabication")` functions and their descriptions:<br>

<TABLE BORDER=2 CELLPADDING=2 CELLSPACING=2 WIDTH="90%">
  <TR> <TD> <b>syllable_count</b> </TD> <TD> Count the number of syllables in a single text string. </TD> </TR>
  <TR> <TD> <b>syllable_sum</b> </TD> <TD> Count the number of syllables per row of text. </TD> </TR>
  <TR> <TD> <b>polysyllable_sum</b> </TD> <TD> Count the number of polysyllables per row of text. </TD> </TR>
  <TR> <TD> <b>combo_syllable_sum</b> </TD> <TD> Count the number of both syllables and polysyllables per row of text. </TD> </TR>
   </TABLE>


`r FT(orange, 5, text="&diams;")` **`r FUN("syllabication", "syllabication")` Examples**`r FT(orange, 5, text="&diams;")`


```{r, comment=NA}
syllable_count("Robots like Dason lie.")
## The text variable for reference
DATA$state
syllable_sum(DATA$state)
polysyllable_sum(DATA$state)
combo_syllable_sum(DATA$state)
```



<h3 id="measures">Word Measures and Scoring</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):    

<form action="http://trinker.github.io/qdap_dev/automated_readability_index.html" target="_blank">
    <input type="submit" value="automated_readability_index"><input type="submit" value="coleman_liau"><input type="submit" value="flesch_kincaid"><input type="submit" value="fry"><input type="submit" value="linsear_write"><input type="submit" value="SMOG"> - `r HR("#readability", "Readability Measures")`
</form>

<form action="http://trinker.github.io/qdap_dev/dissimilarity.html" target="_blank">
    <input type="submit" value="Dissimilarity"> - `r HR("#dissimilarity", "Dissimilarity")`
</form>

<form action="http://trinker.github.io/qdap_dev/diversity.html" target="_blank">
    <input type="submit" value="diversity"> - `r HR("#diversity", "Diversity Statistics")`
</form>

<form action="http://trinker.github.io/qdap_dev/formality.html" target="_blank">
    <input type="submit" value="formality"> - `r HR("#formality", "Formality Score")`
</form>

<form action="http://trinker.github.io/qdap_dev/kullback_leibler.html" target="_blank">
    <input type="submit" value="kullback_leibler"> - `r HR("#kullback", "Kullback-Leibler divergence ")`
</form>

<form action="http://trinker.github.io/qdap_dev/polarity.html" target="_blank">
    <input type="submit" value="polarity"> - `r HR("#polarity", "Polarity Score (Sentiment Analysis)")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_cor.html" target="_blank">
    <input type="submit" value="word_cor"> - `r HR("#wordcor", "Word Associations (Correlations)")`
</form>
</div>

qdap offers a number of word statistics and scores applied by grouping variable.  Some functions are original to qdap, while others are taken from academic papers.  Complete references for statistics/scores based on others' work are provided in the `r HR("http://cran.r-project.org/web/packages/qdap/qdap.pdf", "help manual")` where appropriate.  It is assumed that the reader is familiar, or can become acquainted, with the theory and methods for qdap functions based on the work of others.  For qdap functions that are original to qdap a more robust description of the use and theory is provided.

<h4 id="readability">Readability Scores</h4>

Readability scores were originally designed to measure the difficulty of text.  Scores are generally based on, number of words, syllables, polly-syllables and word length.  While these scores are not specifically designed for, or tested on, speech, they can be useful indicators of speech complexity.  The following score examples demonstrate the use of the following readability scores:

1. `r HR("#ari", "Automated Readability Index")`    
2. `r HR("#coleman", "Coleman Liau")`    
3. `r HR("#smog", "SMOG")`    
4. `r HR("#flesch", "Flesch Kincaid")`     
5. `r HR("#fry", "Fry")`    
6.  `r HR("#linwr", "Linsear Write")`    


<p id="ari">`r FT(orange, 5, text="&diams;")` **Automated Readability Index**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval = FALSE}
with(rajSPLIT, automated_readability_index(dialogue, list(sex, fam.aff)))
```

<pre><code>  sex&fam.aff word.count sentence.count character.count Aut._Read._Index
1       f.cap       9458            929           37474              2.3
2      f.mont         28              4              88             -3.1
3       m.cap       1204            133            4615              1.2
4     m.escal       3292            262           13406              4.0
5      m.mont       6356            555           26025              3.6
6      m.none       3233            250           13527              4.7
7   none.none        156             12             665              5.1
</code></pre>

<p id="coleman">`r FT(orange, 5, text="&diams;")` **Coleman Liau**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval = FALSE}
with(rajSPLIT, coleman_liau(dialogue, list(fam.aff, act)))
```

<pre><code>  fam.aff&act word.count sentence.count character.count Coleman_Liau
1       cap.1       2636            272           10228          4.0
2       cap.2       2113            193            8223          4.4
3       cap.3       3540            339           14183          4.9
4       cap.4       2159            232            8620          4.5
5       cap.5        214             26             835          3.5
6     escal.1        748             36            3259          8.4
</code></pre>

<p id="smog">`r FT(orange, 5, text="&diams;")` **SMOG**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval = FALSE}
with(rajSPLIT, SMOG(dialogue, list(person, act)))
```

<pre><code>         person&act word.count sentence.count polysyllable.count SMOG
1        Benvolio.1        621             51                 25  7.1
2         Capulet.1        736             72                 35  7.1
3         Capulet.3        749             69                 28  6.8
4         Capulet.4        569             73                 25  6.5
5  Friar Laurence.2        699             42                 36  8.4
6  Friar Laurence.3        675             61                 32  7.3
7  Friar Laurence.4        656             42                 25  7.5
8  Friar Laurence.5        696             54                 32  7.5
9          Juliet.2       1289            113                 48  6.9
10         Juliet.3       1722            152                 64  6.8
11         Juliet.4        932             61                 37  7.6
12   Lady Capulet.3        393             39                 15  6.7
13       Mercutio.2        964             82                 43  7.3
14       Mercutio.3        578             54                 19  6.5
15          Nurse.1        599             59                 20  6.5
16          Nurse.2        779             76                 24  6.3
17          Nurse.3        579             68                 14  5.7
18          Nurse.4        250             50                  9  5.6
19          Romeo.1       1158            113                 48  6.9
20          Romeo.2       1289            109                 46  6.8
21          Romeo.3        969             87                 48  7.4
22          Romeo.5       1216            103                 52  7.2
</code></pre>

<p id="flesch">`r FT(orange, 5, text="&diams;")` **Flesch Kincaid**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval = FALSE}
with(rajSPLIT, flesch_kincaid(dialogue, list(sex, fam.aff)))
```

<pre><code>  sex&fam.aff word.count sentence.count syllable.count FK_grd.lvl FK_read.ease
1       f.cap       9458            929          11641        2.9       92.375
2      f.mont         28              4             30       -0.2      109.087
3       m.cap       1204            133           1452        2.2       95.621
4     m.escal       3292            262           4139        4.1       87.715
5      m.mont       6356            555           7965        3.7       89.195
6      m.none       3233            250           4097        4.4       86.500
7   none.none        156             12            195        4.2       87.890
</code></pre>

Note that the Fry score is a graphical display, rather than text as the other readability scores are.  This is in keeping with the original procedures outlined by Fry.

<p id="fry">`r FT(orange, 5, text="&diams;")` **Fry**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval =  FALSE}
with(rajSPLIT, fry(dialogue, list(sex, fam.aff)))
```


```{r, echo = FALSE, fig.width = 10}
set.seed(46)
with(rajSPLIT, fry(dialogue, list(sex, fam.aff)))
```


<p id="linwr">`r FT(orange, 5, text="&diams;")` **Linsear Write**`r FT(orange, 5, text="&diams;")`</p >

```{r, eval = FALSE}
with(rajSPLIT, linsear_write(dialogue, person))
```

<pre><code>           person sent.per.100 hard_easy_sum Linsear_Write
1       Balthasar        9.556           110          4.76
2        Benvolio        4.143           108         12.03
3         Capulet       11.469           115          4.01
4          Chorus        3.071           104         15.93
5  First Watchman       14.222           114          3.01
6  Friar Laurence        4.263           108         11.67
7         Gregory       11.000           100          3.55
8          Juliet        3.446           110         14.96
9    Lady Capulet        7.267           110          6.57
10       Mercutio        5.625           102          8.07
11       Montague        6.000           114          8.50
12          Nurse       12.098           102          3.22
13          Paris        9.091           110          5.05
14          Peter       10.357           110          4.31
15         Prince       10.842           110          4.07
16          Romeo        9.250           114          5.16
17        Sampson        9.421           107          4.68
18        Servant        9.667           104          4.38
19         Tybalt        9.591           112          4.84
</code></pre>


<h4 id="dissimilarity">Dissimilarity</h4>

*Dissimilarity* is another term for distance that is often used in text analysis to measure the pairwise proximity of grouping variables.  The qdap `r FUN("Dissimilarity")` function is a wrapper for the R stats package's `r HR("http://stat.ethz.ch/R-manual/R-devel/library/stats/html/dist.html", "dist")` function designed to handle text.  `r FUN("Dissimilarity")` takes all the same <b><font color="green" face="courier new">method</font></b> types as `r HR("http://stat.ethz.ch/R-manual/R-devel/library/stats/html/dist.html", "dist")` but also includes the default <b><font color="green" face="courier new">method = "prop"</font></b> (1 - "binary") that is focused on the similarity between grouping variables.

<p id="linwr">`r FT(orange, 5, text="&diams;")` **`r FUN("Dissimilarity")` Examples**`r FT(orange, 5, text="&diams;")`</p >

```{r, comment=NA}
with(DATA, Dissimilarity(state, list(sex, adult)))
with(DATA, Dissimilarity(state, person))
with(DATA, Dissimilarity(state, person, method = "minkowski"))
```

```{r eval=FALSE}
dat <- pres_debates2012[pres_debates2012$person %in% qcv(OBAMA, ROMNEY),]
with(dat, Dissimilarity(dialogue, list(person, time)))
```

<pre><code>         OBAMA.1 OBAMA.2 OBAMA.3 ROMNEY.1 ROMNEY.2
OBAMA.2    0.340                                  
OBAMA.3    0.300   0.341                          
ROMNEY.1   0.340   0.287   0.258                  
ROMNEY.2   0.291   0.349   0.296    0.321         
ROMNEY.3   0.264   0.297   0.329    0.290    0.338
</code></pre>


<p id="linwr">`r FT(orange, 5, text="&diams;")` **`r FUN("Dissimilarity")` Clustering (Dendrogram)**`r FT(orange, 5, text="&diams;")`</p >

```{r}
x <- with(pres_debates2012, Dissimilarity(dialogue, list(person, time)))
fit <- hclust(x)
plot(fit)
```

```{r, echo=-1}
plot(fit)
## draw dendogram with colored borders around the 3 clusters 
rect.hclust(fit, k=3, border=c("red", "purple", "seagreen"))
```

<h4 id="kullback">Kullback-Leibler divergence </h4>

The Kullback Leibler is often used as a measure of distance, though the matrix is asymmetrical.  qdap's `r FUN("kullback_leibler")` compares the differences between two probability distributions and often leads to results similar to those from `r FUN("Dissimilarity")`.  Note that unlike many other qdap functions the user must either supply a word frequency matric (`r FUN("wfm", "Word_Frequency_Matrix")`) to  <b><font color="green" face="courier new">x</font></b> or some other matrix format.  This allows the function to be flexibly used with `r FUN("termco")` and other functions that produce count matrices.

```{r include = FALSE}
dat <- pres_debates2012[pres_debates2012$person %in% qcv(OBAMA, ROMNEY),]
KL <- (kullback_leibler(with(dat, wfm(dialogue, list(person, time)))))
```

<p id="linwr">`r FT(orange, 5, text="&diams;")` **`r FUN("kullback_leibler")` Example** - *Compare to `r FUN("Dissimilarity")`*`r FT(orange, 5, text="&diams;")`</p >

```{r eval = FALSE}
dat <- pres_debates2012[pres_debates2012$person %in% qcv(OBAMA, ROMNEY),]
(KL <- (kullback_leibler(with(dat, wfm(dialogue, list(person, time))))))
```

<pre><code>         OBAMA.1 OBAMA.2 OBAMA.3 ROMNEY.1 ROMNEY.2 ROMNEY.3
OBAMA.1    0.000   0.237   0.221    0.195    0.250    0.264
OBAMA.2    0.104   0.000   0.161    0.148    0.142    0.223
OBAMA.3    0.119   0.152   0.000    0.142    0.180    0.168
ROMNEY.1   0.207   0.297   0.279    0.000    0.216    0.224
ROMNEY.2   0.194   0.195   0.262    0.116    0.000    0.234
ROMNEY.3   0.160   0.182   0.141    0.101    0.140    0.000
</code></pre>

```{r, fig.height = 4.5, fig.width = 5.5}
plot(KL, high = "red", values = TRUE)
```


<h4 id="diversity">Diversity Statistics</h4>

Diversity, as applied to dialogue, is a measure of the richness of language being used.  Specifically, it measures how expansive the vocabulary is while taking into account the number of total words used and the different words being used.  qdap's `r FUN("diversity")` function provides output for the Simpson, Shannon, Collision, Berger Parker, and Brillouin measures.

<p id="linwr">`r FT(orange, 5, text="&diams;")` **`r FUN("diversity")` Example**`r FT(orange, 5, text="&diams;")`</p >
```{r, comment =NA}
(div.mod <- with(mraja1spl, diversity(dialogue, person)))
```

<p id="linwr">`r FT(orange, 5, text="&diams;")` **`r FUN("diversity")` Plot Method**`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 10, fig.height = 5}
plot(div.mod, low = "yellow", grid = FALSE, values = TRUE)
```

<h4 id="formality">Formality</h4>

Formality is how contextualize a person's language use is.  In situations involving what may be new content/context for an audience, a speaker may be more formal in their speech (Heylighen & Dewaele, 1999a, 1999b, 2002).  Heylighen & Dewaele (2002) have developed a measure of formality based on categorizing parts of speech into contextual/formal categories.  While qdap is not the creator of the algorithm for calculating `r FUN("formality")`, Heylighen & Dewaele's (2002) F-measure (formality) is less known than other qdap word measures and thus more explanation is provide to the reader than say the `r FUN("Dissimilarity")` measures above. Heylighen & Dewaele's (2002) F-measure is calculated by finding the difference of all of the formal parts ($f$) of speech (noun, adjective, preposition, article) and contextual ($c$) parts of speech (pronoun, verb, adverb, interjection) divided by the sum of all formal & contextual speech plus conjunctions ($N$).  This quotient is added to one and multiplied by 50 to ensure a measure between 0 and 1, with scores closer to 100 being more formal and those approaching 0 being more contextual.
<br><br>

$$ F = 50(\frac{n_{f}-n_{c}}{N} + 1) $$

Where:
<br>

$$ f = \left \{noun, \;adjective, \;preposition, \;article\right \} $$
$$ c = \left \{pronoun, \;verb, \;adverb, \;interjection\right \} $$
$$ N = \sum{(f \;+ \;c \;+ \;conjunctions)} $$
<br>

Note that formality utilize parts of speech tagging. This is computationally expensive.  The user may gain speed by setting <b><font color="green" face="courier new">parallel = TRUE</font></b> if multiple cores are available.  The user can also "recycle" the output from `r FUN("pos")`, `r FUN("pos_by", "pos")`, or `r FUN("formality")` for the same text.  This save considerable time as the parts of speech is saved in the output from these functions as demonstrated in the `r HR("#recform", "Recycled Formality Example")` below.

`r FUN("formality")` also has a plotting method that allows for easy visualization and comparison of formality scores, word counts, formal/contextual parts of speech all by grouping variable(s).  Please note that Heylighen & Dewaele (2002) state, "At present, a sample would probably need to contain a few hundred words for the measure to be minimally reliable. For single sentences, the F-value should only be computed for purposes of illustration" (p. 24).  


```{r, include = FALSE}
#citep(bib["Heylighen1999a"])
#citep(bib["Heylighen1999b"])
#citep(bib["Heylighen2002"])
```

<p>`r FT(orange, 5, text="&diams;")` **`r FUN("formality")` Example**`r FT(orange, 5, text="&diams;")`</p >

```{r, include = FALSE, comment = FALSE}
(form <- with(raj, formality(rajPOS, person)))
```

```{r, eval=FALSE}
form <- with(raj, formality(dialogue, person))
```

<p>`r FT(orange, 5, text="&diams;")` **`r FUN("formality")` Plot Method**`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14, fig.height = 10}
plot(form)
```

<p id="recform">`r FT(orange, 5, text="&diams;")` **Recycling `r FUN("formality")`**`r FT(orange, 5, text="&diams;")`</p > 

```{r, comment = NA}
(form2 <- with(raj, formality(form, act)))
```

```{r, fig.width = 14, fig.height = 3.5}
plot(form2, bar.colors=c("Set2", "RdBu"))
```

<h4 id="polarity">Polarity Score (Sentiment Analysis)</h4>

Polarity assignment, a form of sentiment analysis, is using an algorithm to determine the polarity of a sentence/statement.  While the use polarity scores is applied to many forms of written social dialogue (e.g., Twitter, Facebook, etc.) it has not typically been applied to spoken dialogue.  qdap offers a flexible function, `r FUN("polarity")` to determine polarity at the sentence level as well as to assign an average polarity score to individual groups within the grouping variable(s).  The frame work for `r FUN("polarity")` is flexible in that the user may supply a polarized dictionary and optional weights.  Many dictionaries used in sentiment analysis are designed for written, adult, online interaction.  

<div class="middleDiv">
<b>`r FT(red, 4, text="It is assumed you have run <font face=\"courier\">sentSplit</font> on the data.<br>If this is not the case the counts will not be accurate.")`</b>
</div>

The polarity score generated by `r FUN("polarity")` is dependent upon the polarity dictionary used.  This function defaults to the word polarity dictionary used by `r HR("http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html", "Hu & Liu (2004)")`, however, this may not be appropriate for the context of children in a classroom.  For instance the word <font color="green">"sick"</font> in a high school setting may mean that something is good, whereas <font color="green">"sick"</font> used by a typical adult indicates something is not right or negative connotation.  The user may (is encouraged) to provide/augment/alter the dictionary (see the `r FUN("sentiment_frame", "polarity")` function).  Development of context specific dictionaries, that are better suited for spoken dialogue in a school setting, is an exciting prospect that could lead to greater understanding of the emotional aspects of the spoken word on students.  The user may add a dictionary with optional weights as a dataframe within an environment.  The `r FUN("sentiment_frame", "polarity")` function aides the user in creating the polarity environment. 


```{r, include=FALSE}
#citep(bib["Hu2004"])
```

The equation used by the algorithm to assign value to polarity of each sentence fist utilizes the sentiment dictionary (Hu & Liu, 2004) to tag polarized words.  A context cluster of words is pulled from around this polarized word (default 4 words before and two words after) to be considered as valence shifters.  The words in this context cluster are tagged as neutral ($x_i^{0}$), negator ($x_i^{N}$), amplifier ($x_i^{a}$), or de-amplifier ($x_i^{d}$). Neutral words hold no value in the equation but do affect word count ($n$).  Each polarized word is then weighted $w$ based on the weights from the <b><font color="green" face="courier new">polarity.frame</font></b> argument and then further weighted by the number and position of the valence shifters directly surrounding the positive or negative word.  The researcher may provide a weight $c$ to be utilized with amplifiers/de-amplifiers (default is .8; deamplifier weight is constrained to -1 lower bound).  Last,  these context cluster ($x_i^{T}$) are summed and divided by the square root of the word count ($\sqrt{n}$) yielding an unbounded polarity score 
 ($\delta$).  Note that context clusters containing a comma before the  polarized word will only consider words found after the comma.
 
$$
\delta=\frac{x_i^T}{\sqrt{n}}
$$

 Where:
 
$$
x_i^T=\sum{((1 + c(x_i^{A} - x_i^{D}))\cdot w(-1)^{\sum{x_i^{N}}})}
$$
<br>
$$
x_i^{A}=\sum{(w_{neg}\cdot x_i^{a})}
$$
<br>
$$
x_i^D = \max(x_i^{D'}, -1)
$$ 
<br>
$$
x_i^{D'}=\sum{(- w_{neg}\cdot x_i^{a} + x_i^{d})}
$$ 
<br>
$$
w_{neg}= \left(\sum{x_i^{N}}\right) \bmod {2}
$$


The following examples demonstrate how the `r FUN("polarity")` and `r FUN("sentiment_frame", "polarity")` functions operate.  Here the polarity for the `r FUN("mraja1spl")` data set (Act 1 of Romeo and Juliet).  The gender, family affiliation and binary died/didn't die are used as the grouping variables.

<p id="polarity1">`r FT(orange, 5, text="&diams;")` **`r FUN("polarity")` Example**`r FT(orange, 5, text="&diams;")`</p >

```{r, echo = FALSE, include=FALSE}
poldat <- readRDS("data/polarity.example.rds")
```


<pre><code class="r"> (poldat <- with(mraja1spl, polarity(dialogue, list(sex, fam.aff, died))))
</code></pre>

<pre><code>POLARITY BY GROUP
=================

   sex&fam.aff&died tot.sent tot.word ave.polarity sd.polarity sd.mean.polarity
1       f.cap.FALSE      158     1810        0.076       0.262            0.292
2        f.cap.TRUE       24      221        0.042       0.209            0.204
3       f.mont.TRUE        4       29        0.079       0.398            0.199
4       m.cap.FALSE       73      717        0.026       0.256            0.104
5        m.cap.TRUE       17      185       -0.160       0.313           -0.510
6     m.escal.FALSE        9      195       -0.153       0.313           -0.488
7      m.escal.TRUE       27      646       -0.069       0.256           -0.272
8      m.mont.FALSE       70      952       -0.044       0.384           -0.114
9       m.mont.TRUE      114     1273       -0.004       0.409           -0.009
10     m.none.FALSE        7       78        0.062       0.107            0.583
11  none.none.FALSE        5       18       -0.282       0.439           -0.642
</code></pre>

```{r}
names(poldat)
```

<p id="polarity2">`r FT(orange, 5, text="&diams;")` **`r FUN("polarity")`** - *Sentence Level Polarity Scores*`r FT(orange, 5, text="&diams;")`</p >

```{r}
htruncdf(counts(poldat), 20, 10)
```

<p id="polarity3">`r FT(orange, 5, text="&diams;")` **`r FUN("polarity")` Plot Method**`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 15, fig.height = 11}
plot(poldat)
```

<p id="polarity4">`r FT(orange, 5, text="&diams;")` **`r FUN("polarity")` Plot Group Polarity as Heat Map**`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.height=4}
qheat(scores(poldat), high="blue", low="yellow", grid=NULL, order.b="ave.polarity")
```

<p id="polarity5">`r FT(orange, 5, text="&diams;")` **`r FUN("sentiment_frame")`** - *Specify Your Own Polarity Environment*`r FT(orange, 5, text="&diams;")`</p >

```{r, comment=NA}
(POLENV <- sentiment_frame(positive.words, negative.words))
```

<p id="polarity6">`r FT(orange, 5, text="&diams;")` **Polarity Over Time** `r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">poldat4 <- with(rajSPLIT, polarity(dialogue, act, constrain = TRUE))

polcount <- na.omit(counts(poldat4)$polarity)
len <- length(polcount)

cummean <- function(x){cumsum(x)/seq_along(x)}

cumpolarity <- data.frame(cum_mean = cummean(polcount), Time=1:len)

## Calculate background rectangles
ends <- cumsum(rle(counts(poldat4)$act)$lengths)
starts <- c(1, head(ends + 1, -1))
rects <- data.frame(xstart = starts, xend = ends + 1,
    Act = c("I", "II", "III", "IV", "V"))

library(ggplot2)
ggplot() + theme_bw() +
    geom_rect(data = rects, aes(xmin = xstart, xmax = xend,
        ymin = -Inf, ymax = Inf, fill = Act), alpha = 0.17) +
    geom_smooth(data = cumpolarity, aes(y=cum_mean, x = Time)) +
    geom_hline(y=mean(polcount), color="grey30", size=1, alpha=.3, linetype=2) +
    annotate("text", x = mean(ends[1:2]), y = mean(polcount), color="grey30",
        label = "Average Polarity", vjust = .3, size=3) +
    geom_line(data = cumpolarity, aes(y=cum_mean, x = Time), size=1) +
    ylab("Cumulative Average Polarity") + xlab("Duration") +
    scale_x_continuous(expand = c(0,0)) +
    geom_text(data=rects, aes(x=(xstart + xend)/2, y=-.04,
        label=paste("Act", Act)), size=3) +
    guides(fill=FALSE) +
    scale_fill_brewer(palette="Set1")
</code></pre>

```{r, echo = FALSE, fig.width=14.5, fig.height=6, message=FALSE, warning=FALSE}
poldat4 <- readRDS("data/poldat4.rds")

polcount <- na.omit(counts(poldat4)$polarity)
len <- length(polcount)

cummean <- function(x){cumsum(x)/seq_along(x)}

cumpolarity <- data.frame(cum_mean = cummean(polcount), Time=1:len)

## Calculate background rectangles
ends <- cumsum(rle(counts(poldat4)$act)$lengths)
starts <- c(1, head(ends + 1, -1))
rects <- data.frame(xstart = starts, xend = ends + 1,
    Act = c("I", "II", "III", "IV", "V"))

library(ggplot2)
ggplot() + theme_bw() +
    geom_rect(data = rects, aes(xmin = xstart, xmax = xend,
        ymin = -Inf, ymax = Inf, fill = Act), alpha = 0.17) +
    geom_smooth(data = cumpolarity, aes(y=cum_mean, x = Time)) +
    geom_hline(y=mean(polcount), color="grey30", size=1, alpha=.3, linetype=2) +
    ggplot2::annotate("text", x = mean(ends[1:2]), y = mean(polcount), color="grey30",
        label = "Average Polarity", vjust = .3, size=3) +
    geom_line(data = cumpolarity, aes(y=cum_mean, x = Time), size=1) +
    ylab("Cumulative Average Polarity") + xlab("Duration") +
    scale_x_continuous(expand = c(0,0)) +
    geom_text(data=rects, aes(x=(xstart + xend)/2, y=-.04,
        label=paste("Act", Act)), size=3) +
    guides(fill=FALSE) +
    scale_fill_brewer(palette="Set1")
```

<h4 id="wordcor">Word Association (Correlations)</h4>

It is helpful to finds words associated (or negatively associated) with or correlations between selected words in understanding language selection.  The <a href="http://trinker.github.io/qdap_dev/word_cor.html" target="_blank"><code>word&#095;cor</code></a> function calculates correlations (based on the `r FUN("wfm")` function) for words nested within grouping variables (turn of talk is an obvious choice for a grouping variable).  Running bootstrapping with a random sample can help the researcher determine if a co-occurrence of words is by chance.  `r FUN("wordword&#095;cor")` is even more flexible in that it can actually take a frequency matrix (e.g., the `r FUN("wfm&#095;combine", "wfm")` function or the `r HR("#coding", "cm&#095;")` family of functions).  

<p id="wordcor1">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Single Words*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">library(reports)
x <- factor(with(rajSPLIT, paste(act, pad(TOT(tot)), sep = "|")))
word_cor(rajSPLIT$dialogue, x, "romeo", .45)
</code></pre>

<pre><code>$romeo
     that    tybalt 
0.4540979 0.4831937 
</code></pre>


<pre><code class="r">word_cor(rajSPLIT$dialogue, x, "love", .5)
</code></pre>

<pre><code>$love
 likewise 
0.5013104 
</code></pre>

<p id="wordcor2">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Negative Correlation*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">word_cor(rajSPLIT$dialogue, x, "you", -.1)
with(rajSPLIT, word_cor(dialogue, list(person, act), "hate"))
</code></pre>

<pre><code>$hate
 eyesight    knight    prison    smooth     vex'd 
0.7318131 0.7318131 0.7318131 0.7318131 0.7318131 
</code></pre>

<p id="wordcor3">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Multiple Words*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">words <- c("hate", "i", "love", "ghost")
with(rajSPLIT, word_cor(dialogue, x, words, r = .5))
</code></pre>

<pre><code>$hate
          beasts        beseeming            bills             bred 
       0.6251743        0.6251743        0.6251743        0.6251743 
        canker'd      capulethold            clubs           coward 
       0.6251743        0.6251743        0.6251743        0.6251743 
          crutch        disturb'd       flourishes        fountains 
       0.6251743        0.6251743        0.6251743        0.6251743 
         issuing      mistemper'd neighbourstained        partisans 
       0.6251743        0.6251743        0.6251743        0.6251743 
      pernicious        profaners           purple       rebellious 
       0.6251743        0.6251743        0.6251743        0.6251743 
         streets         subjects            sword           thrice 
       0.5027573        0.6251743        0.6164718        0.6251743 
           throw            wield 
       0.6251743        0.6251743 

$i
      and      have        me        my      thee        to 
0.5150992 0.5573359 0.5329341 0.5134372 0.5101593 0.5533506 

$love
 likewise 
0.5013104 

$ghost
     bone    brains      club      dash     drink      keys kinsman's  methinks 
0.7056134 0.7056134 1.0000000 1.0000000 0.5749090 1.0000000 1.0000000 0.5749090 
     rage  rapier's   seeking    spices      spit 
0.5749090 1.0000000 1.0000000 1.0000000 1.0000000 
</code></pre>

<p id="wordcor4">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Correlations Between Terms: Example 1*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">## Set r = NULL to get matrix between words
with(rajSPLIT, word_cor(dialogue, x, words, r = NULL))
</code></pre>

<pre><code>             hate          i        love       ghost
hate   1.00000000 0.05142236  0.15871966 -0.01159382
i      0.05142236 1.00000000  0.36986172  0.01489943
love   0.15871966 0.36986172  1.00000000 -0.02847837
ghost -0.01159382 0.01489943 -0.02847837  1.00000000
</code></pre>

<p id="wordcor5">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Correlations Between Terms: Example 2*`r FT(orange, 5, text="&diams;")`</p >


<pre><code class="r">dat <- pres_debates2012
dat$TOT <- factor(with(dat, paste(time, pad(TOT(tot)), sep = "|")))
dat <- dat[dat$person %in% qcv(OBAMA, ROMNEY), ]
dat$person <- factor(dat$person)
dat.split <- with(dat, split(dat, list(person, time)))

wrds <- qcv(america, debt, dollar, people, tax, health)
lapply(dat.split, function(x) {
    word_cor(x[, "dialogue"], x[, "TOT"], wrds, r=NULL)
})
</code></pre>

<pre><code>$`OBAMA.time 1`
             america       dollar    people          tax      health
america  1.000000000 -0.005979775 0.6117618 -0.005979775  0.13803797
dollar  -0.005979775  1.000000000 0.1650493 -0.004219409 -0.01092353
people   0.611761819  0.165049280 1.0000000  0.165049280  0.50398555
tax     -0.005979775 -0.004219409 0.1650493  1.000000000  0.20572642
health   0.138037974 -0.010923527 0.5039855  0.205726420  1.00000000

$`ROMNEY.time 1`
           america     dollar    people         tax      health
america 1.00000000 0.07493271 0.2336551  0.07033784  0.14986684
dollar  0.07493271 1.00000000 0.5859944  0.11109650  0.33821359
people  0.23365513 0.58599441 1.0000000  0.20584588  0.61333714
tax     0.07033784 0.11109650 0.2058459  1.00000000 -0.01723713
health  0.14986684 0.33821359 0.6133371 -0.01723713  1.00000000

$`OBAMA.time 2`
            america      dollar     people        tax      health
america  1.00000000 -0.01526328 0.41353310 0.07609871  0.25733977
dollar  -0.01526328  1.00000000 0.11671525 0.51222872 -0.01220067
people   0.41353310  0.11671525 1.00000000 0.03761852  0.11285926
tax      0.07609871  0.51222872 0.03761852 1.00000000  0.03431397
health   0.25733977 -0.01220067 0.11285926 0.03431397  1.00000000

$`ROMNEY.time 2`
            america         debt     dollar     people          tax
america  1.00000000 -0.018370290 0.07531545 0.59403781  0.291238391
debt    -0.01837029  1.000000000 0.53340505 0.02329285 -0.009432552
dollar   0.07531545  0.533405053 1.00000000 0.33346752  0.600125943
people   0.59403781  0.023292854 0.33346752 1.00000000  0.516577197
tax      0.29123839 -0.009432552 0.60012594 0.51657720  1.000000000
health   0.06384509 -0.008308652 0.68299026 0.25536510  0.658231340
              health
america  0.063845090
debt    -0.008308652
dollar   0.682990261
people   0.255365102
tax      0.658231340
health   1.000000000

$`OBAMA.time 3`
            america        debt      dollar    people         tax
america  1.00000000 -0.01224452 -0.02326653 0.1182189 -0.02326653
debt    -0.01224452  1.00000000  0.37361771 0.1765301  0.75525297
dollar  -0.02326653  0.37361771  1.00000000 0.1909401  0.70993297
people   0.11821887  0.17653013  0.19094008 1.0000000  0.19094008
tax     -0.02326653  0.75525297  0.70993297 0.1909401  1.00000000

$`ROMNEY.time 3`
          america      debt    dollar    people
america 1.0000000 0.2130341 0.2675978 0.3007027
debt    0.2130341 1.0000000 0.8191341 0.4275521
dollar  0.2675978 0.8191341 1.0000000 0.4666635
people  0.3007027 0.4275521 0.4666635 1.0000000
</code></pre>

<p id="wordcor6">`r FT(orange, 5, text="&diams;")` **`r FUN("word_cor")`** - *Matrix from `r FUN("wfm_combine", "wfm")`*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">worlis <- list(
    pronouns = c("you", "it", "it's", "we", "i'm", "i"),
    negative = qcv(no, dumb, distrust, not, stinks),
    literacy = qcv(computer, talking, telling)
)
y <- wfdf(DATA$state, id(DATA, prefix = TRUE))
z <- wfm_combine(y, worlis)

word_cor(t(z), word = c(names(worlis), "else.words"), r = NULL)
</code></pre>

<pre><code>             pronouns   negative   literacy else.words
pronouns    1.0000000  0.2488822 -0.4407045 -0.5914760
negative    0.2488822  1.0000000 -0.2105380 -0.7146856
literacy   -0.4407045 -0.2105380  1.0000000  0.2318694
else.words -0.5914760 -0.7146856  0.2318694  1.0000000
</code></pre>

<h3 id="visualization">Visualizing Discourse Data</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more): <br>   

<form action="http://trinker.github.io/qdap_dev/dispersion_plot.html" target="_blank">
    <input type="submit" value="dispersion_plot"> - `r HR("#dispersion", "Lexical Dispersion Plot")`
</form>

<form class="form_left" action="http://trinker.github.io/qdap_dev/gradient_cloud.html" target="_blank">
    <input type="submit" value="gradient_cloud"> 
</form>

<form action="http://trinker.github.io/qdap_dev/trans_cloud.html" target="_blank">
    <input type="submit" value="trans_cloud"> - `r HR("#wordclouds", "Word Cloud")`
</form>

<form action="http://trinker.github.io/qdap_dev/gantt_plot.html" target="_blank">
    <input type="submit" value="gantt_plot"><input type="submit" value="gantt_wrap">  - `r HR("#gantts", "Gantt Plot")`
</form>


<form action="http://trinker.github.io/qdap_dev/qheat.html" target="_blank">
    <input type="submit" value="qheat"> - `r HR("#heatmaps", "Quick Heatmap")`
</form>

<form action="http://trinker.github.io/qdap_dev/rank_freq_mplot.html" target="_blank">
    <input type="submit" value="rank_freq_mplot"><input type="submit" value="rank_freq_plot"> - `r HR("#rankfreq", "Rank Frequency Plot")`
</form>

<form action="http://trinker.github.io/qdap_dev/tot_plot.html" target="_blank">
    <input type="submit" value="tot_plot"> - `r HR("#totplot", "Visualize Word Length by Turn of Talk")`
</form>

<form action="http://trinker.github.io/qdap_dev/trans_venn.html" target="_blank">
    <input type="submit" value="trans_venn"> - `r HR("#venn", "Venn Diagram")`
</form>

<form action="http://trinker.github.io/qdap_dev/word_network_plot.html" target="_blank">
    <input type="submit" value="word_network_plot"> - `r HR("#wordnet", "Word Network Plot")`
</form>
</div>

qdap offers a number of plot methods for various outputs from functions (use <b><font color="green" face="courier new">plot(qdap_FUNCTION_OUTPUT)</font></b>).  In addition to the numerous plot methods qdap also has several functions dedicated solely to plotting purposes.  Many of these functions rely on the `r HR("http://docs.ggplot2.org/current/", "ggplot2 package")` (Wickham, 2009) to produce plots.

```{r, include = FALSE}
#citep(bib["Wickham2009"])
```

<h4 id="dispersion">Lexical Dispersion Plot</h4>

The *lexical dispersion plot* is a useful tool (particularly in the early stage of analysis) for looking at the dispersion of a word throughout the dialogue.  `r FUN("dispersion_plot")` provides the means to look at and compare multiple word dispersions across repeated measures and/or grouping variables.  This can be useful in conjunction with a `r HR("#corr", "correlation analysis")`.  

The `r HR("#match", "search mechanism")` used by `r FUN("dispersion_plot")` is identical to `r FUN("termco")` and `r FUN("term_match", "termco")`.  For example, <b><font color="green" face="courier new">" love "</font></b> will not yield the same search as <b><font color="green" face="courier new">"love"</font></b>.  The `r HR("#disper1", "search example below")` demonstrates the way the search functions.  For more information see the `r HR("#match", "termco search description")` above.

<p id="disper1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("dispersion_plot")`</b> - <i>Understand the Search</i>`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">term_match(raj$dialogue, c(" love ", "love", " night ", "night"))
</code></pre>

<pre><code>$` love `
[1] "love"

$love
 [1] "love"           "love's"         "lovers"         "loved"          "lovely"        
 [6] "lovest"         "lover"          "beloved"        "loves"          "newbeloved"    
[11] "glove"          "lovesong"       "lovedevouring"  "loveperforming" "dearloved"     

$` night `
[1] "night"

$night
[1] "night"       "fortnight"   "nights"      "tonight"     "night's"     "knight"     
[7] "nightingale" "nightly"     "yesternight"
</code></pre>

<p id="disper2">`r FT(orange, 5, text="&diams;")` **`r FUN("dispersion_plot")`** - *Example 1*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 15, fig.height = 3.25}
with(rajSPLIT , dispersion_plot(dialogue, c("love", "night"),
    grouping.var = list(fam.aff, sex), rm.vars = act))
```

<p id="disper2">`r FT(orange, 5, text="&diams;")` **`r FUN("dispersion_plot")`** - *Example 2: Color Schemes*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 15, fig.height = 3}
with(rajSPLIT, dispersion_plot(dialogue, c("love", "night"),
    bg.color = "black", grouping.var = list(fam.aff, sex),
    color = "yellow", total.color = "white", horiz.color="grey20"))
```

Using `r FUN("dispersion_plot")` with `r FUN("freq_terms")`'s <b><font color="green" face="courier new">[["rfswl"]][["all"]]</font></b> can be a useful means of viewing the dispersion of high frequency words after stopword removal.

<p id="disper2">`r FT(orange, 5, text="&diams;")` **`r FUN("dispersion_plot")`** - *Example 3: Using with `r FUN("freq_terms")`*`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">wrds <- freq_terms(pres_debates2012$dialogue, stopwords = Top200Words)

## Add leading/trailing spaces if desired
wrds2 <- spaste(wrds)

## Use `~~` to maintain spaces
wrds2 <- c(" governor~~romney ", wrds2[-c(3, 12)])

## Plot
with(pres_debates2012 , dispersion_plot(dialogue, wrds2, rm.vars = time, 
    color="black", bg.color="white"))
</code></pre>

```{r, echo=FALSE, fig.width = 15, fig.height = 5}
wrds2 <- c(" governor~~romney ", " going ", " that's ", " president ", 
    " we're ", " sure ", " it's ", " we've ", " years ", " jobs ", 
    " don't ", " I'm ", " those ", " got ", " four ", " let ", " middle ", 
    " thousand ", " economy ", " government ", " I've ")
with(pres_debates2012 , dispersion_plot(dialogue, wrds2, rm.vars = time, 
    color="black", bg.color="white"))
```


<h4 id="wordclouds">Word Clouds</h4>

Wordclouds can be a useful tool to help find words/phrases that are used frequently.  It allows for the entire dialogue to be contained in pictorial form.  The word cloud becomes more useful in discovering themes when color can be used in a meaningful way (i.e., the information contained in the word size and word color are not redundant).  qdap has two word cloud functions (both are wrappers for `r HR("http://cran.r-project.org/web/packages/wordcloud/index.html", "wordcloud")` from the wordcloud package).  The `r FUN("trans_cloud")` function produces word clouds with optional theme coloring by grouping variable.  The `r FUN("gradient_cloud")` function produces a gradient word cloud colored by a binary grouping variable.


`r FUN("trans_cloud")` is passed a list of named vectors to <b><font color="green" face="courier new">target.words</font></b> in much the same way as <b><font color="green" face="courier new">match.list</font></b> in `r FUN("termco")`.

<b><em>Format for Named Vectors</b></em>

<pre><code>list(
    theme_1 = c("word1", "word2", "word3"),
    theme_2 = c("word4", "word5"),
    theme_3 = c("word6", "word7", "word8")    
)
</code></pre>

The <b><font color="green" face="courier new">cloud.colors</font></b> argument takes a single color or a vector of colors 1 greater than the number of vectors of <b><font color="green" face="courier new">target.words</font></b>.  The order of <b><font color="green" face="courier new">cloud.colors</font></b> corresponds to the order of <b><font color="green" face="courier new">target.words</font></b> with the extra, final color being utilized for all words not matched to <b><font color="green" face="courier new">target.words</font></b>.


<p id="transcloud1">`r FT(orange, 5, text="&diams;")` **`r FUN("trans_cloud")` Example 1**`r FT(orange, 5, text="&diams;")`</p >

```{r}
## Generate themes/terms to color by
terms <- list(
    I=c("i", "i'm"),
    mal=qcv(stinks, dumb, distrust),
    articles=qcv(the, a, an),
    pronoun=qcv(we, you)
)

with(DATA, trans_cloud(state, person, target.words=terms,
    cloud.colors=qcv(red, green, blue, black, gray65),
    expand.target=FALSE, proportional=TRUE, legend=c(names(terms),
    "other")))
```

<p id="transcloud1">`r FT(orange, 5, text="&diams;")` **`r FUN("trans_cloud")` Example 2** - *Polarity*`r FT(orange, 5, text="&diams;")`</p >

```{r}
## Rearrange the data
DATA2 <- qdap::DATA
DATA2[1, 4] <- "This is not good!"
DATA2[8, 4] <- "I don't distrust you."
DATA2$state <- space_fill(DATA2$state, paste0(negation.words, " "),
    rm.extra = FALSE)
txt <- gsub("~~", " ", breaker(DATA2$state))
rev.neg <- sapply(negation.words, paste, negative.words)
rev.pos <- sapply(negation.words, paste, positive.words)

## Generate themes/terms to color by
tw <- list(
    positive=c(positive.words, rev.neg[rev.neg %in% txt]),
    negative=c(negative.words, rev.pos[rev.pos %in% txt])
)

with(DATA2, trans_cloud(state, person,
    target.words=tw,
    cloud.colors=qcv(darkgreen, red, gray65),
    expand.target=FALSE, proportional=TRUE, legend=names(tw)))
```


<p id="gradient">`r FT(orange, 5, text="&diams;")` **`r FUN("gradient_cloud")` Examples**`r FT(orange, 5, text="&diams;")`</p >


```{r}
## Fuse two words
DATA2 <- DATA
DATA2$state <- space_fill(DATA$state, c("is fun", "too fun", "you liar"))
```

```{r}
gradient_cloud(DATA2$state, DATA2$sex, title="Lying Fun", max.word.size = 5,
    min.word.size = .025)
gradient_cloud(DATA2$state, DATA2$sex, title="Houghton Colors", 
    max.word.size = 8, min.word.size = .01, X ="purple" , Y = "yellow")
```

```{r include = FALSE, echo=FALSE}
DATA2 <- qdap::DATA2
```

<h4 id="gantts">Gantt Plot</h4>

Many of the plot methods utilized by other functions' classes are a wrapper for `r FUN("gantt_plot")` or `r FUN("gantt_wrap")`.  `r FUN("gantt_plot")` wraps the `r FUN("gantt")`, `r FUN("gantt_rep")` and `r FUN("gantt_wrap")` functions to allow for direct input of text dialogue and grouping variables.  The `r FUN("gantt_plot")` function is a fast way to make Gantt charts that can be faceted and filled by grouping variables.  A Gantt plot allows the user to find trends and patterns in dialogue across time.  It essentially allows for a visual representation of an entire exchange of dialogue.  The following examples show the flexibility of `r FUN("gantt_plot")`; many of these techniques can also be utilized in plot methods for qdap classes that utilize `r FUN("gantt_plot")` and `r FUN("gantt_wrap")`.  It is also prudent to be aware of `r FUN("gantt_wrap")`, that is its arguments and how to utilize it, as it is less convenient yet more flexible and powerful than `r FUN("gantt_plot")`.

<p id="ganttplot1">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")`** - *Single Time/Single Grouping Variable*`r FT(orange, 5, text="&diams;")`</p >

```{r}
with(rajSPLIT, gantt_plot(text.var = dialogue,
    grouping.var = person, size=4))
```

<p id="ganttplot2">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")`** - *Single Time/Multiple Grouping Variable*`r FT(orange, 5, text="&diams;")`</p >

```{r}
with(rajSPLIT, gantt_plot(text.var = dialogue,
    grouping.var = list(fam.aff, sex), rm.var  = act,
    title = "Romeo and Juliet's dialogue"))
```

Sometimes the location of the facets may not be ideal to show the data (i.e., you may want to reverse the x and y axis).  By setting <b><font color="green" face="courier new">transform = TRUE</font></b> the user can make this switch.

<p id="transform">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")``** - *Transforming*`r FT(orange, 5, text="&diams;")`</p >

```{r}
with(rajSPLIT, gantt_plot(dialogue, list(fam.aff, sex), act,
    transform=T))
```

Often the default colors are less useful in displaying the trends in a way that is most meaningful. Because `r FUN("gantt_plot")` is a wrapper for ggplot2 the color palettes can easily be extended to use with the output from `r FUN("gantt_plot")`.

<p id="changecols">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")`** - *Color Palette Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14.5}
## Load needed packages
library(ggplot2); library(scales); library(RColorBrewer); library(grid)

## Duplicate a new data set and make alterations
rajSPLIT2 <- rajSPLIT

rajSPLIT2$newb <- as.factor(sample(LETTERS[1:2], nrow(rajSPLIT2),
    replace=TRUE))

z <- with(rajSPLIT2, gantt_plot(dialogue, list(fam.aff, sex),
    list(act, newb), size = 4))
z + theme(panel.margin = unit(1, "lines")) + scale_colour_grey()
z + scale_colour_brewer(palette="Dark2")
z + scale_colour_manual(values=rep("black", 7))
## vector of colors
cols <- c("black", "red", "blue", "yellow", "orange", "purple", "grey40")

z + scale_colour_manual(values=cols)
```

At times it may be useful to fill the bar colors by another grouping variable.  The <b><font color="green" face="courier new">fill.var</font></b> argument allows another coloring variable to be utilized.

<p id="examp1">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")`** - *Fill Variable Example 1*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14.5}
## Generate an end mark variable set to fill by
dat <- rajSPLIT[rajSPLIT$act == 1, ]
dat$end_mark <- factor(end_mark(dat$dialogue))

with(dat, gantt_plot(text.var = dialogue, grouping.var = list(person, sex),
    fill.var=end_mark))
```


<p id="examp2">`r FT(orange, 5, text="&diams;")` **`r FUN("gantt_plot")`** - *Fill Variable Example 2*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14.5}
## Generate an end mark variable data set to fill by
rajSPLIT2 <- rajSPLIT
rajSPLIT2$end_mark <- end_mark(rajSPLIT2$dialogue)

with(rajSPLIT2, gantt_plot(text.var = dialogue,
    grouping.var = list(fam.aff), rm.var  = list(act),
    fill.var=end_mark, title = "Romeo and Juliet's dialogue"))
```

Be wary though of using coloring to show what faceting would show better.  Here is an example of faceting versus the color fill used in the `r HR("#examp1", "Fill Variable Example 1")` above.

<p id="gradient">`r FT(orange, 5, text="&diams;")` **`r FUN("gradient_plot")`** - *Facet Instead of Fill Variable*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14.5}
## Repeated Measures Sentence Type Example
with(rajSPLIT2, gantt_plot(text.var = dialogue,
    grouping.var = list(fam.aff, sex), rm.var  = list(end_mark, act),
    title = "Romeo and Juliet's dialogue"))
```


<h4 id="heatmaps">Quick Heatmap</h4>

Heatmaps are a powerful way to visualize patterns in matrices.  The gradient allows the user to quickly pick out high and low values.  `r FUN("qheat")` (quick heat map) is a heat map function that accepts matrices and dataframes and has some nice pre-sets that work well with the way qdap data is structured.  Two of these assumptions to be aware of is that dataframe is numeric with the exception of a single grouping variable column with the possibility of additional non-numeric columns passed to  <b><font color="green" face="courier new">facet.vars</font></b>.  `r FUN("qheat")` also assumes that matrices are all numeric with row names serving as the grouping variable.  If passing a dataframe, `r FUN("qheat")` the grouping variable column is assumed to be the first column.

The following examples demonstrate various uses of `r FUN("qheat")`.

<p id="heat1">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Basic Example*`r FT(orange, 5, text="&diams;")`</p >

```{r}
## word stats data set
ws.ob <- with(DATA.SPLIT, word_stats(state, list(sex, adult), tot=tot))

# same as `plot(ws.ob)`
qheat(ws.ob) 
```

<p id="heat2">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Color Group Labels Example*`r FT(orange, 5, text="&diams;")`</p >


```{r}
qheat(ws.ob, xaxis.col = c("red", "black", "green", "blue"))
```

<p id="heat3">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Order By Numeric Variable Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r}
## Order by sptot
qheat(ws.ob, order.by = "sptot")
## Reverse order by sptot
qheat(ws.ob, order.by = "-sptot")
```

<p id="heat4">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Cell Labels Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r}
qheat(ws.ob, values = TRUE)
qheat(ws.ob, values = TRUE, text.color = "red")
```

<p id="heat5">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Custom Cell Labels Example*`r FT(orange, 5, text="&diams;")`</p >


```{r}
## Create a data set and matching labels
dat1 <- data.frame(G=LETTERS[1:5], matrix(rnorm(20), ncol = 4))
dat2 <- data.frame(matrix(LETTERS[1:25], ncol=5))
qheat(dat1, high = "orange", values=TRUE, text.color = "black")
qheat(dat1, high = "orange", values=TRUE, text.color = "black", mat2=dat2)
```

<p id="heat6">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Grid Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r}
qheat(ws.ob, "yellow", "red", grid = FALSE)
qheat(ws.ob, high = "red", grid = "black")
```

<p id="heat7">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Facet Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width=14}
qheat(mtcars, facet.vars = "cyl")
qheat(mtcars, facet.vars = c("gear", "cyl"))
```

<p id="heat8">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Transposing Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r}
qheat(t(mtcars), by.column=FALSE)
qheat(mtcars, plot = FALSE) + coord_flip()
```

When plotting a correlation/distance matrix set <b>`r FT(green, courier_new, text="diag.na = TRUE")`</b> to keep these extreme values from effecting the scaling.

<p id="heat9">`r FT(orange, 5, text="&diams;")` **`r FUN("qheat")`** - *Correlation Matrix Examples*`r FT(orange, 5, text="&diams;")`</p >

```{r}
qheat(cor(mtcars), diag.na=TRUE, by.column = NULL)
```

<h4 id="rankfreq">Rank Frequency Plot</h4>

Rank Frequency Plots are a way of visualizing word rank versus frequencies as related to Zipf's law which states that the rank of a word is inversely related to its frequency.  The `r FUN("rank_freq_mplot")` and `r FUN("rank_freq_plot", "rank_freq_mplot")` provide the means to plot the ranks and frequencies of words (with `r FUN("rank_freq_mplot")` plotting by grouping variable(s)).  

`r FUN("rank_freq_mplot")` utilizes the ggplot2 package, whereas, `r FUN("rank_freq_plot", "rank_freq_mplot")` employs base graphics. `r FUN("rank_freq_mplot")` is more general, flexible, and takes text/grouping variables directly; in most cases `r FUN("rank_freq_mplot")` should be preferred (though `r FUN("rank_freq_plot", "rank_freq_mplot")` will render quicker).  The `r FUN("rank_freq_mplot")` family of functions also outputs a list of rank/frequency dataframes used plot the visuals and other related descriptive statistics.

<p id="rank1">`r FT(orange, 5, text="&diams;")` **`r FUN("rank_freq_mplot")`**`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14, fig.height = 10}
## Plot log-log version
x2 <- rank_freq_mplot(mraja1spl$dialogue, mraja1spl$person, ncol = 5,
    hap.col = "purple")
## View output
ltruncdf(x2, 10)
## Plot standard rank-freq version
invisible(rank_freq_mplot(mraja1spl$dialogue, mraja1spl$person, ncol = 5,
    log.freq = FALSE, log.rank = FALSE, jitter = .6,  hap.col = "purple"))
```

<p id="rank2">`r FT(orange, 5, text="&diams;")` **`r FUN("rank_freq_mplot")`** - *Using alpha*`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9}
invisible(rank_freq_mplot(raj$dialogue, jitter = .5, shape = 19, alpha = 1/15))
```

The `r FUN("rank_freq_plot", "rank_freq_mplot")` plots more quickly but does not handle multiple groups and does not take text/grouping variables directly.

<p id="rank3">`r FT(orange, 5, text="&diams;")` **`r FUN("rank_freq_plot", "rank_freq_mplot")` Example** `r FT(orange, 5, text="&diams;")`</p >

```{r}
## Generate data for `rank_freq_plot` via `word_list` function
mod <- with(mraja1spl , word_list(dialogue, person, cut.n = 10,
    cap.list=unique(mraja1spl$person)))

## Plot it
x3 <- rank_freq_plot(mod$fwl$Romeo$WORD, mod$fwl$Romeo$FREQ, 
    title.ext = 'Romeo')
## View output
ltruncdf(x3, 10)
```


<h4 id="totplot">Visualize Word Length by Turn of Talk</h4>

It is often useful to view the lengths of turns of talk as a bar plot, particularly if the bars are colored by grouping variable.  The `r FUN("tot_plot")` function plots dialogue as a bar plot with the option to color by grouping variables and facet by repeated measure variables.  This can enable the entire dialogue to be viewed in a succinct way, possibly leading the researcher to see patterns that may have otherwise escaped attention.  

Within the `r FUN("tot_plot")` function the turn of talk argument (<b><font color="green" face="courier new">tot</font></b>) may be the <b>"tot"</b> column from `r FUN("sentSplit")` output (<b><font color="green" face="courier new">tot = TRUE</font></b>), the row numbers (<b><font color="green" face="courier new">tot = FALSE</font></b>), the character name of a column (<b><font color="green" face="courier new">tot = "COLUMN NAME"</font></b>), or a separate numeric/character vector equal in length to the <b><font color="green" face="courier new">text.var</font></b>.  

<p id="tot1">`r FT(orange, 5, text="&diams;")` **`r FUN("tot_plot")`** - *Examples* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.height = 3}
dataframe <- sentSplit(DATA, "state")
tot_plot(dataframe, text.var = "state")
## Change space between bars
tot_plot(dataframe, text.var = "state", bar.space = .03)
## Color bars by grouping variable(s)
tot_plot(dataframe, text.var = "state", grouping.var = "sex")
```

```{r, fig.width = 14, fig.height = 4}
## Use rownames as tot: color by family affiliation
tot_plot(mraja1, "dialogue", grouping.var = "fam.aff", tot = FALSE)
## Use rownames as tot: color by death
tot_plot(mraja1, "dialogue", grouping.var = "died", tot = FALSE)
```


<p id="tot2">`r FT(orange, 5, text="&diams;")` **`r FUN("tot_plot")`** - *Facet Variables* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9}
rajSPLIT2 <- do.call(rbind, lapply(split(rajSPLIT, rajSPLIT$act), head, 25))
tot_plot(rajSPLIT2, "dialogue", grouping.var = "fam.aff", facet.var = "act")
```

Because `r FUN("tot_plot")` is based on the `r HR("http://docs.ggplot2.org/current/", "ggplot2 package")` (Wickham, 2009) and `r FUN("tot_plot")` invisibly returns the ggplot2 object, the output (of the class "ggplot") can be altered in the same way that another ggplot2 object can be.  In the following examples the color palette is altered.

<p id="tot3">`r FT(orange, 5, text="&diams;")` **`r FUN("tot_plot")`** - *Alter Colors* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14, fig.height = 4}
base <- tot_plot(mraja1, "dialogue", grouping.var = c("sex", "fam.aff"), 
    tot=FALSE, plot=FALSE) 
base + scale_fill_hue(l=40)
base + scale_fill_brewer(palette="Spectral")
base + scale_fill_brewer(palette="Set1")
```

<p id="tot3">`r FT(orange, 5, text="&diams;")` **`r FUN("tot_plot")`** - *Add Mean +2/+3 sd* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 14, fig.height = 4}
base +
    scale_fill_brewer(palette="Set1") +
    geom_hline(aes(yintercept=mean(word.count))) +
    geom_hline(aes(yintercept=mean(word.count) + (2 *sd(word.count)))) +
    geom_hline(aes(yintercept=mean(word.count) + (3 *sd(word.count)))) +
    geom_text(parse=TRUE, hjust=0, vjust=0, size = 3, aes(x = 2, 
        y = mean(word.count) + 2, label = "bar(x)")) +
    geom_text(hjust=0, vjust=0, size = 3, aes(x = 1, 
        y = mean(word.count) + (2 *sd(word.count)) + 2, label = "+2 sd")) +
    geom_text(hjust=0, vjust=0,  size = 3, aes(x = 1, 
        y = mean(word.count) + (3 *sd(word.count)) + 2, label = "+3 sd")) 
```

<h4 id="venn">Venn Diagram</h4>

The Venn diagram can be a useful way to visualize similarity between grouping variables with respect to word use when the number of groups is relatively small.  The `r FUN("trans_venn")` function wraps the `r HR("http://cran.r-project.org/web/packages/venneuler/index.html", "venneuler")` package to produce Venn diagrams.  The user must keep in mind that producing this output is computationally slow, thus consideration must be given with regard to data size and number of groups when using `r FUN("trans_venn")` to avoid over plotting and lengthy plot production.  The use of the <b><font color="green" face="courier new">stopwords</font></b> argument can also be useful to reduce the overlap of common words between grouping variables.

If a data set is larger the user may want to consider representing the data as a `r FUN("Dissimilarity")` matrix or as an adjacency matrix that can be plotted with the `r HR("http://cran.r-project.org/web/packages/igraph/index.html", "igraph package")` as seen in the `r HR("#plotadj2", "presidential examples above")`.

In the following example the reader will notice the centers of the circles (i.e., the person labels) are very similar to the positioning (the distances between nodes) of the same data in the `r HR("#plotadj", "adjacency matrix plot")` of the same data above.


```{r, fig.width = 9, fig.height = 9, echo=2}
set.seed(10)
with(DATA , trans_venn(state, person, legend.location = "topright"))
```


<h4 id="wordnet">Word Network Plot</h4>

Viewing connections between words within grouping variables (particularly turns of talk) are a useful means of examining what words are connected together.  For example, this may be useful to a researcher who is looking at particular vocabulary usage by a teacher.  The researcher may wish to know what other terms are supporting/supported by/connected to the terms of interest.  `r FUN("word_network_plot")` a wrapper for the `r HR2("http://igraph.sourceforge.net/", "igraph")` package.  This approach may be used on concert with `r HR("#corr", "correlations between words")`.  Not that terms could also be combined via the `r FUN("wfm_combine", "Word_Frequency_Matrix")` before running a correlation in order to represent the clustering of words that `r FUN("word_network_plot")` handles. Further analysis of the word correlations can be tested via bootstrapping of the attained correlation against a random correlation. It is worth noting that the `r FUN("word_associate")` function is a wrapper for `r FUN("word_network_plot")` (for mor see `r HR("#assoc", "this example")` above).

<p id="wordnet1">`r FT(orange, 5, text="&diams;")` **`r FUN("word_network_plot")`** - *Between Turns of Talk: All Words* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9, fig.height = 9}
word_network_plot(text.var=DATA$state, stopwords=NULL, label.cex = .95)
```

<p id="wordnet2">`r FT(orange, 5, text="&diams;")` **`r FUN("word_network_plot")`** - *Between People* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9, fig.height = 9}
word_network_plot(text.var=DATA$state, DATA$person)
word_network_plot(text.var=DATA$state, DATA$person, stopwords=NULL)
```

<p id="wordnet3">`r FT(orange, 5, text="&diams;")` **`r FUN("word_network_plot")`** - *Between sex and adult* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9, fig.height = 9}
word_network_plot(text.var=DATA$state, grouping.var=list(DATA$sex,
    DATA$adult))
```

<p id="wordnet4">`r FT(orange, 5, text="&diams;")` **`r FUN("word_network_plot")`** - *`log.labels`* `r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9, fig.height = 9}
word_network_plot(text.var=DATA$state, grouping.var=DATA$person,
    title.name = "TITLE", log.labels=TRUE, label.size = .9)
```


<h3 id="id">ID Sentences</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):  

<form action="http://trinker.github.io/qdap_dev/end_inc.html" target="_blank">
    <input type="submit" value="end_inc"> - `r HR("#incomp", "Test for Incomplete Sentences")`
</form>

<form action="http://trinker.github.io/qdap_dev/end_mark.html" target="_blank">
    <input type="submit" value="end_mark"> - `r HR("#endmark", "Sentence End Marks")`
</form>

<form action="http://trinker.github.io/qdap_dev/id.html" target="_blank">
    <input type="submit" value="id"> - `r HR("#id", "Generate unique ID")`
</form>

<form action="http://trinker.github.io/qdap_dev/imperative.html" target="_blank">
    <input type="submit" value="imperative"> - `r HR("#imper", "Detect and Remark Imperative Sentences")`
</form>

</div>

qdap provies a few general functions for categorizing sentence types.  This section will outline these functions and some of their uses.

<h4 id="incomp">Test for Incomplete Sentences</h4>

It is often helpful to determine if a sentence (a row) is incomplete, as this may effect some forms of analysis. The `r FUN("end_inc")` provides this functionality after `r FUN("incomplete_replace")` has replaced various incomplete sentence notation with the standard qdap notation (<b><font color="green" face="courier new">|</font></b>).

<p id="incomp1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("end_inc")` Examples</b> `r FT(orange, 5, text="&diams;")` </p >


```{r}
## Alter the DATA.SPLIT data set to have incomplete sentences.
dat <- DATA.SPLIT[, c("person", "state")]
dat[c(1:2, 7), "state"] <- c("the...",  "I.?", "threw..")
dat[, "state"] <- incomplete_replace(dat$state)
dat
## Remove incomplete sentences and warn.
end_inc(dat, "state")
## Remove incomplete sentences and no warning.
end_inc(dat, "state", warning.report = FALSE)
## List of logical checks for which are not/are incomplete
end_inc(dat, "state", which.mode = TRUE)
```

<h4 id="endmark">Sentence End Marks</h4>

It is often useful to determine what sentence type (end mark) a sentence is.  The `r FUN("end_mark")` extracts the end marks from a sentence.  The output can also be used logically grab sentence types. 


<p id="endmark1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("end_mark")` Example</b> `r FT(orange, 5, text="&diams;")`</p >


```{r}
end_mark(DATA.SPLIT$state)
```


<p id="endmark2">`r FT(orange, 5, text="&diams;")` <b>`r FUN("end_mark")`</b> - <em>Grab Sentence Types</em>`r FT(orange, 5, text="&diams;")`</p >

```{r}
## Grab questions
ques <- mraja1spl[end_mark(mraja1spl$dialogue) == "?", ] 
htruncdf(ques)
## Grab non questions
non.ques <- mraja1spl[end_mark(mraja1spl$dialogue) != "?", ] 
htruncdf(non.ques, 12)
## Grab ? and . ending sentences
ques.per <- mraja1spl[end_mark(mraja1spl$dialogue) %in% c(".", "?"), ] 
htruncdf(ques.per, 12)
```

<h4 id="id">Generate unique ID</h4>

The `r FUN("ID")` is a shortcut approach to providing row or element IDs on the fly.

<p id="IDfun">`r FT(orange, 5, text="&diams;")` <b>`r FUN("ID")`</b> - <em>Grab Sentence Types</em>`r FT(orange, 5, text="&diams;")`</p >


```{r}
id(list(1, 4, 6))
id(matrix(1:10, ncol=1))
id(mtcars)
id(mtcars, FALSE)
question_type(DATA.SPLIT$state, id(DATA.SPLIT, TRUE))
```

<h4 id="imper">Detect and Remark Imperative Sentences</h4>

qdap allows for the detection of imperative sentences via the `r FUN("imperative")` function.  The function detects and optionally remarks as imperative, an asterisk (<b><font color="green" face="courier new">*</font></b>) is used, however, `r FUN("imperative")` is sensitive to choppy, comma riddled sentences and dialects such as African American Vernacular English.  The algorithm is complex and thus slower.


<p id="imperative1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("imperative")`</b> - <em>Imperative Data</em>`r FT(orange, 5, text="&diams;")`</p >

```{r}
(dat <- data.frame(name=c("sue", rep(c("greg", "tyler", "phil",
    "sue"), 2)), statement=c("go get it|", "I hate to read.",
    "Stop running!", "I like it!", "You are terrible!", "Don't!",
    "Greg, go to the red, brick office.", "Tyler go to the gym.",
    "Alex don't run."), stringsAsFactors = FALSE))
```


<pre><code>   name                          statement
1   sue                         go get it|
2  greg                    I hate to read.
3 tyler                      Stop running!
4  phil                         I like it!
5   sue                  You are terrible!
6  greg                             Don't!
7 tyler Greg, go to the red, brick office.
8  phil               Tyler go to the gym.
9   sue                    Alex don't run.
</code></pre>

<p id="imperative2">`r FT(orange, 5, text="&diams;")` <b>`r FUN("imperative")`</b> - <em>Re-mark End Marks</em>`r FT(orange, 5, text="&diams;")`</p >


<pre><code class="r">imperative(dat, "name", "statement", additional.names = c("Alex"))
</code></pre>

<pre><code>   name                           statement
1   sue                         go get it*|
2  greg                     I hate to read.
3 tyler                      Stop running*!
4  phil                          I like it!
5   sue                   You are terrible!
6  greg                             Don't*!
7 tyler Greg, go to the red, brick office*.
8  phil               Tyler go to the gym*.
9   sue                    Alex don't run*.
</code></pre>


<p id="imperative1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("imperative")`</b> - <em>Handle Incomplete Sentences</em>`r FT(orange, 5, text="&diams;")`</p >


<pre><code class="r">imperative(dat, "name", "statement", lock.incomplete = TRUE, "Alex")
</code></pre>

<pre><code>   name                           statement
1   sue                          go get it|
2  greg                     I hate to read.
3 tyler                      Stop running*!
4  phil                          I like it!
5   sue                   You are terrible!
6  greg                             Don't*!
7 tyler Greg, go to the red, brick office*.
8  phil               Tyler go to the gym*.
9   sue                    Alex don't run*.
</code></pre>

<p id="imperative1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("imperative")`</b> - <em>Warning Report</em>`r FT(orange, 5, text="&diams;")`</p >

<pre><code class="r">imperative(dat, "name", "statement", additional.names = "Alex", warning=TRUE)
</code></pre>

<pre><code>   name                           statement warnings
1   sue                         go get it*|        -
2  greg                     I hate to read.     read
3 tyler                      Stop running*!        -
4  phil                          I like it!        -
5   sue                   You are terrible!        -
6  greg                             Don't*!        -
7 tyler Greg, go to the red, brick office*. 2 commas
8  phil               Tyler go to the gym*.        -
9   sue                    Alex don't run*.     AAVE
</code></pre>


<h3 id="tm">tm Package Compatability</h3>

<div class="funs">
The following functions will be utilized in this section (click to view more):   


<form action="http://trinker.github.io/qdap_dev/as.tdm.html" target="_blank">
    <input type="submit" value="as.tdm"><input type="submit" value="as.dtm"> - `r HR("#as.tdm", "Create term document matrices or document term matrices from raw text or wfm")`
</form>

<form action="http://trinker.github.io/qdap_dev/Word_Frequency_Matrix.html" target="_blank">
    <input type="submit" value="as.wfm"> - `r HR("#as.wfm", "Convert the tm package's TermDocumentMatrix/DocumentTermMatrix to wfm")`
</form>

<form action="http://trinker.github.io/qdap_dev/as.tdm.html" target="_blank">
    <input type="submit" value="apply_as_tm"> - `r HR("#apply_as_tm", "Apply functions intended to be used on the tm package's TermDocumentMatrix to a wfm object")`
</form>

<form action="http://trinker.github.io/qdap_dev/as.tdm.html" target="_blank">
    <input type="submit" value="as.data.frame"><input type="submit" value="as.Corpus"> - `r HR("#as.data.frame", "Convert between tm package Corpus and qdap dataframe")`
</form>

</div>


The `r HR("http://cran.r-project.org/web/packages/tm/index.html", "tm package")` is a heavily regarded and utilized R package for text mining purposes.  The primary data forms for the tm package are <a href="http://www.inside-r.org/packages/cran/tm/docs/Corpus"><b><font color="green", face="courier">Corpus</font></b></a> and <a href="http://www.inside-r.org/packages/cran/tm/docs/http://www.inside-r.org/packages/cran/tm/docs/TermDocumentMatrix"><b><font color="green", face="courier">TermDocumentMatrix</font></b></a>/<a href="http://www.inside-r.org/packages/cran/tm/docs/http://www.inside-r.org/packages/cran/tm/docs/TermDocumentMatrix"><b><font color="green", face="courier">DocumentTermMatrix</font></b></a>.  Because tm is a dependancy for many R text mining packages it is prudent to provide a set of tools to convert between tm and qdap data types.  This section demos a few of the tools designed to achieve qdap-tm compatibility.  For a more thorough vignette describing qdap-tm compatability use `browseVignettes(package = "qdap")` or `r HR2("http://cran.r-project.org/web/packages/qdap/vignettes/tm_package_compatibility.pdf", "Click Here")`.


<h4 id="as.tdm">Create term document matrices or document term matrices from raw text or wfm</h4>

<p id="as.tdm1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("as.tdm")`</b> & <b>`r FUN("as.dtm", "as.tdm")`</b>  - <em>From Raw Text Example 1</em>`r FT(orange, 5, text="&diams;")`</p >

```{r}
as.tdm(DATA$state, DATA$person)
as.dtm(DATA$state, DATA$person)
```

<p id="as.tdm1">`r FT(orange, 5, text="&diams;")` <b>`r FUN("as.tdm")`</b> & <b>`r FUN("as.dtm", "as.tdm")`</b>  - <em>From Raw Text Example 2</em>`r FT(orange, 5, text="&diams;")`</p >

```{r message = FALSE, fig.width = 9, fig.height = 9}
(pres <- as.tdm(pres_debates2012$dialogue, pres_debates2012$person))
```

<pre><code class="r">library(tm)
plot(pres, corThreshold = 0.8)</code></pre>

```{r, echo=FALSE, results='asis'}
uri_embed("imgs/tm_imgs/img1.png")
```

```{r message = FALSE, fig.width = 9, fig.height = 9}
(pres2 <- removeSparseTerms(pres, .3))
```

<pre><code class="r">plot(pres2, corThreshold = 0.95)</code></pre>

```{r, echo=FALSE, results='asis'}
uri_embed("imgs/tm_imgs/img2.png")
```

<p id="as.tdm2">`r FT(orange, 5, text="&diams;")` <b>`r FUN("as.tdm")`</b> & <b>`r FUN("as.dtm", "as.tdm")`</b>  - <em>From `r FUN("wfm")`</em>`r FT(orange, 5, text="&diams;")`</p >

```{r, fig.width = 9, fig.height = 9}
x <- wfm(DATA$state, DATA$person)
as.tdm(x)
as.dtm(x)
```

<pre><code class="r">plot(as.tdm(x))</code></pre>

```{r, echo=FALSE, results='asis'}
uri_embed("imgs/tm_imgs/img3.png")
```


<h4 id="tm2wfm">Convert the tm package's TermDocumentMatrix/DocumentTermMatrix to wfm</h4>

```{r message = FALSE}
library(tm); data(crude)
(dtm_in <- DocumentTermMatrix(crude, control = list(stopwords = TRUE)))
summary(as.wfm(dtm_in))
```


<h4 id="apply_as_tm">Apply functions intended to be used on the tm package's TermDocumentMatrix to a wfm object</h4>

`r FUN("apply_as_tm", "as.tdm")` allows the user to apply functions intended to be used on the `tm` package's `r HR("http://www.inside-r.org/packages/cran/tm/docs/as.TermDocumentMatrix", "TermDocumentMatrix")` to a `r FUN("wfm")` object.  `r FUN("apply_as_tm", "as.tdm")` attempts to simplify back to a `r FUN("wfm")` or `r FUN("wfm_weight", "wfm")` format.  In the examples belows we first create a `r FUN("wfm")` and then apply functions designed for a `r HR("http://www.inside-r.org/packages/cran/tm/docs/as.TermDocumentMatrix", "TermDocumentMatrix")`.


```{r message = FALSE}
library(tm); library(proxy)
## Create a wfm
a <- with(DATA, wfm(state, list(sex, adult)))
summary(a)

## Apply as tm
(out <- apply_as_tm(a, tm:::removeSparseTerms, sparse=0.6))
summary(out)
apply_as_tm(a, tm:::findAssocs, "computer", .8)
apply_as_tm(a, tm:::findFreqTerms, 2, 3)
apply_as_tm(a, tm:::Zipf_plot)
apply_as_tm(a, tm:::Heaps_plot)
```

<pre><code class="r">apply_as_tm(a, tm:::plot.TermDocumentMatrix, corThreshold = 0.4)</code></pre>

```{r, echo=FALSE, results='asis'}
uri_embed("imgs/tm_imgs/img4.png")
```


```{r message = FALSE}
apply_as_tm(a, tm:::weightBin)
apply_as_tm(a, tm:::weightBin, to.qdap = FALSE)
apply_as_tm(a, tm:::weightSMART)
apply_as_tm(a, tm:::weightTfIdf)
```


<h4 id="as.data.frame">Convert between tm package Corpus and qdap dataframe.</h4>

```{r}
## Convert dataframe to a Corpus
(x <- with(DATA2, as.Corpus(state, list(person, class, day))))
library(tm)
inspect(x)
class(x)
## Convert Back
htruncdf(as.data.frame(x), 15, 30)
```

<h3 id="data">Data Sets</h3>

<div class="textbox", style="background-color: #D6EFD6;"> 
The following data sets are included with qdap for demonstration purposes (click to view more)
<form action="http://trinker.github.io/qdap_dev/DATA.html" target="_blank">
    <input type="submit" value="DATA"> - Fictitious Classroom Dialogue
</form>
          
<form action="http://trinker.github.io/qdap_dev/DATA.SPLIT.html" target="_blank">
    <input type="submit" value="DATA.SPLIT"> - Fictitious Split Sentence Classroom Dialogue
</form>

<form action="http://trinker.github.io/qdap_dev/DATA2.html" target="_blank">
    <input type="submit" value="DATA2"> - Fictitious Repeated Measures Classroom Dialogue
</form>

<form action="http://trinker.github.io/qdap_dev/pres_debates2012.html" target="_blank">
    <input type="submit" value="pres_debates2012"> - 2012 U.S. Presidential Debates
</form>

<form action="http://trinker.github.io/qdap_dev/pres_debate_raw2012.html" target="_blank">
    <input type="submit" value="pres_debate_raw2012"> - First 2012 U.S. Presidential Debate
</form>

<form action="http://trinker.github.io/qdap_dev/mraja1.html" target="_blank">
    <input type="submit" value="mraja1"> - Romeo and Juliet: Act 1 Dialogue Merged with Demographics
</form>

<form action="http://trinker.github.io/qdap_dev/mraja1spl.html" target="_blank">
    <input type="submit" value="mraja1spl"> - Romeo and Juliet: Act 1 Dialogue Merged with Demographics and Split
</form>

<form action="http://trinker.github.io/qdap_dev/raj.act.1.html" target="_blank">
    <input type="submit" value="raj.act.1"> - Romeo and Juliet: Act 1
</form>

<form action="http://trinker.github.io/qdap_dev/raj.act.2.html" target="_blank">
    <input type="submit" value="raj.act.2"> - Romeo and Juliet: Act 2
</form>

<form action="http://trinker.github.io/qdap_dev/raj.act.3.html" target="_blank">
    <input type="submit" value="raj.act.3"> - Romeo and Juliet: Act 3
</form>

<form action="http://trinker.github.io/qdap_dev/raj.act.4.html" target="_blank">
    <input type="submit" value="raj.act.4"> - Romeo and Juliet: Act 4
</form>

<form action="http://trinker.github.io/qdap_dev/raj.act.5.html" target="_blank">
    <input type="submit" value="raj.act.5"> - Romeo and Juliet: Act 5
</form>

<form action="http://trinker.github.io/qdap_dev/raj.demographics.html" target="_blank">
    <input type="submit" value="raj.demographics"> - Romeo and Juliet Demographics
</form>

<form action="http://trinker.github.io/qdap_dev/raj.html" target="_blank">
    <input type="submit" value="raj"> - Romeo and Juliet (Unchanged & Complete)
</form>

<form action="http://trinker.github.io/qdap_dev/rajPOS.html" target="_blank">
    <input type="submit" value="rajPOS"> - Romeo and Juliet Split in Parts of Speech
</form>

<form action="http://trinker.github.io/qdap_dev/rajSPLIT.html" target="_blank">
    <input type="submit" value="rajSPLIT"> - Romeo and Juliet (Complete & Split)
</form>

<form action="http://trinker.github.io/qdap_dev/hamlet.html" target="_blank">
    <input type="submit" value="hamlet"> - Hamlet (Complete & Split)
</form>
</div>

```{r eval=FALSE, echo = FALSE}
path <- "C:/Users/trinker/GitHub/trinker.github.com/qdapDictionaries"
#  path <- "C:/Users/trinker/GitHub/trinker.github.com/qdap"
URL <- "http://trinker.github.io/qdapDictionaries/"


inds <- readLines(file.path(path, "index.html"))
h3s <- grep("<h3", inds)
h2s <- grep("<h2", inds)

inds <- inds[head(h3s, 1):(tail(h2s, 1) - 1)]
inds <- inds[7: tail(grep("</ul>", inds), 1)]
#h3s <- grep("<h3", inds)
#dat2 <- data.frame(start = h3s + 4, end = c(tail(h3s, -1) - 1, length(inds)))

inds <- inds[grep("<code>", inds)]
inds <- substring(inds, 9)

library(qdap)

dat <- data.frame(x = unlist(genXtract(inds, ".html\">", "</a>")),
    y = unlist(genXtract(inds, "<br />", "</li>")), row.names = NULL)

m <- lapply(1:nrow(dat), function(i) dat[i, ])

rws <-  lapply(m, function(x) {
  paste0("<form action=\"", file.path(URL, paste0(x[[1]], ".html\"")), 
    " target=\"_blank\" \">\n", "    <input type=\"submit\" value=\"", x[[1]], "\"> - ", x[[2]], "\n</form>", "\n")
})

cat(paste(unlist(rws), collapse="\n"))
```

<h3 id="dict">Dictionaries and Word Lists</h3>

qdap utilizes the following dictionaries/wordlists from the `r HR("http://trinker.github.io/qdapDictionaries", "qdapDictionaries")` package.

<div class="textbox", style="background-color: #D6EFD6;"> 
The following dictionaries/word lists are utilized by qdap (click to view more)

<form action="http://trinker.github.io/qdapDictionaries/abbreviations.html" target="_blank" ">
    <input type="submit" value="abbreviations"> - Small Abbreviations Data Set
</form>

<form action="http://trinker.github.io/qdapDictionaries/action.verbs.html" target="_blank" ">
    <input type="submit" value="action.verbs"> - Action Word List
</form>

<form action="http://trinker.github.io/qdapDictionaries/adverb.html" target="_blank" ">
    <input type="submit" value="adverb"> - Adverb Word List
</form>

<form action="http://trinker.github.io/qdapDictionaries/BuckleySaltonSWL.html" target="_blank" ">
    <input type="submit" value="BuckleySaltonSWL"> - Buckley & Salton Stopword List
</form>

<form action="http://trinker.github.io/qdapDictionaries/contractions.html" target="_blank" ">
    <input type="submit" value="contractions"> - Contraction Conversions
</form>

<form action="http://trinker.github.io/qdapDictionaries/DICTIONARY.html" target="_blank" ">
    <input type="submit" value="DICTIONARY"> - Nettalk Corpus Syllable Data Set
</form>

<form action="http://trinker.github.io/qdapDictionaries/Dolch.html" target="_blank" ">
    <input type="submit" value="Dolch"> - Dolch's list of 220 Most Commonly Used Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/emoticon.html" target="_blank" ">
    <input type="submit" value="emoticon"> - Emoticons Data Set
</form>

<form action="http://trinker.github.io/qdapDictionaries/key.syl.html" target="_blank" ">
    <input type="submit" value="key.syl"> - Syllable Lookup Key
</form>

<form action="http://trinker.github.io/qdapDictionaries/key.syn.html" target="_blank" ">
    <input type="submit" value="key.syn"> - Synonyms Lookup Key
</form>

<form action="http://trinker.github.io/qdapDictionaries/Fry_1000" target="_blank" ">
    <input type="submit" value="Fry_1000"> - Fry's 1000 Most Frequent Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/increase.amplification.words.html" target="_blank" ">
    <input type="submit" value="increase.amplification.words"> - Amplifying Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/interjections.html" target="_blank" ">
    <input type="submit" value="interjections"> - Interjections
</form>

<form action="http://trinker.github.io/qdapDictionaries/labMT.html" target="_blank" ">
    <input type="submit" value="labMT"> - Language Assessment by Mechanical Turk (labMT) Sentiment Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/Leveled_Dolch.html" target="_blank" ">
    <input type="submit" value="Leveled_Dolch"> - Leveled Dolch List
</form>

<form action="http://trinker.github.io/qdapDictionaries/NAMES.html" target="_blank" ">
    <input type="submit" value="NAMES"> - First Names and Gender (U.S.)
</form>

<form action="http://trinker.github.io/qdapDictionaries/NAMES_SEX.html" target="_blank" ">
    <input type="submit" value="NAMES_SEX"> - First Names and Predictive Gender (U.S.)
</form>

<form action="http://trinker.github.io/qdapDictionaries/NAMES_LIST.html" target="_blank" ">
    <input type="submit" value="NAMES_LIST"> - First Names and Predictive Gender (U.S.) List
</form>

<form action="http://trinker.github.io/qdapDictionaries/negation.words.html" target="_blank" ">
    <input type="submit" value="negation.words"> - Negating Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/negative.words.html" target="_blank" ">
    <input type="submit" value="negative.words"> - Negative Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/OnixTxtRetToolkitSWL1.html" target="_blank" ">
    <input type="submit" value="OnixTxtRetToolkitSWL1"> - Onix Text Retrieval Toolkit Stopword List 1
</form>

<form action="http://trinker.github.io/qdapDictionaries/positive.words.html" target="_blank" ">
    <input type="submit" value="positive.words"> - Positive Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/preposition.html" target="_blank" ">
    <input type="submit" value="preposition"> - Preposition Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/SYNONYM.html" target="_blank" ">
    <input type="submit" value="SYNONYM"> - Synonyms Data Set
</form>

<form action="http://trinker.github.io/qdapDictionaries/Top100Words.html" target="_blank" ">
    <input type="submit" value="Top100Words"> - Fry's  100 Most Commonly Used English Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/Top200Words.html" target="_blank" ">
    <input type="submit" value="Top200Words"> - Fry's 200 Most Commonly Used English Words
</form>

<form action="http://trinker.github.io/qdapDictionaries/Top25Words.html" target="_blank" ">
    <input type="submit" value="Top25Words"> - Fry's 25 Most Commonly Used English Words
</form>
</div>

<h3 id="install">Installation Issues</h3>

<h4 id="java">Java Issues</h3>
  
<p>If there is a discrepancy between the <a href="https://dl.dropbox.com/u/61803503/rjava_warning.txt">R and Java architectures</a> you will have to <a href="http://www.java.com/en/download/manual.jsp">download</a> the appropriate version of Java compatible with the version of R you're using.    

For more see <a href="http://www.r-statistics.com/2012/08/how-to-load-the-rjava-package-after-the-error-java_home-cannot-be-determined-from-the-registry/" target="_blank">Tal Galili's blog post</a> regarding rJava issues.


<h3 id="connect">Recommended Packages (Extending qdap)</h3>

qdap is designed to be a bridge between qualitative text and the quantitative tools that R and its massive collection of add on packages have to offer.  The following categorized list of packages are tools that I have used to further extend the text analysis of qdap.

<h4 id="plotting">Plotting</h4> 

1. `r HR("http://docs.ggplot2.org/current/", "ggplot2")`         
2. `r HR("http://cran.r-project.org/web/packages/gridExtra/index.html", "gridExtra")`             
3. `r HR("http://igraph.sourceforge.net/", "igraph")`               
4. `r HR("http://cran.r-project.org/web/packages/wordcloud/index.html", "wordcloud")`               


<h4 id="reshaping">Reshaping/Aggregating</h4> 

1. `r HR("http://datatable.r-forge.r-project.org/", "data.table")`             
2. `r HR("http://cran.r-project.org/web/packages/plyr/index.html", "plyr")`              
3. `r HR("http://cran.r-project.org/web/packages/reshape2/index.html", "reshape2")`             

<h4 id="topics">Semantics</h4> 

1. `r HR("http://cran.r-project.org/web/packages/ca/index.html", "ca")`             
2. `r HR("http://cran.r-project.org/web/packages/lda/index.html", "lda")`                              
3. `r HR("http://cran.r-project.org/web/packages/mallet/index.html", "mallet")`              
4. `r HR("http://cran.r-project.org/web/packages/textir/index.html", "textir")`                 
5. `r HR("http://cran.r-project.org/web/packages/topicmodels/index.html", "topicmodels")`               

<h4 id="hlm">Multi Level Modeling/Structural Equation Modeling</h4>  

1. `r HR("http://cran.r-project.org/web/packages/lme4/index.html", "lme4")`               
2. `r HR("http://cran.r-project.org/web/packages/nlme/index.html", "nlme")`          
3. `r HR("http://cran.r-project.org/web/packages/lavaan/index.html", "lavaan")`              
4. `r HR("http://cran.r-project.org/web/packages/sem/index.html", "sem")`                 


<h4 id="textmining">Text Mining</h4>

1. `r HR("http://cran.r-project.org/web/packages/tm/index.html", "tm")`              
2. `r HR("http://cran.r-project.org/web/packages/koRpus/index.html", "koRpus")`             
3. `r HR("http://cran.r-project.org/web/packages/openNLP/index.html", "openNLP")`              
4. `r HR("http://cran.r-project.org/web/packages/zipfR/index.html", "zipfR")`             

For more on natural language processing see the related `r HR("http://cran.r-project.org/web/views/NaturalLanguageProcessing.html", "CRAN NLP task view")`.


<hr>
## Acknowledgements

The qdap package was my first R package and a learning process. Several people contributed immensely to my learning. I'd like to particularly thank `r HR2("https://github.com/Dasonk/", "Dason Kurkiewicz")` for his constant mentoring/assistance in learning the R language, GitHub and package development as well as collaboration on numerous qdap functions. Thank you to `r HR2("https://twitter.com/bryangoodrich", "Bryan Goodrich")` for his teaching, feedback and collaboration on several qdap functions. Thank you to `r HR2("https://github.com/hadley", "Dr. Hadley Wickham")` for roxygen2, ggplot2, devtools and GitHub repos which I referenced often. I'd also like to thank the many folks at `r HR2("http://www.talkstats.com/", "talkstats.com")` and `r HR2("http://stackoverflow.com/questions/tagged/r", "stackoverflow.com")` for their help in answering many R questions related to qdap.

<hr> 

## Improvements

If the reader spots an error in this Vignette or would like to suggest an improvement please contact me @ Tyler Rinker&lt;<a href="mailto:tyler.rinker@gmail.com" target="_blank">tyler.rinker@gmail.com</a>&gt;.  To submit bug reports and feature requests related to the qdap package please visit `r HR2("https://github.com/trinker/qdap/issues?state=open", "qdap's GitHub issues page")`.

<hr> 

*<em><font size="3">Vignette created with the reports package (<a href="http://github.com/trinker/reports">Rinker, 2013b</a>)</font><em>


```{r css, echo = FALSE}
options(markdown.HTML.stylesheet = "css/style.css")
```

## References
<ul>
<li> Fox, J. (2005). Programmer&#39;s niche: How do you spell that number?.  <em>R News, 5</em>(1), 51-55.</li>
<li> Heylighen, F. (1999). Advantages and limitations of formal expression.  <em>Foundations of Science, 5</em>, 25-56. <a href="http://dx.doi.org/10.1023/A:1009686703349">10.1023/A:1009686703349</a></li>
<li>Heylighen, F. &  Dewaele, J.-M. (1999). Formality of language: Definition, measurement and behavioral determinants.  <a href="http://pespmc1.vub.ac.be/Papers/Formality.pdf"><a href="http://pespmc1.vub.ac.be/Papers/Formality.pdf">http://pespmc1.vub.ac.be/Papers/Formality.pdf</a></a></li>
<li>Heylighen, F. &  Dewaele, J.-M. (2002). Variation in the contextuality of language: An empirical measure.  <em>Foundations of Science, 7</em>(3),  293-340. <a href="http://dx.doi.org/10.1023/A:1019661126744">doi: 10.1023/A:1019661126744</a></li>
<li>Hu, M. & Liu, B. (2004). Mining Opinion features in customer reviews.  <a href="http://www.cs.uic.edu/~liub/publications/aaai04-featureExtract.pdf"><a href="http://www.cs.uic.edu/%7Eliub/publications/aaai04-featureExtract.pdf">http://www.cs.uic.edu/~liub/publications/aaai04-featureExtract.pdf</a></a></li>
<li>Rinker,  T. (2013). qdap: Quantitative discourse analysis package.  <a href="http://github.com/trinker/qdap"><a href="http://github.com/trinker/qdap">http://github.com/trinker/qdap</a></a></li>
<li>Rinker, T. (2013). reports: Package to asssist in report writing.  <a href="http://github.com/trinker/reports"><a href="http://github.com/trinker/reports">http://github.com/trinker/reports</a></a></li>
<li> Wickham, H. (2009). ggplot2: Elegant graphics for data analysis.  <a href="http://had.co.nz/ggplot2/book"><a href="http://had.co.nz/ggplot2/book">http://had.co.nz/ggplot2/book</a></a></li>
</ul>


