<!--
%\VignetteEngine{knitr::knitr}
%\VignetteIndexEntry{fulltext introduction}
%\VignetteEncoding{UTF-8}
-->

```{r echo=FALSE}
knitr::opts_chunk$set(
  comment = "#>",
  collapse = TRUE,
  warning = FALSE, 
  message = FALSE
)
```

fulltext introduction
======

`fulltext` is a package to facilitate text mining. It focuses on open access journals. This package makes it easier to search for articles, download those articles in full text if available, convert pdf format to plain text, and extract text chunks for vizualization/analysis. We are planning to add bits for analysis in future versions. The steps in bullet form:

* Search
* Retrieve
* Convert
* Text
* Extract

## Load fulltext

```{r}
library("fulltext")
```

## Search for articles

Search for the term _ecology_ in PLOS journals.

```{r}
(res1 <- ft_search(query = 'ecology', from = 'plos'))
```

Each publisher/search-engine has a slot with metadata and data

```{r}
res1$plos
```

## Get full text

Using the results from `ft_search()` we can grab full text of some articles

```{r}
(out <- ft_get(res1))
```

Dig in to the PLOS data

```{r}
out$plos$data$path[[1]]
```

## Extract text from pdfs

Ideally for text mining you have access to XML or other text based formats. However, 
sometimes you only have access to PDFs. In this case you want to extract text 
from PDFs. `fulltext` can help with that. 

You can extract from any pdf from a file path, like:

```{r}
path <- system.file("examples", "example1.pdf", package = "fulltext")
ft_extract(path)
```

Let's search for articles from arXiv, a preprint service. Here, get pdf from 
an article with ID `cond-mat/9309029`:

```{r}
res <- ft_get('cond-mat/9309029', from = "arxiv")
res2 <- ft_extract(res)
# the first page
res2$arxiv$data$data$`cond-mat/9309029`[1]
```

## Extract text chunks

We have a few functions to help you pull out certain parts of an article. 
For example, perhaps you want to get just the authors from your articles, 
or just the abstracts. 

Here, we'll search for some PLOS articles, then get their full text, then
extract various parts of each article with `ft_chunks()`.

```{r}
res <- ft_search(query = "ecology", from = "plos")
(x <- ft_get(res))
```

Extract DOIs

```{r}
x %>% ft_collect() %>% ft_chunks("doi")
```

Extract DOIs and categories

```{r}
x %>% ft_collect() %>% ft_chunks(c("doi", "title"))
```

`ft_tabularize` attempts to help you put the data that comes out of 
`ft_chunks()` in to a `data.frame`, that we all know and love. 

```{r}
x %>% ft_collect() %>% ft_chunks(c("doi", "history")) %>% ft_tabularize()
```

