<!--
%\VignetteEngine{knitr::knitr}
%\VignetteIndexEntry{Getting full text}
%\VignetteEncoding{UTF-8}
-->

```{r echo=FALSE}
knitr::opts_chunk$set(
  comment = "#>",
  collapse = TRUE,
  warning = FALSE, 
  message = FALSE
)
```

The main interface to fetching full text is through `ft_get()`.

## Load fulltext

```{r}
library("fulltext")
```

## Structure of the returned object from ft_get()

Simple call, pass in a DOI and say where you want to get data from (by default, it's _plos_)

```{r}
res <- ft_get('10.1371/journal.pone.0086169', from = 'plos')
```

The article text and metadata is stored in the output object.

The `res` object is a list, with slots for each of the data sources, b/c you can request 
data from more than 1 data source.

```{r}
names(res)
```

Let's dig into the `plos` source object, which is another list, including metadata the 
text data itself (in the `data` slot).

```{r}
res$plos
```

Indexing to the `data` slot takes us to another list with metadata and the article

```{r}
res$plos$data
```

Going down one more index gets us the data object, using the DOI searched to get the text. 
However, since we're dealing with full text, this is a bit messy (just a snippet here)

```{r eval=FALSE}
res$plos$data$data$`10.1371/journal.pone.0086169`
#> [1] "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE article\n  
#> PUBLIC \"-//NLM//DTD Journal Publishing DTD v3.0 20080202//EN\"
#> \"http://dtd.nlm.nih.gov/publishing/3.0/journalpublishing3.dtd\">\n<article
#> xmlns:mml=\"http://www.w3.org/1998/Math/MathML\" xmlns:xlink=\"
...
```

## Fetching many articles

You can get a bunch of DOIs first, e.g., from PLOS using the `rplos` package

```{r}
library("rplos")
(dois <- searchplos(q = "*:*", fl = 'id',
   fq = list('doc_type:full', "article_type:\"research article\""), limit = 5)$data$id)
ft_get(dois, from = 'plos')
```

## Different data sources

### Articles from eLife

One article

```{r}
ft_get('10.7554/eLife.04300', from = 'elife')
```

Many articles

```{r}
ft_get(c('10.7554/eLife.04300','10.7554/eLife.03032'), from = 'elife')
```

### Articles from Frontiers in Pharmacology (publisher: Frontiers)

```{r}
doi <- '10.3389/fphar.2014.00109'
ft_get(doi, from = "entrez")
```

## Search using ft_search()

For example, search entrez, get some DOIs, then fetch some articles

```{r}
(res <- ft_search(query = 'ecology', from = 'entrez'))
res$entrez$data$doi
```

Get articles

```{r}
ft_get(res$entrez$data$doi[1:3], from = 'entrez')
```

## Collect full text from file on disk

When using `ft_get()` you write the files to disk, and you have to pull text out of them as a 
separate step.

```{r}
(res <- ft_get('10.1371/journal.pone.0086169', from = 'plos'))
```

One way to do that is with `ft_collect()`. Before running `ft_collect()` the `data` slot is `NULL`.

```{r}
res$plos$data$data
```

Run `ft_collect()`

```{r}
res <- res %>% ft_collect
```

After running `ft_collect()` the `data` slot has the text. If there's more than one article they are named
by the identifier

```{r}
res$plos$data$data
```

