# Xan zine: September 2024 Edition

As usual, upgrade to the latest version (at least 0.18.0) using:

```bash
cargo install xan
```

So many things to say and so little time... This newest entry of the zine will most likely look like a changelog of sorts, rather than the usual detailed chronicles.

## Summary

- [--source-column](#--source-column)
- [xan tokenize](#xan-tokenize)
- [xan vocab](#xan-vocab)
- [xan top](#xan-top)
- [Grouping flag for some commands](#grouping-flag-for-some-commands)
- [Static evaluation of expressions](#static-evaluation-of-expressions)
- [Miscellaneous features](#miscellaneous-features)
- [Miscellaneous moonblade](#miscellaneous-moonblade)

## --source-column

The `xan cat rows` command and the `xan merge` commands now both support a `--source-column` flag that takes the name of a column to prepend containing the name of the original file of each row.

## xan tokenize

Thanks to latest development in our [paltoquet](https://docs.rs/paltoquet/latest/paltoquet/) library, `xan` is now able to tokenize text!

Here is the command's help:

```txt
Tokenize the given text column by splitting it into word pieces (think
words, numbers, hashtags etc.). This command will therefore emit one row
per token written in a new column added at the end, all while dropping
the original text column unless --sep or --keep-text is passed.

For instance, given the following input:

id,text
1,one cat eats 2 mice! 😎
2,hello

The following command:
    $ xan tokenize text -T type file.csv

Will produce the following result:

id,token,type
1,one,word
1,cat,word
1,eats,word
1,2,number
1,mice,word
1,!,punct
1,😎,emoji
2,hello,word

You can easily pipe the command into "xan vocab" to create a vocabulary:
    $ xan tokenize text file.csv | xan vocab id token > vocab.csv

You can easily keep the tokens in a separate file using the "tee" command:
    $ xan tokenize text file.csv | tee tokens.csv | xan vocab id token > vocab.csv

This tokenizer is able to distinguish between the following types of tokens:
    - word
    - number
    - hashtag
    - mention
    - emoji
    - punct
    - url
    - email

Usage:
    xan tokenize [options] <column> [<input>]
    xan tokenize --help

tokenize options:
    -c, --column <name>      Name for the token column. Will default to "token" or "tokens"
                             if --sep is given.
    -N, --ngrams <n>         If given, will output token ngrams using the given n or the given
                             range of n values using a comma as separator e.g. "1,3".
                             This cannot be used with -T, --token-type.
    -T, --token-type <name>  Name of a column to add containing the type of the tokens.
                             This cannot be used with -N, --ngrams.
    -S, --simple             Use a simpler, more performant variant of the tokenizer but unable
                             to infer token types, nor handle subtle cases.
    -D, --drop <types>       Types of tokens to drop from the results, separated by comma,
                             e.g. "word,number". Cannot work with -k, --keep.
                             See the list of recognized types above.
    -K, --keep <types>       Types of tokens to keep in the results, separated by comma,
                             e.g. "word,number". Cannot work with -d, --drop.
                             See the list of recognized types above.
    -m, --min-token <n>      Minimum characters count of a token to be included in the output.
    -M, --max-token <n>      Maximum characters count of a token to be included in the output.
    --stoplist <path>        Path to a .txt stoplist containing one word per line.
    --sep <char>             If given, the command will output exactly one row per input row,
                             keep the text column and join the tokens using the provided character.
                             We recommend using "§" as a separator.
    --ngrams-sep <char>      Separator to be use to join ngrams tokens.
                             [default: |]
    --keep-text              Force keeping the text column in output.
    -p, --parallel           Whether to use parallelization to speed up computations.
                             Will automatically select a suitable number of threads to use
                             based on your number of cores. Use -t, --threads if you want to
                             indicate the number of threads yourself.
    -t, --threads <threads>  Parellize computations using this many threads. Use -p, --parallel
                             if you want the number of threads to be automatically chosen instead.

Common options:
    -h, --help             Display this message
    -o, --output <file>    Write output to <file> instead of stdout.
    -n, --no-headers       When set, the first row will not be interpreted
                           as headers.
    -d, --delimiter <arg>  The field delimiter for reading CSV data.
                           Must be a single character.
```

## xan vocab

Since we can now tokenize text easily, why stop here? Let's compute vocabulary statistics with the newest `xan vocab` command!

Here is the command's help:

```txt
Compute vocabulary statistics over tokenized documents. Those documents
must be given as a CSV with one or more column representing a document key
and a column containing single tokens like so (typically an output from
the "tokenize" command):

doc,token
1,the
1,cat
1,eats
2,hello

This command can compute 4 kinds of differents vocabulary statistics:

1. corpus-level statistics (using the "corpus" subcommand):
    - doc_count: number of documents in the corpus
    - token_count: total number of tokens in the corpus
    - distinct_token_count: number of distinct tokens in the corpus
    - average_doc_len: average number of tokens per document

2. token-level statistics (using the "token" subcommand):
    - (token): some distinct token (the column will be named like the input)
    - gf: global frequency of the token across corpus
    - df: document frequency of the token
    - idf: inverse document frequency of the token
    - gfidf: global frequency * idf for the token
    - pigeonhole: ratio between df and expected df in random distribution

3. doc-level statistics (using the "doc" subcommand):
    - (*doc): columns representing the document (named like the input)
    - token_count: total number of tokens in document
    - distinct_token_count: number of distinct tokens in document

4. doc-token-level statistics (using the "doc-token" subcommand):
    - (*doc): columns representing the document (named like the input)
    - (token): some distinct document token (the column will be named like the input)
    - tf: term frequency for the token in the document
    - tfidf: term frequency * idf for the token in the document
    - bm25: BM25 score for the token in the document
    - chi2: chi2 score for the token in the document

Usage:
    xan vocab corpus <doc-cols> <token-col> [options] [<input>]
    xan vocab token <doc-cols> <token-col> [options] [<input>]
    xan vocab doc <doc-cols> <token-col> [options] [<input>]
    xan vocab doc-token <doc-cols> <token-col> [options] [<input>]
    xan vocab --help

vocab doc-token options:
    --k1-value <value>     "k1" factor for BM25 computation. [default: 1.2]
    --b-value <value>      "b" factor for BM25 computation. [default: 0.75]

Common options:
    -h, --help             Display this message
    -o, --output <file>    Write output to <file> instead of stdout.
    -n, --no-headers       When set, the first row will NOT be included
                           in the frequency table. Additionally, the 'field'
                           column will be 1-based indices instead of header
                           names.
    -d, --delimiter <arg>  The field delimiter for reading CSV data.
                           Must be a single character.
```

## xan top

It is now possible to use `xan top` to efficiently find the top n rows of some CSV file, as per the values contained in a given column.

It works using a custom made fixed reverse heaps, that is able to keep hold of ties if necessary.

Here is the command's help:

```txt
Find top k CSV rows according to some column values.

Runs in O(N * log k) time, consuming only O(k) memory.

Usage:
    xan top <column> [options] [<input>]
    xan top --help

dedup options:
    -l, --limit <n>       Number of top items to return. Cannot be < 1.
                          [default: 10]
    -R, --reverse         Reverse order.
    -g, --groupby <cols>  Return top n values per group, represented
                          by the values in given columns.
    -r, --rank <col>      Name of a rank column to prepend.
    -T, --ties            Keep all rows tied for last. Will therefore
                          consume O(k + t) memory, t being the number of ties.

Common options:
    -h, --help               Display this message
    -o, --output <file>      Write output to <file> instead of stdout.
    -n, --no-headers         When set, the first row will not be evaled
                             as headers.
    -d, --delimiter <arg>    The field delimiter for reading CSV data.
                             Must be a single character.
```

## Grouping flag for some commands

Some commands, at least `freq`, `top` and `view`, now accept a `-g/--groupby` flag that group the aggregation/visualization by some column selection.

## Static evaluation of expressions

`moonblade` is now able to evaluate statically analyzable parts (when function arguments are only strings, integers, floats etc.) of evaluated expression to avoid computing them once per row.

This means that in the following command:

```bash
xan map 'read_json("color-to-hex.json")[color]' hex nodes.csv > nodes-with-hex-color.csv
```

The `read_json("color-to-hex.json")` part is static and does not depend on information found in each row, which means the JSON file will only be read and parsed once!

## Miscellaneous features

- the `from` command now support json dicts as input.
- the `foreach` command is now just a version of `map` producing side-effects (copying files, deleting files etc.).
- `dedup` now has a `--keep-last` flag. It consumes more memory and delays the output until everything has been read of course.
- the `frequency` command now has a `freq` alias.
- all grouping commands (`freq`, `top`, `groupby` etc.) now produce deterministic output where the order of input file will explain the order of emitted groups.
- all grouping commands relying on hashmaps have been optimized to minimize lookups when the input is already sorted.
- added a `blank` command that "blanks" down consecutive identical values for presentation or compression purposes (similar to what Open Refine does).
- added `dedup --external`, relying on a transient on-disk BTree index, for cases where the set of considered values does not hold in memory.
- added multiple indices support to the `slice` command, e.g. `xan slice -i 345,657,99897`.

## Miscellaneous moonblade

- `moonblade` should be more performant, and use less memory.
- `&&` and `||` semantics are now aligned on most interpreted programming languages (they short-circuit and return the last relevant value, not cast to boolean).
- added `min`, `max`, `argmin` (I blame @jimenaRL), `argmax` functions.
- added the `mean` function, working with lists.
- added the `match` function to capture groups from a regular expression.
- adding `cols` and `headers` functions, so you can do `argmin(cols(), headers())` (again, I blame @jimenaRL).
- `contains` function and `in` operator now works for lists and maps, as expected.
- the `write` function is now fully threadsafe.
- added `copy` and `move` functions to move files around.
- added `keys` and `values` functions to work with maps.
- it is now possible to index strings, list, maps etc. `map["key"]`, `list[-1]`, for instance.
- it is now possible to slice strings and list like with python `list[1:3]`, `list[1:]` etc.
- `get` now accepts a nested path as a list.
- some functions are now able to accept named arguments, e.g. `read(path, encoding="latin-1")`.
- added `top`, `argtop`, `most_common`, `most_common_counts` aggregating functions, to be used with the `agg` and `groupby` commands.
- added `read_csv` and `read_json` functions.
- added the `index_by` function to work with lists of maps.