# emuR (development version)

# emuR 2.5.2

## new features / performance tweaks / improvements

* `add_signalVia...()`: Minor iteration on the function signatures

## bug fixes

# emuR 2.5.1

## new features / performance tweaks / improvements

* Documentation: Include package anchors in links for CRAN’s new HTML reference manuals
* `add_signalVia...()`: Drafting a new family of functions that allows users to use signal procissing from outside Wrassp; starting with Matlab

## bug fixes

# emuR 2.5.0

## new features / performance tweaks / improvements

* CRUD operations: `create_itemsInLevel()` can now be used in more ways than before
* CRUD operations: Improved documentation for `create/update/delete_itemsInLevel()`; this is ongoing work

## bug fixes

* remove one instance of is.R() because it is no longer allowed on CRAN
* CRUD operations: `delete_itemsInLevel()` was heavily flawed and is now usable

# emuR 2.4.2

## new features / performance tweaks / improvements

* experimental: signal tracks can be stored in Rda format as well as SSFF

## bug fixes

* passing a data frame or tibble as language option to BAS Web Services calls was broken (closes #272, thanks to @johannacronenberg)
* fix error in create_itemsInLevel where end of last segment in a level could be miscalculated

# emuR 2.4.1

## new features / performance tweaks / improvements

## bug fixes

* worked around a problem in vroom_write_lines that could materialize on Windows in rare cases
* fix loading databases with empty sessions (thanks to @FredrikKarlssonSpeech via #268, #271)

# emuR 2.4.0

## new features / performance tweaks / improvements

* cache for onTheFly computations in `get_trackData()` (thanks to @samgregory via #254)
* improved a small number of error messages

## bug fixes

* `add_files` not correctly matching file extension (thanks to @samgregory via #254)
* fixed level attribute mixup in generation of seglist (closes #263)
* workaround for `wrassp::readDataObj()` crashing R session. Temporary fix for issue #261.
* generic functions Math2/round and Math2/signif did not accept digits parameter for legacy trackdata objects
* adaptations to Tidyverse 2.0.0 and other adapations to make unit tests and cran checks smooth again

# emuR 2.3.0

## new features / performance tweaks / improvements

- added explicit checks for all `dir.create()` (closes #49)
- checking dir parameter in `import_mediaFiles()` (closes #108)
- `add_files()` now works with strings like `_annot.json` as `fileExtension` (closes #246)
- now prechecking user defined options in `.onLoad()` (closes #240)
- added new `sessionPattern` and `bundlePattern` args to `list_bundles()` and `list_sessions()` (closes #201)
- better output for warning in `get_trackdata()` regarding heterogeneous sampling rates (closes #189) 
- using `cli` package to improve output of `summary(emuDBhandle)`
- slight performance increase in EQL by avoiding full table scans due to REGEXP
- checking for repeating sequences in `normalize_length()` (closes #211)
- added explicit error message when bad column name is used in DBconfig (closes #212)
- removing linkDefinition on error if it was added be4 (closes #226)

## bug fixes

- fixed bug in requery workflows like: `sl = query(ae, "[Phonetic== V]"); requery_hier(ae, sl[1:2,], level = "Text")`
- added missing loading of DBconfig to `write_bundleList()` (fixes #248)
- fixed bad handeling of dir paths containgin RegExp chars in  `convert_TextGridCollection()` (fixes #207)

# emuR 2.2.0

## new features / performance tweaks / improvements

- compete rewrite of hierarchical queries + requeries (now using CTEs) + clean-up of the EQL engine
- `serve()` -> save bundle performance bump due to SQLite transaction improvement + using prep. statements 
- `serve()` function now downloads current release .zip instead of cloning repo to save time and bandwidth
- fixed a few more `readr::` file vs path parameter warnings
- using `readr::write_lines()` function in `export_TextGridCollection()` to better support UTF-8 symbols
- added `create_spectrogram_image_as_raster` function that used the EMU-webApp
JavaScript code to draw a spectrogram as a raster 

## bug fixes

- fixed invalid TextGrid export on empty levels

# emuR 2.1.1

## new features / performance tweaks / improvements

- `load_emuDB()` performance bump due to SQLite transaction improvement + using prep. statements 
- `create_itemsInLevel()` now supports items of type SEGMENT
- added deprecated/not recommended texts to \code{eplot()}, \code{dplot()} and \code{dur()} help pages (fixes \#234)

## bug fixes

- implemented first version of `delete_itemsInLevel()`
- `requery_seq()` now returns correct attribute label vs. the level attribute
- fixed `In rbind(names(params), unlist(params, recursive = F)): number of columns of result is not a multiple of vector length (arg 1)` warning in `runBASwebservice_maus()` with `httr::upload_file()`
- added missing `()` to `return` statement
- no emuDBcache needed for `rename_emuDB()`


# emuR 2.1.0

## new features / performance tweaks / improvements

- implemented `write_bundleList()` and `read_bundleList()` functions
- new `bundleListName` `serve()` parameter implemented
- added warning to `serve()` when either bundleComments or bundleFinishedEditing is set to true and no bundleListName was set (closes \#268)
- implemented `onTheFlyFunction` parameter for `get_trackdata()` which allows users to implement their own functions (input: path to wav file; output tibble/data.frame that has a column called `"frame_time"`)
- converting factors into characters in `get_trackdata()` and `normalize_length()` (closes \#224 and \#223) 
- `add_perspective()` now sets `"restrictions"` -> `"showPerspectivesSidebar"` to `true` to make the side bar visible
- `normalize_length()` now doesn't throw tibble 3.0.0 warnings any more
- `query()` RegEx operators `=~` now using SQLites native support (wasn't available in earlier versions)
- replaced RCurl with httr dep
- moved some dependencies to `Suggests` field in DESCRIPTION file
- removed vignette stubs

## bug fixes

- try catching rounding errors of sample rate and start time in `get_trackdata()`


# emuR 2.0.4

## new features / performance tweaks / improvements

- `export_seglistToTxtCollection()` now zero pads file names (should fix #219)
- implemented `write_bundleList()` and `read_bundleList()` functions
- new `bundleListName` `serve()` parameter implemented

## bug fixes

- using `rstudioapi::translateLocalUrl()` for ws connection url to permit RStudio to connect
- fixed bug in duplicate levels (bad level name insert into items table) that was causing the items array in the JSON file to stay empty
- `serve()` working outside of RStudio again (overlooked that `rstudioapi::translateLocalUrl()` needs RStudio)
- improved error formatting in `load_emuDB()` and `export_TextGridCollection`
- fixed `rename_bundles()` issue with sub-string matching of names (fixes \#220)
- `requery_hier()` works with non-main attributes again (always returned levels main attribute labels) 

# emuR 2.0.2

## new features / performance tweaks / improvements

* changed default `resultTypes` to `"tibble"` (`query()`, `requery_hier()`, `requery_seq()`, `get_trackdata()`)
* changed url in printed output of serve to https
* `serve()` doesn't block the R console any more
* suppressing requery differing length warnings in `runBASwebservice_*` functions
* `requery_hier()` now persists the input segment list length by inserting NA rows for missing segments
* depricated vignettes are now stubs only (== empty containing notice pointing to the EMU-SDMS manual)
* implemented `rename_bundles()` function
* better `times_norm` calculation for `normalize_length()`
* better error message when empty or non existing session is passed into `add_files()`
* `serve()` `useViewer` parameter now implemented and the default.
* `serve()` now uses a single server to host the EMU-webApp and provide the websocket server for the emuDB
* `serve()` now works within RStudio when it is run as a web application

## bug fixes

# emuR 1.1.2

## new features / performance tweaks / improvements

* rewrite of `list_files()` that leads to massive speed bump
* implemented `convert_wideToLong()` function to convert trackdata tibble objects to their long form representation (useful for spectral analysis)
* `query()` sub-function now sets `perl = TRUE` arguments in `regexp` call when using `sessionPattern` and `bundlePattern` (useful for e.g. for negative look-around `bundlePattern = "^((?!msajc003).)*$"`)
* implemented `export_seglistToTxtCollection()` 
* normalize length now also using `list()` with `do.call(rbind, res_list)`
* implemted new `runMINNI` boolean parameter in `runBASwebservice_all()`

## bug fixes

* https default on `serve()`
* fixed problems of `calcFreqs = T` in `convert_wideToLong()` (possibly caused by newer version of dplyr?)


# emuR 1.1.1

## new features / performance tweaks / improvements

## bug fixes

* fixed bad column init. in `normalize_length()` column
* removed detritus LaTex files (e.g. .log, .aux) from `vignette/pics` directory (as requested by CRAN maintainer)

# emuR 1.1.0

## new features / performance tweaks / improvements

* URL encoding of bundle and session names to allow for URL string reserved characters in bundle and session names
* error message of `get_trackdata()` now contains seglist row index if "Can not extract following"
* `normalize_length()` now allows for additional non-numeric columns
* changed `stop()` to `warning()` in `get_trackdata()` when samplerates are inconsistent (closes \#190)
* better error message when there is a naming mismatch of `_emuDB` dir `_DBconfig.json`
* using `sub()` instead of `tools::file_path_sans_ext()` to handle `_` in file extensions
* `query()`, `requery_seq()` and `requery_hier()` now supports the `resultType` `"tibble"` (and `get_trackdata()`, `requery_seq()`, `requery_hier()` and `serve()` support them as input)
* `serve()` function now uses `seglist$start` and  `seglist$end` instead of `seglist$sample_start` and `seglist$sample_end`
* implemented first version of `update_itemsInLevel()` (only label updates for now)
* now setting the `sample_start` and `sample_end` values in query results when EVENT levels are queried (previously only `start` was set)
* implemented first versions of `create_links()` (currently not checking for anything), `create_itemsInLevel()` (only EVENTs and ITEMs) and `update_itemsInLevel()` (only labels) and `delete_itemsInLevel()`
* `list_bundles()` outside of loop for performance bump in `get_trackdata()`
* better error message in `add_files()` if no files are found
* `consistentOutputType` of `get_trackdata()` is not set to `TRUE` and is reset to `T` if `resultType` is `"emuRtrackdata"` or `"tibble"` (fixes \#203)
* avoiding negative `times_rel` and `times_norm` values in `create_emuRtrackdata()` by setting them to 0 (caused by string to numeric conversion precision errors)
* `requery_seq()` now inserts NA values for the out of bounds rows instead of dropping them.
* implemented `list_sampleRates()` function
* `get_trackdata()` is now iteratively appending to a `list()` instead of into a SQLite temp table. This is a fairly large performance boost and also fixes \#206.
* added deprecation warnings to vignettes (added links to manual chapters)

## bug fixes

* propper fix for "now ordering by `items_idx` not by `start_start_seq_idx` which led to bad label sequences (fixes \#140)"
* fixed bad indexing in `normalize_length()` when sl_rowIdx values are not a `c(1, 2, 3, 4, ...)` sequence
* fixed `staticContours` SSFF tracks not being sent to EMU-webApp (fixes \#195)
* fixed bug with completely empty levels that caused a bad resort of levels in `_annot.json`s in `rewrite_allAnnots()`
* fixed bug in `add_files()` that was using the wrong variable (fixes \#196)
* added error message when querying levels without time-bearing sub-levels (closes \#150)
* fixed bug in `create_emuRtrackdata()` with handling trackdata object of class `spectral`
* correct recalculation of ITEM IDs of missing levels in .hlb files (== only present in ESPS files) in `convert_legacyEmuDB()`
* fixed handling of completely empty levels in .hlb files
* `normalize_length()` now handles various additional column types (not just `"numeric"`)

# emuR 1.0.0

## new features / performance tweaks / improvements

* implemented new `consistentOutputType` parameter for `get_trackdata()` to always return a `trackdata` or `emuRtrackdata` object independent of what the `cut` and `npoints` arguments are set to
* now removing `levelCanvasOrder` entry in `remove_levelDefinition()` (fixes \#156)
* `serve()` method now uses GET to deliver media files to the EMU-webApp. This avoids the base64 conversion overhead and is a quite significant load time improvement
* explicit error message in `convert_legacyEmuDB()` when invalid redundant links are found
* better error message in BPF parser
* `convert_legacyEmuDB()` automatically converts `.ssd` media files to `.wav` and normalizes the annotations to start at 0 (only if attr(ssd,'startTime') is not 0).
* added `sort()` S3 method for `emuRsegs` objects
* checking for badly sorted `emuRsegs` in `requery_hier()` and `requery_seq()` functions
* `create_emuRtrackdata()` returns a simple `data.frame` object not a `data.table` object
* `emuRtrackdata` object now contains a `times_norm` (normalized time values between 0 and 1 for each segment) column by default
* added note to `print.emuRsegs()` to give the user a hint about missing columns
* implemented `print.emuRtrackdata()` to avoid overly verbose output
* implemented `normalize_length()` function as S3 function to normalize the length of each segment in an `emuRtrackdata` object
* added `absolute_file_path` column to output of `list_files()`
* query engine does not rely on label index in label array any more (updated `convert_queryResultToEmuRsegs()` to use `resultAttrDef` instead of `labelIdx`). Closes \#164.
* added `browser` argument to `serve()` function which is passed on to `utils::browseURL()` function
* `requery_seq()` now uses `start_item_seq_idx` and `end_item_seq_idx` of seglist instead of `start_item_id` and `start_item_id` to simplify function
* implemented `check_emuDBhandle()` function that is used on every exported function that takes a `emuDBhandle` as an argument to check if the handle is still valid (closes \#176)
* implemented `"tibble"` as `resultType` option in `get_trackdata()`. This will probably replace the `"emuRtrackdata"` option in future (it contains exactly the same data/columns).
* prechecking if attribute definition is already defined (closes \#182)
* `get_trackdata()` now uses temporary SQL tables to store the intermediate results (massive performance gains!). Removed `nrOfAllocationRows` parameter as this is no longer needed as no matrix is used to store the intermediate results. (also closes \#125)
* `convert_TextGridCollectio()` using `dir.exists()` instead of `file.exists()` to check dirs 
* all read operations now use the readr package (avoids encoding problems like \#187)
* `list_attributeDefinitions()` now allows for a name vector to be passed in
* rewrite of `rewrite_allAnnots()` functions for faster rewrites of `_annot.json` files to disk
* improved cleanup in testing DBconfig functions
* now ordering by `items_idx` not by `start_start_seq_idx` which led to bad label sequences (fixes \#140)

## bug fixes

* fixed problem of updating cache. Didn't handle `data.frame` object that was thought to be a vector correctly!
* fixed a bug in the BPF export function, which meant that WAVE files were only copied into one session
* added missing `$` in pattern arguments in `list.files` call in `list_files` (fixes \#170)
* not adding ssffTrackDefinition to DBconfig if user input is no (closes \#171)
* fixed bad `seq_start_seq_idx` and `seq_seq_idx` returned by `requery_seq()` (fixes \#183) 
* fixed bad `seq_start_seq_idx` returned by internal `query_databaseHier()` function
* fixed bad sorting of `requery_seq()/requery_hier()` when `calcTimes = F` (still sorted by `start_sample` instead of the correct `seq_idx`)
* added `readr::parse_character()` to data received in `serve()` as this is recoded in windows (fixes \#188).

# emuR 0.2.3

## new features / performance tweaks / improvements

* tweaked `runBASwebservice_maus()`; improved performance for presegmented bundles
* performance bump for `fapply()` by preallocating result matrix
* performance bump for `trapply()` by preallocating result matrix
* performance bump for `mel.spectral()` by preallocating result matrix
* performance bump for `bark.spectral()` by preallocating result matrix
* updated DBI calls to comply with the latest best practices (using `DBI::dbExecute()` instead of `DBI::dbGetQuery()` for non-`SELECT` queries)
* BPF collection exporter documented and now public

## bug fixes

* `export_TextGridCollection()` now handles partial includes of bundle and session names correctly (issue \#147)
* added missing check if `anagestConfig` is defined to `rename_attributeDefinition()`
* setting useBytes to T to avoid reencoding under windows
* fixed bug in `add_ssffTrackDefinition()` that was trying to access `fp` which was renamed in a refactor to `filesDf`
* fixed export to autodetect S3 methods (cbind & rbind for trackdata)

# emuR 0.2.2

## new features / performance tweaks / improvements

* some changes to the parameter names in the BAS webservice functions
* convert_txtCollection and convert_BPFCollection now name topmost item "bundle"
* added functions to set and get level descriptions in DBconfig
* BAS webservice functions now perform a cache update prior to departure
* added multiple perspectives to ae demo database
* choosing explicit paths with intersecting hierarchies now possible
* remove levelDef & linkDef now implement force parameters
* new function convert_txtCollection converts plain text collections into single-node emuDB
* new functions runBASwebservice_* that call various BAS webservices from inside emuR
* NULLing out empty DFs on list_level/linkDefs for more consistent API
* `newLinkDefType` argument implemented in `autobuild_linkFromTimes()` to generate linkDefinition if so desired 
* automatically removing superlevel from `levelCanvasOrder` if `convertSuperlevel` is set to `TRUE` in `autobuild_linkFromTimes()`

## bug fixes

* wrapped `readChar`s in `enc2utf8` to avoid encodings issues on windows
* updating label table correctly on add_attributeDefinition() (#138)
* runBASwebservice_maus / minni / all now no longer ignore unlinked items (idx -1) but treat them as linkless segments
* commented out `cat()` in `train()` function be be less verbose
* BAS webservice calls now get their own temp directories (UUID based). This avoids race conditions when several scripts are running in parallel.
* convert_txtCollection now treats perspectives as array (as it should)

# emuR 0.2.1

## new features / performance tweaks / improvements

* added new `EMUwebAppConfig -> perspectives -> signalCanvases -> minMaxValLims` config option to emuDB vignette
* requery_hier + requery_seq now implement the same timeRefSegmentLevel parameter as query (#135)

## bug fixes

* fixed requery\_hier() bug of requery on same attribute definition
* fixed requery\_hier() bug of requery on same level but different attribute definition

# emuR 0.2.0

## new features / performance tweaks / improvements

* rewrite of query engine to not require links_ext table any more (== redundant links)
* calcTimes parameter added to query() / requery\_seq() / requery\_hier() to make calculating times optional (extreme performance boost if no times have to be calculated)
* rewrite of annotJSONcharToBundleAnnotDFs() for faster loads emuDBs containing large annotJSONs
* replaced tidyjson as annot.json parser with own solution at tidyjson didn't scale well on larger annotation files
* added verbose parameter to export\_TextGridCollection()
* improved pre-check of dir exists in export\_TextGridCollection()
* added new replace\_itemLabels function
* improved export\_TextGridCollection() doc
* improved replace_itemLables() speed
* implemented rename\_emuDB() (\#116)
* implemented duplicate\_level() (\#113)
* implemented linkDuplicates parameter in duplicate\_level()
* autobuild\_linkFromTimes() speed improvements
* FUNCQ queries (start(),end(), medial()) now additionally support TRUE & FALSE and T & F values (vs. 0 & 1) 
* added attrDefNames column to list\_levelDefinitions() output
* can now deal with read only emuDBs by copying the cache to tempdir() and making it writable for the user
* added start\_item\_seq\_idx and end\_item\_seq\_idx to emuRsegs object
* added start\_item\_seq\_idx and end\_item\_seq\_idx type values to all intermediate result tables
* added optional function to reduce hierarchical query results to left and right most children only (large performance gain on calcTimes = T)
* rewriting annot.json files now updates MD5 sums as well (avoids unnecessary reload on next load\_emuDB)
* rewriting annot.json files now writes all (including empty / missing) attributeDef. labels

## bug fixes

* fixed bad DBconfig gen. on add_perspective
* fixed list\_linkDefinitions() returning strings as factors
* fixed bad error message when passing in ITEM levels to autobuild\_linkFromTimes()
* fixed incorrect handling of DBconfig when writeToFS was set to FALSE (writeToFS is now called rewriteAllAnnots)

# emuR 0.1.9

## new features / performance tweaks / improvements

* also allowing "time = " in TextTiers
* "levels of type 'EVENT' are not allowed to be super levels (== parents) in a domination relationship" constraint enforced in add_linkDefinition
* added "MEDIAFILE\_SAMPLES" as constant name to access audio samples to get\_trackdata() function
* improved error message to include tgPath in create_DBconfigFromTextGrid function
* no integer return value returned by create_emuRdemoData() any more! It was implicitly returned from wrassp function call...
* improved the slow overlap checking function in the BPF parser (is now O(n) instead of O(n^2))
* fixed col naming problems for new (unreleased) RSQLite version
* added export_TextGridCollection() function
* improved doc for get_trackdata
* constant naming of EMU-SDMS vs EMU_SDMS in various files
* rewriting all annotation file on add\_levelDefinition, remove\_levelDefinition 

## bug fixes

* fixed problem in conversion to JSON with empty items array (object '{}' vs array '[]')
* fixed problem of keywords "number" | "time" | "xmin" | ... in labels causing TextGrid parser to fail
* fixed problem with to lax RegEx in TextGrid parser
* fixed validation problem with missing levels regarding types

# emuR 0.1.8

## new features / performance tweaks / improvements

* get_trackdata with onTheFly calculation now reuses AsspDataObj if the current utterance is the same as the previous (large performance gain especially on long audio files)
* checking if DBconfig exists for better error message if 'name' field is not set correctly in DBconfig
* setting PRAGMA temp_store = 2; for SQLite connections
* not extracting tables to R if no RegEx needed to create filtered_tmp tables (performance gain when querying large emuDBs)
* convert_BPFCollection can now assigns the same label to more than one item when unifying tiers
* newline at the end of load_emuDB if no redundant links are built
* queries using dominates operator '^' don't use linksExt table anymore -> large performances benefits 
* only using \_filtered\_tmp tables if RegEx patterns are used
* changed primary key on items table which leads to massive performance gains (deleting _emuDBcache.sqlite required)

## bug fixes

* fixed error handling of create_emuRtrackdata + added @export to roxygen doc
* invalid annotJSONs generated by import_mediaFiles fixed
* convert_TextGridCollection can now handle nested folders again
* invalid UUIDs in DBConfig produced by convert_BPFCollection. Also added additional unit test to detect this.
* list_bundles uses session argument again
* fixed "Expression tree is too large (maximum depth 1000)" error in get_trackdata with long emuRsegs lists

# emuR 0.1.7

* R depends version bump to 3.2.0 (as requested by CRAN maintainer)
* updated testthat::expect\_less\_than to expect\_lt calls (due to deprecated warnings)
* Using new .keep_all = T parameter of dplyr
* removed legacy version of EQL vignette (overlooked as inst/doc was in .gitignore)

# emuR 0.1.6

* skipping in-depth thorough tests on CRAN for query and autobuild SQL functions 

# emuR 0.1.5

* fixed problem of interm\_res\_tables already being present with queries that have multiple recursion depth on both sides
of either -> or ^ operand (e.g. query (ae ,  "[[[Phonetic = n -> Phonetic =z] -> Phonetic = S ] ^ [Text = friends -> Text = she]]")) 
* fixed bad URL in README.md
* added CITATION file

# emuR 0.1.3.9000

* renamed SQL tables & columns from camel case to underscore notation 
* variable SQL backend implementation

# emuR 0.1.2.9000

* multiple check fixes on various platforms

# emuR 0.1.1.9000

* `serve` problem with internalVars bug fixed
* file locking problem that caused vignettes to fail under windows problem fixed

# emuR 0.1.0.9000

* massive refactor of all functions that used to refer to an emuDB by 
  name and optionally by its UUID. They now use the new emuDBhandle object
  that is now returned by the `load_emuDB()` function.
* `convert_XXX_to_emuDB()` functions renamed to `convert_XXX()`
