tokenize
now warns rather than throws an error when an invalid input is given during partial parsing. With this change,tokenize
is no longer entirely aborted even if an invalid string is given. Parsing of those strings is simply skipped.
- Corrected probabilistic IDF calculation by
global_idf3
. - Refactored
bind_tf_idf2
.- Breaking Change: Changed behavior when
norm=TRUE
. Cosine nomalization is now performed ontf_idf
values as in the RMeCab package. - Added
tf="itf"
andidf="df"
options.
- Breaking Change: Changed behavior when
- Added wrappers around dictionary compiler of MeCab.
- Removed audubon dependency for maintainability.
pack
now preservesdoc_id
type when it's factor.
- Updated Makevars for Unix alikes. Users can now use a file specified by the
MECABRC
environment variable or~/.mecabrc
to set up dictionaries.
- Removed unnecessary C++ files.
- Prepare for CRAN release.
- For performance,
tokenize
now skips resetting the output encodings to UTF-8.
- Breaking Change: Changed numbering style of 'sentence_id' when
split
isFALSE
. - Added
grain_size
argument totokenize
. - Added new
bind_lr
function.
- Use
RcppParallel::parallelFor
instead oftbb::parallel_for
. There are no user's visible changes.
- Fix documentations. There are no visible changes.
tokenize
can now accept a character vector in addition to a data.frame like object.gbs_tokenize
is now deprecated. Please use thetokenize
function instead.
- Refactored
is_blank
.
- Added the
partial
argument togbs_tokenize
andtokenize
. This argument controls the partial parsing mode, which forces to extract given chunks of sentences when activated.
- More friendly errors are returned when invalid dictionary path was provided.
- Added new
posDebugRcpp
function.
- Revert some missing examples.
- Functions added in version '0.5.1' was moved to 'audubon' package (>= 0.4.0).
- Added some new functions.
bind_tf_idf2
can calculate and bind the term frequency, inverse document frequency, and tf-idf of the tidy text dataset.collapse_tokens
,mute_tokens
, andlexical_density
can be used for handling a tidy text dataset of tokens.
- gibasa now includes the MeCab source, so that users do not need to pre-install the MeCab library when building and installing the package (to use
tokenize
, it still requires MeCab and its dictionaries installed and available).
tokenize
now preserves the original order ofdocid_field
.
- Added
bind_tf_idf2
function andis_blank
function.
- Updated dependencies.
- Changed build process on Windows.
- Added a vignette.
prettify
now can extract columns only specified bycol_select
.
- Added a
NEWS.md
file to track changes to the package. tokenize
now takes a data.frame as its first argument, returns a data.frame only. The former function that gets character vector and returns a data.frame or named list was renamed asgbs_tokenize
.