Ucto: Unicode Tokenizer

Ucto: Unicode Tokenizer

https://languagemachines.github.io/ucto https://github.com/languagemachines/ucto http://webservices-lst.science.ru.nl/ucto/ GNU Public License v3

Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation.