ForBo7 // Salman Naqvi
  • Home
  • ForBlog
  • Playground
  • Dictionary
  • About

Tokenization

Splitting a document into its component words.

Note

If a word is too long or very uncommon, the word itself may be split. Take the word “supercalifragilisticexpialidocious” as an example. It could be split into “super”, “cali”, “fragilistic”, “expi”, “ali”, and “docious”.

ForBo7 // Salman Naqvi © 2022 and ForBlog™ by Salman Naqvi | Site Version 2.0.3.1 | Site Feedback | Website made by me!