category

`tokenizer_rules`

Reusable portable rules for tokenization and sentence splitting. Importing objects supply language-specific abbreviation and word punctuation hooks. URL and email address recognition delegates to the url library.

Availability:

logtalk_load(tokenization(loader))

Author: Paulo Moura
Version: 1:0:0
Date: 2026-09-05

Compilation flags:

static

Implements:

public tokenizer_language_protocol

Extends:

public options

Uses:

list
number_grammars(Format)
string(Representation)
url(Representation)

Remarks:

(none)

Inherited public predicates:

check_option/1 check_options/1 default_option/1 default_options/1 option/2 option/3 split_sentence_codes/3 tokenize_codes/3 valid_option/1 valid_options/1

Public predicates 

(no local declarations; see entity ancestors if any)

Protected predicates 

`abbreviation/1`

Enumerates canonical lowercase abbreviations, including their periods.

Compilation flags:

static

Template:

abbreviation(Abbreviation)

Mode and number of proofs:

abbreviation(?atom) - zero_or_more

`non_terminal_abbreviation/1`

Enumerates abbreviations that do not terminate a sentence when followed by another token.

Compilation flags:

static

Template:

non_terminal_abbreviation(Abbreviation)

Mode and number of proofs:

non_terminal_abbreviation(?atom) - zero_or_more

`internal_apostrophe/1`

Enumerates apostrophe character codes accepted inside words.

Compilation flags:

static

Template:

internal_apostrophe(Code)

Mode and number of proofs:

internal_apostrophe(?character_code) - zero_or_more

`internal_hyphen/1`

Enumerates hyphen character codes accepted inside words.

Compilation flags:

static

Template:

internal_hyphen(Code)

Mode and number of proofs:

internal_hyphen(?character_code) - zero_or_more

Private predicates 

(no local declarations; see entity ancestors if any)

Operators 

(none)

tokenizer_rules

`tokenizer_rules`