`language_detection`

This library provides lightweight language detection for text represented as atoms, character lists, or character code lists. It currently supports the most common languages and returns the corresponding ISO 639-1 language codes. It requires a backend with Unicode support.

API documentation

Open the ../../apis/library_index.html#language-detection link in a web browser.

Loading

To load all entities in this library, load the loader.lgt file:

| ?- logtalk_load(language_detection(loader)).

Testing

To test this library predicates, load the tester.lgt file:

| ?- logtalk_load(language_detection(tester)).

Tests cover all included languages and detection strategies using the atom, chars, and codes text representations.

Usage

The language_detector(Representation, Strategy) parametric object accepts the atom, chars, and codes representations. The strategy parameter must be an object implementing the language_detection_strategy_protocol protocol.

To detect the most likely language:

| ?- language_detector(atom, ngram_language_detector)::detect(
         'The quick brown fox jumps over the lazy dog.', Language
     ).
Language = en
yes

To also return a relative detection score:

| ?- language_detector(atom, hybrid_language_detector)::detect(
         'Questo testo è scritto in italiano.', Language, Score,
         [min_length(10)]
     ).
Language = it,
Score = ...
yes

The detect_all/2-3 predicates return all candidate languages ordered by decreasing score and then by increasing language code. The scores are relative to the selected candidate set; they are not calibrated probabilities and should not be compared across different candidate sets.

The input is normalized to NFC, converted to lowercase, stripped of control characters, and normalized for whitespace before detection. Diacritics are preserved because they provide useful language evidence.

Strategies

The library includes the following strategy objects:

ngram_language_detector compares character-trigram count vectors using cosine similarity.
stopword_language_detector tokenizes the input and scores discriminative stop-word matches.
script_language_detector scores candidates using the Unicode scripts observed in the input. Languages sharing the same script receive the same script evidence and are therefore expected to tie when used alone.
hybrid_language_detector first combines the n-gram and stop-word scores using weights of 0.75 and 0.25, respectively, and then combines the result with script scores using weights of 0.80 and 0.20. When all strategies find evidence, the effective weights are therefore 0.60 for n-grams, 0.20 for stop words, and 0.20 for scripts. When either lexical or script evidence is unavailable, the available scores are returned unchanged.

Strategy selection is performed by the second object parameter. It is not an option of the language_detector(Representation, Strategy) object.

Script analysis

The language_detection_scripts object exposes the script analysis used by the script strategy. It accepts normalized character codes and returns all observed Unicode scripts with ratios ordered by decreasing ratio:

| ?- atom_codes('abc éè', Codes),
     language_detection_scripts::script_ratios(Codes, Ratios).
Ratios = ['Latin'-0.8333333333333334, 'Common'-0.16666666666666666]
yes

Script names are the values provided by the Unicode Character Database, such as 'Latin', 'Cyrillic', and 'Arabic'. The scripts_to_languages/2 predicate maps one or more meaningful scripts to the sorted set of compatible registered languages:

| ?- language_detection_scripts::scripts_to_languages(['Cyrillic'], Languages).
Languages = [bg, ru, uk]
yes

The script strategy ignores the 'Common', 'Inherited', 'Unknown', and 'Zzzz' values when scoring. For mixed-script text, evidence from every other observed script is retained. If no meaningful mapped script is observed, the strategy returns no scores and detection fails normally.

Options

The detect/4 and detect_all/3 predicates accept:

min_length(N), defaulting to 20, sets the minimum normalized text length.
candidates(all|Languages), defaulting to all, selects either all built-in profiles or a nonempty list of supported ISO 639-1 language codes.
min_score(Score), defaulting to 0.30, sets the minimum best relative score.
min_margin(Margin), defaulting to 0.05, sets the minimum difference between the two highest scores.

Duplicate candidate codes, unsupported candidate codes, duplicate options, and invalid option values are rejected.

The detect/2-4 predicates fail when the normalized input is shorter than the configured minimum or when the available evidence does not satisfy the score and margin thresholds. The detect_all/2-3 predicates return an empty list in those cases.

Language profiles

The library includes the several profile objects named XX_language_profile in the profiles sub-directory where XX is the ISO language code.

The character-trigram vectors are deterministically derived from the vendored stop-word inventories. Words are padded with spaces before trigram extraction so that word boundaries contribute evidence. The stop-word source files record their upstream repositories, commits, licenses, and generated entry counts.

Adding strategies and profiles

A custom strategy implements the language_detection_strategy_protocol protocol. Its scores/3 predicate receives normalized character codes and a validated list of candidate language codes and returns one Language-Score pair for every candidate, or an empty list when it finds no evidence.

A custom profile imports the language_profile_common category. Profiles are registered by defining the language_profiles::custom_profile/2 multifile predicate. For example:

:- multifile(language_profiles::custom_profile/2).
language_profiles::custom_profile(it, it_custom_language_profile).

The profile protocol and registry allow replacing the bundled vectors with corpus-derived profiles without changing the detector or strategy APIs.

A custom profile can override or add its expected scripts by defining the language_detection_scripts::custom_language_scripts/2 multifile predicate:

:- multifile(language_detection_scripts::custom_language_scripts/2).
language_detection_scripts::custom_language_scripts(sr, ['Cyrillic', 'Latin']).

Limitations

Accuracy decreases for short, code-switched, transliterated, or highly specialized text. Only the most common languages are currently included. The bundled n-gram profiles are intentionally lightweight and are not a substitute for profiles trained and evaluated on balanced corpora. Script-run segmentation, calibrated probabilities, and runtime profile training are outside the scope of this library.

language_detection