`stop_words`

This library provides predicates for enumerating, recognizing, and filtering stop words represented as atoms, character lists, or character code lists. Language providers store canonical lowercase words as indexed atom facts.

API documentation

Open the ../../apis/library_index.html#stop-words link in a web browser.

Loading

To load the library main files and the default English language file, load the loader.lgt file:

| ?- logtalk_load(stop_words(loader)).

To load also all the language provider objects, load the loader_all.lgt file:

| ?- logtalk_load(stop_words(loader_all)).

Testing

To test this library predicates, load the tester.lgt file:

| ?- logtalk_load(stop_words(tester)).

Tests load and run smoke tests on all included language files.

Usage

The stop_words(Representation, Language) object accepts atom, chars, and codes representations. The language parameter is an object implementing the stop_words_language_protocol protocol. The library includes a large number of language provider objects in the languages sub-directory named stopwords_XX where XX is the two letters ISO language code.

For example, to enumerate canonical English stop words as atoms:

| ?- stop_words(atom, stopwords_en)::stop_word(Word).
Word = 'll'
...

To test membership, including ASCII case normalization:

| ?- stop_words(atom, stopwords_en)::is_stop_word('THE').
yes

To remove stop words while preserving the original values and order of the remaining tokens:

| ?- stop_words(atom, stopwords_en)::exclude(['The',quick,and,'BROWN',fox], Filtered).
Filtered = [quick,'BROWN',fox]
yes

stop_word/1 on a language provider enumerates canonical lowercase atoms. stop_word/1 on the parametric facade enumerates those words in the configured representation. is_stop_word/1 and exclude/2 lowercase ASCII letters for matching. Tokenization, punctuation stripping, stemming, and Unicode normalization are the caller’s responsibility.

Stop-word sources and license

The default language objects are generated from the corresponding stopwords-iso repositories:

https://github.com/stopwords-iso

Each source snapshot is pinned to a full commit identifier under scripts/sources/stopwords-LANGUAGE/. The source collections and generated objects are copyright 2016 Gene Diaz and licensed under the MIT License. Each original license is preserved beside its source snapshot and copied verbatim into the generated object. The remaining library files are licensed under Apache-2.0.

The upstream entries are preserved without correction, except for four malformed non-ASCII entries in the English source (herse”, himse”, itse”, and myse”). Their intended reflexive pronouns already occur in the source, so the malformed duplicates are omitted from the generated ASCII-only English object.

To update or add a language object, pass the individual repository name and a full commit identifier to the generic updater. For example:

$ ./scripts/update_stopwords.sh stopwords-pt 84249b7ca37005f8cd19df70d86f44dc1f1936a5

The repository must use the stopwords-LANGUAGE naming convention and provide stopwords-LANGUAGE.txt plus an MIT LICENSE file. The generated object and file are named stopwords_LANGUAGE. Language files other than the default stopwords_en.lgt file include an encoding('UTF-8') directive in the first line.

Adding languages

A language provider implements stop_words_language_protocol with indexed lowercase atom facts:

:- object(example_stop_words,
    implements(stop_words_language_protocol)).

    stop_word(a).
    stop_word(the).

:- end_object.

stop_words