Skip to main content

%iKnow.Stemming.DecompoundUtils

class %iKnow.Stemming.DecompoundUtils

This class contains utility methods to manage the word list used by the decompounding algorithm. Decompounding is about identifying the words making up a compound term, such as the words "thunder" and "storm" in the compound term "thunderstorms". It is used primarily for search purposes, allowing you to find records containing compounds of the search terms too. Lanugages like German, where compounding happens often, require decompounding support for a good search experience.

Training the decompounder

The decompounding algorithm supplied here requires a list of candidate words it will try to recognize in to-be-decompounded terms. These candidate words can be added through training the algorithm using any of the following methods, which accept free text that will be cut into candidate terms and then stripped of any recognizable compounds:

Alternatively, individual words can be added and removed through the AddWord() and RemoveWord() methods. Words that should never be separated (returned as a single word) can be registered through the NeverSeparate().

Invoking the decompounder

Decompounding is used by iFind indices who have their INDEXOPTION set to 2 (see also %iFind.Index.Basic). When subsequently adding records to such an indexed table, all words will be checked for compounding and additional index structures will be populated to allow retrieving records based on the compounding words.

The algorithm can also be invoked directly through a %iKnow.Stemmer object, should there be any requirement to find the compounding words of a given term (ie for debug purposes).

 // simple training
   do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "thunder")
   do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "storm")
   // invoke decompounder
   write ##class(%iKnow.Stemmer).GetDefault("en", .tStemmer)
   write tStemmer.Decompound("thunderstorms", .tWords)
   zwrite tWords

Method Inventory

Methods

classmethod AddWord(pLanguage As %String, pWord As %String, pFrequency As %Integer = 1, pClean As %Boolean = 1, pVerbose As %Boolean = 0) as %Status

Adds a word to the compound dictionary for the supplied language. The supplied word will be treated as a valid compound element the algorithm will no longer try to split in smaller elements. Optionally supply a positive integer frequency value to increase its weight when multiple options are available.

If pWord is also present in the list of strings never to split off through a call to NeverSeparate(), it will be removed from that list.

When performing a lot of manual updates, it is recommended to set pClean=0 and only run the CleanWords() method once after all additions, to verify if these new additions indicate particular existing words should be removed as they turn out to be compounds themselves.

classmethod AppendTrainingDataFromDomain(pDomainName As %String, pLanguage As %String = "en", pEntType As %Integer = $$$ENTTYPEANY, pClean As %Boolean = 1, pVerbose As %Boolean = 1) as %Status

Appends word frequency information drawn from an existing iKnow domain to the word dictionary for decompounding in this namespace. When pEntType=$$$ENTTYPEANY (default), the full sentence values (with literal info) will be used to derive words. To restrict this to concepts or relations only, use $$$ENTTYPECONCEPT resp. $$$ENTTYPERELATION.

Multiple calls to this method (for different resultsets) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.

When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.

classmethod AppendTrainingDataFromFiles(pDirectory As %String = "", pLanguage As %String = "en", pClean As %Boolean = 1, pVerbose As %Boolean = 1) as %Status

Appends word frequency information drawn from the *.txt files in pDirectory to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different directories) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.

When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.

classmethod AppendTrainingDataFromQuery(pResultSet As %ResultSet, pLanguage As %String = "en", pClean As %Boolean = 1, pVerbose As %Boolean = 1) as %Status

Appends word frequency information drawn from the first column of the supplied %ResultSet to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different resultsets) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.

When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.

classmethod CleanWordList(pLanguage As %String = "en", pVerbose As %Boolean = 0, pOutputFile As %String = "", pFilter As %String = "") as %Status
Clears any identifiable compounds from the current decompound dictionary for pLanguage. This method should be run at least once between appending data to the training set through any of the Append* methods in this class and using the Decompound() method in a %iKnow.Stemmer object.
classmethod ClearTrainingData(pLanguage As %String) as %Status
Drops ALL training data for a given language. Use with care.
classmethod NeverSeparate(pLanguage As %String, pString As %String) as %Status
Marks pString as a character sequence that should never be split off and therefore never be returned as a compound element of its own. If this string was also part of the compound dictionary as a candidate, it will be removed automatically as if calling RemoveWord()
classmethod RemoveWord(pLanguage As %String, pWord As %String) as %Status
Removes a word from the compound dictionary for the supplied language. This word will no longer be treated as a valid compound element. Use this to clear the list of eventual composite words added previously.
FeedbackOpens in a new tab