%iKnow.Stemming.DecompoundUtils
deprecated class %iKnow.Stemming.DecompoundUtils
This class contains utility methods to manage the word list used by the decompounding algorithm. Decompounding is about identifying the words making up a compound term, such as the words "thunder" and "storm" in the compound term "thunderstorms". It is used primarily for search purposes, allowing you to find records containing compounds of the search terms too. Lanugages like German, where compounding happens often, require decompounding support for a good search experience.
Training the decompounder
The decompounding algorithm supplied here requires a list of candidate words it will try to recognize in to-be-decompounded terms. These candidate words can be added through training the algorithm using any of the following methods, which accept free text that will be cut into candidate terms and then stripped of any recognizable compounds:
- AppendTrainingDataFromQuery() loads candidate words from a query result set
- AppendTrainingDataFromFiles() loads candidate words from plaintext files
- AppendTrainingDataFromDomain() loads candidate words from an iKnow domain
Alternatively, individual words can be added and removed through the AddWord() and RemoveWord() methods. Words that should never be separated (returned as a single word) can be registered through the NeverSeparate().
Invoking the decompounder
Decompounding is used by iFind indices who have their INDEXOPTION set to 2 (see also %iFind.Index.Basic). When subsequently adding records to such an indexed table, all words will be checked for compounding and additional index structures will be populated to allow retrieving records based on the compounding words.
The algorithm can also be invoked directly through a %iKnow.Stemmer object, should there be any requirement to find the compounding words of a given term (ie for debug purposes).
// simple training
do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "thunder")
do ##class(%iKnow.Stemming.DecompoundUtils).AddWord("en", "storm")
// invoke decompounder
write ##class(%iKnow.Stemmer).GetDefault("en", .tStemmer)
write tStemmer.Decompound("thunderstorms", .tWords)
zwrite tWords
Method Inventory
- AddWord()
- AppendTrainingDataFromDomain()
- AppendTrainingDataFromFiles()
- AppendTrainingDataFromQuery()
- CleanWordList()
- ClearTrainingData()
- NeverSeparate()
- RemoveWord()
Methods
Adds a word to the compound dictionary for the supplied language. The supplied word will be treated as a valid compound element the algorithm will no longer try to split in smaller elements. Optionally supply a positive integer frequency value to increase its weight when multiple options are available.
If pWord is also present in the list of strings never to split off through a call to NeverSeparate(), it will be removed from that list.
When performing a lot of manual updates, it is recommended to set pClean=0 and only run the CleanWords() method once after all additions, to verify if these new additions indicate particular existing words should be removed as they turn out to be compounds themselves.
Appends word frequency information drawn from an existing iKnow domain to the word dictionary for decompounding in this namespace. When pEntType=$$$ENTTYPEANY (default), the full sentence values (with literal info) will be used to derive words. To restrict this to concepts or relations only, use $$$ENTTYPECONCEPT resp. $$$ENTTYPERELATION.
Multiple calls to this method (for different resultsets) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.
When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.
Appends word frequency information drawn from the *.txt files in pDirectory to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different directories) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.
When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.
Appends word frequency information drawn from the first column of the supplied %ResultSet to the word dictionary for decompounding in this namespace. Multiple calls to this method (for different resultsets) will append to the existing info. Use ClearTrainingData() if you want to drop all existing data.
When pClean=1, the generated word list will automatically be cleaned after loading the new data through a call to CleanWordList(). You may use pClean=0 and only call CleanWordList() after appending training data from multiple sources, but it should be called once before decompounding any new words through the %iKnow.Stemmer object.