docs.intersystems.com
Home  /  Application Development: Analytics Options  /  Using InterSystems IRIS Natural Language Processing (NLP)  /  Blacklists

Using InterSystems IRIS Natural Language Processing (NLP)
Blacklists
[Back]  [Next] 
 InterSystems: The power behind what matters   
Search:  

A Blacklist is a list of entities that you do not want a query to return. For example, if your source texts include greetings and salutations, you might want to put “many thanks”, “best regards”, and other stock phrases with no real information content in your Blacklist. A Blacklist might also be used to suppress top concepts that are too general and widespread to be of interest when analyzing query results.
Note:
When displaying results of a query that applied a Blacklist, it is important to note that use of a Blacklist silently changes the query results by suppressing some information. Make sure the Blacklists used are relevant for the data contents, for the user looking at the query results, and for the context in which query results are displayed.
Creating a Blacklist
You can define a Blacklist that is assigned to a specific domain, or define a Blacklist that is domain-independent (cross-domain) and can be used by any domain in the current namespace. You can define a Blacklist in two ways:
You can use the CopyBlackLists() method of the %iKnow.Utils.CopyUtils class to copy all defined Blacklists in a domain to another domain.
The Domain Explorer and the Basic Portal user interfaces support the use of Blacklists.
The following example creates a domain-specific Blacklist and populates it with elements. It then lists all Blacklists for the domain, and all of the elements in this domain. Finally, it deletes the blacklist.
DomainGetOrCreate
     SET domn="mydomainwithbl"
     SET domId=##class(%iKnow.Domain).GetOrCreateId(domn)
     SET domoref=##class(%iKnow.Domain).%OpenId(domId)
     WRITE "The ",domn," domain with domain ID ",domId,!
CreateBlackList
  SET blname="AviationBlacklist"
  SET blId=##class(%iKnow.Utils.MaintenanceAPI).CreateBlackList(domId,blname,
       "Aviation non-mechanical terms Blacklist")
PopulateBlackList
  SET black=$LB("aircraft","airplane","flight","accident","event","incident","pilot",
                "student pilot","flight instructor","runway","accident site","ground","visibility","faa")
  SET ptr=0
  FOR x=0:1:100 {
   SET moredata=$LISTNEXT(black,ptr,val)
    IF moredata=1 {
      SET stat=##class(%iKnow.Utils.MaintenanceAPI).AddStringToBlackList(domId,blId,val)
    }
    ELSE { WRITE x," entities in Blacklist",!!
           GOTO ListBlacklist }
  }
ListBlacklist
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).GetBlackLists(.bl,domId,0)
  SET i=1
  WHILE $DATA(bl(i)) {
       WRITE $LISTTOSTRING(bl(i),",",1),!
       SET i=i+1 }
  WRITE "Printed the ",i-1," Blacklists",!
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).GetBlackListElements(.ble,domId,blId)
   /* IF stat=1 {WRITE "success",!}
  ELSE {WRITE "GetBlackListElements failed" QUIT } */
  SET j=1
  WHILE $DATA(ble(j)) {
       WRITE $LISTTOSTRING(ble(j),",",1),!
       SET j=j+1 }
  WRITE "Printed the ",j-1," Blacklist elements",!
CleanUp
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).DropBlackList(domId,blId)
  IF stat=1 {WRITE "Blacklist deleted",!}
  ELSE {WRITE "DropBlackList failed" QUIT }
The CreateBlackList() method allows you to specify both a name and a description for your Blacklist. A Blacklist name can be any valid string of any length; Blacklist names are case-sensitive. The name you assign to a Blacklist must be unique: for a domain-specific Blacklist it must be unique within the domain; for a cross-domain Blacklist it must be unique within the namespace. Specifying a duplicate Blacklist name generates ERROR #8091. The Blacklist description is optional; it can be a string of any length.
Blacklists and Domains
Each Blacklist you create can either be specific to a domain, or can be cross-domain (domain-independent) and usable by any domain in the current namespace:
To populate a domain-specific Blacklist, you can use either AddEntityToBlackList() or AddStringToBlackList(). To populate a cross-domain Blacklist, you can only use AddStringToBlackList().
GetBlackListElements() returns the empty string for the entUniId value for a cross-domain Blacklist.
The following example creates and populates two Blacklists, a domain-specific blacklist (AviationTermsBlacklist) and a cross-domain blacklist (JobTitleBlacklist). The GetBlackLists() method returns both blacklists, because the pIncludeCrossDomain boolean is set to 1. Note that GetBlackLists() returns the blacklist Id for the cross-domain blacklist as a negative integer. 
DomainGetOrCreate
     SET domn="mydomainwithbl"
     SET domId=##class(%iKnow.Domain).GetOrCreateId(domn)
     SET domoref=##class(%iKnow.Domain).%OpenId(domId)
     WRITE "The ",domn," domain with domain ID ",domId,!
CreateBlackList1
  SET blname="AviationTermsBlacklist"
  SET blId=##class(%iKnow.Utils.MaintenanceAPI).CreateBlackList(domId,blname,
       "Common aviation terms Blacklist")
PopulateBlackList1
  SET black=$LB("aircraft","airplane","flight","accident","event","incident","airport","runway")
  SET ptr=0
  FOR x=0:1:100 {
   SET moredata=$LISTNEXT(black,ptr,val)
    IF moredata=1 {
      SET stat=##class(%iKnow.Utils.MaintenanceAPI).AddStringToBlackList(domId,blId,val)
    }
  }
  WRITE "Blacklist ",blname," populated",!
CreateBlackList2
  SET bl2name="JobTitleBlacklist"
  SET bl2Id=##class(%iKnow.Utils.MaintenanceAPI).CreateBlackList(0,bl2name,
       "Aviation personnel Blacklist")
PopulateBlackList2
  SET jobblack=$LB("pilot","copilot","student pilot","flight instructor","passenger")
  SET ptr=0
  FOR x=0:1:100 {
   SET moredata=$LISTNEXT(jobblack,ptr,val)
    IF moredata=1 {
      SET stat=##class(%iKnow.Utils.MaintenanceAPI).AddStringToBlackList(0,bl2Id,val)
    }
  }
  WRITE "Blacklist ",bl2name," populated",!!
ListBlacklists
  SET pIncludeCrossDomain=1
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).GetBlackLists(.bl,domId,pIncludeCrossDomain)
  SET i=1
  WHILE $DATA(bl(i)) {
     IF $LIST(bl(i),1)<0 {
        WRITE "cross-domain:",!,$LISTTOSTRING(bl(i),",",1),! }
     ELSE { WRITE "domain-specific:",!,$LISTTOSTRING(bl(i),",",1),! }
  SET i=i+1 }
  WRITE "Printed the ",i-1," Blacklists",!!
CleanUp
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).DropBlackList(domId,blId)
  IF stat=1 {WRITE "domain Blacklist deleted",!}
  ELSE {WRITE "first DropBlackList failed" }
  SET stat=##class(%iKnow.Utils.MaintenanceAPI).DropBlackList(0,bl2Id)
  IF stat=1 {WRITE "cross-domain Blacklist deleted",!}
  ELSE {WRITE "second DropBlackList failed" }
Queries that Support Blacklists
The following query methods provide a parameter to specify Blacklists. You can specify multiple Blacklists to any of these methods by specifying the Blacklist Ids as elements of a %List structure, using the $LISTBUILD function. You specify a domain-specific blacklist as a positive integer blacklistId value; you specify a cross-domain blacklist as a negative integer blacklistId value.
Entity Queries:
Sentence Queries:
Source Queries:
Blacklist Query Example
Refer to A Note on Program Examples for details on the coding and data used in the examples in this book.
The following example suppresses non-mechanical aviation terms that are too general to be of interest. It uses CreateBlackList() to create a Blacklist, uses AddStringToBlackList() to add entities to the Blacklist, then supplies the Blacklist to the GetTop() method: 
#Include %IKPublic
DomainCreateOrOpen
  SET dname="mydomain"
  IF (##class(%iKnow.Domain).Exists(dname))
     { WRITE "The ",dname," domain already exists",!
       SET domoref=##class(%iKnow.Domain).Open(dname)
       GOTO DeleteOldData }
  ELSE 
     { WRITE "The ",dname," domain does not exist",!
       SET domoref=##class(%iKnow.Domain).%New(dname)
       DO domoref.%Save()
       WRITE "Created the ",dname," domain with domain ID ",domoref.Id,!
       GOTO ListerAndLoader }
DeleteOldData
  SET stat=domoref.DropData()
  IF stat { WRITE "Deleted the data from the ",dname," domain",!!
            GOTO ListerAndLoader }
  ELSE    { WRITE "DropData error ",$System.Status.DisplayError(stat)
            QUIT}
ListerAndLoader
  SET domId=domoref.Id
  SET flister=##class(%iKnow.Source.SQL.Lister).%New(domId)
  SET myloader=##class(%iKnow.Source.Loader).%New(domId)
CreateBlackList1
  SET blname="AviationTermsBlacklist"
  SET blId=##class(%iKnow.Utils.MaintenanceAPI).CreateBlackList(domId,blname,
       "Common aviation terms Blacklist")
PopulateBlackList
  SET black=$LB("aircraft","airplane","flight","accident","event","incident","pilot","airport",
                "student pilot","flight instructor","runway","accident site","ground","visibility","faa")
  SET ptr=0
  FOR x=0:1:100 {
   SET moredata=$LISTNEXT(black,ptr,val)
    IF moredata=1 {
      SET stat=##class(%iKnow.Utils.MaintenanceAPI).AddStringToBlackList(domId,blId,val)
    }
  }
  WRITE "Blacklist ",blname," populated",!

QueryBuild
   SET myquery="SELECT TOP 100 ID AS UniqueVal,Type,NarrativeFull FROM Aviation.Event"
   SET idfld="UniqueVal"
   SET grpfld="Type"
   SET dataflds=$LB("NarrativeFull")
UseLister
  SET stat=flister.AddListToBatch(myquery,idfld,grpfld,dataflds)
      IF stat '= 1 {WRITE "The lister failed: ",$System.Status.DisplayError(stat) QUIT }
UseLoader
  SET stat=myloader.ProcessBatch()
      IF stat '= 1 {WRITE "The loader failed: ",$System.Status.DisplayError(stat) QUIT }
SourceCountQuery
  SET numSrcD=##class(%iKnow.Queries.SourceQAPI).GetCountByDomain(domId)
  WRITE "The domain contains ",numSrcD," sources",!
TopEntitiesQuery
  DO ##class(%iKnow.Queries.EntityAPI).GetTop(.result,domId,1,20,"",0,0,0,0,$LB(blId))
  WRITE "NOTE: the ",blname," Blacklist",!,
        "has been applied to this list of top entities",!
  SET i=1
  WHILE $DATA(result(i)) {
       SET outstr = $LISTTOSTRING(result(i),",",1)
         SET entity = $PIECE(outstr,",",2)
         SET freq = $PIECE(outstr,",",3)
         SET spread = $PIECE(outstr,",",4)
       WRITE "[",entity,"] appears ",freq," times in ",spread," sources",!
       SET i=i+1 }
  WRITE "Printed the top ",i-1," entities"

[Home] [Back]  [Next]  [Top of Page] 
Copyright © 1997-2018, InterSystems Corporation   
Content for this page loaded from GIKNOW_blacklist.xml on 2018-02-01 14:30:12