Grammalecte  Check-in [b5ff33ce96]

Overview
Comment:[core] token neutralization
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | core | rg
Files: files | file ages | folders
SHA3-256: b5ff33ce96fa4a652c64d2891a44bcafec39f06f628d9640e21f0d30b47771ef
User & Date: olr on 2018-06-26 09:58:37
Other Links: branch diff | manifest | tags
Context
2018-06-26
12:34
[core] gc engine: new char to merge tokens check-in: 4e6d599062 user: olr tags: core, rg
09:58
[core] token neutralization check-in: b5ff33ce96 user: olr tags: core, rg
09:57
[fr] conversion: regex rules -> graph rules check-in: 1f0c86fd07 user: olr tags: fr, rg
Changes

Modified gc_core/py/lang_core/gc_engine.py from [990cf356cf] to [f034e8dbe9].

   899    899           if sWhat == "*":
   900    900               # purge text
   901    901               if nTokenRewriteEnd - nTokenRewriteStart == 0:
   902    902                   self.lToken[nTokenRewriteStart]["bToRemove"] = True
   903    903               else:
   904    904                   for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
   905    905                       self.lToken[i]["bToRemove"] = True
   906         -        elif sWhat == "_":
          906  +        elif sWhat == "=":
   907    907               # merge tokens
   908    908               self.lToken[nTokenRewriteStart]["nMergeUntil"] = nTokenRewriteEnd
   909    909           elif sWhat == "!":
   910    910               # immunity
   911    911               if nTokenRewriteEnd - nTokenRewriteStart == 0:
   912    912                   self.lToken[nTokenRewriteStart]["bImmune"] = True
   913    913               else:
   914    914                   for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
   915    915                       self.lToken[i]["bImmune"] = True
          916  +        elif sWhat == "_":
          917  +            # neutralized token
          918  +            if nTokenRewriteEnd - nTokenRewriteStart == 0:
          919  +                self.lToken[nTokenRewriteStart]["sNewValue"] = "_"
          920  +            else:
          921  +                for i in range(nTokenRewriteStart, nTokenRewriteEnd+1):
          922  +                    self.lToken[i]["sNewValue"] = "_"
   916    923           else:
   917    924               if sWhat.startswith("="):
   918    925                   sWhat = globals()[sWhat[1:]](self.lToken)
   919    926               bUppercase = bUppercase and self.lToken[nTokenRewriteStart]["sValue"][0:1].isupper()
   920    927               if nTokenRewriteEnd - nTokenRewriteStart == 0:
   921    928                   sWhat = sWhat + " " * (len(self.lToken[nTokenRewriteStart]["sValue"])-len(sWhat))
   922    929                   if bUppercase: