From: armelasselin@hotmail.com   
      
   > there is an argument for introducing such a gap to "segment"   
   > tokens into smaller chunks for both performance and expressibility   
   > reasons.   
      
   could you elaborate on this segmentation mechanism?   
      
   In my lexer generator, the developer can introduce start states by himself   
   and 'cut' complex expressions into smaller expressions which still respect   
   AFD capabilities and introduce dynamic regular expressions where absolutely   
   necessary, for languages allowing dynamic string delimiters for example   
   (e.g. doc-strings like << END_OF_STR_MARKER, some lines then a line with   
   END_OF_STR_MARKER only on the line), languages such as ruby are very funny   
   from that point of view if I remember well.   
      
   Regards   
   Armel   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)