From: kaz@kylheku.com   
      
   On 2011-12-07, Andrew Tomazos  wrote:   
   > Summary: We want to find out how often a given token appears in a   
   > random stream formed by concatenating randomly chosen strings from a   
   > given set of strings.   
      
   > (Note hits can overlap each other)   
      
   But tokens do not overlap, so you're not actually extracting tokens.  Using C   
   tokens as an example, the C token >>= is one hit, not four.  The longest match   
   calls for extracting three characters and moving on.   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)