53b2867d   
   From: BloodyShame@hotmail.com   
      
   There could be two modes for your application.   
      
   Mode 1:   
      
   Send all stock data.   
      
   Mode 2:   
      
   Send only the stock information that changed.   
      
   Mode 1 could be used when clients connect for the first time.   
      
   Mode 2 could be used from then on.   
      
   A huffman table could be used per stock price to compress the stock prices   
   themselfes... they will probably fluctuate or stay pretty much the same   
   around only a few values... so it might not require the full 8 bit, or 16   
   bit or 24 bit or 32 bit or 64 bit's that might be used for floats, doubles,   
   currency or what not.   
      
   I would round the stock prices to cents... something like: 60.50 becomes:   
   6050   
      
   The values themselfes can be stored/encoded/decode in/from/to 64 bit but   
   when compressed can be as little as a few huffman bits.   
      
   Kinda fun to do it that way.   
      
   Only thing left necessary to add is an identifier to indicate to which stock   
   the price belongs... and maybe add a date/time stamp too.   
      
   The date/time stamp could be inserted into the packet/data just once... the   
   identifiers per stock/price pair.   
      
   To benefit from huffman compression the packet/data layout could be as   
   follows:   
      
   date/time stamp   
   count (indicates how many identifiers/price pairs there are)   
   identifiers   
   huffman prices   
      
   This way the variable bit of the data can be all stuffed at the end of the   
   packet/data so it compresses nicely.   
      
   The identifiers themselfes can simply be 8 bit, 16, 32 bit, 64 bit whatever   
   is needed.   
      
   I will end with a little example:   
   date/timestamp: 5 march 2009, 16:04  (stored in binary)   
   count: 7 (stored in binary)   
   identifiers: 5, 7, 100, 200, 5400, 200, 1230 (stored in binary)   
   prices: 50.40, 20.40, 10.65, 1.34, 1.34, 64.22, 1.97 (stored in bits/huffman   
   codes)   
      
   So the idea's for you are:   
      
   1. Switch from text to binary which is much more efficient.   
   2. Exploit what is known about the data to design/make a better compression   
   scheme (than just a general compression scheme like gzip).   
   3. Use huffman for further compression of prices.   
   4. Use 8 bit, 16, 32, or 64 bit for identifiers depending on how many bits   
   they needed... a mode field could be added as well to switch between   
   8,16,32,64 bit mode.   
   5. Count field can probably be 8 bit, at most 16 bit.   
   6. Send the majority of the data once in mode 1. Send any additional   
   changes/info in mode 2.   
      
   Extra idea's include:   
      
   7. If this is for lan only, udp/multicast might be interesting.   
      
   Bottom line is:   
      
   This idea requires excellent programming skills, something you might lack ?   
   ) :) Or maybe not :)   
      
   This idea requires a lot of time to implement and might have short comings   
   not visibile in this design ? ;)   
      
   This idea is complex and might not be easy adjustable to changing   
   requirements ! ;)   
      
   This idea is binary based and could require byte swaps for little endian/big   
   endian systems.   
      
   If others are to interface with your systems they might not like such a   
   complex design and might not be able to pull it off ;)   
      
   Finally it remains to be proven if this idea would give better compression   
   then gzip and text what you are doing now.   
      
   Bye,   
     Skybuck ;)   
      
   --- SoupGate-Win32 v1.05   
    * Origin: you cannot sedate... all the things you hate (1:229/2)