-=> Maurice Kinal wrote to Tony Langdon <=-

 MK> -={ Thursday, 28 April 2016, 01:33:37.560060883 +1000 }=-

 MK> Hey Tony!

 TL> mainly to discuss a project with Maurice.

 MK> Sounds great.  How about trying this just to get us started out;

Looks like I've just brought an echo to this part of the world, as my uplink
didn't have it, but found it quickly from his uplink. :)

 MK> -------------- test raw pkt processing start
 MK> start_regexp="[0-9]{2} [[:alpha:]]{3} [0-9]{2}
 MK> [0-9]{2}:[0-9]{2}:[0-9]{2}\x00"

 MK> tr '\n' '\37' < raw.pkt | sed -r "s/${start_regexp}/\n&/g" | \
 MK>   sed -r -n "/^${start_regexp}/ p" | \
 MK>   gawk -F'\0' 'BEGIN { OFS = "\0" } { print $1, $2, $3, $4, $5 }' | \
 MK> temp.msg
 MK> -------------- test raw pkt processing end

 MK> The start_regexp variable is the ftn datetime stamp and I am using that
 MK> as the start of the message.  'tr' should replace any fake linefeeds
 MK> with unit seperators to keep offsets the same.  The first sed call
 MK> places a linefeed in front of all start_regexp it finds and then the
 MK> second sed prints all the lines that start with a start_regexp. gawk
 MK> prints out the resulting null delimited fields.  Finally the last sed
 MK> step if that information is wanted/needed).  The end result should be a
 MK> file with each line contatining exactly one message, null delimited
 MK> fields.  'wc -l < temp.msg' will yield exactly how many messages are in
 MK> that file.

It will take me some time (and probably some playing with real data and
scripts) to fully catch on.  It takes me a little while to understand the
nuances of scripts.

 MK> I think this is a great place to start and I'd appreciate any input
 MK> about the format of temp.data as well as anything that needs
 MK> troubleshooting, especially sed which can and does vary across
 MK> different systems.

The format is good for delineating messages, but do we want to keep track of
any individual components such as headers, kludge lines, etc?  If you want to
be able to rapidly process those, you might need to modify the format a bit. 
Of course, if they can be easuly obtained by grep (as I do with some
configuration files), then that's a non issue.  Again, I need to play with some
real data. :)



... Kitty litter: created by exploding catnip.
--- MultiMail/Win32 v0.49
 * Origin: Freeway BBS - freeway.apana.org.au (3:633/410)