From: david@edenroad.demon.co.uk
On Mon, 24 Oct 2005 13:21:51 -0500, Dick Silk - The Computer Tutor wrote:
> "David Mitchell" wrote in message
> news:pan.2005.10.24.08.56.46.725580@edenroad.demon.co.uk...
>> On Mon, 24 Oct 2005 00:30:16 -0500, Dick Silk - The Computer Tutor wrote:
>>
>>> "David Mitchell" wrote in message
>>> news:pan.2005.10.21.08.49.14.860838@edenroad.demon.co.uk...
>>>> On Fri, 21 Oct 2005 03:15:40 -0500, Dick Silk - The Computer Tutor
>>>> wrote:
>>>
>>> DM wrote:
>
>> Oh sure, you can do keyword searches (as they do in the example Janice
>> provided from dream research); but large-scale semantic analysis is not,
>> as far as I know, being performed anywhere.
>
> I see where you're getting now: you want to mass-feed large amounts of text
> data (of whatever classification) into a program and get data spat out.
> There is a program that does that, but I don't remember the name of it and I
> have NOT used it so I can't even speak for its validity. However, it uses
> basic grammar rules to determine if sentences are stating positive values or
> negative values, and goes from there.
It's not that easy. The CIA alone have spent quite a few million trying
to develop systems whch could read e-mails with any real degree of
semantic "comprehension".
AFAIK they're still trying.
>
> As to what I *can* tell you about what I did personally, it runs like this:
>
> 1) construct your surveys so that you ask for respondents to give you ONLY
> positive feedback in section A, and only NEGATIVE feedback in section B,
> etc. This helps you to automatically group the results into two or more
> categories for the presentation of your results / findings.
>
> 2) (this is the hard part) you need to *wash* the data. When washing data,
> you must be extremely careful to preserve the spirit and intent as much as
> humanly possible AND inform whomever you share results with that this data
> IS washed. (We always provide a copy of the original source text in case of
> any doubt or questions.) When your data is washed, it is:
> a) grammatically digestible. Rednecks and Ebonics and other
> bastardizations of the English language are edited into exact grammar when /
> where ever possible. If you can't translate it, leave it alone.
> b) spell checked to perfection, and converted to all upper-case in order
> to minimize / negate redundant data
>
> 3) you now have a data base of spellchecked upper case words that can be
> alpha-sorted and quantified. From THIS step, you can present the client(s)
> with, say, the top 100 themes that are being presented by the respondents.
> (Also, in step 2b, you can eliminate words of no significance, such as "the"
> "this" "that", etc., but I confess, there is an art to this step. "WELL",
> for instance, has *several* different meanings.)
>
So it's, essentially, keyword based, stripping the grammatical information
out. That's what I thought.
Very different to semantic comprehension of anecdotes.
> Now here you get into a straw-man argument -- and that is -- exactly what do
> you mean by "large" scale?
It's not what _I_ mean, it'a what _you_ meant when you said:
"There is a rather large rise these days in large-scale qualitative data
mining -- extremely lucrative, when applied correctly."
--
=======================================================================
= David --- If you use Microsoft products, you will, inevitably, get
= Mitchell --- viruses, so please don't add me to your address book.
=======================================================================
--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)
|
