It is convenient to look at string edit distance as a string alignment problem. The example below should provide some insight into this:
String 1: p a r k s String 2: s p a r k Operation: i c c c c d
The operation is the character operation needed to align or transform string 1 into string 2. i is insert, c is copy, r is replace, and d is delete. (You may find other references using s (substitute) instead of r (replace).) The space at the beginning of string 1 or at the end of string 2 is called a gap.
For a given set of strings, there may be many ways to align the strings. For example, the strings abc123 and abcqwertabc123 have two obvious alignments:
a b c 1 2 3 a b c q w e r t a b c 1 2 3
and
a b c 1 2 3 a b c q w e r t a b c 1 2 3
The particular application would determine which alignment is preferred.
Different distance metrics are used to select the desired alignment. For example, the measurement used in the original example of the cost between parks and spark is the Levenshtein distance. In this metric the cost of replacing, adding or deleting a character is one whereas the cost of equal (or copied) characters is zero. (In the conversion of parks to spark, the cost of copying letters p a r k is zero.) The second alignment above (where abc123 has no gap) can be obtained using the Smith-Waterman distance.
For simple strings it is relatively easy to understand how to transform one string into another but with longer strings or very dissimilar strings it is not so clear. For example:
See Spot run We work and play
There are brute force techniques that consider all possible alignments and find the one(s) with the minimum cost. I refer you to a few good references on string edit distance and the associated dynamic programming algorithms for more information.
]]>Search for "book" on Amazon.com. You should see a page with a number of results / data records. Placing your mouse over each of the titles shows many similar urls, some of which I have pasted below:
| http://www.amazon.com/Harry-Potter-Half-Blood-Prince-Book/dp/0439785960/ref=pd_bbs_1?ie=UTF8&s=books&qid=1196364228&sr=8-1 |
| http://www.amazon.com/Little-Green-Book-Getting-Your/dp/0131576070/ref=pd_bbs_2?ie=UTF8&s=books&qid=1196364228&sr=8-2 |
| http://www.amazon.com/Learning-Curve-LC97727-Lamaze-Caterpillar/dp/B00009IMD8/ref=pd_bbs_3?ie=UTF8&s=baby-products&qid=1196364228&sr=8-3 |
| http://www.amazon.com/Book-General-Ignorance-John-Mitchinson/dp/0307394913/ref=pd_bbs_sr_4?ie=UTF8&s=books&qid=1196364228&sr=8-4 |
| http://www.amazon.com/The-Daring-Book-for-Girls/dp/B000UZJQNM/ref=sr_1_13?ie=UTF8&s=books&qid=1196364228&sr=8-13 |
| http://www.amazon.com/Inconvenient-Book-Solutions-Biggest-Problems/dp/B000WJVLLG/ref=sr_1_16?ie=UTF8&s=books&qid=1196364228&sr=8-16 |
We can easily see a pattern (the template) in those urls:
http://www.amazon.com/<title>/dp/<book id>/ref=<ref id>?ie=UTF8&s=<product type>&qid=1196364228&sr=8-<index>
Our job is to find an algorithm that will allow us to automatically discover that template.
(I should mention that if one continues to other pages in the search results that there are more differences in the URLs that leads to a different pattern. In order to keep the discussion simple, I will only use the first page's URLs. This demonstrates that a single page, even one that contains multiple records, might not contain enough information to deduce a complete template.
I also should mention that the ability to label the fields in the template as I did in the example above will not be covered (in the near future) in this series of postings. I will only state that there are techniques available in the literature on web wrappers for labeling fields.)
We are going to take two approaches to extract templates from URLs. The first will use traditional string edit distance algorithms and the second will exploit the fact that there is underlying structure in the url (i.e. the scheme, authority, path, query, and fragment).
]]>By visually looking at a rendered Google search results page, for example, a human can easily see similar, repeated sections. Although it is a bit more difficult, someone familiar with HTML can look at the source to that same Google page and find the repeated sections. Similarly, there are two general approaches to automatic template discovery -- one that relies on the rendered representation of the page and one that relies on the source of the page. A survey of the research literature does not show a distinct advantage of one technique over the other and in many cases the underlying algorithms are very similar. I am going to focus on the latter approach since that is where my experience lies.
]]>Much research has gone into extracting data (either structured or unstructured) from a web page using web wrappers. A web wrapper is a tool for "converting information implicitly stored as an HTML document into information explicitly stored as a data-structure for further processing" [W4F]. One particular type of automatically generated web wrapper uses template extraction. Template extraction is the inverse of creating a web page from a template -- for a given web page, attempt to deduce the template that was used to generate the page. If a template can be generated for any (template derived) web page, then the data that populates that template can be easily extracted.
Over the next few weeks I am going to focus on machine learing and other automatic web wrapper technologies in a series of postings.
]]>One quick example of all of this is AOP. I'm enamored with AOP but I cannot and will not use it in production software. The reason is that AOP simply does not have a maintenance model at all. In other words, I cannot take an AOP'ified application and future apply AOP on it (i.e. maintain it) and have understandable and determinable effects.
Editor's note: this is a stream of consciousness posting to get an idea down and is not complete or thorough in any way. But as always, comments are welcome.
]]>For interested parties, the discussion archives are located here rather than the broken link from the Live Clipboard site.
]]>I could elborate further on this topic but I'm still in the head shaking phase (i.e. "denial"). Yargh!
]]>My wife is a public high school science teacher and she will periodically ask the students to submit what they think that their grade is. Her findings match those of the article: poorer performing students tend to believe they are earning a grade significantly higher than is the case. This results in the odd implication that one cannot expect these students to contribute more effort to achieving a higher grade since they already believe that they're done what is necessary to achieve that higher grade. Without constant vigilance, one-on-one instruction, and continual family support to effectively force the student to exert the necessary effort to achieve the higher grade, the student will never know where the various rungs of the ladder lie and therefore will never know what subject mastery means.
I too have witnessed the same phenomenon working with developers over the years. Developers that produce code with the most logical errors that requires the majority of effort to maintain tend to be those that believe highly in their abilities. I have taken a few of these developers by the hand and walked them through a few full life-cycles of development demonstrating what is necessary to produce quality code (and to see the implications of poor quality code). Most of my efforts were greeted with incredulous responses and outright denial of the effort involved but there has been an individual or two that has gained more understanding and used that experience to take their craft to a new level.
]]>Check out StreamCruncher for more info.
]]>