This example takes two texts as input and outputs one merged text marked with what was deleted and what was added. Take a look and feel free to download the source code. This also inlcludes the Levenshtein algorithm source code.

Character sets

Other character sets have been defined. The International Organization for Standardization came up with the ISO-8859 series to satisfy this shortcoming and defined some different character sets using 8 bits per character. Microsoft developed its own schemes too like cp-1252 for instance, along with some others (like IBM). Also some local institutes needed to create encoding to facilitate the needs of their native language which the standardized sets still lacked. This introduced the problem of multiple interpretations of numbers. What characters are they mapped to? What character set do I need to use to decode 65 for instance? Does 65 mean A or a or ä or ç or R or…

With the ISO 8-bit character sets, 256 characters were possible (190 characters without control characters, etc). This was sufficient as the sets covered the top ten most used languages. Still for instance Chinese and Japanese we not at all covered. This is where the Unicode Transformation Format (UTF) comes into play, developed by the Unicode Consortium. Unicode are multi-byte character sets which means that per character one, two, three or four 8-bit bytes are used to identify that character. This UTF-8 scheme is nowadays the most commonly used set. It is backward compatible with ASCII and with over 30.000 characters it is able to represent most of the living languages with a single code. UTF contains information of how to convert lowercase characters to uppercase and vice verse which is not the same or even symmetric for every character, and it has sorting rules. E.g. traditional Spanisch knows a single ch character which is sorted between the c and d and in Greek the uppercase Σ in lowercase is a σ, but if it is the last character of a word the lowercase of Σ is ς.

Problems or challenges

When programming web applications for instance a programmer often has to work with a database, file system, web server and one or several browsers. These systems and the data traffic between them (protocols) need to be tuned so they use the same character sets. If this is not the case errors can occur. For instance, if you send “Hellø world” from the database in UTF-8 to the browser, which interprets the bit stream it is receiving as CP-1252, the string is displayed as “Hell? world” and you have probably seen the € sign appear as � because of this error. The solution to this problem, of course is to keep all character sets the same when different systems (applications) communicate with each other.

Copyright sign, it's Numeric Character Reference and alias

A way to prevent this problem from occurring is to obey the HTML standard which requires characters not defined in certain character sets to be converted to their Numeric Character Reference (HTML entity). That doing this is mandatory for element attribute values is fairly unknown. This is done by code points denoted by &#[code]; where [code] is replaced by a number. For most used characters an alias is available. E.g. the equivalent for & is &, ë has ë, a space is denoted by   and the copyright sign has equivalent © with alias ©. The codes (digits) and aliases itself only contain characters from the ASCII set, meaning an entire SGML (including HMTL, XML, etc) document can be composed of only ASCII characters. PHP has a built in function that searches and converts such characters in an input. Also, a nice tool to look up entities is available at LeftLogic.com.

The are several ways to compare texts and find differences and similarity scores. For this article the similarity scores are not relevant because these scores are just numbers. We are interested in what is added, deleted or substituted in the transformation from text A to text B. In other words, we would like to mark the minimal number of primitive operations needed to transform A to B. To do this we’ll need the basics from the classic computer science problem: the longest common subsequence problem. The technique described hereunder which is derived from this problem is the Levenshtein distance algorithm. The algorithm was developed by Vladimir Levenshtein to replace the Hamming distance. The result of Levenshtein’s algorithm is exactly the minimal number of operations, but you can use the unpolished  result of this algorithm to determine what parts of text were added, deleted or substituted. Originally this is done per character, but with a little tweak this can be changed to a per word, line or paragraph level function. When you’ve read this article you will know how this works (and the demo, which is referred to at the end). This demo takes two texts as input and outputs what was added, deleted and replaced.

For this article and the demo I used search results for some inspiration. A nice explanation already there is this description of the Levenshtein algorithm, as well as the Wiki page on it. For this article, let’s use two sample lines: “The brown dog jumped away from the sprinkler” and “The dog ran towards the green sprinkler”. Now we want to know with words were added, deleted or replaced in the transition from the first sentence to the second. To do this, let’s take a closer look on how the iterative process of the Levenshtein algorithm is executed.

1. The first step is to contruct a matrix of n+1 by m+1, where n is the number of words in the first line and m the number of words in the second line.
2. Secondly, fill the first row and column with (from top left to bottom or right) zero to respectively m and n.
3. Now for each n, evaluate each m. If  the evaluated word of n matches m, the cost is 0, otherwise it’s 2.
4. Fill out cell (n, m) having (n, m) is the minumum of:
   • The value of the cell above + 1
   • The left neighbour cell value + 1
   • The above left cell value + cost

For the Levenshtein distance it stops right here, when the iteration is completed. The distance is the value in the lower right cell. Here we are not interested in the Levenshtein distance itself, but in the matrix we’ve just constructed. The lowest cost route from bottom right to top left reveals information on what words have been added, deleted and/or substituted. To show how, we need to costruct the matrix with the algorithm above.

For the first cell (*), the words “The” versus “The” are equal, so the cost is 0. Now the minumum of the cell above + 1 (2), the cell to the left + 1 (2) and the above left + cost (0), is the latter one.

The cell underneath it (**) has cost 1 (“The” versus “dog”) and gets a value equal to the minimum of the cell above + 1 (1), the cell to the left + 1 (3) and the above left + cost (2), which is 1.

Continue to fill this out and the table will look like this:

Now we need to find the lowest cost path from the bottom right to the zero in the top left. To do this simply jump to the cell with the lowest value adjacent to the current cell (to left, above or diagonal). Jumping diagonal is only allowed if the words are the same (column and row). If two or more have the same (lower) value, the priority of choosing a route is to try diagonal first, then either left or above. So, from the bottom right 5 we start the route to the diagonally adjacent 5 (because ’sprinkler’ equals ’sprinkler’). From the 5  the next step would be the lower 4 above it, then the diagonally adjacent 4, etc… The route table will look like this:

After the route is calculated, every step in it tells something about the operations needed (from top left to bottom right).

• Every diagonal step (not increasing the score) tells us nothing happened. E.g. the first step from 0 to 0 tells us “The” from the first line stays “The” in the second line.
• Every horizontal step means a word is deleted. E.g. the step from 0 to 1 tells us “brown” was deleted at this point.
• Every vertical step means a word is added. E.g. the step from 4 to 5 tells us “green” was added at this point.
• Every diagonal step having the score increased means a word is substituted (added and deleted) at this point. E.g. the step from 2 to 3 tells “away” is substituted by “ran”. This is an illegal opperation in the detection of addition and deletion of words.

This way a text indicating the operations can be constructed:

The brown dog jumped ranaway towardsfrom the green sprinkler.

Red indicates deletion, green for insertion and a red and green pair indicates substitution.

Of course some optimalizations can be performed. The above for instance does give a good indication of what happened to the text. Imagine a larger text than just these lines and the relevance of changes are marked this way will become more obvious. But because only the primitive operations are detected at the word level, word groups are not taken into account. In this example for instance, the algorithm would be better if it marked “jumped away from” as replaced by “ran towards” instead of each seperate word as it does now:

The brown dog ran towardsjumped away from the green sprinkler.

This operation is not that hard to implement, simply replace subsequent differing operations by substitute operations.

An implementation of this algorithm, with the optimalization patch suggested here, is now available as an example. Source code (PHP) is available as well.

And about the featured picture on top: there are 12 differences to spot.

At the tester links to PCRE documentation (Perl Compatible Regular Expressions) are available. Also I would like to point to a handy cheat sheet on general regular expressions by Dave Child and another cheat sheet on JavaScript regular expressions from Visibone.

This tool is accessible again at regex.edesign.nl and can be downloaded there as wel (use the src link).