eDesign.nl http://www.edesign.nl Thoughts and concepts on software development Fri, 08 Apr 2011 13:35:29 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 HTML5 Canvas experiments http://www.edesign.nl/2011/03/23/html5-canvas-experiments/ http://www.edesign.nl/2011/03/23/html5-canvas-experiments/#comments Wed, 23 Mar 2011 21:45:11 +0000 Jurgen http://www.edesign.nl/?p=841

HTML5 Canvas element

HTML5 is supported by more and more browsers. It has a nice feature called the ‘canvas’ element. It is an element on which you can ‘paint’ using e.g. JavaScript. I experimented with this new canvas element. You can view the experiments (clock, 3D-axis and rotation, ray tracer) and view the source too! The ray tracer should be activated in Chrome or Safari as Firefox and Internet Explorer tend to be (very) slow.

]]> http://www.edesign.nl/2011/03/23/html5-canvas-experiments/feed/ 0 Avoid becoming a spam source http://www.edesign.nl/2010/08/21/avoid-becoming-a-spam-source/ http://www.edesign.nl/2010/08/21/avoid-becoming-a-spam-source/#comments Fri, 20 Aug 2010 22:03:17 +0000 Jurgen http://www.edesign.nl/?p=818 The number one rule in programming is not to trust external data, especially not user data. One of the reasons not to trust user data is the threat of hackers searching for injection possibilities. When you start develloping web applications you’ll soon learn the dangers of SQL injection. This might be the most important form of injection, but by far it is not the only one out there: cross site scripting is another good example. If your application sends out mail, you might want to make sure you sanitize user input as well as the application could be a magnet for spammers who’ll send spam mail using your IP.

Mail is formatted using MIME, a content discription format. In this format, all properties of an email are specified in the header of the message. Examples include ‘From:’, ‘Subject:’ and ‘Date:’ for instance and also ‘To:’ and ‘Cc:’ are headers in the MIME.

Mail is generally sent using SMTP. In contrast to MIME, the message format description, SMTP is a protocol. One can tell a mail server (SMTP) the recipients of an email using this protocol. Therefor these recipients set with SMTP can be different than the ones specified in the MIME message (ever noticed a header like ‘undisclosed recipients’, which is clearly not a valid email address).

Because it is redundant at a high programming level, like PHP, to set recipients twice (once in MIME and once in SMTP), methods are defined for sending mail. PHP’s mail() function for example accepts ‘to’, ’subject’ and ‘message’ parameters. Also, since the number of different MIME headers are virtually infinite, it also accepts an ‘additional headers’ parameter. This is one part of introducing a potential security breach.

This actually is a disputable ‘violation’ of the headers. As you might have noticed, this way you’ll have to add Bcc recipients to the message’s header. As by definition, you don’t want other recipients to be able to see Bcc recipients as part of the message they recieve so the smtp client will have to strip it before sending.

The other part is in the function’s handling of the additional headers parameter. Of course it has to be possible to send mail using recipients specified as Cc or Bcc instead of To. To this end mail() examines the parameter for these headers and extracts them. As headers are specified per line in the MIME format, you can inject headers to the ‘additional headers’ parameter if one has access to a field in it, let’s say ‘Date:’ or ‘From:’. Headers are separated using a line feed character (LF), \n in PHP or %0A url encoded.

A MIME email usually has the following layout:

To: you@yourdomain.com
From: me@mydomain.com
Subject: Hi there

I'm just mailing to say hello!

Now if you have your website visitors send you mail and have them set the ‘From:’ header contents for instance, they might fill out an arbitrary email address and add ‘%0ABcc: extra@email.com; extra2@email.com;… (etc)’. Now also an Bcc header is set and your mail form just became a spam robot. Workaround is simple: don’t allow newlines in MIME header fields! Another post about this common pittfall can be found on websiterepairs.net.

]]> http://www.edesign.nl/2010/08/21/avoid-becoming-a-spam-source/feed/ 0 Decoration aid http://www.edesign.nl/2010/08/03/decoration-aid/ http://www.edesign.nl/2010/08/03/decoration-aid/#comments Tue, 03 Aug 2010 14:36:28 +0000 Jurgen http://www.edesign.nl/?p=783 When it comes to designing websites choosing what colors to use is one of the first steps. Nowadays online applications often let users personalize the look and feel of the site by offering a way to customize the colors the website wears. JavaScript and HTML can be used effectively to this end. A problem is arising for component developers using JavaScript: what framework to use? There is MooTools, ProtoType, YUI library, jQuery, etc, etc… This is the reason I developed a standalone color picker. It also works with every JavaScript framework, as it does not depend on one or interferes with one. It is one single .js file, no extra images, css, javascript or whatsoever.

Using it is easy. Basic usage has tree steps.

  1. Include the colorpicker.js in the <head> of the HTML file.
  2. Add the class ‘colorpicker’ to the <input> tag you use in a form for the color to choose.
  3. On document load, initialize the color picker with: ColorPicker.init();

It is as simple as that. Example:

<hmtl>
 <head>
  <title>ColorPicker example</title>
  <script type="text/javascript" src="colorpicker.js" />
 </head>
 <body onload="ColorPicker.run();">
  <form>
   Choose a color: <input type="text" name="color" class="colorpicker" />
   <input type="submit" />
  </form>
 </body>
</html>

Multiple pickers can be added by inserting more <input> tags with the classname ‘colorpicker’ which the standard ColorPicker.init() method all transforms into color pickers.

The manual way to initialize color pickers is by calling:

new ColorPicker(element);

You’ll need to repeat that line for every color picker you have if you don’t want to use ColorPicker.init().

Customizing

The look and feel of the color picker can be modified by adding an configuration hash table as argument to the init() function. This way you can change the colors, borders and language to your needs. For instance, in Dutch the word ‘Cancel’ means ‘Annuleren’. So if I want a color picker on a Dutch website I would change the init() call to:

ColorPicker.init({'language': {'cancel': 'Annuleren'}})

This way you can set parts of the configuration (e.g. only the word ‘cancel’) or the entire configuration by extending the hash table.

Download

You can get the latste version of the ColorPicker from http://projects.edesign.nl/colorpicker/latest/colorpicker.js. The project page is available at http://projects.edesign.nl/colorpicker/.

]]> http://www.edesign.nl/2010/08/03/decoration-aid/feed/ 0 No robots beyond this line http://www.edesign.nl/2010/04/23/no-robots-beyond-this-line/ http://www.edesign.nl/2010/04/23/no-robots-beyond-this-line/#comments Fri, 23 Apr 2010 17:43:30 +0000 Jurgen http://www.edesign.nl/?p=729 Customs passport checkOnline communities are hot. Globally recognized examples are easy to give: websites like Facebook, LinkedIn and are very popular, manufacturers have online fora to have their customers support each other, newspapers let you leave comments on their articles on their websites and you can share everything with tools like Delicious, Digg and Reddit. This development on the Internet supports new possibilities which were unknown before. Of course this also counts for rogues. Spam is a commonly known phenomenon and global annoyance. Beside spamming unwanted messages by mail, spamming the comment boxes and fora is an issue web programmers have to deal with too. Spamming often is automated and this is a feature which can be used to counter spam. The goal is to identify a messenger being human or robot.

For this purpose the captcha was invented. Besides the fact that captcha sounds nice enough to be a buzz word it actually is short for Completely Automated Public Turing test to tell Computers and Humans Apart, although this is a bit contrived. This means that a captcha is a challenge response mechanism but it doesn’t need to be in the form of an image depicting distorted text which has to be copied in a text box which is the most common form of captchas. Creative new captchas can be found, like a transistor image which has to be read.

Technology

Wikipedia mentions a couple of features a captcha must have to qualify as one. A captcha is a challenge-response test between a system and a user of which

  1. current software is unable to solve accurately,
  2. most humans can solve, and
  3. does not rely on the type of CAPTCHA being new to the attacker.

The first one remarks a captcha as temporal. This means that with increasing processing power and increasing insight in artificial intelligence challenges we now consider to be captchas might not be in the (near) future. Philosophically, this means that captchas are a temporary phenomenon because mankind will eventually be able to build robots which are at least as intelligent as humans are. But for now they’ll do.

The second one emphasises the differences we see between humans and robots. This is actually quite an interesting point because mankind actually admits its current limitations in its own intelligence being unable to write software which is able to solve ‘puzzles’ which are easy to solve for humans.

Which bridges to the last bullet on the list. Of course you could just add a simple checkbox labeled “Do not check this box if you are human”. No attacker would think of a spam protection this weak but because of that it might just work. The robot stumbling across your comment submission form does not expect such a protection and therefor cannot bypass it. Although this does not qualify as a captcha because the novelty of the protection will only make the attacker look into it to solve it in an instance. Of course a captcha can be a captcha when it is an innovative challenge but it should not rely on being unknown. Security through obscurity is not security at all.

Processing power and algorithms

As time passes by technology advances resulting in more processing power in both processor quantity and quality and more mathematical developments. These are the engine propelling artificial intelligence development. On the other hand, having this field of computer science developing the struggle for making captchas to tell humans and computers apart becomes harder and harder. Who knows when software becomes as advanced as to be able to not only solve puzzles or identify puzzle types but to be really intelligent and thereby be able to find ways to solve a puzzle without knowing the puzzle’s rules in advance? 20 years? 30? 5?!

Usefulness

Captcha’s can be useful too. The Recaptcha program for instance helps digitizing books by showing snippets scanned from books which they are unable to parse with their OCR software. This way the snippets are ‘decyphered’ by hundreds of people insuring accuracy and helping the system in which it is implemented to be bot-free.

Other examples of captchas might be usefull to the website’s theme such as a math class forum’s captcha challenging users with simple math like 3 + 5 = or 4 * 3 =. Another example of such a situated captcha is Adafruit’s. Adafruit is a website and webshop on the Arduino, which is a do it yourself programmable breadboard. You’ll need to ‘read’ the resistor’s value in order to post a comment.

Adafruit's resistor captcha

Adafruit's resistor captcha makes you slide the four sliders to match the color code on the resistor above.

]]> http://www.edesign.nl/2010/04/23/no-robots-beyond-this-line/feed/ 0 The Game of Office Decoration http://www.edesign.nl/2010/04/22/the-game-of-office-decoration/ http://www.edesign.nl/2010/04/22/the-game-of-office-decoration/#comments Thu, 22 Apr 2010 17:48:17 +0000 Jurgen http://www.edesign.nl/?p=730 Life LogoThis week I had a discussion with a colleague of mine about decoration for the new office rooms we hired. We thought of an Arduino based project, because it would just be cool to have some fun with the gadget.

We could make a giant board filled with buttons with a led in them. You’d press a button to toggle it. This board could be controlled by an Arduino and display several things on it. A clock for instance or news headlines from an arbitrary RSS feed or… since they’re buttons, Conway’s Game of Life!

Conway’s Game of Life is a cellular automaton. It is a discrete model which can used in studies like mathematics, theoretical biology and physics. The rules are pretty simple. There is a grid filled with cells which are alive or dead. In each ‘turn’ (iteration) the number of alive neighbors are counted for every cell. If a cell is currently dead and it has exactly three alive neighbor cells that cell will be alive in the next iteration. If a cell is alive and it has less than two or more than three alive neighbors it dies. This way you can model certain patterns with it, run the simulation game for a couple of iterations and see what it does.

I found implementations in Java and C#. I made one in JavaScript using HTML for the gui. Check it out and play with it. There is a lot of information on this game on Wikipedia, including all kinds of ‘famous patterns‘ and many links.

Conway’s Game of Life example is viewed best with Google Chrome or Safari (because of WebKit). It does work in other browsers too but way slower, especially Internet Explorer isn’t very good in processing the Javscript game.

UPDATE: I found another site explaining the mathematical interesting nature of Conway’s Game of Life.

]]> http://www.edesign.nl/2010/04/22/the-game-of-office-decoration/feed/ 0 Math behind a world sunlight map http://www.edesign.nl/2009/05/14/math-behind-a-world-sunlight-map/ http://www.edesign.nl/2009/05/14/math-behind-a-world-sunlight-map/#comments Thu, 14 May 2009 00:59:29 +0000 Jurgen http://www.edesign.nl/?p=434 World sunlight map fractionMy neighbour has a map of the world on the wall. You can see it from the street in front of his house. It has a backlight but that only illuminates half of the map. The transition from day to night is shaped like a sine wave most of the time. It actually is a physical world sunlight map. Of course, you can simulate this with a computer too. There even is an instance using Google maps.

As many roads lead to Rome multiple ways are possible to this simulation. One could model the sun, earth, maybe more and start ray tracing. This approach would include solar eclipses but is quite heavy by means of the load on the processor. Because of the number of calculations involved in ray tracing is quite high. The way I choose to describe fully in this article is one close to it. Using vectors pointing from a sphere (earth) to a point (sun) I map a Mercator projected map of the world on the sphere. The challenges included are the yearly orbit of earth around the sun and it’s 23.5° tilted 24 hour spin.

About a decade ago I made a similar program in highschool. I didn’t know vectors and trigonometry then as I do now. Back then I used to have the map of the world where for each longitude (the ones parallel to the equator) I calculated sunset and sunrise times for that longitude. This way I knew when to start painting night over the daylight map I had. This result was pretty accurate and an algorithm to determine those solar times is not hard to implement. This time I used another more advanced approach to the problem.

Space

First let us define a mathematical 3D space, with three axis: x (pointing from left to right), y (pointing up) and z (pointing into your monitor). Pointing directions are given to help your imagination. Axis point from small (negative infinite) to big (positive infinite), they intersect at O_{x,y,z} = (0, 0, 0).

Earth

Define a sphere on point O, with a radius of 1. This will be the earth. The funny thing is that for every point\vec{p}(x,y,z coordinate) on the earth, because the earth is centered around O, automatically is the same a the vector pointing orthoganally away from the earth starting on that point\vec{p}. This comes in handy when we do calculations later on.

\vec{p} = (x, y, z)

Sun

We don’t need the sun for this method. We only need the direction to it from the earth. So define a vector S which points in the direction of the sun from O. To do this we must use sine and cosine. On time t = 0 the direction of the sun will be (1, 0, 0), which is a vector pointing to the right. On t = 12 (hour) that should be pointing right (-1, 0, 0). The same goes for 6 o’clock am and pm, being respectively (0, 1, 0) and (0, -1, 0). This gives us insight that the direction \vec{d} to the sun can be given as (t should be scaled to a value from 0 to 1, so we need to divide t by 24):

\vec{d} =(\cos{\frac{2\pi t}{24}}, 0, \sin{\frac{2\pi t}{24}})

With this approach we actually make the sun rotate around the earth.

Tilted earth

Earth tilted axis

Tilted orbit of the spinning earth around the sun. The method discussed here turns it around as it models the sun orbitting the earth.

The above formula still is not complete. The axis around which the earth is spinning is not orthogonal to the plane defined by the orbit of the earth around the sun. It is tilted 23.5°. This is how we get the four seasons on earth of course. To model this a nice trick can be applied to d. We know that on or closely around June 21th the sun is right above the 23.5° latitude (Tropic of Cancer), the summer solstice. From here it takes a sine with a period of a year to go to -23.5° and back again. June 21th is the 172th day of the year. The standard cosine function has the form:

y = a \cdot \cos{(bx + c)} + d

We can simply fill this out. The a must be 23.5, b is \frac{2\pi}{365}, c is -172 and d equals 0. If you define y the observable tilt (because the real tilt will always be 23.5) and x is the day of the year:

observableTilt = 23.5 \cos{(\frac{2\pi}{365} ({dayOfYear} - 172))}

Now this observable tilt is the angle there is between what is the real direction to the sun and the already calculated direction d. To calculate the vector (y component, pointing up or down) which should be added to \vec{d} to get the real direction you can use tangens.

\vec{tiltCorrection} = \vec{d} \cdot \tan{ 2\pi \frac{observableTilt}{360}}

Then the real direction vector \vec{rd} is the addition of \vec{tiltCorrection} to \vec{d}:

\vec{rd} = \vec{tiltCorrection} + \vec{d}

Calcultating illumination and mapping

Using the dot product (or scalar product or inner product) you can calculate the angle between two vectors. Or, if the vectors are both normalized (length = 1) it is the projection of the one vector onto the other. So if we calculate the dot product for every normal vector from the surface of the earth (\vec{p}) with \vec{rd} we should get some meaningful results.

illumination = \vec{rd} \cdot \vec{p}

If the illumination < 0 the sun is in another direction so \vec{p} is pointing to the night side of the earth. If illumination \geq 0 that piece of earth recieves solar light.

We have a 2D map (Mercator projection) we need to map on the 3D sphere. We need to do this to determine which we need to calculate dot products for. To avoid confusion, lets call x and y from the 2D map u and v. We can iterate for each u and each v for each u and map it to a \vec{p} in 3D. To do this we need to determine phi (\phi, logitude: v)and theta (\theta, latitude: u). In the iteration (maxV is the height of the map and maxU is the width of the map):

\phi = 2\pi (\frac{v}{maxV})

\theta = 2\pi (\frac{u}{maxU})

Now u and v map to \vec{p} as follows:

\left\{\begin{matrix} \vec{p}_x = \sin{\phi} \cos{\theta} \\ \vec{p}_y = \cos{\theta} \\ \vec{p}_z = \sin{\phi} sin{\theta} \end{matrix}\right.

Now we can for each coordinate (u, v) on the map if it should be displayed as day or night.

Fun with shading

Because you know the angle the sunlight makes with the earth’s surface you can make add some shading by making it increasingly dark on the edges of the day. It the dot product of \vec{p} and \vec{rd} is smaller than 0.1 (illumination < 0.1) the point is in dusk or dawn and you could mix day and night as a gradient to make the difference between them look more fluently. Another joke is the reflection of the sun: if the product is greater than, let’s say, 0.95 (illumination > 0.95) the sun is approximately orthogonally above that coordinate an you could make it more white to have it look like the sun is reflecting in the map.

Demo and discussion

A demontration of this method is available at the edesign example site. There are a number of assumptions done and restrictions set while creating this demo.

Live sunlight map generated using the algorithm discussed in this article. Click the image to go to the example page where you can override current time.

  • I assumed the solar rays are parallel. The distance from the sun to the earth is huge, but it is actually wrong to assume rays are parallel. They are only by approximation so it is good enough for this simulation.
  • Because of this assumption, the night ’starts’ exactly at the half of the world in the shade while actually this is different. The sun is far bigger than the earth and therefor some rays should be able to reach the shadow half of the earth (near the limit depicted in the first image). By approximation I eliminated this too.
  • Timing is not very accurate. As well as for the 24h clock as for the solstices rough estimates are used.
  • A perfect sphere is taken as world. No fattening is modelled (polar radius should be smaller than equatorial radius).
  • The ‘fun with shading’ should implement something to make land mass not reflect sunlight.
]]> http://www.edesign.nl/2009/05/14/math-behind-a-world-sunlight-map/feed/ 8 Textual difference detector http://www.edesign.nl/2009/05/07/textual-difference-detector/ http://www.edesign.nl/2009/05/07/textual-difference-detector/#comments Thu, 07 May 2009 15:55:13 +0000 Jurgen http://www.edesign.nl/?p=397 comparedifflargeToday I uploaded my textual difference detector to the eDesign examples. This is an example application demonstrating the theory of applying the Levenshtein algorithm to detect differences between two versions of the same text. Also, the ‘Find the differences‘ post is updated with a link to this example.

This example takes two texts as input and outputs one merged text marked with what was deleted and what was added. Take a look and feel free to download the source code. This also inlcludes the Levenshtein algorithm source code.

]]> http://www.edesign.nl/2009/05/07/textual-difference-detector/feed/ 0 Challenge Hash http://www.edesign.nl/2009/05/05/challenge-hash/ http://www.edesign.nl/2009/05/05/challenge-hash/#comments Tue, 05 May 2009 07:47:51 +0000 Jurgen http://www.edesign.nl/?p=202 Hide a keyThe Internet is a crowd and everybody in it can potentially hear what you say. Methods have been developed to prevent this and ensure identity, integrity and authenticity. Often these three can be seen as properties of encryption. Encryption implies the possibility of decryption. Passwords are precious things you don’t want others to decrypt and read. With a technique called challenge hashing you don’t need to have any worries about it. Challenge hashing is a technique used to verify a password on site B which was sent from site A without sending the password in plain text. This article covers how.

Hash

A cryptographic hash function is a deterministic procedure that takes an arbitrary block of data and returns a fixed-size bit string, the (cryptographic) hash value, such that an accidental or intentional change to the data will change the hash value. The data to be encoded is often called the “message”, and the hash value is sometimes called the message digest or simply digest. (Wikipedia, retreived may 2009). In other words a hash (digest) is the result of a hashing function from a certain input (password, file, etc.).

Challenge

The challenge is a question presented to a party who needs to provide the correct answer. A common form of this algorithm is where the challenge is asking for the password and the valid response is the correct password. Also CAPCHAs are a well known implementation.

One step further

When you combine these two an intuitive way of keeping a password secret while being sent along a publicly accessible area and still being valid for authentication checks emerges.

A system has stored user information (username, password, email, etc.) in a database and has the password stored as an MD5 hash. MD5 is the name of the function as there are more hashing functions. When a user requests a login prompt, the server generates a random string (the challenge) and sends it along with the login prompt. Also it stores the string in the session of that request.

The user enters his username and password and hits ‘login’. Just before submitting, a client side script is triggered which calculates the MD5 hash of the password, concatenates the challenge to the digest and hashes that result. This is submitted as the ‘password’ in code.

Now the server has to verify the password. As there is no way to reverse the MD5 digest, the coded password is matched agains the database in a special way. The database needs to concatenate the previously generated challenge to the stored digests and calculate the MD5 hash of that. When the result is the same as the submitted coded password a login is successful.

Discussion

A downside to this technique is the database server processing capacity is required as password digests need to be hashed every login attempt. Worst case (most processing time) is when such an attempt fails or the last hit is a success as every password in the database needs to be checked. Therefore this system is not really scalable to systems aiming for masses of users.

Client scripting must be available. This is not really a critical downside as e.g. JavaScript is common, but you can not assume everybody supports it.

Demonstration

A demonstration of challenge hashing is available in JavaScript and PHP for you to investigate.

]]> http://www.edesign.nl/2009/05/05/challenge-hash/feed/ 1 Character entities http://www.edesign.nl/2009/05/04/character-entities/ http://www.edesign.nl/2009/05/04/character-entities/#comments Mon, 04 May 2009 09:40:58 +0000 Jurgen http://www.edesign.nl/?p=180 Character encodingAs in real life characters that build written language differ from system to system. Ελληνικά characters differ from Русский,  汉语 and Latin characters. Fortunately these character sets have been standardized and called alphabets. The same goes for character sets in the digital world. As computers can only process binary data, all characters are mapped to a number. In the early days such a mapping of the Latin alphabet, along with some other graphical ‘characters’, digits and control characters (e.g. escape, tab, line feed, carriage return) was standardized. This standard is known as  the American Standard Code for Information Interchange (ASCII) and was developed by the American Standards Association (currently: ANSI). This 7-bit encoding lacked digital representations for many characters of e.g. foreign characters (as respectively Greek, Russian and Chinese are mentioned above) but also accents like å, è, ï, ó and û were not represented in the set. But as you can see in this paragraph, improvements have been made to facilitate such ’special’ characters.

Character sets

Other character sets have been defined. The International Organization for Standardization came up with the ISO-8859 series to satisfy this shortcoming and defined some different character sets using 8 bits per character. Microsoft developed its own schemes too like cp-1252 for instance, along with some others (like IBM). Also some local institutes needed to create encoding to facilitate the needs of their native language which the standardized sets still lacked. This introduced the problem of multiple interpretations of numbers. What characters are they mapped to? What character set do I need to use to decode 65 for instance? Does 65 mean A or a or ä or ç or R or…

With the ISO 8-bit character sets, 256 characters were possible (190 characters without control characters, etc). This was sufficient as the sets covered the top ten most used languages. Still for instance Chinese and Japanese we not at all covered. This is where the Unicode Transformation Format (UTF) comes into play, developed by the Unicode Consortium. Unicode are multi-byte character sets which means that per character one, two, three or four 8-bit bytes are used to identify that character. This UTF-8 scheme is nowadays the most commonly used set. It is backward compatible with ASCII and with over 30.000 characters it is able to represent most of the living languages with a single code. UTF contains information of how to convert lowercase characters to uppercase and vice verse which is not the same or even symmetric for every character, and it has sorting rules. E.g. traditional Spanisch knows a single ch character which is sorted between the c and d and in Greek the uppercase Σ in lowercase is a σ, but if it is the last character of a word the lowercase of Σ is ς.

Problems or challenges

When programming web applications for instance a programmer often has to work with a database, file system, web server and one or several browsers. These systems and the data traffic between them (protocols) need to be tuned so they use the same character sets. If this is not the case errors can occur. For instance, if you send “Hellø world” from the database in UTF-8 to the browser, which interprets the bit stream it is receiving as CP-1252, the string is displayed as “Hell? world” and you have probably seen the € sign appear as � because of this error. The solution to this problem, of course is to keep all character sets the same when different systems (applications) communicate with each other.

Copy sign sgml entity

Copyright sign, it's Numeric Character Reference and alias

A way to prevent this problem from occurring is to obey the HTML standard which requires characters not defined in certain character sets to be converted to their Numeric Character Reference (HTML entity). That doing this is mandatory for element attribute values is fairly unknown. This is done by code points denoted by &#[code]; where [code] is replaced by a number. For most used characters an alias is available. E.g. the equivalent for & is &amp;, ë has &euml;, a space is denoted by &nbsp; and the copyright sign has equivalent &#169; with alias &copy;. The codes (digits) and aliases itself only contain characters from the ASCII set, meaning an entire SGML (including HMTL, XML, etc) document can be composed of only ASCII characters. PHP has a built in function that searches and converts such characters in an input. Also, a nice tool to look up entities is available at LeftLogic.com.

]]> http://www.edesign.nl/2009/05/04/character-entities/feed/ 0 Sudoku Logic – part I http://www.edesign.nl/2009/05/01/sudoku-logic-part-i/ http://www.edesign.nl/2009/05/01/sudoku-logic-part-i/#comments Fri, 01 May 2009 12:51:25 +0000 Jurgen http://www.edesign.nl/?p=262 vinyl-leolan-puzzle-largeIf you haven’t heard of Sudoku puzzles (数独, sūdoku) you’ve either been sleeping under a rock or been space traveling for quite a while. These 9×9 square puzzles originating from around 1900 became an international hit in 2005. Sudokus appear in newspapers, online and special sudoku puzzle books all-over-the-world. And as if that is not yet enough Sudoku TV shows and all kinds of variants of the puzzle are made. One can solve a sudoku using logic only. Because of this computational algorithms to solve every possible Sudoku must exist. This is part one in the series on such algorithms.

We must speak of multiple algorithms (strategies) because one algorithm would consist of multiple solve strategies. The most simple strategy not included in this article is a brute force or trail and error ‘attack’ on given input. This is not logic and could take quite a while as there are 70759827985602812313600000000 possibilities to a default 9×9 sudoku, not considering variations (in Dutch) of e.g. 16×16 hex or jigsaw sudokus. The strategies discussed by Andrew Stuart are based on the default 9×9 sudoku and some are limited to those puzzles and some block only strategies overlap with row column strategies. What I want to try with this article is programming a more generic solution which implements some strategies making the result cover all strategies. I think this is possible if you imagine a sudoku board not as a set of cells, divided in rows, columns and blocks, but as a set of equal sections which might have one or more cells in common. This way it will also be easy to implement variants simply by extending the model for a sudoku, resulting in e.g. the X-sudoku or hypersudoku and strategies need to be implemented only once.

Sudoku model

First of all, a model is needed which represents every possible sudoku. That model should include the character set (numbers 1 to 9), the cells (81) and the sections (27). Defaults for a 9×9 sudoku are filled out between brackets. If not mentioned otherwise, every example will concern such a default 9×9 sudoku.

In the model, each cell is identified by a number (0-80). This array keeps an array of possibilities for every cell. On initilization, every possibility is set to true, as everything still is possible but while solving the sudoku more and more will toggle to false, finally leaving only one to true.

$possibilities = array(
false,                //false if unsolved, if solved integer of solution (redundant*)
true, true, true, true, true, true, true, true, true //1 - 9 are possible (true)
);
 
for($i = 0; $i &lt; 81; $i++) {
   $cells[$i] = $possibilities;
}

* The first key in the ‘possibilities’ array is redundant: it could be determined by the other keys (if only one is true, it is solved and has that value). But this way, the array keys match the value they represent and it might increase speed by avoiding determining if the cell is solved or not over and over again.

The sections are probably the most important piece of the model as they define how cells relate to each other. There are 27 sections, each containing nPossibilities (9) cells. There are the rows, columns and blocks. Example resp. top row, left column and top left block with the cell identifiers:

$section = array();
$section[] = array(0, 1, 2, 3, 4, 5, 6, 7, 8); //top row
//... 8 more ...
$section[] = array(0, 9, 18, 27, 36, 45, 54, 63, 72); //left column
//... 8 more ...
$section[] = array(0, 1, 2, 9, 10, 11, 18, 19, 20); //top left block
//... 8 more ...

Also, some default actions (methods) can be added to this model (class). A way to set a cell to a value, one to set a possibility to false and one to count instances of possible locations for a digit in a section. They can call each other to implement recursion:

  • setCellValue() calls removeFromCell() for all other cells for the same value
  • removeFromCell() calls countPossibleLocations() for that value on every section in which that cell occurs, and if the count equals one
  • countPossibleLocations() calls setCellValue() for that value

Solving loop

When you defined the model and filled out some initial digits the algorithm should include a loop searching for the solution. I have included a ‘changed’ property to the sudoku model. This boolean is initially set to true. The first line in the while(sudoku.changed) loop is setting this boolean to false. Next, when iterating over every section trying to find solutions on every change (setting or deleting a number from a cell), this boolean is set to true causing another iteration to occur (a kind of recursion).

Now we have a setup where we can add solve strategies to the loop. In this article, the first strategy will be explained.

Simple elimination strategy

This first strategy to implement in the section iteration is elimination. This is pretty straight forward. As a digit can only occur once in a section (row, column or block), other cells still having this digit set to true can toggle it. This iterative process continues while this resolves possibilities in a single cell to be unique for a section, all other possibles have been set to false.

In Sudoku Logic – part II more strategies will be discussed. Also some sudoku extensions, to simulate sudoku variants and a live demonstration of an implementation of this algorithm in PHP will be included.

]]> http://www.edesign.nl/2009/05/01/sudoku-logic-part-i/feed/ 1