The big picture

The major takeaway from the book is that, contrary to conventional wisdom, reading is not like a standard string search algorithm that one might program into a computer (e.g. to look up the word “time”, one might start with all words that begin with t, narrow down to words containing i as the second letter, and so on until there is only one word left).

Instead, reading is a massively parallel and hierarchical process whereby neurons at the bottom of the pyramid are allowed to vote “yes” or “no” to a neuron one level up the hierarchy. This aggregation neuron tallies the votes of all neurons that report to it, dutifully passing its findings further up the pyramid.

So how does this work in practice? Let’s take our example of the word “time”.

At the very bottom of the pyramid are neurons which respond to very basic elements in an image: vertical lines, horizontal lines, diagonal lines, intersections, right angles, etc. For our  ’t’ in  ”time”, there is perhaps a neuron somewhere which detects the intersection of the vertical line and horizontal line in the letter ‘t’. This neuron then proceeds to vote “yes” in favor of every possible letter that contains this intersection. It also votes “no” for letters which do not contain this intersection, like ‘s’.

The aggregation neuron in charge of the letter ‘t’ receives votes from below, tallies them, and then sends a vote to the next level in the pyramid that the word probably contains a ‘t’.  Meanwhile, this higher neuron is simultaneously receiving “no” votes for letters like ‘c’, ‘y’, and ‘u’.  Eventually, the tallied votes for ‘t’, ‘i’, ‘m’, and ‘e’ win the election, and the chaotic process very quickly converges on the right word – somewhere near the top of the pyramid.

What this means for usability

I am certainly not suggesting that these new findings will turn the field of usability on its head. However, they do shed some light on some long-held usability best practices.

The brain’s visual system is biased to the right of one’s focus

Most of the text processing capabilities of the brain reside in the left hemisphere. This means that visual stimuli appearing on the right side of the visual field have a decided advantage.

In fact, information on the left side of the visual field has to enter the right hemisphere of the brain and be transferred through the corpus callosum to the left hemisphere, through two centimeters of callosal cable.  As a result, words on the left side of the visual field are recognized more slowly and are subject to more errors.

In the English-speaking world, our left to right reading style seems to be well adapted to this limitation. However, I think this idea could be generalized to more than just words.

It seems to follow that not only are pages optimally processed from left to right, but that actions regarding the current area of focus are best placed on the right.  This approach could certainly apply to context menus, action buttons, and error states.

Word familiarity matters, length doesn’t

Because of the massively parallel operation of the brain’s neural network, the brain processes all of the letters in a word at once, and common words receive stronger votes in the voting process.

This finding holds for words of up to eight letters, where presumably the eye has to move forward to process the rest of the letters.  Therefore, it is often better to use a longer and more familiar word than a shorter and less common one.

Words are not recognized by their shape

The image from the retina is initially processed in what is called the “letterbox” area of the left hemisphere, and it turns out that this area processes uppercase and lowercase letters in parallel.  Indeed, according to the book, the small difference in reading performance for the sentences “It was a dark and stormy night” and “It WaS a DaRk AnD sToRmY nIgHt” is due simply to the fact that capital letters are less familiar than mixed case.

It is not immediately clear how far this finding extends into font choices. How unusual must a font be before a lack of familiarity slows reading speed?  The book suggests that our text processing system is highly tolerant of errors, giving examples of situations where transposed letters, missing letters, or misshapen letters have no effect on reading performance.

So it seems usability best practices were right for the wrong reason. Yes, it is slightly slower to read all capital letters, but it is only because we see mixed case more often.

——

As User Experience practitioners, we are always learning rules. Sometimes we are fortunate enough to know the ‘why’ behind these rules, but often we do not.

Especially, now that eyetracking  and A/B testing is widely used, it is becoming commonplace to know with certainty the best design  for a task empirically without touching the why question at all.

However, with more research like we find in this book, I hope we can continue to make sense of these sometimes arbitrary, yet indisputable conclusions.

Upon visiting the site for the first time, one notices that the initial appearance is friendly,with a soft color scheme and edges.  The appearance is entirely appropriate for a site such as this. Normally, a media focused site would be sharper with bolder colors, but since watchparty is selling community, the friendly appearance is a big asset.

Home page

• The featured shows slider is a good focal point on the homepage for new users, however the hover menu should use a more readable version of the date, preferably the month spelled out with the day. The year is really not necessary here.
• Auto complete would be nice in the search box (see Hulu)
• Separating the bottom part of the homepage into current parties and upcoming parties is a little confusing because, essentially, they’re all points on the same timeline. A better solution would be to split the page into thirds, used two thirds for the left side to display currently active parties and upcoming parties in a scrollable format, and use the right third of the page to highlight the most popular parties at the moment. The idea here is to use the mental model of an online TV Guide.
• The visual focal point of the listings is the “create private” button.  However,it is worth thinking about whether users would more likely be looking to join a public watch party. If so, a join button should be the primary call to action, with the create private button being a secondary action.
• Regardless, simply turning the show name into linkable text is too subtle for a call to action. A button is needed here.
• Also as a logged in user, the message link on the global navigation bar seems out of place and doesn’t actually live anywhere in the information architecture. The message section should actually live under the my profile section, and instead of a navigation button, there should be a new message identifier (see Gmail notifier).

Create Private Pop-up

Clicking on the create private button results in a pop-up with two options: create a password and invite people.

• 6 to 10 characters is a bit too secure for what amounts to a few people watching a TV show. A better option would be to auto generate a four digit pin instead of a password. This approach gives essentially 1000 combinations, and is  unlikely to be brute forced before the show is completed.
• An even better option would be to just auto generate a short URL, ending in a four or five digit alphanumeric phrase. This URL could serve as both a bookmark and a password.
• The invite people box should accept any delimiter that is not a valid character in an e-mail. This should be very easily done with the proper regular expression. At the very least, it should be able to handle commas, semicolons, spaces,  and new lines.

Register

• The registration process is nice and short, but could be made even shorter by removing the password confirmation blank. There will clearly be those who mistype their password, but an easy to use password recovery feature should mitigate any problems.
• The line of action links at the bottom is duplicative and confusing. The sign in link is duplicated from the global menu, the forgot password link actually should not be on this page, and the Facebook and twitter logins are duplicated from just above. Also, the e-mail confirmation help link should be located on the register confirmation page and also the login confirmation page (only if they have not e-mail confirmed), but not this page.
• After registering, the user is presented with an error message: (could not sign in, user is unconfirmed). This is not an error, because the user did nothing wrong. Instead, this should be presented as a normal step in the register process. Even better, auto log the user in and worry about e-mail confirmation at their second login.
• My confirmation e-mail was sent to my spam folder (a common problem). Have you ensured that your e-mail infrastructure has all the proper verifications?

Register confirm

• The phrase “must set up the following data” is technical sounding and unfriendly, and it is not true that one must fill out the data because the page also says that they can optionally log in.
• Ideally, the user should have already been logged in, and this page simply asks them to select their time zone before proceeding. Also, the terms of service agreement should probably happen on the register page (It does, so why make the user do it twice?)
• After clicking “save and accept terms”, there should be some sort of final confirmation that the register process completed successfully.Currently, the user is redirected to the home page. There are two solutions here: the first is to redirect the homepage but provide a nice confirmation message and call to action or redirect to the my profile page where the user can find  content specific to them.
• As an additional note, after logging in the user is presented with a confirmation message  in orange. A less alarming color should be used for confirmations.

Create new private watch party

• The back button is in the wrong place. It should either be moved to the top left or replaced with the cancel button next to the confirm button at the bottom of the form.
• The combined time and date input form elements are a bit daunting. In this case, it would be easier to comprehend two separate date and time input fields. The calendar could have the standard calendar pop-up, and the time input could be done iPhone style or with drop downs (think about limiting the options to either 15 min. intervals or 30 min. intervals).
• After the start time is entered, the end times should automatically populate with 30 min. or one hour later. It is also unnecessary to say that the end time has to be after the start time, as this can be caught with a form check.
• See note from above about the password field and the invite field.

Watchparty event page

The first impressions when initially viewing this page are that there are a lot of options, there are some unfamiliar words, and the page feels quite a bit more static than it should.

The eye begins in the top left and observes the name of the current watch party (good), which is followed by a rather large button to see the program details.  This button is perhaps a little bit large for what it does.  After the title, the eyes fall upon the top left box called “flips”.  It is immediately unclear what this word means, but using the context of the box contents one can deduce that they have something to do with expressing emotions.

The eye continues to the right from flips, landing on the “how do you feel” box, which is fairly clear as to its function. Moving down the page, the “blips” box does a fairly good job of communicating that it is some sort of short comment field. Finally, we move back to the left column to the “traks” box.  It is completely unclear what this box is for, especially if there  are no previous traks. Attendees is, of course, self-explanatory.

Significant shortcomings

• My initial expectation was that the page would be more dynamic. This, of course, is not due to bad design  but instead reflects the fact that the product is still in beta. Every effort should be made to Ajaxify all of the boxes on the page so that the page does not have to be refreshed for updates.
• Boxes are not located correctly for their importance. For example, blips should be the most important thing on the page, and thus in the top left.
• It is a little too ambitious to try to coin three new words. I was constantly experiencing varying degrees of cognitive dissonance trying to square the fact that a trak is really a poll, and a flip is really an emoticon.
• It is not clear why one must be logged in to “attend” a watch party. One should be able to observe, without logging in or receiving an error message. The blip box could be replaced by a login box explaining that one must log in in order to contribute. I would go so far as to list anonymous observers in the “attendees” box, but listed after the registered users.

Minor shortcomings

• The JavaScript character counter is misplaced, and really should be invisible until one is approaching the limit.
• The “how do you feel” box should really be a dynamic graph  (see Frank Luntz’s debate feedback sessions for an example)
• There are too many different types of buttons and links on the page.
• Traks have constraints that don’t make sense. Why two character minimum; why not five or 10? Why does the trak have to have four options?

Design suggestions

The wireframe below is a suggestion on how one might reorganize the page to overcome these shortcomings.

• The words “trak” and “flip” have been removed completely.
• Breadcrumbs have been added in place of the “leave watch party” link.
• Blips have been moved “front and center” because they are the main focus of activity.
• Flips have been integrated with the blip box, invoking the web HTML editor mental model.
• The character counter has been moved to the top right of the text box, and should be invisible until one approaches the limit.
• The motion tracking box in the upper right has been changed such that the slider is now vertical and the display is a dynamically changing graph.
• The program details button is now a link.
• Traks are now called polls, for less confusion, and are moved to the right. Past polls should still be visible and should move down when a new poll is created. There should be a hard limit on active polls to ensure that no active  poll falls below visibility.
• The social networking features and invite features are moved to the attendees box which stretches across the bottom of the screen. This feature is important, but  sharing is provided immediately upon entering the page with a modal pop-up.
• Creating a new emotion (previously a flip) can be accomplished with an in-line text field, or optionally hidden behind a “create new” link.  There is no need to label these at all, as they are self-explanatory.
• For added simplicity, the blip button can be removed, and blips are simply published after pressing the enter key ( Facebook does this).
• Black triangles now represent infinitely scrolling content boxes.

Home Page

The site has a coherent and appropriate color scheme, a good layout, and feels generally friendly, if a bit unpolished. The large promotional area in the center left is a good idea, and the limited browse functionality in the bottom half of the screen works well for users who aren’t sure what they’re looking for. I was able to find out very quickly what the site was about and what things I can do as a buyer or seller.

For the purpose of this analysis, I will be focusing only on areas of needed improvement:

• URL – The address bar shows the homepage address as being /default.aspx. The homepage should show only the domain name.
• Logo – The main sans serif font is good, but the points on the two arrows are too sharp, and I would worry that the green (the color of tension) is not the right color for them. The font for the smaller tagline looks as though the text has been resized imperfectly (see the S in sell).
• The blue text at the top in the header bar (Sell Your Used Stuff Today! It’s Free) is unnecessary and only serves to clutter the interface. The blue text on blue background lacks contrast, and the exclamation point is unnecessary.
• Removing it and moving the search box up and to the right would improve the visibility of this important feature.
• In the top right, the main call to action is “Upload Video”. I understand that the emphasis of the site is on video, but what the user is actually trying to do is sell an item, and the button label should perhaps reflect this.
• The location drop-down at the very top of the site gets lost in the page. This feature should be moved down with the search.
• As this is a fairly standard layout for this type of site, box labels like “promotion”, “what’s new”, etc. are unnecessary and clutter up the interface. “Sell your stuff in three easy steps” is important, but perhaps should be placed inside of the box instead of as a box header.
• The social media buttons seem to have been placed arbitrarily. There is also some confusion about whether the like button and the tweet button apply to the video above or to the site itself. A better solution would be to group these together with the invite friends buttons.
• The Starbucks coffee promotion looks like an advertisement instead of a promotion, and I would question the effectiveness of a free coffee on getting new users signed up (what do your numbers say?). Ads cheapen the look of your site, and I would try to avoid the appearance of them if possible.
• Throughout the site, you use red text to emphasize prices. Red text is associated with danger or error, and I would highly recommend that you use a combination of font size and font weight (or use the color green) instead of the color red to point out prices.
• There are a lot of social media buttons at the bottom, but some appear to be duplicates (even though they have different functions). The purpose of these buttons should be made more clear.
• The community box at the bottom of the front page is confusing and ineffective. All of the posts are by “admin” which makes the section appear like an announcement section. Further, as a user I am not clear as to what I would use a community section for. For example, I just want to sell an iPod, and I have nothing in common with someone trying to sell a pair of jeans. It appears that there may be two different startups here: a marketplace and a general topic message board. Best to focus on the former.

Search Listing Pages

• It is unnecessary to repeat the category for each result. Either the user got this page by clicking on a category (in which case they will all be the same) or the user got here with a custom search in which case the category is irrelevant.
• The item ID is also unnecessary.
• The time format should be “X days ago” instead of the full timestamp. At minimum, there is no need for seconds to be listed.

Item Listing Page

• Again, box headings are unnecessary. It is obvious what information appears in the “posting information box” and the “more info box”. Better to remove these and cleanup the interface.
• The posting information box is cluttered with too much information. Cleanup the date and remove the item ID. The “view times” should also be removed. At this early stage, it only serves to work against you.
• The condition rating system is confusing. I have no idea what “used 70%” means.
• The blue text style for links and the styling for the item title are too similar, making the item title appear clickable.
• Avatars in comments make the site appear MySpace-y.
• Clicking on the respond button gave me a pop-up box. Better to just redirect the user to the login page with an explanation as to why they have been sent there.

Login Page

• The phrase “welcome to 123 exchanges” is unnecessary.
• The phrase “join our community now” appears to be a link to join the community, not a lead in to the sign-up link.
• “Sign-up here” should be a button instead of a link with arrows on each side.

Sign-up Page

This page is very important because it is the barrier between an interested user and a registered user. Every attempt should be made to make this page appear and be as simple and effortless as possible.

• According to Jakob Nielsen, a two column layout is less effective than a one column layout. Regardless, there is no reason why the last name field should be the only field in the right column.
• There is no reason to have a country drop-down if there is only one country.
• Eliminate the city field and state field, and instead pull those from the zip code.
• Forcing the user to enter their e-mail twice, while helping to guard against typos, reduces your conversion rate. Furthermore, a great deal of users simply copy and paste what they had just typed. Better to remove the second e-mail address blank.  There is some controversy over this, but the number of people who mistyped their e-mail and do not correct it will be mitigated by the number of people now completing the form who would not have otherwise.
• User id sounds like it should be a number. Better to label the field “username”, or even better remove it and auto generate a user name (while allowing the user to change it in his preferences).
• Again, I would recommend the removal of the second password field. Ideally, the password recovery feature should be easy to use for those who mis-type the password initially.
• Captchas should only be used when absolutely needed. While in such an early-stage, it is better to remove the captcha and either deal with the spam sign-ups or implement either the dummy form elements method, GET request detection, or separate the sign-up process into two steps.
• Even better to implement a lazy registration process where you allow the user to upload a video and metadata first, and then ask them for e-mail and password as a last step.

Sign-up Confirmation Page

• There is no confirmation that my registration actually went through.
• I would be curious to see numbers on whether the social media invite is used on this page or whether some other content would be better.
• “Next Step” is not prominent enough.
• There is no option to resend the confirmation e-mail if it was not received.
• To cut friction, the user could be automatically logged in at this point, and the e-mail confirm could simply be required for the next login.
• The confirmation e-mail is too wordy, and it leads with red text which says “danger”. There are also several typos. Ideally, this e-mail should contain the activation link in enough text to explain what it is and nothing more.
• After clicking the activation link, congratulations is unneeded, and the phrase “wishing you the best” is unprofessional.
• The user should ideally be automatically signed in and redirected to the my account home page, but if not at least the sign-up information should pre-seed.

My Account

• There is no link to logout. This link should be located near where the login button was.
• The search function appears to be duplicated, and the drop-down for search type is unnecessary since it is easy to tell the difference between a product name and ID.
• In general, there are too many menu items for the amount of things that one can do on this page. Messages should be integrated with the buy and sell list, home is unnecessary. Also, I would be careful about repurposing the main navigation for the my account section, while also having a my account navigation bar at the very top. It appears at first glance, that the new bar at the very top is navigation for the my account section, but in fact it duplicates navigation in the main nav.
• There are several unnecessary columns, including product ID and view times.
• The “view top 10 link”, from a user standpoint is useless. Why do I care who the top 10 users are?
• “Request a Starbucks coffee” should be automatic, without having to request (send the coupon via e-mail)
• Get rid of the refresh page button, all browsers have a refresh button.
• Pagination is a little awkward (page count)
• Upload video functionality is duplicated, but the second button is labeled “post video”.

Upload Page

• This page has far too much text, and the red text is jarring. Most of the text refers to troubleshooting steps, which the user should not be bothered with until they’re actually having trouble.
• The choices on this page should be upload or cancel. “Back to selling list” is awkward phrasing, and there is no reason why the user would need to go to the homepage.
• Above the video upload area, there is a sentence with buttons explaining that the user can use one of two video converting applications. What is a video converting application? Is it required? How do I know whether I need to use one or not? Best to place this in the troubleshooting section
• The notice explains that the maximum video duration is 3 min., but a little bit later 3 min. is listed as a recommended length
• This whole page is really unnecessary, as the video upload area could be fit into a box in the my account area with a small link below labeled “troubleshooting”, or “problems?”. After upload, the my account area would refresh the new video listed in an option to enter metadata.

Video Metadata Page

• Blue is not a good color to use for the confirmation message at the top. It blends in with the background, and the user will skip over it straight to the input fields.
• Too many fields crammed together, looks daunting. Use some white space to break up the fields.
• It is unclear what the “call” checkbox does.
• Again, city and state are not needed.
• Tags are hard to explain to users, and there is no indication of what format is needed here, whether they should be separated spaces or commas, or whether multiple word tags are accepted. Due to the varied entry by users, this is not likely to be useful for navigation or searching (better to just emphasize a good item description)
• Other information is deemphasized and would better be called “item description”.

SELECT AddGeometryColumn('public', 'table','column', 4269, 'POINT', 2)

This will add a new column to the table and add an entry to the metadata table (the 4269 specifies my reference system, more on that below). It will also add a column to “table” and several constraint checks on the new column.

Next, you’ll need to know about spacial reference systems. You’ll notice that the letters SRID are scattered throughout the documentation. SRID stands for “Spacial Reference ID”, and it describes the system to use to transform a round world into a flat x,y coordinate system. (Interestingly enough, Latitude and Longitude are not x,y coordinates but instead radians.) There are literally thousands of them.

I chose to store my geocoded addresses in two formats: SRID 4269 (Lat/Lon) and SRID 2163 (US National Atlas – meters). That way I can do my distance calculations quickly using the x,y coordinates (for approximate accuracy) and i’ll have the lat/lon to feed to, say, Google Maps.

So now i’ll need to add the second column:

SELECT AddGeometryColumn('public', 'table','column2', 2163, 'POINT', 2)

Inserting Data

My data is actually coming from the Yahoo geocoding api. I’m feeding them an address and they are returning a latitude/longitude. I used the following SQL to update my addresses table:

UPDATE addresses SET
coordinates_ll = ST_SetSRID(ST_MakePoint(" . $point['longitude'] . "," . $point['latitude'] . "),4269),
coordinates_proj_m = ST_Transform(ST_SetSRID(ST_MakePoint(" . $point['longitude'] . "," . $point['latitude'] . "),4269), 2163)
WHERE id=" . $id;

What we are doing here is, from inside out:

1. Create a point.
2. Set its SRID to 4269 (lat/lon)
3. AND for the second query, the point is converted to SRID 2163
4. Store it.

Note, the longitude comes first in these functions.

A quick distance query

Show me the ids of records that are within 5 miles (8000 meters) of a point.

SELECT id
 FROM addresses
 WHERE Distance(coordinates_proj_m, ST_SetSRID('POINT(-1952333 -571399)', 2163)) < 8000

To get this to work, you must transform your point into the same SRID as the column you're querying.

It also helps to put an index on the column we're querying on:

CREATE INDEX idx_coordinates_proj_m
ON addresses
USING gist(coordinates_proj_m);

Note: This does not scale.

References

• http://www.bostongis.com/?content_name=postgis_tut02#21
• http://unserializableone.blogspot.com/2007/02/using-postgis-to-find-points-of.html
• http://forum.geonames.org/gforum/posts/list/727.page
• http://www.bostongis.com/PrinterFriendly.aspx?content_name=postgis_tut03
• http://postgis.refractions.net/support/wiki/index.php?ExamplesFindNearby

Share/Bookmark

]]> http://www.jeremytunnell.com/posts/postgis-basics-part-2-hello-world/feed 0 http://www.jeremytunnell.com/posts/postgis-basics-part-1-installation http://www.jeremytunnell.com/posts/postgis-basics-part-1-installation#comments Thu, 05 Aug 2010 00:44:54 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=19 My OS is CentOS, so I used yum to handle installs from the rpm database. The thing with CentOS is that it is purposefully one step behind the cutting edge, so the most recent version of certain applications are not available (PHP, Postgres, etc). To get around this limitation, you have to instruct CentOS to look elsewhere for updated RPMs.

One location is the Postgresql repository: http://www.postgresql.org/download/linux

Another location for these is DAG. To set this as an additional rpm source, run this command as root (this works for RHEL 5 and CentOS):

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

Now we have the ability to get more up-to-date rpms.

Next, we must work on getting all of the dependencies for PostGIS installed, which amount to:

• ProJ4
• GEOS 3
• The PL/pgSQL procedural language

Proj4

Unfortunately, rpms for the proj-4 application still weren’t available from the repository, so I found an rpm here: Proj-4 rpms. This installed with a quick rpm -U software-2.3.4.rpm

GEOS

Geos is thankfully available from Dag, so yum should find it and install it. There may be some weird dependencies you will have to work through. The below packages should help, but they’ll all have to be installed eventually, so you may as well go ahead and do it.

• php-pear
• php-devel

* At this point, if you are using PDO and haven’t installed the pgsql PDO drivers, you can do it now: “pecl install pdo_pgsql”

PL/pgSQL

You’ll need to run the following commands from a command line (either as root or the postgres user, I can’t recall which one is needed by which). Please note all of these paths seem to be unique to each installation, so yours might be different.

/usr/bin/initdb -D /var/lib/pgsql/data_dir    //Create the database

psql create language plpgsql  

psql -d database_name -f /usr/share/pgsql/contrib/lwpostgis.sql
psql -d database_name -f /usr/share/pgsql/contrib/spatial_ref_sys.sql

The last two lines load the spacial functions into the database for use.

Share/Bookmark

]]> http://www.jeremytunnell.com/posts/postgis-basics-part-1-installation/feed 0 http://www.jeremytunnell.com/posts/transforming-an-address-into-latitude-and-longitude-using-php-and-curl http://www.jeremytunnell.com/posts/transforming-an-address-into-latitude-and-longitude-using-php-and-curl#comments Wed, 05 Aug 2009 00:44:04 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=17 Step 2 – Construct the query

Yahoo offers a REST interface to handle the conversion so we can use either a plain HTTP GET request, or if you want to do it inside of a script that is doing many other things you can use CURL, a set of very useful functions for querying and “scraping” web pages and applications.

We must construct the url according to the api guidelines, which boils down to something like this:

	$url =  "http://local.yahooapis.com/MapsService/V1/geocode";
	$args = "?appid=your_key_";
	$args .= "&street=" . urlencode(street_address);  //We must urlencode variables that will have spaces/etc
	$args .= "&city=" . urlencode(city);
	$args .= "&state=" . state;
	$args .= "&output=php";  //Optional parameter to return a php object
	$url = $url . $args;

The code above basically asks yahoo to use the current key to return the latitude and longitude as a serialized php object (the other option is xml, but I don’t like parsing it).

Step 3 – Send the query

Sending the query via CURL works something like this:

	$ch = curl_init();
	curl_setopt($ch, CURLOPT_TIMEOUT, 5000);
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	$address_data = unserialize(curl_exec($ch));
	curl_close($ch);

The script is doing the following:

1. Initializing a CURL object.
2. Setting the timeout to 5 seconds.
3. Setting the url to query.
4. Setting CURL to return the results to a variable rather than displaying it to the screen.
5. Reading the results, unserializing them, and storing the results in our $address_data object, which is really just a bunch of nested arrays.

Step 4 – Get the latitude and longitude

You can retrieve the latitude and longitude this way:

	$latitude = $address_data['ResultSet']['Result']['Latitude'];
	$longitude = $address_data['ResultSet']['Result']['Longitude'];

That’s all there is to it.

Share/Bookmark

]]> http://www.jeremytunnell.com/posts/transforming-an-address-into-latitude-and-longitude-using-php-and-curl/feed 0 http://www.jeremytunnell.com/posts/really-hairy-problem-with-seemingly-random-crlf-and-spaces-inserted-in-emails http://www.jeremytunnell.com/posts/really-hairy-problem-with-seemingly-random-crlf-and-spaces-inserted-in-emails#comments Mon, 05 Jan 2009 01:42:38 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=15 Troubleshooting

I isolated the process into three components: the email builder (my code), the mailer application, and the sending application (in this case, smtp through postfix). I followed the below process to troubleshoot:

1. Changed the mailer application to use the built in PHP mail() function, and I also tried sending using a completely different SMTP box. The problem was persistent no matter what process was used to send the mail.
2. I found the place in my code where the completed HTML was being handed off to the mailer application. Adding in a file_put_contents, I could then examine the exact data before it went to the mailer. Taking a look at it through a hex editor revealed that the body did not contain the offending characters
3. Finally, I found another mailer application (I was using phpmailer). Swiftmailer is an alternative to phpmailer. I figured that if it was obscure bug with phpmailer, changing the mailing application would clear it up. This also did not solve the problem.

So if everything was working correctly, where could the problem crop up? The only place left to look was the encoding scheme for the emails. Scouring the internet, I found a few suggestions that the problem may be related to the message encoding. In this case, it appeared the problem stemmed from (possibly) a 75 character limit in quoted-printable encodings. This didn’t help me, as my emails were being sent using 8-bit encoding.

The solution

Just to see what happened, I changed the encoding settings on phpmailer to use base64 encoding using $mail->Encoding=”base64″. This solved my problem, though I don’t admit to knowing exactly why.

Note: don’t confuse this with character set issues. In the past I have seen situations where utf-8 text was being read into an email and sent as ISO-8859-1. When having weird characters showing up in your emails, this is most often the problem (or vice versa).

Archived Comments

Hello,

Your problem is rather similar to what I am having, here is an example of what my text might look like: “he would perf! orm his operation”.

You see the “! ” inserted in the middle of “perform”. How odd. It is messing up my html emails because it’ll interrupt the source code.

I do not fully understand your solution- would you mind illustrating in some more detail what it was you did to fix this very similar problem you were having?

I am only sending my html mails using the mail() function built into PHP, and I am encoding that email using charset=iso-8859-1 though it doesn’t seem to matter if I use utf-8.

Very hopefully in SLC,

Nik

Posted by Nik on January 18th, 2008

---

Nik,

I think you’re confusing character encoding with mime content transfer encoding.

First, if you’re sending using ISO-8859-1, you need to make sure your database or wherever you’re getting the text from is delivering it is in the same encoding format (see my earlier posts).

If so, try looking at the mime transfer encoding (base64, 8-bit, quoted-printable, etc). I use phpmailer (http://phpmailer.codeworxtech.com/) to send emails, and i’d recommend it.

A quick look at the mail() php manual page, it appears that the following setting may help you:

“Apparently if using the mbstring library, and using overriden functions, the “mail” command appears to use “Content-Transfer-Encoding: BASE64″ by default. Which if you combine it with PEAR Mail_Mime you’ll get mails that, although they appear RFC compliant, not mailer can read correctly.

To fix this it appears you should add a fixed header in the mail command (this one assumes the pear mime module is 7bit clean, perhaps 8bit would also be fine)

$headers['Content-Transfer-Encoding'] = ’7bit’;”

Good luck,

Jeremy

Posted by Jeremy on February 04th, 2008

---

Nik,

Also, you might take a look at the email content in a hex editor to figure out whether the exclamation point is really an exclamation point or just confusion over the text encoding of the email.

Posted by Jeremy on February 04th, 2008

---

Have you tried inserting your own line breaks? How long of an email are we talking about? The SMTP server may be inserting it’s own line breaks at will to keep any single line from being over a certain limit. Base64 has a limit of 76 characters per line, so if you look at the character output you should see a nicely aligned block of text.

Posted by James on February 28th, 2008

---

rtyey

Posted by Anonymous on June 02nd, 2008

---

Hi guys,

I also had the rogue “!” problem Nik was having, i.e. “perform” comes out as “per !form”

It was driving me crazy until I googled and found this.

I am using phpmailer as Jeremy is (it’s good I recommend it – no pont re-inventing the wheel) and my problems were solved by adding the suggested fix:

$mail->Encoding=”base64″;

Thanks guys, you saved my weekend and health!

cheers

Alex

Posted by Alex Seymour on February 22nd, 2009

---

I think the reason setting $phpmailer->Encoding = “base64″; cures the random spaces is because it causes the data to get encoded into a form which adheres to the 998 octets per line limit. For me the problem was happening when the HTML in the email had lines which exceeded 998 chars before a line break. See http://en.wikipedia.org/wiki/MIME#Content-Transfer-Encoding

Posted by Jon Nott on June 17th, 2009

---

Thanks a mil! Worked perfectly!

Posted by Jeff on August 28th, 2009

---

Absolutely genius – just the problem I was having. A white space being inserted in phpMailer scripts randomly after a certain char count. I had tried to resolve this for AGES! This should be documented in the phpMailer files. Pat on back to all.

Posted by Lewis on December 15th, 2009

---

Awesome, thanks for the solution. I did change phpmailer class from 8bit to base64 and it worked like a charm. I’d been changing all sorts of variables for the past few hours trying to get those exclamation marks to go away.

Posted by Victor on January 13th, 2010

---

I had this same problem using the Pear Mail_Mime class. By using base64 encoding when preparing the email the problem was fixed. You need to set the encoding with the $mime->get() method like so.. $body = $mime->get(array(‘html_encoding’=>’base64′));

Hope this helps

Posted by Freed on March 30th, 2010

---

This was very helpful. I dropped in the line of code with the base64 encoding change and it fixed my problem. Thank you very much for posting.

Posted by L on July 13th, 2010

---

Adding a /n/r into your PHP code at every 990 characters or less fixes the problem if you’re using PHP’s standard mail() function. The extra space (which in reality is a linebreak and a space was breaking longs links in my messages. Not anymore! Thanks for the posts everyeone!

Posted by Zoltan on August 05th, 2010

---

I have exactly the same problem. Sometimes it a space in the middle of the word, sometimes it messes up the html syntax (eg. doesn’t close a )…

adding /n/r does not seem to help, and I am not quite sure where/how to add $mail->Encoding=”base64″;.

Using the standard php mail command, in this syntax:

if (mail($to,$subject,$msg,$headers)) {

header(‘Location: complete.php’);

} else

{

header(‘Location: incomplete.php’);

}

NB. These are the headers:

$headers = “MIME-Version: 1.0\r\n”;

$headers .= “Content-type: text/html; charset=iso-8859-1\r\n”;

Any help would be truly appreciated!

Posted by Richard Crampton on October 27th, 2010

---

Thankyou, thankyou, thankyou. Saved my day and a lot of debugging with a lot of potential error sources! Thanks for taking the time to write this up.

Posted by Kristian on October 27th, 2010

---

Thnx for the tip, it works 100%

Posted by Marcel on November 06th, 2010

Share/Bookmark

]]> http://www.jeremytunnell.com/posts/really-hairy-problem-with-seemingly-random-crlf-and-spaces-inserted-in-emails/feed 0 http://www.jeremytunnell.com/posts/how-to-configure-your-web-application-to-correctly-deal-with-character-setencoding-issues http://www.jeremytunnell.com/posts/how-to-configure-your-web-application-to-correctly-deal-with-character-setencoding-issues#comments Wed, 05 Nov 2008 01:41:48 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=13 Background

I’ll just link to the best places i’ve found to read up on the background. Read these first if you want to understand whats going on:

• Excellent intro to character encodings
• Joel On Software, on Unicode

The Details

In order to get your website working harmoniously with one character set, you first have to pick one. I picked ISO-8859-1 (sometimes referred to as latin1). It’s the most popular English (Latin) language set, though it doesn’t display many foreign characters.

Unfortunately, despite the fact that Unicode is the end-and-be-all PHP is really horrible at dealing with Unicode (specifically multi-byte strings). Since I’ve got enough to worry about without having to also check every function i’m using for multi-byte compatibility, ISO-8859-1 is right for me, for now. I hear PHP6 will fully support mb strings.

In order to get your web application working correctly with one character set, there are basically three parts you must take care of:

1. The database
2. The web server
3. The web page itself

Database config

In a perfect world, you would compile your database to default to your preferred character set. I won’t cover that here.

Assuming you cannot do that, you need to create your tables using the correct set. Keep in mind, at least on MySQL, the character set can be configured all the way down to the column level. Here we will just set it at a database level. (See here for much more detail):

CREATE DATABASE database CHARACTER SET utf8 COLLATE utf8_general_ci;

Next, we need to configure how the database delivers the results of queries to the calling application. Assuming you don’t (or can’t) ensure this is done at compile time, you can use the following command before sending a query:

SET NAMES charset

According to the linked article above, this is shorthand for setting the character_set_client, character_set_results, and collation_connection variables in MySQL. A good place to put this query is in the constructor of your database access class.

Web Server

Assuming you are using apache, you need to edit a setting in your httpd.conf file (the primary apache config file).

AddDefaultCharSet charset

Set this to be what you prefer apache to tell the client that the default character set is in case one is not specified. (This is important, as in some cases the client will trust this even if one is specified in the document.)

Also, there are options in whatever server side language you’re using to set this variable in outgoing headers. In PHP you use the header() function. See the PHP documentation for details.

The web page

Finally, you should specify the character set of the page you’re serving with a meta tag at the top below the tag. It looks like this:

<meta http-equiv="Content-Type" content="text/html; charset=charset" />

Getting obsessive

Most browsers support specifying the character set that a form should accept (and what the browser will convert the text to if it is not correct). To manually set this, include the following attribute in the


tag:
 accept-charset='ISO-8859-1'
Doing the above should ensure that you operate using the same character set throughout your web application, and hopefully garbage characters and question marks will be forever in the past.
Reference
• DBMS and charsets – Settings for various databases.
• More charset settings for databases.
• MS Word and Web Development – Short and specific to MS Word.
• Turning MySQL data in latin1 to utf8 – Case study that gets down to the minute details.
• Recode: Character set conversion tool
• Fixing MYSQL charset issues on existing data.
Share/Bookmark
]]> http://www.jeremytunnell.com/posts/how-to-configure-your-web-application-to-correctly-deal-with-character-setencoding-issues/feed 0 http://www.jeremytunnell.com/posts/more-modrewrite-using-a-dbm-file-instead-of-a-txt-file http://www.jeremytunnell.com/posts/more-modrewrite-using-a-dbm-file-instead-of-a-txt-file#comments Fri, 05 Sep 2008 00:41:14 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=11 The most confusing thing is when you use the perl script on the official apache site here it’s written for the NDBM format (which isn’t available on my box).
Here’s the perl script:
#!/path/to/bin/perl
##
##  txt2dbm -- convert txt map to dbm format
##

use NDBM_File;
use Fcntl;

($txtmap, $dbmmap) = @ARGV;

open(TXT, "<$txtmap") or die "Couldn't open $txtmap!\n";
tie (%DB, 'NDBM_File', $dbmmap,O_RDWR|O_TRUNC|O_CREAT, 0644)
  or die "Couldn't create $dbmmap!\n";

while () {
  next if (/^\s*#/ or /^\s*$/);
  $DB{$1} = $2 if (/^\s*(\S+)\s+(\S+)/);
}

untie %DB;
close(TXT);
So after some trial and error, I found out that my version of apache works with SDBM, and it appears that if you just change NDBM to SDBM it works…
#!/usr/bin/perl -w
##
##  txt2dbm -- convert txt map to dbm format
##

use SDBM_File;
use Fcntl;

($txtmap, $dbmmap) = @ARGV;

open(TXT, "<$txtmap") or die "Couldn't open $txtmap!\n";
tie (%DB, 'SDBM_File', $dbmmap,O_RDWR|O_TRUNC|O_CREAT, 0644)
  or die "Couldn't create $dbmmap!\n";

while () {
  next if (/^\s*#/ or /^\s*$/);
  $DB{$1} = $2 if (/^\s*(\S+)\s+(\S+)/);
}

untie %DB;
close(TXT);
So problem 1 fixed. Then I find out the output of this thing is two files instead of one…wtf? The files are named: rewritemap.dbm.dir and rewritemap.dbm.pag
Here’s the short answer: If you just ignore the additional extensions, it will work. Here’s the httpd.conf line:
RewriteMap redirects dbm:/.../.../.../rewrite/rewritemap.dbm
Archived Comments
Great article! Thanks for clearing up this otherwise poorly-documented issue.
Posted by Nick on October 18th, 2007
---
Yes, thank you very much. I ran in to the exact same problem. From what I gather you can only use
use NDBM_File;
With Perl 5.10
And I couldn’t upgrade my system to this version.
Many thanks!
Posted by Kevin Smith on July 31st, 2008
---
One small typo.
while ()
Should be:
while (<TXT>)
-Peter
Posted by Peter Burkholder on April 09th, 2009
Share/Bookmark
]]> http://www.jeremytunnell.com/posts/more-modrewrite-using-a-dbm-file-instead-of-a-txt-file/feed 0 http://www.jeremytunnell.com/posts/mod_rewrite-attempting-to-bend-rewritemap-rewritecond-and-rewriterule-to-my-will http://www.jeremytunnell.com/posts/mod_rewrite-attempting-to-bend-rewritemap-rewritecond-and-rewriterule-to-my-will#comments Fri, 22 Aug 2008 00:39:10 +0000 Jeremy Tunnell http://www.jeremytunnell.com/?p=6
RewriteEngine On
RewriteMap redirects txt:/home/.../.../rewriterules.map
RewriteRule ^(.*)$ ${redirects:$1|$1} [R,L,NC]
How does it work? (See here for details.)
The first line turns on mod_rewrite. The next line sets the location of the map file (the two column list of source/destinations). The third line sets a RewriteRule that says “take everything in the request uri” (.*), and search the RewriteMap redirects for a match. If we have a match, the user’s browser gets a redirect and is forwarded to the new uri. If we don’t get a match, i’ve specified with the $1 backreference that the user is to be transparently passed on to the uri he requested.
The problem is that when Apache doesn’t find a match in the rewritemap, we get a new request for the same uri…which runs through the process, ending in a request for the same uri…and so on. Because I want to redirect the user instead of handling the change transparently, I keep running into an infinite loop problem.
So we use a RewriteCond.
RewriteEngine On
RewriteMap redirects txt:/home/.../.../rewriterules.map
RewriteCond ${redirects:$1}   >"" [NC]
RewriteRule ^(.*)$ ${redirects:$1|$1} [R,L,NC]
We basically say “Look up the uri from the following RewriteRule ($1 means look ahead to the next RewriteRule. We could look behind to the last RewriteCond by using %1.) and if it is lexically greater than the empty string then do the Rewrite. Otherwise, skip it.
Version 2
I never got the [NC] switches to work so I just converted everything to lowercase before I do the comparisons (You must make sure your map is all lowercase).
RewriteEngine On
RewriteMap redirects txt:/home/.../.../rewriterules.map
RewriteMap lowercase int:tolower
RewriteCond ${lowercase:%{REQUEST_URI}|NONE} ^(.+)$
RewriteCond ${redirects:%1}   >""
RewriteRule ^(.*)$ ${redirects:%1} [R,L]
Links
• Check out here for more good mod_rewrite explanations
• Mod Rewrite Tutorial
• No more endless loops
• Lots of rewrite examples
Archived Comments
Thanks for this tip, was having a problem with an infinite loop on a rewrite map – this helped.
Posted by James on September 14th, 2007
---
Found this via Google. Thanks for the tip!
Posted by foo on February 10th, 2008
---
Thank you for sharing this missing part of the Apache documentation. Worked great for me.
Posted by Fredrik Nygren on March 17th, 2008
---
Thank you very much
Posted by sohbet on December 13th, 2008
---
thx for this great article, i searched for hours!
Posted by Felix on January 23rd, 2009
---
Thanks for this article, solved a little typo in my script and only noticed it after looking at this article.
Posted by Web Design Sydney Melbourne Brisbane on March 17th, 2009
---
Oh, can I kiss your toes?
Seriously, I didn’t grok the bit about using rewritemaps in a RewriteCond before. Thanks!
Peter
Posted by Peter Burkholder on July 14th, 2009
---
Thanks for this!
Just one thing – the syntax for the “greater than the empty string” is wrong – >”" fails to match certain things (for example, if the result of your rewritemap is the number 9). The correct syntax is “> ” (note the order of the quotes), which matches any ascii value lexically greater than space (char(32)).
Posted by Andy Theyers on July 20th, 2009
---
Here’s a little solution I came up with for making my URLs prettier and getting search engines to repoint to the new URL. I have a products db that has uppercase word keys for each product that points to the numeric product id. Then I have a reverse db that contains the same in the opposite order. So far this seems to work..

RewriteMap uppercase int:toupper

RewriteMap products dbm:conf/products.dbm

RewriteMap rproducts dbm:conf/rproducts.dbm
# 301 redirect ugly product URL to user-friendly word based URL.

RewriteCond %{QUERY_STRING} ^partNumber=(\d*)$

RewriteCond ${rproducts:%1} >”"

RewriteRule ^/product\.html$ /product/${rproducts:%1|%1}? [R=301,L]
# Convert user-friendly word based product URL internally to ugly URL with part number.

RewriteRule ^/product/(\D.*)$ /product.html?partNumber=${products:${uppercase:$1}} [E=PAGE:product.html]
Posted by MikeFM on July 28th, 2009
---
Hi, I am working on the rewritemap and rewriterule for my website. actually requirement is like i have one website whose address is http://www.myside1.com now since i am going to retire this website so i need to redirect most of the pages to new website. i want to use map file for this. i have created this mapping file where i want to keep maping like /folder1/page1.html http://newwebsite.com/content/view/hello.html in this way i want to use the redirection. i dont want to hardcode the each redirect rule. I can have /folder2/page1.html also. Could you please help me to have the correct rule.
Posted by yashjimmy on July 09th, 2010
Share/Bookmark
]]> http://www.jeremytunnell.com/posts/mod_rewrite-attempting-to-bend-rewritemap-rewritecond-and-rewriterule-to-my-will/feed 0