Smartphone Talk

The last few weeks saw some big announcements in the smartphone world:

  • Palm released the phone that they’ve been promising us for years, the Palm Pre, with it’s new WebOS, to reviews that were mostly favorable and summed up as “The iPhone’s baby brother“.
  • Apple stole some of Palm’s thunder by dominating the press two days later with news of their relatively unexciting new phones and 3.0 software.
  • In the weeks prior, news came out that about 18 more Android phones should be out in calendar 2009 and that, by early 2010, all of the major carriers will have them.
  • And Nokia’s E71 hit our shores, an incredibly full-featured phone that you can get for just over $300 unlocked, and use the carrier of your choice. While this isn’t a touchscreen, and is therefore suspect in terms of it’s ease of use, it is an amazingly full-featured product.

Left in the wings were Blackberry, who keep producing phones, including their iPhone competitor, the Storm — to yawns from the press, and Microsoft, who are talking a lot about Windows Mobile 6.5 and 7.0, but seem to have really been decimated by the ugliness of their mobile OS when compared to just about anyone else’s.

What’s clear is that a few things differentiate smartphones these days, and the gap between the ones that get it and the ones that don’t are huge. They are:

Responsive Touchscreen Interfaces. The UI’s of the iPhone, Android and Palm’s WebOS get around the sticky problem that phones were just to small to support anything but simple functionality without requiring an oppressive amount of taps and clicks. This is why Microsoft has fallen down the smartphone food chain so far and fast — their mobile OS is just like their desktop OS, with no flagship phone that does the touchscreen nearly as well as the new competition.

Desktop-Class Web Browsers. This is where Apple and Google have drawn a huge line, and it looks like Palm might have joined them. All three use browser’s based on Webkit, the same technology that fuels Safari and Chrome. On a 3G phone, this makes for a fast and complete experience that puts the Blackberry, Mobile Internet Explorer and the Treo’s hideous Blazer. Add Google’s voice activation (native on Android and available for iPhone), and their smartphone-optimized results (which don’t work on the non-webkit browsers) and the task of finding a Starbucks or hotel on the road takes seconds, instead of the average ten to 15 minutes on the old, lousy browsers, which simply choke on the graphics.

Push Email. If you connect to Exchange servers, the iPhone and Pre have Activesync built in. If your mail is with Google, you’re connected to it as soon as you tell an Android phone your login and password. And the Android phone app is the best out there, with Apple’s mail running close behind it. What’s ironic is that Microsoft targeted their biggest threat with Activesync — the Blackberry’s kludgy, but, at the time, unparalleled email forwarding — and gave it wings by licensing it to Palm, Apple and others. This is fueling corporate acceptance of the iPhone and Pre, meaning that this Blackberry-beating strategy might have worked, but more likely it did it for Apple and Palm, not Microsoft.

Music. The iPhone is an iPod; everything else isn’t, meaning that, if having a high quality phone and music experience on one device is a priority, you’re not going to go wrong with the iPhone. I love my G1, but I weigh my value of the real keyboard and awesome, open source OS on T-Mobile over the iPhone’s built-in iPod and Activesync on AT&T. As OSes go, Android is only marginally better than Apple, but the Apple hardware is much better than the G1. Newer Android phones are going to show that up.

People make a lot of noise about the apps available for the iPhone (and Windows/Blackberry) as opposed to the newer Android and Pre. I think that’s a defining question for the Pre, but it looks like companies are jumping on board. For Android, it’s quite arguably a wash. All of the important things are available for Android and, given that it’s open source, most of them are free. And with those 18 phones due out by year end on every carrier, the discrepancies will be short-lived.

I have to wonder how long it will take Microsoft to “get” mobile. They have a heavy foot in the market as the commodity OS on the smartphones that can’t get any buzz. But the choice to bring the worst things about the Windows Desktop experience to their mobile OS was unfortunate. Should I really get a pop-up that has to be manually dismissed every time I get an email or encounter a wireless network? Do I have to pull out the stylus and click on Start every time I want to do anything? What’s even more worrisome is that Windows Mobile is a separate OS from Windows, that merely emulates it, as opposed to sharing a code base. Apple’s OS is the same OSX that you get on a MacBook, just stripped down, and Google’s OS is already starting to appear on Netbooks and other devices, and will likely fuel full desktops within a year or two — it is, after all, Linux.

So, the state of the smartphone market is easily broken into the haves and have-nots, meaning that some phones have far more usable and exciting functionality, while most phones don’t. There’s a whole second post dealing with the choice of carriers and their rankings in the race to offer the most customer disservice, and it does play into your smartphone decision, as Verizon might be a very stable network, but their phone selection is miserable, and AT&T might have the best selection but, well, they’re AT&T. I love Android, so, were I looking, I’d hold out until four or five of those new sets are out. But I don’t know anyone with an iPhone who’s unsatisfied (and I know lots of people with iPhones).

Regular (Expression) Magic

Let’s get a bit geeky. Many Idealware visitors come here for advice on purchasing and deploying data management systems, such as donor databases, constituent relation management systems and content management systems. And, more often than not, are replacing older systems with new ones, meaning that one of the trickiest tasks is data migration. If any of this work has ever fallen to you, then you might have found yourself doing tedious editing and corrections in Excel, pouring over data screens or rows in Access trying to formalize non-formalized data entry, and generally settling for some lost or incorrect data moving from old system to new.

Wouldn’t it be great to have a magic wand that can instantly reformat the data to the proper format? Well, I have one for you. But, just as Harry Potter had to go to school before he could effectively wave his wand, mine comes with a lesson or two as well.

The wand in question is a search/replace language called regular expressions. Regular expressions are a set of terms that can be used, in supported software, to perform advanced search and replace functions. They were originally popularized in the Unix Stream Editor (SED), but are now standardly found in text editors, word processors, scripting languages (such as PHP) and other software, usually as an advanced option.

The reason to use them instead of a regular search and replace function is simple: they can search for things that regular search tools can’t. For example:

  • the first three characters at the beginning of each line
  • the three at the end of each line
  • one or more spaces

Regular expressions can also do multiple replacements in one phrase, allowing you to either remove the first comma encountered in a sentence, or all commas. Here are the basics:

A regular expression takes the form of /Search Phrase/Replacement/. A simple search to replace all instances of the word “fish” with the word “bird” would look like:


But regular expressions only prove their worth when you learn their special characters:

. (any character)

* (one or more characters)

^ (the beginning of a line

$ (the end of a line)

() (parentheses surrounding characters in the search phrase can be recalled in the replacement)

$1, $2 (substitute in the replacement for characters saved by parentheses in the search phrase)

\ (backslashes treat the next character literally, even if it’s a Regular expression special character)

[a-z], [0-9], [A-Za-z] (groupings search for all of the characters specified between the brackets, using dashes to identify ranges


If you have a text printout of a document that you want to whittle into something more useful, like a CSV file, step one might be to remove any dead space.

/ */ /

will search for one or more spaces (the asterisk means “any number of the preceding character) and replace them with one space.


will remove all blank lines (lines with nothing between the beginning and the end of the line)

If you are moving data from one system to another, you might have to reformat dates for the new system. Say the old system exports dates as MM/DD/YYYY and the SQL database you’re importing them to expects YYYY-MM-DD. This Regular Expression will convert all dates to the new format:


Let’s break this down:

/ – a slash starts the search phrase section.

( – parentheses surround things that we want to remember, so this starts a section we’ll remember.

[01][0-9] – a month (MM) will be a number between 1 and 12, so, if your system is exporting dates with leading zeros (if not, you can do this with a series of regular expressions to get around that), then the [01] set will match either a leading zero or a one. The [0-9] set will match any digit following that one or zero.

) – this will be remembered in the replacement as $1, because it’s the first thing we remembered.

\/ – since the slash is a regular expression special character (the delimiter), we precede it with a backslash, telling the parser to treat it a a slash, not a delimiter.

([0-3][0-9]) – this will find any pair of numbers between 01 and 39, which we know as the day, and remember it as $2, because it’s enclosed in parentheses.

\/ – next slash

([12][0-9][0-9][0-9]) – this catches the year. You see how, right? It is specifying that the year will be in this millennia or the last by limiting the first character to one or two. We use parentheses to remember this as well.

/ – this slash signifies that the search phrase is done, and the replacement will follow.

$3-$1-$2 – this takes our three remembered phrases and reorders them from month, day, year to year ($3), month ($1), day ($2), placing dashes in-between them.

/ – finally, we close the command with a slash.

One of my standard uses is to take a list – which could be an Excel spreadsheet, or a database dump, or a Word table — clean it up, and then format it into SQL statements that can then be pulled into a database. Most databases can import in CSV files, but Excel, while good at doing some reformatting, can’t do the fancy cleanup tasks that my regular expression-enabled editor can. Once my specific clean-up chores are done, if I’m left with a tab-delimited file, I can do the following three simple searches to turn it into a SQL input file that can just be run in my SQL interpreter.

/\t/’,'/ — searches for all tabs (\t is a symbol that means “tab”) and replaces them with ‘,’

/(.)$/$1′);/ – searches for the last character in a line and replaces it with that character followed by a close quote, close parens and semi-colon.

/^(.)/insert into players (name, title, company) values (‘$1/ – searches for the first character in any line and prepends the front end of the SQL statement.

If we had an input file with lines like this:

Joe NamathQuarterbackForty-niners

It would become

insert into players (name, title, company) values (‘Joe Namath’,'Quarterback’,'Forty-niners’);

There are plenty of excellent resources for learning about regular expressions on the web, but many of them are targeted at programmers, making them a bit thick to read through. For more friendly introductions, I recommend The quickstart. While many text-processing tools, including Microsoft Word, support regular expression search and replace, I recommend using a good text editor over a word processor, because it will likely include supporting functionality, such as block copying/pasting, and they’ll handle very large files with far more speed and grace. I’ve been happy using TextPad and EditPlus on Windows, and TextMate and TextWrangler on the Mac. Wikipedia publishes an incomplete list of applications that include regular expression functionality.

Does Your Data have a Bad Reputation?

notepad.jpgPhoto by StarbuckGuy

As you probably know, the U.S. Congress has been having a big debate about what went on behind closed door briefings on the treatment of detainees in the war on terrorism. At issue is whether House Leader Nancy Pelosi was told about the use of harsh interrogation tactics, which many of us define as torture, in 2002 and 2003 briefings, when the tactics were actually in use. Rep. Pelosi maintains that they weren’t discussed; The CIA, responsible for the briefings, maintains that they were, but neither of them has yet provided documentation that might settle the matter. Meanwhile, Rep. Pelosi’s Democratic colleague, Rep. Bob Graham, who, as head of the Senate Intelligence Committee, was also to be briefed on such actions, reports that the CIA’s assertions are in error. Dates that they claim he was in briefings on the subject are wrong. His his meticulous notes, which he has traditionally been kidded about keeping, establish that only one of four CIA-alleged meetings actually occurred, and, in it, the harsh interrogation tactics weren’t discussed.

At this point, you might well be asking why I’m bringing this up on the Idealware blog. And the answer is, because it’s about data, or, more to the point, the integrity of data and data keeping systems, and that’s a topic close to our hearts here at Idealware. This example was inspired by some great reporting by the frivously-named, but thought-provoking blog BoingBoing, and a post of theirs on May 21st titled “Bob Graham’s much-scoffed-at little notebooks are more reliable than the CIA’s records“. They quote Gary Wolf’s post (which I highly recommend reading) about the intriguing fact that the CIA backed off of their record keeping claims rather quickly upon learning that they didn’t jibe with Graham’s personal notes. Consider this for a minute: Bob Graham’s personal note-taking has more authority than the record keeping of the Central Intelligence Agency. The killer line from Wolf’s post is:

“Personal data, kept by a dedicated and interested party, even using yesterday’s technology, will trump large scale collection systems managed by bureaucrats.”

You can find some really excellent advice here at Idealware on what to buy and how to implement the software that will manage the critical information that your organization lives and dies by. You can spend hundreds of thousands of dollars deploying it. But it, too, might be outclassed by the scribbling of a person who’s scribble-keeping habits are far less impeachable (to keep the political allegory going) than the data integrity securing processes that you build around your system.

When you deploy that software, one thing to consider is “who owns this data? Who has the most respect for it?”. Distribute the data entry duties in ways that insure that the people who first put that data into the system care about it, and are invested in seeing that it goes in correctly. Then, integrate your systems in ways that eliminate duplicate entry of that data. Set up triggers that push data from the authoritative systems of record (the ones that the people who care enter the data into) to the auxiliary systems, insuring that no donor or client’s name is misspelled one place, but correct in another; and that a $50 donation via the web site isn’t recorded as a $500 entry in your donor database.

Doing this will insure that your data-keeping systems have the upstanding reputations that your organization depends on.