Tag Archives: webdev

Web Development

Year-end Reflections

This post was originally published on the NTEN Blog on December 24th, 2015.

As years go, 2015 was a significant one in my career. The work of a CIO, or IT Director, or whatever title you give the person primarily responsible for IT strategy and implementation, is (ideally) two parts planning and one part doing. So in 2015—my third year at Legal Services Corporation—we did a couple of the big things that we’d been planning in 2013 and 2014.

First and foremost, we (and I do mean we—I play my part, but I get things done with an awesome staff and coworkers) rolled out the first iteration of our “Data Portal.” The vision for the Data Portal is that, as a funder that works primarily with 134 civil legal aid firms across the U.S. and territories, we should be able to access the relevant information about any grantee quickly and easily without worrying about whether we have the latest version of a document or report. To reach this vision, we implemented a custom, merged Salesforce/Box system. This entailed about a year of co-development with our partner, Exponent Partners, and a move from in-house servers to the Cloud. We’ll complete our Cloud “trifecta” in early 2016, when we go to Microsoft’s Office 365.

This was particularly exciting for me, because I have been envisioning and waiting for technology to reach a level of maturity and… collegiality that makes the vision of one place where documents and databases can co-exist a reality. Integration, and one-stop access to information, have always been the holy grails that I’ve sought for the companies that I’ve worked for; but the quests have been Monty Python-esque through the days when even Microsoft products weren’t compatible with each other, much less compatible with anything else. What we’ve rolled out is more of a stump than a tree; but in the next year we’ll grow a custom grants management system on top of that; and then we’ll incorporate everything pertinent to our grantees that currently hides in Access, Excel, and other places.

I’m working on a much more detailed case study of this project for NTEN to publish next year.

Secondly, we revamped our website, doing a massive upgrade from Drupal 7 to… Drupal 7! The website in place when I came to LSC was content-rich, navigation-challenged, and not too good at telling people what it is that we actually do.The four separate websites that made up our entire site weren’t even cross-searchable until we addressed that problem in early 2014. Internal terminology and acronyms existed on the front page and in the menus, making some things incomprehensible to the public, and others misleading. For example, we often refer to the law firms that we fund as “programs.” But, in the funding world, a “program” is a funding category, such as “arts” or “environment.” Using that terminology. along with too buried an explanation that what we actually do is allocate funding, not practice law ourselves, led many people to assume that we were the parent office of a nationwide legal aid firm, which we aren’t.

The new site, designed by some incredibly talented people at Beaconfire-RedEngine (with a particular call out to Eve Simon, who COMPLETELY got the aesthetic that we were going for and pretty much designed the site in about six hours), tells you up front who we are, what we do, and why civil legal aid is so important, in a country where the right to an attorney is only assured in criminal cases. While civil cases include home foreclosures, domestic violence, child custody, and all sorts of things that can devastate the lives of people who can’t afford an attorney to defend them. This new site looks just as good on a phone as on a computer, a requirement for the Twenty-Teens.

My happiness in life directly correlates to my ability to improve the effectiveness of the organizations that I work for, with meaningful missions like equal justice for all, defense against those who pollute the planet, and the opportunity to work, regardless of your situation in life. At my current job, we’re killing it.

RFPs GOOD. Fixed Bids BAD.

It occurs to me that my signature rant these days is not clearly posted on my own blog. Let’s fix that!

As I’ve mentioned before. Requests for Proposals (RFP’s) are controversial in the nonprofit sector. Vendors hate them. dollar-163473_640Nonprofits struggle with developing them. I’ve been on a multi-year mission to educate and encourage the community to rethink RFPs, as opposed to throwing them out. In particular, nonprofits need to break away from fixed bid requests when hiring web developers, programmers, and people who implement CRMs. Here’s why:

Done correctly, RFP’s are an excellent practice. A good RFP informs potential vendors about the organization, their current condition, and their project goals. A questionnaire can focus on vetting the expertise of the consultant, examples of prior work, stability of the company, etc. All good things to know before investing serious time in the relationship. The RFP can also request billing rates and the like, but, in my experience, the cheaper rates don’t always correlate with ultimate project cost. Some higher hourly consultants do the work in half the time of some moderately priced ones.

The problem is that many nonprofits want to get that fixed bid and then hire the lowest bidder. But, for a web design or CRM project, the odds that the nonprofit knows how many hours the project is going to take are practically nil and, what’s more, they absolutely shouldn’t know. With a good consultant, you’re going to learn a lot in the process about what you should be doing. With a wild guess-based fixed bid, you are likely to suffer from one of two problems:

  • The project will be seriously underbid (very likely) and the vendor relationship will get worse and worse as they keep expending more hours without being compensated;
  • Or the vendor will finish up in half or two thirds of the hours and there you’ll be, donating to their charity.

You can vet the fiscal competence of a consultant.  Check their references and ask good questions like:

  • “Did the project come in at or under budget?”
  • “Was the vendor able to scale the project to your budget?”
  • “Can you tell me about a time that you had a billing disagreement with them, and how well it was resolved?”

Also, check their reputation in the nonprofit sector, because we have lots of mailing lists and forums where you can do that.

I hire consultants based on their expertise, reputation, and compatibility with my organization’s goals and work style. I stress that vendor interviews should be with the staff that I will most likely be working with. I’ll often break a project into two phases, one for discovery and then another for implementation. With the great consultants that I work with, this does not result in over-budget implementation bids. Instead, it helps us define what we can do and stay within budget. Because this is all about taking away the guess work.

So, RFPs are good things, as long as they are making realistic requests of the vendors. The crisis with them in our sector is based simply on the fact that most of our RFPs ask questions that can not, and should not, be answered, such as “how much will you charge me to do this undetermined amount of work?”

Architecting Healthy Data Management Systems

This article was originally published in the NTEN eBook “Collected Voices: Data-Informed Nonprofits” in January of 2014.

tape-403593_640Introduction

The reasons why we want to make data-driven decisions are clear.  The challenge, in our cash-strapped, resource-shy environments is to install, configure and manage the systems that will allow us to easily and efficiently analyze, report on and visualize the data.  This article will offer some insight into how that can be done, while being ever mindful that the money and time to invest is hard to come by.  But we’ll also point out where those investments can pay off in more ways than just the critical one: the ability to justify our mission-effectiveness.

Right off the bat, acknowledge that it might be a long-term project to get there.  But, acknowledge as well, that you are already collecting all sorts of data, and there is a lot more data available that can put your work in context.  The challenge is to implement new systems without wasting earlier investments, and to funnel data to a central repository for reporting, as opposed to re-entering it all into a redundant system.  Done correctly, this project should result in greater efficiency once it’s completed.

Consider these goals:

  • An integrated data management and reporting system that can easily output metrics in the formats that constituents and funders desire;
  • A streamlined process for managing data that increases the validity of the data entered while reducing the amount of data entry; and
  • A broader, shared understanding of the effectiveness of our strategic plans.

Here are the steps you can take to accomplish these goals.

Taking Inventory

The first step in building the system involves ferreting out all of the systems that you store data in today.  These will likely be applications, like case or client management systems, finance databases, human resources systems and constituent relationship management (CRM) systems.  It will also include Access databases, Excel spreadsheets, Word documents, email, and, of course, paper.  In most organizations (and this isn’t limited to nonprofits), data isn’t centrally managed.  It’s stored by application and/or department, and by individuals.

The challenge is to identify the data that you need to report on, wherever it might be hidden, and catalogue it. Write down what it is, where it is, what format it is in, and who maintains it.  Catalogue your information security: what content is subject to limited availability within the company (e.g., HR data and HIPAA-related information)? What can be seen organization-wide? What can be seen by the public?

Traditionally, companies have defaulted to securing data by department. While this offers a high-level of security, it can stifle collaboration and result in data sprawl, as copies of secured documents are printed and emailed to those who need to see the information, but don’t have access. Consider a data strategy that keeps most things public (within the organization), and only secures documents when there is clear reason to do so.

You’ll likely find a fair amount of redundant data.  This, in particular, should be catalogued.  For example, say that you work at a social services organization.  When a new client comes on, they’re entered into the case management system, the CRM, a learning management system, and a security system database, because you’ve given them some kind of access card. Key to our data management strategy is to identify redundant data entry and remove it.  We should be able to enter this client information once and have it automatically replicated in the other systems.

Systems Integration

Chances are, of course, that all of your data is not in one system, and the systems that you do have (finance, CRM, etc.) don’t easily integrate with each other.  The first question to ask is, how are we going to get all of our systems to share with each other? One approach, of course, is to replace all of your separate databases with one database.  Fortune 500 companies use products from Oracle and SAP to do this, systems that incorporate finance, HR, CRM and inventory management.  Chances are that these will not work at your nonprofit; the software is expensive and the developers that know how to customize it are, as well.  More affordable options exist from companies like MicroSoft, Salesforce, NetSuite and IBM, at special pricing for 501(c)(3)’s.

Data Platforms

A data platform is one of these systems that stores your data in a single database, but offers multiple ways of working with the data.  Accordingly, a NetSuite platform can handle your finance, HR, CRM/Donor Management and e-commerce without maintaining separate data stores, allowing you to report on combined metrics on things like fundraiser effectiveness (Donor Management and HR) and mail vs online donations (E-commerce and Donor Management).  Microsoft’s solution will incorporate separate products, such as Sharepoint, Dynamics CRM, and the Dynamics ERP applications (HR, Finance).  Solutions like Salesforce and NetSuite are cloud only, whereas Microsoft  and IBM can be installed locally or run from the cloud.

Getting from here to there

Of course, replacing all of your key systems overnight is neither a likely option nor an advisable one.  Change like this has to be implemented over a period of time, possibly spanning years (for larger organizations where the system changes will be costly and complex). As part of the earlier system evaluation, you’ll want to factor in the state of each system.  Are some approaching obsoletion?  Are some not meeting your needs? Prioritize based on the natural life of the existing systems and the particular business requirements. Replacing major data systems can be difficult and complex — the point isn’t to gloss over this.  You need to have a strong plan that factors in budget, resources, and change management.  Replacing too many systems too quickly can overwhelm both the staff implementing the change and the users of the systems being changed.  If you don’t have executive level IT Staff on board, working with consultants to accomplish this is highly recommended.

Business Process Mapping

BPM_Example

The success of the conversion is less dependent on the platform you choose than it is on the way you configure it.  Systems optimize and streamline data management; they don’t manage the data for you.  In order to insure that this investment is realized, a prerequisite investment is one in understanding how you currently work with data and optimizing those processes for the new platform.

To do this, take a look at the key reports and types of information in the list that you compiled and draw the process that produces each piece, whether it’s a report, a chart, a list of addresses or a board report.  Drawing processes, aka business process mapping, is best done with a flowcharting tool, such as Microsoft Visio.  A simple process map will look like this:

In particular, look at the processes that are being done on paper, in Word, or in Excel that would benefit from being in a database.  Aggregating information from individual documents is laborious; the goal is to store data in the data platform and make it available for combined reporting.  If today’s process involves cataloguing data in an word processing table or a spreadsheet, then you will want to identify a data platform table that will store that information in the future.

Design Considerations

Once you have catalogued your data stores and the processes in place to interact with the data, and you’ve identified the key relationships between sets of data and improved processes that reduce redundancy, improve data integrity and automate repetitive tasks, you can begin designing the data platform.  This is likely best done with consulting help from vendors who have both expertise in the platform and knowledge of your business objectives and practices.

As much as possible, try and use the built-in functionality of the platform, as opposed to custom programming.  A solid CRM like Salesforce or MS CRM will let you create custom objects that map to your data and then allow you to input, manage, and report on the data that is stored in them without resorting to actual programming in Java or .NET languages.  Once you start developing new interfaces and adding functionality that isn’t native to the platform, things become more difficult to support.  Custom training is required; developers have to be able to fully document what they’ve done, or swear that they’ll never quit, be laid off, or get hit by a bus. And you have to be sure that the data platform vendor won’t release updates that break the home-grown components.

Conclusion

The end game is to have one place where all staff working with your information can sign on and work with the data, without worrying about which version is current or where everything might have been stored.  Ideally, it will be a cloud platform that allows secure access from any internet-accessible location, with mobile apps as well as browser-based.  Further considerations might include restricted access for key constituents and integration with document management systems and business intelligence tools. But key to the effort is a systematic approach that includes a deep investment in taking stock of your needs and understanding what the system will do for you before the first keypress or mouse click occurs, and patience, so that you get it all and get it right.  It’s not an impossible dream.

 

Making Your Website More Useful For More People

This post was originally published on the LSC Technology Blog in January of 2014. LSC is Legal Services Corporation, my employer.

At LSC, we’ve been taking a critical look at our web site, to see if we can make it a more useful web site by factoring in all of the ways that people might want to view or use our information. In these days of big data and small screens, we realize that we have to be much more attentive to the ways that we present data than we have in the past.

Identifying the different visitors who frequently use our site, we took a closer look at their needs, and how we could improve our delivery of information to them. For example, visitors to LSC’s web site could be:

  • reporters or Hill staffers looking for a quick cut and paste of data on the site that is hard to get out of a linked PDF;
  • general public looking for data to pull into a spreadsheet, who would also be disappointed to find that data in a PDF;
  • visually or physically impaired, and therefore not able to view web content that isn’t compliant with the standards that their specialized software requires;
  • accessing the site on a mobile device that doesn’t display flash or video and has no capability to display a PDF

The PDF Problem

Adobe has done great things with the Portable Document Format, opening it up as a public standard and continually improving the functionality of the format. But this is not an optimal format for web-based content, because PDFs require additional software in order to be viewed, and they need to be created with a solid understanding of how PDFs need to be prepared, so that they are compatible with accessibility standards. Our goal is to ensure content is delivered optimally, and in a format that makes it easy to access for anyone and everyone visiting our site.

In the past, we’ve relied heavily on publishing web content via PDF, and we now have a backlog of documents that aren’t as widely usable as we would like. Our plan is to immediately make two changes:

  1. Use PDF sparingly and thoughtfully as we move forward. Use PDFs as optional downloads for content that is also displayed in HTML, or as appropriate downloads for white papers and legal reports that aren’t the types of things that users will want to quote or edit; design PDFs that are compatible with the section 508 standards for web accessibility.
  1. Determine which of our existing PDFs need to be republished in more accessible formats and convert them. We don’t have the resources to fix everything, but we have good statistical data from Google Analytics to tell us which PDFs our visitors look at and a good idea how to prioritize this content.

Open Web

As a nonprofit that allocates federal funds, we have a responsibility to make data available to the public. But a commitment to open data means more than just making the data available; it needs to be available in formats that people can easily use. Data stored in an HTML table can be copied and pasted into Excel. Data in PDF and image formats can’t be, at least, not easily. As David Ottewell recently tweeted, a PDF of a spreadsheet is not a spreadsheet. These efforts dovetail with our broader efforts to make data available in manipulatable formats.

Wild, Wild Web

It is also important that our web site deliver the same user experience on smartphones or a tablets as it would when viewed on desktop or laptop browsers. This wasn’t high on our radar in 2011, when we redesigned our website in the Drupal content management system. At the time, we developed a mobile site as a separate, fractional copy of our main site.

Looking ahead

A  modest revamp of LSC.GOV is planned for second half of 2014 to improve the site navigation and responsiveness on multiple devices (e.g. one site that alters it’s navigational elements and appearance to properly utilize the screen that it’s displayed on). We also won’t forget the visitors that don’t have smart phones and how best to make information available to them.

Having a website that anticipates their diverse needs of our online visitors is our goal. What’s yours? What are your current challenges?

Hacking For Justice

This post was originally published on the LSC Technology Blog in May of 2013. Note that “LSC” is Legal services Corporation, my current employer, and “TIG” stands for “Technology Initiative Grants”.

Welcome to the new LSC Technology blog, hosted here on the TIG site, and written by TIG and Information Technology staff. To kick this off, I wanted to report on a fun, exciting, and long overdue initiative we’re on: making our non-confidential data available to hackers.  Let me be clear here, for those of you who have any bad associations with the word, that  a “hacker” is not a computer criminal or spy.  The term has been misused to connote such things, but the original and current definition of a hacker is simply someone who likes to take things apart and rebuild them better, or take things apart and make new things out of them. Most recently, hacking and hackers have been tied to the community of civic-minded web application developers who want to take publicly available data and make it accessible and relevant to their communities. And that’s the group of hackers that we’re discussing.

Hackers hold Hackathons, extended sessions where hackers get together to collaborate on projects. At the first LSC Tech Summit, United States Chief Technology Officer Todd Park addressed the group and urged us to model the behavior of the Department of Health and Human Services by holding hackathons and letting developers build the rich demographic applications that tell our story.

June 1st is the National Day of Civic Hacking.  Across the United States, “Hackathons” will be held in cities of towns, and the attendees will show up with their laptops, connect to the wifi, and create map mashups using tools like Google Maps and a collection of public data sets. The About section of the website describes it like this:

“The event will bring together citizens, software developers, and entrepreneurs from all over the nation to collaboratively create, build, and invent new solutions using publicly-released data, code and technology to solve challenges relevant to our neighborhoods, our cities, our states and our country.”

We’re busy analyzing our data sets, many of which are already available via our web site, but not in the most flexible formats. We’re also working with friends and partners like ProBono.Net to identify more legal aid data, on the assumption that the richer the data set, the more inspiring it will be for the hackers to work with. And I’m looking into other ways to make this information available, such as submitting it to the U.S. open data repository at Data.Gov. A big tip of our hat is due to Kate Bladow, who alerted me to the Civic day of Hacking to begin with, aware of how great it would be if we could get our data sets there on time.

Two questions for you:

  1. What kind of mapping mashups would you like to see done with LSC and related data? We can’t tell the developers what to do, but we should be able to tell them what people would love to see, and hopefully inspire them.
  2. Are you a developer? Whether you’re a C++ maven or just somebody who figured out how to save a Google Map, you might enjoy and benefit from participating in the hackathon.  Do consider it.  I’ll be attending the Baltimore day on June 1st. See you there?

Delicious Memories

This article was originally published on the Idealware Blog in December of 2010.

Like many of my NPTECH peers, I was dismayed to learn yesterday that Delicious, the social bookmarking service, was being put to pasture by Yahoo!, the big company that purchased the startup five years ago.  Marshall Kirkpatrick of ReadWriteWeb has written the best memorial,  But the demise of Delicious marks a passing of significant note to our community of nonprofit staff that seek innovative uses of technology.  So let me talk quickly about how Delicious brought me into this community, and, along the way, a bit about what it meant to all of us.

In 2002, I was wrapped up in my job as VP of Information Technology at San Franciscco Goodwill.  At that time, the buzz term was “Web 2.0”, and it was all over the tech press with about a thousand definitions.  We all knew that “Web 2.0” meant the evolution of the web from a straight publisher to consumer distribution medium to something more interactive, but nobody knew exactly what. Around that time, I started reading columns by Jon Udell about RSS, technology that would, as a simpler, subset of XML, helps us share web-based information the way that newspapers share syndicated content, such as comic strips and columns.  I was really intrigued.  The early adopters of RSS were bloggers, and what I think was very cool about this is that RSS was free technology that, like the web, advanced the opportunities of penniless mortals to become global publishers.  People who couldn’t tell an XML feed from an XL T-Shirt were championing an open standard, because it served as the megaphone in front of their soapboxes.

I kept my eye out for innovative uses of RSS,a nd quickly discovered Joshua Schacter’s del.icio.us website.  This was a social bookmarking service where, by adding a little javascript link to your web browsers bookmark bar (or quick links, or whatever), you could quickly save any web page you enjoyed to an online repository for later retrieval.  That repository was public, so others could see what you found valuable as well.  But this is where Schacter jumped the gun, and championed two information technology strategies that have, since that time, significantly changed the web: tagging and rss.

Tagging

In addition to the link and a brief description, you could add keywords to each bookmark, and then later find related bookmarks by that keyword.  You could just find the bookmarks that you tagged with a word, or you could find the tags that anyone using Delicious tagged with that word.  So, if you were studying the russian revolution, you could search Delicious for russia+revolution and find every bookmark that anyone had saved,   This was different than searching for the same terms in Google or yahoo, because the results weren’t just the most read; they were the sites that were meaningful enough to people to actually be saved.  Delicious became, as Kirkpatrick points out,  a mass-curated collection of valuable information, more like wikipedia than, say, Yahoo Directory.  Delicious was the lending library of the web.

RSS

In addition to searching the site for tags by keyword and/or user, any results your searching found could be subscribed to via RSS.  This was crazy powerful! Not only could you follow topics of interest, but, using PHP add-ons like MagpieRSS or aggregation functions like those built into Drupal, Joomla, and pretty much any major Content Management System, you could quickly incorporate valuable, easily updated content into your website.  I immediately replaced my static “Links” page on my website to one that grabbed items witha  particular keyword from Delicious, so that updating that Links page was as easy as bookmarking a site that I wanted listed there.

NPTECH

I wasn’t the only nonprofit strategist taking note of these developments.  One day, while browsing items that Delicious termed Popular (e.g., bookmarks that multiple people had saved to the site), I noted a blog entry titled “The Ten Reasons Nonprofits Should Use RSS“.  The article was written by one Marnie Webb of CompuMentor (now better known as TechSoup, where she is one of the CEOs).  A week or so later, while following the office email mailing lis for Delicious, I encountered Marnie again, and, this time, emailed her and suggested that we meet for lunch, based on our clearly common interest in nonprofits and RSS.  Marnie told me about the NPTech Tagging Project, and effort she started by simply telling her friends to tag websites related to nonprofit technology with the tag “nptech” on Delicious, so that we could all subscribe to that tag in our RSS readers.

Marnie and I believe that what we started was the first mass information referral system of this type.  In 2005 we took it up a level by creating the nptech.info website, which aggregates items tagged with nptech from Delicious, Twitter, Flicker and numerous other sources across the web. Nptech.info is now more widely read via it’s Twitter feed, @nptechinfo.

I think it’s safe to say that the nptech tagging project grew from a cool and useful idea and practice into a community, and a way that many of us identify who we are to the world.  I’m a lot of things, but nptechie sums most of them up into one simple word.  I know that many of you identify yourselves that way as well.

An offshoot of meeting Marnie on the Delicious mailing list was that she introduced me to NTEN, and brought me into the broad community of nptech, and my current status as a blogger, writer, presenter, Idealware board member and happy member of this broad community ties directly back to the Delicious website.  I stopped using the site as a bookmarking service some time ago, as efforts that it inspired (like Google Reader sharing)  became more convenient.  But I still subscribe to Delicious feeds and use it in websites.  It’s demise will likely be the the end of nptech,info.  Efforts are underway to save it, so we’ll see.  But even if this article is the first you’ve heard of Delicious, it’s important to know that it played a role in the evolution of nonprofit technology as the arbiter of all things nptech.  It’s ingenuity and utility will be sorely missed.

The Years Of The Kat

goodhunting

This is a memorial post for Krazy.com, a domain that I registered in February of 1995, back when Network Solutions was the only domain registrar and the annual registration fee was $0. I had recently closed my computer bulletin board system, which was called the Coconino County BBS, after the home of George Herriman‘s classic comic character, Krazy Kat. In it’s place, I put up a web site that grew to be the most complete and best known source of information on the now somewhat obscure, but dearly loved early 20th century newspaper strip about a Kat, named Krazy, who loved a Mouse, named Ignatz. This Ignatz found Krazy quite silly, and showed his disdain by throwing bricks at his/her head (Krazy’s gender was never identified). Offisa Pup, the local Kanine Konstable, who was in love with Krazy, arrested Ignatz every time he caught the mouse in the act. And all of this action took place against a surreally fluid landscape of mesas, monuments and moons inspired by Herriman’s love for eastern Arizona Navajo country, with it’s painted desert and monument valley.

pupslept
As my nptech crowd knows, I just got too busy over the years with other things to properly grow and manage this web site. As much as I love Krazy Kat (and my son’s middle name is Ignatz, no lie!), I have to prioritize my current pursuits. I am blessed with the opportunity to do meaningful work at Earthjustice, to blog, and to help out the nonprofit community where and when I can, as a board member at Idealware, a contributor to Techsoup, and a steadfast supporter of NTEN. There are only so many hours in a day.

Krazy.com had the distinction of being a short, catchy, .com domain name, which means that it’s sale value ain’t hay, and, while my life’s pursuits are pretty rich, I’m not. I got an offer that matched what the domain is professionally valued at, and I couldn’t afford to turn it down. It’s a melancholy moment — one of those decisions that isn’t difficult to make, but is sad all of the same, like trading in a beloved car that will cost too much to keep running.

In the more than 15 years that Krazy.com got steady traffic, from visitors that included Herriman’s great grand-daughter and Krazy Kat book cover artist Chris Ware, I built my career, got married, had a child, built a house, and lived a life that continues to be happy and rewarding. Krazy Kat is fond of singing “There is a heppy lend, fur, fur away”. My heppy lend is right here, and I’m sorry that I have to move away from my beloved Coconino County.

PLAQUE

Adventures In Web Site Migration

This post was first published on the Idealware Blog in April of 2010.

I recently took on the project of migrating the Idealware articles and blog from their old homes on Idealware’s prior web site and Google’s Blogger service to our shiny, new, Drupal-based home. This was an interesting data-migration challenge. The Idealware articles were static HTML web pages that needed to be put in Drupal’s content database. And there is no utility that imports Blogger blogs to Drupal. Both projects required research and creativity.

The first step in any data migration project is to determine if automating the task will be more work than just doing it by hand. Idealware has about 220 articles published; cutting and pasting the text into Drupal, and then cleaning up the formatting, would be a grueling project for someone. On the other hand, automating the process was not a slam dunk. Database data is easier to write conversion processes for than free form text. HTML is somewhere in the middle, with HTML codes that identify sections, but lots of free form data as well.

Converting HTML Articles with Regular Expressions

My toolkit (of choice) for this project was Sed, the Unix Stream Editor, and a generic installation of Drupal. Sed does regular expression searching and replacing. So I wrote a script that:

  1. Deleted lines with HTML tags that we didn’t need;
  2. stored data between title and body tags;
  3. and converted those items to SQL code that would insert the title and article text into my Drupal database.

This was the best I could do: other standardized information, such as author and publishing date, was not standardized in the text, so I left calling those out for a clean-up phase that the Idealware staff took on. The project was a success, in it that it took less than two days to complete the conversion. It was never going to be an easy one.

Without going too far, the sed command to delete, say, a “META” tag is:

/\<meta/d

That says to search for a literal “less than” bracket (the forward slash implies literal) and the text meta and delete any line that contains it. A tricky part of the cleanup was to make sure that my search phrases weren’t ones that might also match article text.

Once I’d stripped the file down to just the data between the “title” and “body” tags, I issued this command:

s/\<title\>(.*)\<\/title\>.*\<body\>(.*)\<\/body\>/insert into articles (title, body) values (‘\1’, ‘\2’);/

This searches for the text between HTML “title” tags, storing it in variable 1, then the text between “body” tags, storing it in variable 2, then substitutes the variable data into a simple SQL insert statement in the replacement string. Iterating a script with all of the clean-up commands, culminating in that last command, gave me a text file that could be imported into the Drupal database. The remaining cleanup was done in Drupal’s WYSIWYG interface.

Blog Conversion

As I said, there is no such thing as a program or module that converts a Blogger Blog into Drupal format. And our circumstance was further complicated by the fact that the Idealware Blog was in Blogger’s legacy “FTP” format, so the conversion options available were further limited.

There is an excellent module for converting WordPress blogs to Drupal, and there were options for converting a legacy Blogger blog to WordPress. So, then the question was, how well will the blog survive a double conversion? The answer was: very well! I challenge any of you to identify the one post that didn’t come through with every word and picture intact.

I had a good start for this, Matthew Saunders at the Nonprofits and Web 2.0 Blog posted this excellent guide. If you have a current Blogger blog to migrate, every step here will work. My problem was that the Idealware blog was in the old “FTP” format. Google has announced that blogs in their original publishing format must be converted by May 1st. While this fact had little or no relationship to the web site move to Drupal, it’s convenient that we made the move well in advance of that.

To prep, I installed current, vanilla copies of WordPress and Drupal at techcafeteria.com. I tracked down Google’s free blog converters. While there is no WP to Drupal converter, most other formats are covered, and I just used their web-based Blogger to WordPress tool to convert the exported Idealware blog to WP format. The conversion process prompted me to create accounts for each author.

To get from WordPress to Drupal, I installed above-mentioned WordPress-import module. As with the first import, this one also prompted me to create the authors’ Drupal accounts. It also had an option to store all images locally (which required rights to create a public-writeable folder on the Drupal server). Again, this worked very well.

With my test completed, I set about doing it all over again on the new Idealware blog. Here I had a little less flexibility. I had administrative rights in Drupal, but I didn’t have access to the server. Two challenges: The server’s file upload limit (set in both Drupal and PHP’s initialization file) was set to a smaller size than my WordPress import file. I got around this by importing it in by individual blogger, making sure to include all current and former Idealware bloggers. The second issue was in creating a folder for the images, which I asked our host and designer at Digital Loom.com to do for me.

Cleanup!

The final challenge was even stickier — the posts came across, but the URLs were in a different format than the old Blogger URLs This was a problem for the articles as well. How many sites do you think link to Idealware content out there? For this, I begged for enough server access to write and run a PHP script that renamed the current URLs to their former names — a half-successful effort, as Drupal had dramatically renamed a bunch of them. The remainder we manually altered.

All told, about two hours research time, three or four hours conversion (over a number of days) and more for the clean-up, as I wasted a lot of time trying to come up with a pure SQL command to do the URL renaming, only to eventually determine that it couldn’t be done without some scripting. A fun project, though, but I’d call it a success.

I hope this helps you out if you ever find yourself faced with a similar challenge.

Why Google Buzz Should Be Your Blog

Buzzcafeteria
Now, you might think that’s a crazy idea, but  I think Buzz is about 80% of the way there. Last week, in my Google’s Creepy Profiles post, I made a suggestion (that someone at Google has hopefully already thought of) that it wouldn’t take much to turn a Profile into a full-fledged biography/lifestreaming site.  Just add some user-configurable tabs, that can contain HTML or RSS-fed content, and add some capability to customize the style of the profile.  Since I wrote that, I’ve been using Buzz quite a bit and I’ve really been appreciating the potential it has to deepen conversations around web-published materials.I think some of my appreciation for Buzz comes from frustration with Google’s previous, half-hearted attempts to make Google Reader more social. If you use Reader heavily, then you know that you can share items via a custom, personal page and the “People You Follow” tab in Reader. You also know that you can comment on items and read others comments in the “Comments View”.  But it’s far from convenient to work with either of these sharing methods.  But, once you link your reader shared items to Buzz, then you aren’t using Reader’s awkward interface to communicate; you’re using Buzzes.  And Buzz, for all of Google’s launch-time snafus, is an easy to use and powerful communications tool, merging some of the best things about Twitter and Facebook.

So, how is Buzz suitable for a blog?

  • It’s a rich editing environment with simple textile formatting and media embedding, just like a blog.
  • Commenting — way built-in.
  • RSS-capable – you can subscribe to anyone’s Buzz feed.
  • Your Google Profile makes for a decent public Blog homepage, with an “About the Author”, links and contact pages.
  • It’s pre-formatted for mobile viewing

What’s missing?

  • Better formatting options.  The textile commands available are minimal
  • XML-RPC remote publishing
  • Plug-ins for the Google Homepage
  • As mentioned, more customization and site-building tools for the Google Homepage.

Why is it compelling?

  • Because your blog posts are directly inserted into a social networking platform.  No need to post a link to it, hope people will follow, and then deal with whatever commenting system your blog has to respond.
  • Your blog’s community grows easily, again fueled by the integrated social network.
  • Managing comments – no longer a chore!

This is the inverse of adding Google or Facebook’s Friend Connect features to your blog.  it’s adding your blog to a social network, with far deeper integration that Twitter and Facebook currently provide. Once Google releases the promised API, much of what’s missing will start to become available.  At that point, I’ll have to think about whether I want to move this island of a blog to the mainland, where it will get a lot more traffic.  I’ll definitely be evaluating that possibility.

Drupal 101: Look and Feel

I’m wrapping up the Drupal 101 series with some talk about Drupal themes, and some additional info on topics that we’ve already covered. The goal of these posts is to give new Drupal administrators an idea about how Drupal works, and some pointers to the key add-ons and resources in the broad Drupal ecosystem. For reference’ sake, we started with an intro, moved on to Modules, and then covered navigation. So, now that we have a functional web site, what does it look like?

Getting Themes

Drupal comes with five or six themes to choose from, and, if you use them, then your site will look very, um, uninspired. This might not be a problem if your goal is not to impress your visitors, but simply provide information or functionality, but, if you’re putting up a website for your organization, you want one that stands out from the crowd. So you have two choices: you can find a better, less common theme, or you can customize one of the default themes.

The first place to go is to Drupal Theme Garden. This is where many Drupal theme designers share their work. Here, you can either find a theme to use (or customize for your use), or get a good idea about the types of things you can do with your theme.

Customizing Themes

From the Administration menu, you can modify any theme’s main text elements, deciding whether or not to display your site’s mission or slogan, name or logo. And you can replace the default “droplet” logo with your own logo (a no-brainer!). Assuming that you’ve started with a theme that you really like, this might be enough. But, if you want to do more serious customizations, such as moving the logo to the center of your header or changing the site colors, you’re going to need basic web 4.0 programming skills and, most likely, some level of comfort with the PHP scripting language.

Most themes consist of one or more style sheets, a number of “tpl” files with PHP/HTML code laying out various page elements, such as blocks, footers and sidebars, and one called page.tpl.php that establishes the overall page layout. The main styles are usually stored in styles.css, and you can make a lot of changes to your site’s appearance here, modifying default background colors and images, placing and resizing content.

If that’s not enough, most customizations can be done using WordPress’s internal macros and functions, meaning that you won’t have to worry about assigning variables or what goes into the foreach loops. WordPress has simple commands that you can insert into a page to loop through your posts and display them or list your categories in the sidebar. A nice breakdown of the WordPress functions can be found at WpExplorer.com.

If you do modify the stylesheets and templates, make sure that you are storing your themes in sites/default folder and that you’re properly backing up whenever you do an upgrade. If you modify theme files in the main themes folder, and then upgrade to, say, a Drupal security fix, your modifications will be overwritten. In general, themes remain functional from dot release to dot release (e.g., what worked for Drupal 6.1 still works in 6.9), but the Drupal maintainers often make dramatic changes in number versions, so don’t assume that your theme in Drupal 6.9 will not be messed up if you upgrade to Drupal 7 (coming soon).

More Installation Options

In the first Drupal 101 post, I mentioned Fantastico, a two-click installer for Drupal available on most hosting services that use the cPanel site management interface. I subsequently ran into this useful article about Elefante and Simplescripts. These are packages that you can use to install a variety of popular open source applications, including Drupal.

In addition to application installers, there are other options for installing Drupal:

Customized Drupal installations like Open Atrium and Acquia come with more modules and functionality.There’s been some development and discussion about Installation Profiles, a Drupal add-on functionality that lets you define additional installation details, such as module defaults and inclusion of additional modules and data for distributing custom Drupal installations.

Conclusion

What I hope this Drupal 101 series has done is to offer some context and guidance for people new to Drupal who are about to give it a try, and some backing to my initial proposition that Drupal’s strength is it’s flexibility. Along the way, I’ve received tweets asking “Why Drupal?” and my answer is that Drupal isn’t the only CMS out there, or necessarily the best one for your web site. There are a huge variety of commercial and open source options. In fact, my personal website runs on a combination of Frog CMS and WordPress, because I wanted a simple tool for integrating RSS feeds, which Frog provides, and a powerful blogging platform. On the other hand, last week the White House ditched their commercial CMS for Drupal. So this series might also inspire you to look elsewhere, particularly if a more traditional, tree-structured content management interface will work better for you than Drupal’s layout by association model. Whichever way you go, we suffer more from a surfeit of good options than a lack of same.