In this article I’m hoping to present some of the options available to you to assess the usability of a CMS you have on your shortlist. This article discusses steps you can take to help you build a picture of how the CMS functions and how it might be used to fulfil your needs.

I thought about writing this article because :

  1. CMS usability is very difficult for vendors to attain, most CMS are very complex consisting of suite of applications catering for users of all abilities (from junior editors to system administrators).
  2. CMS usability is difficult to assess. You may not have the time or resources to give the product a proper test drive, or be overwhelmed by the sheer amount of functions available.

Developing use case scenarios

Stephen Krug’s book “Don’t make me think” tells you precisely how you should go about usability testing. You’ll need a small number of authors and editors (at least 3-4 to make it worthwhile) and a list of scenarios which encompass the most common tasks that you will be performing, these scenarios might include:

  1. Creating new content. Creating a new article (with images) and promoting it on the home page, relating articles – either directly or via meta data (such as a taxonomy)
  2. Updating and approving existing content (include user generated content). This would demonstrate workflow and approval processes and versioning capabilites.
  3. Updating design. Modifying layout of templates and page design elements

Getting your hands dirty

There are a number of options presented below, each depends on the time, budget and resources you have at your displosal. You are likely to be looking at more than one CMS during the vendor and online demos, and looking at going on training and a procuring a proof of concept only for your preferred solution.

  1. Vendor demo
  2. You go through the product with the vendor, a well-managed vendor demonstration should provide a pretty good idea about the usability of the product. These demos usally last around two hours which is usually not enough to get an in-depth understanding of the product, but it’s a good start, and should help you narrow the field.

  3. Online demo
  4. Most vendors will be able to provide you with an online demo site for their CMS. This offers a simple way of playing with the product in your time and using your own equipment. The demo won’t be populated with your content or have your site structure, however it could give you a good feel for how everything fit’s together. A word of warning though as almost every CMS requires training, simply playing with a product could be potentially misleading and can become very easily frustrating!

    Ideally you would want to run through your most common scenarios on the demo without too much input from the vendor. If you can without getting confused or disoriented then it bodes well for how others will be receptive to it.

  5. CMS training
  6. Ask the vendor about end user training for the CMS. This will help to answer many questions: is training available? Some vendors don’t provide user training as they believe that their product doesn’t need it! If it’s needed how long is it? 1 day or 5 days? Some authors could be sent on training to find out how simple the product was to use? Can the authors understand the product at the end of training?

  7. Proof of concept
  8. You should ask the vendor to create and deploy a proof of concept based on your scenarios for you to work on. You can then conduct scripted tests with trained users to help identify any major issues.

Even with using the techniques presented here, CMS usability can still be difficult to assess e.g. what is simple for one content editor could be difficult for another. Nevertheless, I believe it is vital to conduct an evaluation of usability as part of the CMS selection process, and it could be the difference between project success and failure.

Is usability testing a part of your CMS selection process? What has worked for you?


Don’t make me think (usability testing by Stephen Krug)

Practical ways to assess CMS Usability