This is one of the questions I ask potential clients and usually the response is:

  1. “don’t know” or
  2. “our site can support up to x hundred/thousand concurrent users and there are a number of issues we are trying to iron out”

I was surprised to learn that most prospective clients *don’t know* the capacity of their website or service and have no effective plans in place for when there is a sharp increase in site visitors and it crashes (and “to panic” or “reboot” is not a plan)! It is usually after events like a site crashing regularly that web managers begin to consider what they should do about it, of course by which time it is too little too late as the reputational damage has already been done with those visitors affected.

What do we mean by website capacity?

The website capacity is the number of visitors that it can support with each visitor performing representative tasks (browsing, searching, navigating etc.) within specified Service Level’s.

When is it important?

Sites grow in two directions, with increasing visitor numbers and with the amount of content, both of these increases affect the performance of your site. Increasing the number of visitor could slow down the user experience and your visitors won’t hang around if your site is slow or irresponsive, it takes just seconds to arrive at a website and even less to leave it! Increasing the amount of content could also mean that your website doesn’t scale well and page rendering performance is inadvertently affected e.g. bread crumbs and navigation systems may not be generated as quickly.

So *before* launching your next site ensure that you are reducing risk of service failure by specifying and testing a robust, resilient, scalable and redundant architecture. Knowing that your web site is able to support 100’s of users concurrently and also knowing where the bottlenecks are will inform you:

  1. how and when the service is likely to fail, you will recognise this behaviour when and if it reoccurs
  2. mitigate and reduce risk of service failure, and
  3. allow you to build an appropriate business case to improve ongoing service delivery.

What can you do?

It useful to set a benchmark so you can demonstrate what effects any performance improvements your investment produces, there are usually some quick wins that can make a huge difference to your website capacity and in turn the user experience.

All of our clients now perform “stress testing” and ongoing capacity planning for their projects where service delivery within acceptable parameters is crucial. However this hasn’t always been the case.

Even with baked pages generated by some CMS systems there are a number of opportunities to improve your site capacity, e.g. by optimising your client side cache settings and will significantly reduce the number of connections that a user has to make to download each page.

Do you know the capacity of your website? What would you do if your visitor numbers due to a breaking news story suddenly increased 10 fold? 1000 fold? Could your website cope? Could you cope?

In this article I’m hoping to present some of the options available to you to assess the usability of a CMS you have on your shortlist. This article discusses steps you can take to help you build a picture of how the CMS functions and how it might be used to fulfil your needs.

I thought about writing this article because :

  1. CMS usability is very difficult for vendors to attain, most CMS are very complex consisting of suite of applications catering for users of all abilities (from junior editors to system administrators).
  2. CMS usability is difficult to assess. You may not have the time or resources to give the product a proper test drive, or be overwhelmed by the sheer amount of functions available.

Developing use case scenarios

Stephen Krug’s book “Don’t make me think” tells you precisely how you should go about usability testing. You’ll need a small number of authors and editors (at least 3-4 to make it worthwhile) and a list of scenarios which encompass the most common tasks that you will be performing, these scenarios might include:

  1. Creating new content. Creating a new article (with images) and promoting it on the home page, relating articles – either directly or via meta data (such as a taxonomy)
  2. Updating and approving existing content (include user generated content). This would demonstrate workflow and approval processes and versioning capabilites.
  3. Updating design. Modifying layout of templates and page design elements

Getting your hands dirty

There are a number of options presented below, each depends on the time, budget and resources you have at your displosal. You are likely to be looking at more than one CMS during the vendor and online demos, and looking at going on training and a procuring a proof of concept only for your preferred solution.

  1. Vendor demo
  2. You go through the product with the vendor, a well-managed vendor demonstration should provide a pretty good idea about the usability of the product. These demos usally last around two hours which is usually not enough to get an in-depth understanding of the product, but it’s a good start, and should help you narrow the field.

  3. Online demo
  4. Most vendors will be able to provide you with an online demo site for their CMS. This offers a simple way of playing with the product in your time and using your own equipment. The demo won’t be populated with your content or have your site structure, however it could give you a good feel for how everything fit’s together. A word of warning though as almost every CMS requires training, simply playing with a product could be potentially misleading and can become very easily frustrating!

    Ideally you would want to run through your most common scenarios on the demo without too much input from the vendor. If you can without getting confused or disoriented then it bodes well for how others will be receptive to it.

  5. CMS training
  6. Ask the vendor about end user training for the CMS. This will help to answer many questions: is training available? Some vendors don’t provide user training as they believe that their product doesn’t need it! If it’s needed how long is it? 1 day or 5 days? Some authors could be sent on training to find out how simple the product was to use? Can the authors understand the product at the end of training?

  7. Proof of concept
  8. You should ask the vendor to create and deploy a proof of concept based on your scenarios for you to work on. You can then conduct scripted tests with trained users to help identify any major issues.

Even with using the techniques presented here, CMS usability can still be difficult to assess e.g. what is simple for one content editor could be difficult for another. Nevertheless, I believe it is vital to conduct an evaluation of usability as part of the CMS selection process, and it could be the difference between project success and failure.

Is usability testing a part of your CMS selection process? What has worked for you?

Links

Don’t make me think (usability testing by Stephen Krug)

Practical ways to assess CMS Usability

When speaking to clients and CMS vendors we have been discussing what is the most important criteria that CMS buyers are looking for when selecting a CMS. A year ot two ago I would have said it was cost or how well a CMS met requirements. However in our most recent projects usability has been as important as cost or how well the CMS met a clients’ requirements. So why is it important and why does it feature at all in CMS selection projects now and what do we mean by CMS usability?.

What is CMS usability anyway?

Usability means making sure something works well, and that a person of average ability or experience can use it for its intended purpose without getting hopelessly frustrated. Therefore a usable CMS will :

  1. Be easy to learn
  2. Be easy to remember (particularly for occaisional users)
  3. Have a simple content management model (content types, ‘pages’) and follow a users’ mental model
  4. Require minimal training and be easy to explain to non web savvy editors.
  5. Be fast, resilient and error proof.
  6. Provide helpful error messages and tips when things go wrong. (wishful thinking!)

Clearly these things are not always covered in a statement of requirements, and have a significant impact on a successful CMS project.

Why is usability so important?

  1. Project failure is not an option (particularly now). If a CMS has been selected based on primarily technical or value for money criteria then there is a siginificant additional risk the project will fail!
  2. If  your team does not understand the CMS they will not use it.
  3. You will have to factor in additional hidden costs of training (of existing and new staff) and support. Your team will be inundated with support requests from users who will get stuck and then your team will spend time helping them.
  4. Devolved content management will not be possible, without significant additional investment.

What do you think? Does usability feature in your CMS selection? If not why not?

Useful posts on CMS usability:

  1. 11 usability principles for CMS products (Step Two Designs)
  2. The 5 hidden costs of running a CMS (Paul Boag)