Finding evidence

There are a couple of things that make patterns distinct from other forms of design guidance. One is that they are routed in practice – they are not simply “good ideas” or theories (not that there is anything wrong with those) but they are drawn from real examples of things that have worked to solve problems in specific contexts. Another is that they cannot just be based on a single example – a pattern can only really be called a pattern if it can be demonstrated to have worked in at least three distinct cases. We call this the rule of three. We recently reviewed all the “patterns” that have arisen from our workshops and most fall into the “candidate pattern” category – we think they might be patterns, we have one – occasionally two examples of the pattern – but we don’t yet have sufficient evidence to be confident that this really is a pattern.

So we are looking for evidence in the stories of successful practice that we all have as educators. Our Elluminate story telling session on Tuesday with the EXTEND project brought up several examples which – at least on the surface – seem to be evidence of one or other of our candidate patterns. We plan to hold more open story telling sessions over the next couple of weeks. Prior to that we will be publishing summaries of our existing candidate patterns so that people can easily review them and let us know if they have had any experiences relevant to any of them.

Pattern elicitation is an ongoing community process – and we expect the Planet patterns to be evolving and developing long after the of the project through other initiatives that are already in progress. But we’d like to make the most of the wealth of experience in Emerge to gather evidence! Watch this space.

An inconvenient thesis?

Christine Elizabeth Wania’s phd thesis questions the empirical evidence for the benefits of pattern languages in HCI:

For more than two decades much of the pattern language literature, within the field of Human Computer Interaction (HCI), has focused on the possible benefits pattern languages may provide, but there has been very little empirical work to support these claims. It has been suggested that interaction patterns or pattern languages in HCI may address some of the problems inherent in designing interactive systems by supporting reuse, capturing design knowledge, enabling the sharing of design knowledge, and facilitating communication among designers and users. This study examined the impact of a pattern language on the design of information retrieval interfaces, in terms of the quality of the interfaces and the time to design the interfaces. Participants created paper and pencil interfaces based on the given design task. Participants were exposed to either a pattern language, guidelines, or no structuring technique. There were no statistically significant differences between the three groups in terms of the quality of the interfaces and time to design the interfaces. The results of this study suggest that the value of pattern languages in HCI may not be in reuse, at the early stages of design, or in terms of the quality of the resulting designs, in domains familiar to designers. Although there was no apparent impact of the pattern language on the early stage designs, the results of a follow-up study suggest there is a significant correlation between the existence of patterns in commercial systems and the overall usability of those systems. Therefore, we suggest that we, as a community, very closely examine the current state of pattern languages in HCI before continuing to move forward. As a community, we need to shift our focus away from discussing the possible benefits of pattern languages and trying to build pattern collections. And instead, focus on trying to fully understand the value of pattern languages in HCI. In doing so, the HCI community, will then begin to see the benefits from all the great efforts in this area.

About time someone asked this question. I mean, I’ll swear by my favourite patterns and pattern languages, defend them as if they where my children flesh and blood, but do we have any evidence, in the scientific sense? In a way, this brings back the discussion on validity, resonance and aggregation. Yes, patterns work great for those who believe in them. But shouldn’t we aim higher?