Take 1: Fusing Prototyping & Evaluation

This post is a starting point, a personal perspective on fusing rapid prototyping and impact evaluation. It has been influenced from reading & research, reflecting on my own processes, and conversations with thought leaders. Incomplete, and probably naive, this perspective is something I aim with which to experiment, break, and evolve.

“We know two things with absolute certainty:
(1) that in twenty years, even ten, our world will look different, and (2) that the decisions and actions we take today will significantly shape our emergent future. However, we can have no certainty about what the future will be. It is not a good time for control freaks.”   — Eric Young

As we navigate this emergent future, one of the most critical tools we have is curiosity. We need to be able to ask questions and learn enough to take a next step. We need to understand the relationships among all of our systems—and ourselves—and the subtle rules of engagement that make one outcome possible over another.

Towards this end I’m inspired by two practices: rapid prototyping and impact evaluation.

Rapid Prototyping: quickly learn from real situations

As an Interaction Designer, rapid prototyping is one of the most powerful practices in our toolkit. Rather than base our designs on assumptions, we give people a chance to experience an idea first-hand. By observing and working with them, we understand the underlying behaviours and values, the subtleties of what people will do, not just what they say they will do. Depending on where we are in the design process, we create prototypes that range from a quick paper mockup to high-fidelity experiences. The key is that these are learning tools, research tools that we can use to inspire final concepts.

Impact Evaluation: structure and evaluate hypotheses

Impact evaluation is a practice primarily used within the professions of environmental planning, social programs, and development. It allows us to take a more scientific approach to learning how a project or program could create change within a complex system; giving us structure and tools to, for example, evaluate the efficacy of a farming training program in India, or a nutrition program in sub-saharan Africa, or a early child literacy program in New York. Recently it has also become widely used in impact investing and philanthropy.

What magic can happen if we fuse rapid prototyping and impact evaluation?

As I’ve participated and led design projects—especially projects that are in the realm of ‘social good’—I struggle to understand the long term impact of a concept. Yes, we can design a delightful user experience to nudge heating consumption behaviour, but given Copenhagen’s system of heating, is this the most effective leverage point? Will users engage for long enough periods of time to actually make a difference? What are the trade-offs in this concept that create harm in other systems (i.e. production of electronics).

I believe there are key moments when designers can pull in evaluation thinking and tools and, on the flip side, I believe there are ways in which evaluators can take advantage of lean, iterative processes. Indeed, there are a few trends that hint at these fields becoming more interlocked: development evaluation, adaptive evaluation, lean data gathering, etc.

A (simplistic) view of change

As a starting point, it is key to first have a perspective on what ‘change’ is and how it can be influenced. There are undoubtably many types of change, the type I often think about is systems change: how can actions lead to reverberating changes throughout systems.

If we have any chance of adapting to climate change, we need to set in place radically different systems. These will have to be designed, adopted, and built amidst our existing systems, creating an emerging future where actions occur in different systems as the effects of climate change shift our base environmental systems.

In thinking about this type of systems change there are three principles which stand out to me.

Social and environmental change unfolds in ripples across time.

Change doesn’t happen in nice linear lines. An action opens new possibilities, communicates new information, and influences subsequent actions and reactions over multiple directions.

Neither does change happen at a consistent rate. One action in at one point in time can lead to rapid events, but the exact same action at another place or moment of time can fail to spark movement. Our actions tug on the systems we are connected to, but whether those systems are rooted in water or molasses can drastically change the vectors of impact.

“Change is rarely straightforward. Sometimes it’s as complex as chaos theory and as slow as evolution. Even things that seem to happen suddenly arise from deep roots in the past or from long-dormant seeds.”   — Rebecca Solnit

There are first-order outcomes in the short term, with long term impact over time.

We can observe the ridges of change as they unfold and the ripples spread. Immediate outcomes, first-order change, are easier to identify. Subsequent ridges, and longer term impact, become unclear and faint as they have more time to merge with other actions.

Rapid prototyping is a great tool to evaluate those first-order outcomes, to immediately understand emotions that are triggered and behaviours that are influenced. Impact evaluation, on the other hand, is better at identifying the connections to those blurred, longer term changes.

…and change is never straightforward

Actions don’t happen in a vacuum. Change is the result of a multitude of actions, colliding, accelerating, and forging new possibilities. It is never straightforward, never as simple at looking at one action. We must work together, to interact, to create outcomes. Past experiences will serve only as learning moments, not blueprints for all future actions.

“We must move from seeing the world as simple, or even merely complicated. To understand social innnovation we must see the world in all its complexity…unpredictable, emergent, evolving and adaptable—not the least bit machine-like”
Getting to Maybe

Three Type of Questions

There’s never going to be a perfect playbook, a toolkit, a or framework for achieving change in a complex world. We have to be responsive to the emerging situations, to continue to ask questions and learn from the answers.

Perhaps, however, there are moments where specific questions and methods are better suited to getting relevant answers. In reflecting about fusing rapid prototyping with impact evaluation, there seem to be three types of moments that have potential: Map, Sprint, Gauge.

Map: When you don’t know where to start, map the terrain.

Before taking any action it can be helpful to get the lay of the land and evaluate potential strategic opportunities. At this stage you want to develop a map, and identify the patterns and principles around which you wish to design.

The resulting learnings can be used to create a Theory of Change and identify key opportunities and actors.

Potential Activities:

  • Review existing theory and past examples
  • Interview experts
  • Conduct field research
  • Conduct environmental analyses
  • Map any legal / civic constraints
  • Map the economic & business landscape
  • Identify key actors
  • Develop a Theory of Change

Sprint: When you have many ideas, get in the field quickly.

Sprints can bring clarity around the first-order outcomes and help inform the design of longer-term prototypes. These types of questions are best when you are just at the beginning of a new concept, or when you want to focus on the nuances of a given design. Ideally, the prototyping for sprints should be quick and cheap moments of field work.

The resulting learnings can be used to set-up and plan for live, longer-term prototypes, or to refine an existing concept.

Potential Activities:

  • Review Theory of Change and opportunities
  • Ideate and sketch
  • Prototype specific moments in the field
  • Evaluate and iterate prototypes
  • Re-map the theory of change
  • Develop synthesised concept scenarios
  • Plan longer-term prototypes (& identify indicators)
  • Refine design of a concept

Gauge: When you have a few directions, prototype live and learn from indicators.

Longer term prototypes can illuminate experiences and behaviours beyond the initial excitement over a new concept or project. It is key to have a method to—as much as possible—gauge indicators which can help you compare different prototypes to each other and a ‘control’.

Learnings from these questions can raise confidence in the impact vector and provide clarity on next steps and design decisions. They can provide a window into those first few ripples of change, an opportunity to evaluate and shift course before we commit to a longer-term plan or scaling.

Potential Activities:

  • Turn scenarios into live prototypes
  • Conduct baseline studies, or establish a ‘control’
  • Launch and monitor the live prototypes
  • Conduct exit analyses
  • Evaluate outcomes
  • Synthesise, re-map the Theory of Change
  • Take action on the learnings

If you’ve made it to the end of this post, first, thank you. And secondly, this is just a first stab—I’m continually testing and expanding on the themes explored above.

If you’ve worked with relevant case studies, have thoughts you’d like to share, or if you’re generally curious—I’d love to hear from you!

Reach out by email or twitter!