Simple Research Loop

If you are relatively new to the field of Design Research, then it may be time to consider other factors that contribute to the success of a research study beyond your expertise with a particular method or tool. Let’s take a step back from any individual method and look at the context within which research is carried out. Specifically, let’s look at the activities immediately surrounding the actual research study itself. In this article, I represent these surrounding activities in a very simple model and offer some considerations for each step of the model that may contribute to the success of your research study.

A Simple Loop

The model is not revolutionary, but rather logically straight-forward.

You can change it to suite your needs; however, it needs to be granular enough to reveal its value: the considerations and questions you ask yourself at each step of the model.

 

How often have you been approached by a team member or stakeholder and told “I need you to test a prototype” or “I need you to survey our customers”? Do you suppose these same people go to their doctors and say “I need you to do surgery on my spleen”? Effectively, they are pushing you to start two steps into the process at the methods step. Perhaps they want to leverage an existing prototype or body of work, but no matter what their intentions or how much they know, it’s incumbent on you to push them back to the proper starting point: “What questions do you have, and what do you need to learn?”

Some stakeholders may have trouble formulating questions, or at least the right questions. Even if they come to you with questions, it still may be prudent to validate by starting with an understanding of their goals, actions, and/or decisions. From there, you can work with them to identify the appropriate questions ultimately affecting and informing their impending decisions and actions.

Some considerations for step 1, “Questions”:

• Have any of these questions been asked and/or answered before? If so, what’s different now?
• Should you contact other stakeholders and give them an opportunity to review the questions collected so far and add or edit them?
• Do you fully understand what will happen after you complete the research loop and answer the questions (i.e., how will your answers/data be used)?
• Is there agreement amongst the stakeholders as to how to respond to various possible outcomes?
• If the questions are very specific and detailed, do you understand enough of the broader context to enable creation of larger themes and bigger picture thinking and analysis?
  • In addition to considering the immediate questions for the work at hand, what if you also took a long-term view to the overall project, then what big-picture questions would you and/or your stakeholders ask?
• If the questions are very broad and general, can you successively break them down into enough detail to help you with the next step?
• If there are multiple questions, can you prioritize or stack rank them? This may come in handy in future steps involving tradeoffs.
• Have you challenged and documented prevailing assumptions (yours and your stakeholders’)?
  • “Researchers tend to form their research questions around their assumptions, so our assumptions are the foundation for all the knowledge we gain through user research. If your assumptions are incorrect and you never realize it, everything else that you discover will be flawed.” – from Getting Good Intel: How User Research Can Go Wrong

 

The next step is to identify data needed to address the collected questions (and assumptions).

Some considerations for step 2, “Data Needed to Answer Questions”:

• Have you considered all the different types of data and which may have higher validity?
• What additional data should you collect to help round out your research?
  • Be sure to identify not only the data needed to address the questions and assumptions, but also the data to address the inevitable follow-up question, “why?”
  • What additional data could enable meta-analysis across a series of studies?
• Target sample size (and confidence interval)?

 

Some considerations for step 3, “Methods to Collect Data”:

• Is primary research necessary, or will secondary research suffice?
  • Have you actually searched various publication types for information related or relevant to the larger issues at hand? Even when you know you have to do primary research, information from secondary research may reveal new questions and/or ideas that may be very helpful.
• Are multiple methods required to capture all the needed data?
  • If so, can they be integrated into one study or should they be executed separately?
  • Target sample size for each datum?
• Is there other research, past, present, or future, with which you need to align?
• What project factors will influence or narrow your method selection?
  • Time (analysis deadline)?
  • Cost (facilities, recruiting, gratuity, etc.)?
  • Resources (people, labs, materials, etc.)?
  • Skills and tools?
• If you are over budget (i.e., not enough time, money, or resources), could dropping one or more lower-priority questions help?
• Be watchful of your own bias towards favorite methods.

 

Some considerations for step 4, “Design the Study”:

• Is there other research, past, present, or future, with which you need to align?
• Does the study have “face validity” with your key stakeholders?
• What data collection process or tools can be setup ahead of time to facilitate data analysis?
• Can you avoid these 7 deadly sins?

 

Some considerations for step 5, “Conduct the Study”:

• Conduct a pilot or dry run, always; test and refine everything from start to finish (including data logging and any automated data analysis)
• If observation or participation is possible, who should you invite?
  • Have you sufficiently prepped observers or participants?
  • Do your best to prevent the observer who watches one session from walking away with a to-do list of things to change!
• Have you scheduled stakeholder touch-points to provide progress updates and avoid “going-dark” during the study?

 

Some considerations for step 6, “Analyze the Data”:

• Can “key takeaways” be identified quickly?
  • If time is of the essence, perhaps key takeaways can be shared first, quickly and informally.
• Can someone not involved with your project scrub your conclusions for bias (i.e., ensure the data supports your conclusions)?
• Are you proposing design recommendations?
  • If so, involve the appropriate designer in crafting the recommendation. If you can’t do that, then focus the recommendation on requirements and objectives and offer your “solution” as an example.
• Beyond answering the initial set of questions, does the data reveal anything more?
  • Does is relate to any previous research?
  • On its own or viewed in conjunction with other research, does it lead to any new findings, theories, or questions?

 

All of your hard work comes down to this last step. A failure to properly communicate and connect with your stakeholders at this point may very well render your work invisible.

Some considerations for step 7, “Answer the Questions”:

• Take a broadening circles approach to information dissemination.
  • Refine and improve what you communicate and how with each successive round, while sharing information with key stakeholders first.
  • Share your results first with your inner-most circle, your own design research team; a good opportunity to get both critical feedback as well as ensuring the entire team is aligned.
  • Share your results with next outer circle, your project team members; then go broader as appropriate.
• Are you answers clear to your stakeholders (are you speaking their language), not just to other researchers?
  • For example, for your business-minded stakeholders share your results and its business implications (saying 60% of users failed a specific task is insufficient).
  • For example, for your designers create links to design principles and objectives where possible (it’s harder to ignore or forget a message that says an agreed design principle is not being met).
• What have you done to make your key messages memorable for your stakeholders?
  • Be creative in engaging your stakeholders; eye-charts and text-filled slides will not convey the message well nor make it memorable.
  • You want your stakeholders to become empathetic; find a way to make them feel, not just understand. This will motivate and make key messages more memorable.
• Be concise and focus on key messages.
  • Take a “headline news” approach: main message first followed by backup details for support and Q&A.
  • Scrutinize your presentation and avoid the temptation to share every little detail. Yes, you worked hard, but the goal is not to show how hard you worked. Don’t risk losing key messages within too much noise.
  • Can you provide an executive summary?
• Are next steps and recommendations clear?
  • Consider a workshop instead of a presentation in order to promote and facilitate action and decisions.
  • Can you produce other work products from the results of your study and use them as sparks and catalysts for next steps (e.g., Experience Maps)?
• Are the results, presentation, and/or next steps available for all stakeholders to access at any time?
• Can you follow-up and measure the impact of your research?

Once this last step is done and the initial questions have been answered, the process should be complete. So, why have I laid it out as a loop? In many cases, research studies answer initial questions and in doing so lead to a new set of questions for further research; hence, the final arrow from step 7 back to step 1. Those new questions will be added to a new collection, which may drive another round of research.

Lastly, there is a theme throughout most of the considerations presented here. A friend and past co-worker of mine said it best: “There is something about having the team involved in research planning and execution. If that is done, Step 7 is not a surprise to anyone. In fact, the best outcome of the research is when the team feels they have contributed to the final analysis and report.” Thanks, Ananth, I couldn’t agree more.