In our prior post, we discussed balancing participant burden and motivation in experience sampling studies. This article focuses on designing a protocol that strikes this balance. You might think of the ESM protocol as the design blueprint for a study. Researchers can optimize their ESM protocol design for a coherent theoretical and empirical approach, ensuring better accuracy and generalizability. By designing intentionally, researchers can also enhance participant motivation and minimize perceived burden.
Designing Protocol for Relevance
As researchers work to design a protocol that increases participant motivation, relevance is one of the key considerations. Begin by assessing the unique aspects of the intended study population with your research aims in mind. You might find some helpful things to consider in the context of your own research from Vachon et al.’s (2019) meta-analysis on compliance and retention in experience sampling method studies of individuals with mental disorders. Some questions to consider when optimizing your study for relevance are:
- At what times are participants unavailable or less likely to respond? (e.g., adolescents and teens during school hours)
- What practical burdens may be more prominent in your participant population? (i.e., using an electronic collection method with a small font size or certain controls that require greater manual dexterity with participants over the age of 65)
- What is a reasonable enrollment period and daily load given your population and research question? (i.e., measuring change in appetite may not need more than 7 days with frequent questionnaires, but measuring seasonal affective disorder will require a longer study period with fewer questionnaires)
- What unique or ethical privacy considerations may arise with the type of data or the study population (i.e., collecting health data from HIV patients or GPS data from those using illicit substances)
- Are there ways to make responding more engaging or fun for your participants? (i.e., a unique user interface, gamification, bring your own device)
One way of addressing these questions is by getting feedback from non-researchers in the study population who can review, pilot, and inform protocol development for your study (Salganik, 2018). An outside perspective can help to answer the above questions and highlight potential weak points in study protocol.
Incentives in Experience Sampling Protocol Design
Another key consideration in protocol design involves incentives. Research shows that perceived participant burden increases with longer questionnaires, but not as much with added frequency of assessment. So, especially for studies that require longer questionnaires, researchers will need to find ways to increase participant motivation. A future post will be devoted specifically to incentive benefits and options.
Questionnaires in ESM Protocol Design
Some studies suggest that study duration (number of days gathering data) has not been shown to affect compliance or retention. However, the frequency of within-day assessment (number of sessions in a day) is inversely correlated with compliance. When possible, researchers should try to “space” their experience sampling sessions so that they are assessing individuals over a longer period of time with lower daily intensity. With that in mind, the top priority when designing a questionnaire will be brevity. How can you address your research aims with the fewest number of questions and with the most clear and concise wording? While drafting, it’s also important to consider question order (or randomization), and response scale options. Because experience sampling questionnaires are not like cross-sectional surveys, you should also ensure that you’re using ESM-appropriate measures, which you can read about in this study. You can also peruse the repository of ESM-appropriate measures curated by KU Leven. It is important to use validated measures from similar studies, or to develop and pilot test your own measures to ensure they work well in an experience sampling context.
Customization and Timing Considerations for ESM Protocol
It is vital to consider unique attributes of your sample population when setting up sampling timing and frequency. Typically, allowing participants some control over their preferred sampling frequency and/or timing is a good idea. If participant self-selection of times is not an option for your study, then be attentive to good and bad times to prompt your sample population and schedule accordingly. Considerations included in this article remind us that participants are not a homogenous group, and so individualization can help with retention and completion rates. Consider the nature of the construct being studied. Is the behavior/emotional state/thought pattern more likely to occur at certain times of day? If not, does it occur fairly frequently or consistently? Finally, as you are deciding on the level of flexibility with timing, you will also need to be wary of convenience sampling, or the bias that can occur when participants only answer assessments on their own terms. Selecting waking hours and then delivering at fixed or random times within those set hours is a commonly used and successful approach.
Plan for Feedback and Support
When establishing your study protocol, it’s important to consider the ways in which you and your team can provide feedback and support to participants. Although personal interaction is not possible with some experience sampling study designs, for those studies where it is possible, including personal interaction can encourage participant engagement that will persist throughout the study. One way to set up positive interaction is by proactively identifying participants who are struggling. Dedicate ownership of compliance tracking and follow-up to someone on your team to keep these participants from falling behind or dropping out of the study completely. Provide early positive reinforcement to participants who are responding well and consider instituting regular check-in calls, texts, or emails. Other ways to be helpful include familiarizing yourself with your chosen data collection platform (for example, a smartphone app or SaaS platform) and preparing to address technical issues. By interacting positively with participants, motivation and overall satisfaction increases, which helps with retention and completion rates.
Considering UI in Experience Sampling Protocol Design
Along with being prepared to provide tech support or refer participants to the right resources, researchers should consider the user interface of the data collection method they’ve chosen. Currently, electronic data collection devices—especially the Bring Your Own Device method—are most popular in conducting an ESM study. When choosing which device or app you will use to collect data, remember that user experience matters. A complex or buggy user interface will lead to greater participant burden, frustration, and dropout rates.
Here are some other considerations when considering the user experience of your data collection method:
- What are the user reviews? How frequently do users report crashes, and how are they handled?
- Does the questionnaire design offer enough different question types to support your study?
- Is the design of the app intuitive and easy to follow?
- Do they use native controls or external controls? What learning curve will be required?
- Does the app support multiple languages?
- Is the app able to capture photo, video, audio, GPS data, or accept uploads? Depending on the study, these components may be necessary.
Spend time speaking with colleagues, reading reviews, meeting with platform providers, and testing various solutions to find the right fit for your study. Opt for a solution that allows participants to easily submit data without creating additional burden.
Ethical Considerations at the Protocol Design Stage
Ethical considerations are always a key component to designing a strong study protocol. Consent, deidentification of data (including photos, videos, audio recordings, or GPS data), privacy and security matters, and accessibility issues that may exclude a group from your research are among these key considerations. See this resource for more information on ethics in ESM studies.
Whether you need to decrease burden or increase motivation, at the protocol design stage of research, considering the participant experience is of utmost importance. A solid experience sampling protocol will include considerations on both sides of the participant experience balance (burden and motivation) through careful planning around relevance, incentives, feedback & support, questionnaire design, customization, user-friendly data collection solutions, and ethics. With these considerations in place, researchers can construct a successful experience sampling study.