Participant Daily Burden’s Impact on Momentary Ecological Studies

In our last post, we addressed incentives and their impact on an experience sampling study. In this article, we will discuss another key domain in drafting a successful ESM study: daily burden. Burden might be defined as the length of experience sampling sessions plus the frequency of prompting, and the impact of these daily interruptions on participant motivation. Too much burden can impact participant compliance. Lesser burden can improve compliance but leave researchers without enough data. As we’ve discovered with many domains in ESM research, it’s all about balance.

The Dimensions of Daily Burden in Experience Sampling

Experience sampling researchers continually face a trade-off between methodological ideals and feasibility; there is no one-size-fits-all approach when it comes to determining the daily burden of an experience sampling study (Myin-Germeys & Kuppens, 2022). Several interconnected factors influence the daily burden placed on participants, and all of them are intimately tied to the specific research question, the construct(s) being measured, and the population being assessed. As a result of this web of connected factors, researchers face the uniquely challenging task of getting as near as possible to, without exceeding, the point at which study parameters begin to harm compliance and decrease statistical power. With that in mind, below are a few factors to consider as you search for the balance between gathering enough data and avoiding participant burnout.

1. Question Quantity & Length of Daily Assessment

As researchers construct questionnaires, they must also consider the participant burden resulting from both the quantity of questions asked and the average length of time required to complete each instance of assessment.

Generally, a higher number of questions included in a momentary assessment predicts lower compliance and data quality (see: Eisele et al., 2020, Morren et al. 2009). This is understandable when we consider that a higher density of questions translates to greater burden in terms of time spent and disruption to a participant’s daily life. A higher quantity of questions can lower participant motivation to respond and can increase the participant’s desire to rush through data submission, impacting data validity.

While each research question and population being studied is unique, an analysis of 125 substance use ESM studies found that the average length of an assessment instance was approximately 4 minutes, with a standard deviation of just under 2.5 minutes (Jones et al., 2018).This was an average of 125 unique studies, and so does not represent the ideal assessment length.Consider your unique population and research question, and work to keep your question quantity to the absolute minimum for each assessment instance. If certain questions are only relevant based on a particular condition, using branching or display logic to make them conditional will help to reduce overall quantity.

Key Considerations for Assessment Length

Here is a list of key considerations when thinking through assessment length in protocol design:

• Construct Attributes: What is the minimum number of items required to accurately capture the dimensions of the construct?

• Time Required: How much time do the questions take to answer? Question quantity and length should be established considering the type of question.

For example, open-ended questions require more time than Likert scale questions. Fewer, more labor-intensive questions may impose more burden than a larger number of less labor-intensive questions.

• Planned Analyses: Will the included items in assessments provide the required measures for any planned analyses? The research question and associated analyses should shape protocol design.

• Sample Attributes: How may the unique qualities of the population being studied impact compliance?

For example, youth participants generally yield lower response rates (see: (Rintala et al., 2019; Ono et al., 2019).

Another example, the proportion of male respondents has been found to be associated with lower compliance (Vachon et al., 2019).

• Branching & Display Logic: Could conditional questions be implemented with display logic or branching? This can reduce burden by only collecting data for particular items when relevant.

For example, smoking cessation studies may begin each assessment by asking if a participant has recently smoked. If the answer is “No,” the assessment might end. If the answer is “Yes,” more questions would be displayed.

2. Participant Burden and Prompting Frequency

Another factor to consider in drafting assessments is the frequency of assessments assigned throughout your study. Research has shown that a higher frequency of prompting does not generally have a negative impact on compliance (Eisele et al., 2020; Jones et al., 2018 , Soyster et al. 2019). However, unique considerations for the population being prompted can impact compliance. Here are some studies to read for further information about unique population considerations:

• This 2019 meta-analysis of experience sampling of those with severe mental disordersfound that compliance was positively associated with fewer assessments per day.

• This 2017 systematic analysis of intensive longitudinal studies on youth, which examined differences between clinical and non-clinical studies with high-frequency prompting. They found that clinical studies with high-frequency prompting yielded higher compliance, whereas non-clinical studies with similar frequency yielded lower compliance.

• This 2022 meta-analysis of 477 studies using intensive longitudinal methods, which revealed that the total number of assessments was not predictive of either compliance or drop-out rates; however, generally, studies with more daily assessments had a shorter study duration.

Key Considerations for Assessment Frequency

Here is a list of key considerations when thinking through assessment frequency in protocol design:

• Time-of-Day & Routines: Is the construct being measured expected to occur at particular times of the day or during the week? Ex: A measure of job satisfaction would probably be most successful when grouped during or shortly after work hours Monday-Friday.

Seek to minimize assessment frequency to avoid disrupting daily routines. At which times of the day or week would an assessment be more burdensome to a participant? For example, would answering during work or school hours potentially add to a participant’s burden? This study found that compliance varies significantly across time of day of an assessment. A high assessment rate can introduce reactivity bias, i.e., begin to change participants’ daily routines, impacting ecological validity.

• Construct Attributes: How fluctuating is the construct of interest? Constructs, like mood, that are expected to vary from moment to moment will best be captured by high frequency assessment (see: Myin-Germeys & Kuppens, 2022). Less dynamic constructs, like sleep quality, require fewer assessments (see: Kasanova et al., 2020).

• Sample Attributes: Would the unique qualities of the population being assessed cause assessment frequency to add disproportionately to burden?

For example: Some evidence suggests that participants with clinical diagnoses may struggle with missing more assessments (see: Rintala et al., 2019; Vachon et al., 2019).

• Impact of Conditional Surveys: In studies where a series of follow-up questions may be triggered by a condition such as drinking or smoking, how might conditional follow-up questions prompt a participant to change or choose not to report a behavior? Be wary of adding too many questions (via length or quantity) unlocked by a conditional response.

Overall, the literature indicates that the number of questions in an assessment is generally inversely associated with compliance. The relationship between assessment frequency and compliance appears to be more context specific. Further research is required around high-frequency assessment over long-running studies. Prevailing best practice is to lower the frequency of assessment as study duration increases.

The Relationship of Study Duration to Daily Burden in Experience Sampling

As we touched on with prompting frequency, study duration can be a key factor to consider in assessing participant burden. Study duration involves the number of days a participant is expected to respond to ESM prompts. The literature is not always conclusive on whether or notstudy duration impacts participant burden. Here’s what we found:

• This 2018 meta-analysis found no association between study duration and overall compliance.

• This 2022 meta-analysis found that studies with longer duration were predominantly adjusted to have lower frequency of assessment.

• Conversely, this 2009 study, 2019 meta-analysis, and another 2019 meta-analysis show that completion rates generally decline over time in studies.

• This more recent 2022 meta-analysis collected mean data for study duration. The studies they reviewed lasted from 1 to 150 days, with a mean duration of 11.2 days. 68% of the studies surveyed lasted between 2 and 10 days.

Key Considerations for Study Duration

Here is a list of key considerations when thinking through study duration in protocol design:

• Construct Attributes: How might the nature of the research question and the construct(s) being studied impact study duration? Does prior research indicate that the construct of interest is more likely to vary over a longer or shorter period?

• Time of Week: Is there a time during the week (ex: weekends vs. weekdays, one week compared to the next) that the construct of interest is more likely to vary? If the construct will not vary greatly from one week to the next, a shorter duration may naturally reduce participant burden.

• Planned Analyses: Will the ESM protocol yield sufficient and appropriate data for the planned analyses? Consider study duration, question quantity, and frequency of prompting with intended analyses in mind (Germeys & Kuppens, 2022).

In designing ESM study protocols, it is essential for researchers to accurately gauge the degree of participant burden by keeping in mind the assessment length, assessment frequency, and study duration in mind. Seek the minimum effective dose for these parameters to minimize burden and maximize study effectiveness. Integrate participant feedback and aspects of previously successful protocols for your best chance at success. Existing ESM literature can be helpful in this step of the process, especially literature dealing with your intended population of study, pilot protocols, or participant feedback. All of this can be woven into a study protocol to ensure an effective experience sampling study.