(C) PLOS One
This story was originally published by PLOS One and is unaltered.
. . . . . . . . . .



Acceptability and playability of an organization training videogame for young adolescents with ADHD: The development of ATHEMOS [1]

['Brandon K. Schultz', 'Department Of Psychology', 'East Carolina University', 'Greenville', 'North Carolina', 'United States Of America', 'Steven W. Evans', 'Ohio University', 'Athens', 'Ohio']

Date: 2023-12

Abstract An estimated 8.7% to 9.8% of school-age children in the United States have attention deficit hyperactivity disorder (ADHD), affecting 4.3 to 4.9 million public school students. ADHD is a costly disorder that often goes untreated, especially among adolescents. Accessible computer-based programs have emerged to address the neurocognitive deficits of ADHD, but results to date have been disappointing. In this study, we tested the acceptability, playability, and user satisfaction of a novel planning/organization skills training game, called “ATHEMOS,” based on an established psychosocial treatment package (i.e., Challenging Horizons Program). We conducted eight focus groups during a three-year development period, using feedback from 72 young adolescents with ADHD to iteratively improve the game. Then, during a pilot study in the fourth year, we collected data from 16 young adolescents who played the game as part of a 6- to 16-week school-based intervention. Our findings suggest that the serious game resulted in acceptability and playability ratings only moderately below that of recreational games (δ = −0.40). Critically, average perceptions remained positive when delivered within a school-based ADHD intervention over several weeks or months, with strong overall user satisfaction. Boys found the game more acceptable than girls, with ratings near that of recreational games (δ = −0.23). We conclude that computer-assisted behavior interventions appeal to adolescents with ADHD and offer a potentially promising treatment delivery method in schools.

Author summary Attention-Deficit Hyperactivity Disorder (ADHD) causes academic impairments, and yet often goes untreated. The Challenging Horizons Program (CHP) was developed to address this need by teaching specific coping skills (e.g., organization, planning, study strategies) to reduce the impairing aspects of ADHD. Although effective, the CHP is too costly for many school districts. In this development project, we examined the acceptability and playability of a videogame-based alternative that trains the organization and planning skills of the CHP in a playful manner. Over a three-year development period we created the game, called ATHEMOS, using feedback from 72 young adolescents with ADHD. Then, in a small pilot study, we deployed ATHEMOS as part of a teacher-led intervention in middle schools. The results suggest that young adolescents with ADHD enjoy ATHEMOS almost as much as leisure games and were satisfied with the overall intervention package. We conclude that serious games teaching organization and planning skills are acceptable and might offer a promising new direction for affordable school-based treatment.

Citation: Schultz BK, Evans SW, Bowditch J, Carter K, Rogers EE, Donelan J, et al. (2023) Acceptability and playability of an organization training videogame for young adolescents with ADHD: The development of ATHEMOS. PLOS Digit Health 2(11): e0000374. https://doi.org/10.1371/journal.pdig.0000374 Editor: Hamish S. Fraser, Brown University, UNITED STATES Received: March 22, 2023; Accepted: September 22, 2023; Published: November 13, 2023 Copyright: © 2023 Schultz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: Data are available at https://osf.io/ew7qk/. Funding: The research described in this article was supported by a grant (R324A180219) from the US Department of Education Institute of Education Sciences (IES) to East Carolina University ($773,968 to BKS) and Ohio University ($615,456 to SWE). IES had no role in the research methodology, data collection or analysis, or preparation of this manuscript. Competing interests: The first two authors, BKS and SWE, are exploring options to potentially redesign and commercialize the videogame and intervention package described herein but have no active commercial interests at the time of submission.

Introduction An estimated 8.7% to 9.8% of school-age children in the United States have attention deficit hyperactivity disorder (ADHD), affecting 4.3 to 4.9 million public school students [1]. ADHD is characterized by developmentally inappropriate levels of inattention and/or hyperactivity/impulsivity, beginning prior to age 12, and leading to functional impairments at school and home [2]. Research shows that ADHD is a chronic disorder, with 70% to 87% of children with ADHD continuing to exhibit symptoms and impairments in adolescence [3] and a substantial proportion developing comorbid conduct problems, especially among boys [4]. School-based behavior interventions can alleviate some of the academic and social impairments of ADHD, but there are persistent implementation challenges that require interventions to be designed and tested specifically in those settings [5]. The Challenging Horizons Program (CHP) is an evidence-based treatment for ADHD designed for use in middle schools. The training interventions (TI) that comprise the CHP—organization, assignment tracking, note taking, study skills, and social skills—were initially developed and tested in a 4.5-hour per week after-school program [6]. The program is staffed by trained paraprofessionals who deliver TI in accordance with a treatment manual, typically under the supervision of a clinical or school psychologist. Studies show that the after-school version of the CHP is effective, resulting in meaningful improvements in organization, homework completion, academic progress, and attention-related behavior when compared to typical school-based practices (ds = .24 to .63) [7] Participants who comply with the CHP—defined as 80%+ attendance during 1-year of intervention—show especially large gains when compared to statistically matched students in a treatment-as-usual control group (ds = 0.56 to 2.00) [8]. Although effective, staffing costs for the CHP after-school program are prohibitive for most schools [9]. For this reason, a cost-effective consultation version of the program (CHP-C) was developed for use during the school day. In the CHP-C, trained behavioral consultants support teachers as they implement CHP interventions in their classrooms, meeting with those teachers regularly to discuss the interventions, track intervention data, and troubleshoot implementation challenges. The CHP-C has been shown to be efficacious in a prospective randomized control trial of middle school students with ADHD, with cumulative benefits over time when compared to typical practices [10]. For example, parent ratings of inattention, hyperactivity, and social functioning suggested small to large benefits (ds = 0.24 to 1.05) for CHP-C participants. Students with ADHD who did not receive the CHP-C were roughly four times more likely than the treatment group to have their grade point averages drop below the threshold for failing in each grading period [11]. But a tradeoff of the CHP-C is that participants receive relatively low doses of intervention, and some benefits (e.g., ADHD symptom reduction) can take up to three years to materialize [10]. There are several reasons why the CHP-C delivers a lower dose than the CHP. First, most teachers cannot devote several hours each week to one-to-one meetings with individual students. Second, many teacher consultees report that participants either skip meetings or fail to bring the requisite materials (e.g., bookbags, binders). So, although dropouts are rare (below 3%), participants may not engage well with teachers [12]. Third, teachers find some of the procedures aversive because student performance is typically poor at first, requiring the teacher to repeatedly correct errors, which can strain the student-teacher relationship [13]. Fourth, like all forms of school consultation, the CHP-C is susceptible to poor implementation fidelity. For example, teacher consultees are often slow to start interventions, in some cases for months, because they are unfamiliar with TIs for ADHD. Early CHP-C consultation sessions are often spent coaching and reassuring teacher consultees [7]. Thus, we explored options to deliver the CHP in a manner that is less burdensome for teachers and acceptable to adolescents with ADHD. Computer-based programs have emerged to support and strengthen school-based intervention, but most options for ADHD to date target the neurocognitive correlates of the disorder (e.g., working memory deficits), with disappointing results [14]. Designers have assumed that gains in neurocognitive functioning can alleviate ADHD impairments in real-world settings, but the research to date has fallen short of expectations [15, 16]. As an alternative, serious games might deliver TIs that directly remediate functional impairments, and then those new skills might be transferred to a target setting with the active assistance of interventionists (e.g., teachers, coaches). However, we are aware of only one game in the research literature that delivers behavioral skill-training interventions, and it is only available for Dutch-speaking players [17, 18]. Our goal was to develop an intervention package consisting of an engaging videogame and real-world intervention, which we refer to as computer-assisted behavior intervention (CABI). The game, called “ATHEMOS,” is an effort to train players in the planning, organization, and note-taking skills taught in the CHP, but ideally introduced in an engaging and playful manner. After players are exposed to the concepts in the game, teacher mentors help players transfer these skills to the classroom. The work of the teacher mentors is outlined in a treatment manual that describes the game and provides strategies for helping players apply the skills in the real world. To our knowledge, ATHEMOS is the first serious game in English that trains organization and planning skills for students with ADHD, paired with school-based intervention to achieve skill transfer. Given the novelty of the CABI concept, it was not clear if adolescents with ADHD would find a gamified training intervention engaging, or whether interest in the game might wane once delivered within a school-based intervention package. There is no advantage to gamifying an intervention like the CHP if the product is unacceptable to the target population. Thus, our focus in this development study was the degree to which ATHEMOS and its broader school-based intervention package appeals to young adolescents with ADHD. Our specific aims were to (a) compare focus group results to the average total score values of a validated videogame satisfaction scale to assess the degree to which the playability and acceptability of ATHEMOS approximates that of recreational videogames; and (b) examine player perceptions of the game when implemented within a school-based intervention package to determine if playability, acceptability, and satisfaction with the game are negatively impacted. The second aim was achieved in a small, randomized pilot study (described below), and for our purposes here we will focus on the pilot study participants who received the game. We anticipated that boys and girls might have differing impressions of an impairment-focused intervention, given the differing experiences of boys and girls with ADHD, but also given that boys are generally more likely than girls to report videogame interest and use [19, 20].

Results We examined results from our eight focus groups as those data were collected. There were no clear patterns of increasing or decreasing satisfaction with ATHEMOS across the eight focus groups, despite iterative game revisions during that same period. Participants were generally satisfied with ATHEMOS as measured on the 7-point GUESS total score scale (total M = 5.28; SD = 1.07) beginning with the first focus group. Overall, 68% of focus group respondents’ total scores were 5.0 or higher, and 24% of respondents’ total scores were 6.0 or higher. The highest user ratings were found on the Enjoyment subscale (M = 5.52; SD = 1.24), and the lowest ratings were found on the Narratives subscale (M = 5.10; SD = 1.17). To assess how strongly the total score results approximate recreational games, we compared our focus group data to the GUESS reference data average [26]. We noted that our focus group data were negatively skewed, as evidenced by boxplots and the Kolmogorov-Smirnov test (D[72] = .132, p = .003), so we applied a one-directional Bayesian Wilcoxon signed-rank test, anticipating the game would perform below the recreational games standard. Because we were unsure how far below the standard ATHEMOS would perform, we applied a nearly uniform prior (Cauchy width = 2.0). By the eighth focus group, participant responses provided strong evidence (BF = 11.94) that ATHEMOS did not perform up to the recreational games standard. The average effect size of the difference settled at -0.40 SD (95% CI = -0.61–-0.16), or moderately below the reference data mean (see Table 2). To check the robustness of these results, we tested several alternative priors, as well as a non-directional test, but came to nearly identical conclusions by the eighth focus group (i.e., convergence). We then conducted a similar single-sample t-test with the pilot study data using a fresh prior on δ (Cauchy width = 2.0), given the uniqueness of this group’s experiences with the game. The results of the total score analysis provided very weak evidence that the pilot study impressions were also below that of the recreational games standard (BF = 1.04; δ = -0.51 [95% CI = -1.00–-0.07]). PPT PowerPoint slide

PNG larger image

TIFF original image Download: Table 2. Focus Group and Pilot Study Total Score Means Compared to the GUESS Reference Sample Mean (5.71) using Bayesian Single Sample T-Tests. https://doi.org/10.1371/journal.pdig.0000374.t002 Next, we compared the pilot study and focus group results on the GUESS total score to see if there were any between-group differences, with a special focus on whether participant perceptions in the pilot study deviated meaningfully from the eight focus groups. Prior to this analysis, we applied a square root transformation to correct model misspecification (i.e., negative skew). We then performed a Bayesian ANCOVA on the GUESS total score, with group designations entered as fixed factors and biological sex of the participant (i.e., boys versus girls) as a covariate. This analysis compared the relative power of four models—group, biological sex, group and biological sex combined, and a null model—when predicting GUESS total score. Results suggest that the data increased the model odds for only biological sex (BF M = 7.79), which was also the most probable model, P(M∣data) = 0.72. After performing Bayesian model averaging, it appeared that the data were 15.63 times more likely for models containing biological sex as a predictor, but only 0.31 times as likely when including group designation. From this, we conclude that only biological sex influenced GUESS total score, and group designation (focus groups or pilot study) had no effect. In other words, participants in the pilot study had impressions of ATHEMOS that were virtually identical to the focus group participants. The results from the satisfaction questionnaire assessing the overall CABI intervention package appeared to confirm these results, with pilot study participants reporting a high degree of satisfaction (M = 3.34; SD = 0.54; range = 2.13 to 4.00), consistent with responses like most of my needs have been met and mostly satisfied.

Discussion We developed a novel videogame (ATHEMOS) and intervention package for ADHD based on the training interventions used in the Challenging Horizons Program. We then examined user acceptability and satisfaction with the game and the broader intervention approach across eight focus groups and a small pilot study during a four-year development project. We found that middle school students with ADHD are engaged by the game, even when it is delivered within a school-based intervention. Although participants did not rate the game as strongly as gamers rate purely recreational games, the differences were only moderate. We believe this is encouraging because educational games can score much lower than recreational games, in the rare instances when this comparison is examined [31]. We noted that players tended to rate ATHEMOS relatively high on the Enjoyment subscale of the GUESS, and relatively low on items measuring Narratives (i.e., the story, characters, and events in the game). One potential explanation is that we designed the game with the expectation that a teacher mentor would help the player understand the game narrative without lengthy in-game exposition, but this might have proved insufficient. It is also possible that players were only moderately engaged by the space invaders storyline, and future modifications might involve distributing the narrative throughout the game in a more satisfying way, or developing the NPCs to make those interactions more meaningful [32]. Importantly, boys and girls expressed differing opinions of ATHEMOS, with boys appearing more interested than girls. This finding was not surprising, as previous research has shown that boys tend to play videogames at a significantly higher rate than girls, with both sexes playing most often during their middle school years (i.e., ages 11-13) [19]. Both boys and girls enjoy the competitive aspect of videogames, playing against others or their own previous performances, but boys are particularly attuned to the competition provided within “physical” game genres (i.e., fighters, shooters) [20], like ATHEMOS. We conducted a post-hoc test examining only the boys’ responses to the GUESS (n = 62) and found weak evidence (BF = 0.80) suggesting that their responses approximate the recreational games standard (δ = -0.23 [95% CI = -0.48–0.01]), but these results were not robust. We conclude from these data that perceptions of ATHEMOS are not dramatically different from purely recreational games. Young adolescents with ADHD, especially boys, appear to be engaged by the game almost as much as they are leisure games. Girls were especially critical of ATHEMOS on the Usability/Playability subscale of the GUESS, which measures how easy it is to learn to play the game. This impression might be attributable to prior experiences with similar videogames (e.g., dual-joystick gamepad); and indeed, the baseline RCGAS data suggested that boys (M = 4.02; SD = 0.82) entered the study moderately more confident than girls (M = 3.55; SD = 0.89) with videogames (BF = 2.86; δ = 0.50 [95% CI = 0.06–0.96]). Moving forward, we believe that modifications to appeal to girls would improve ATHEMOS, perhaps by offering an alternative minigame to acquire the game’s intel or by simplifying the game controls. Perhaps most encouraging, player perception of ATHEMOS did not meaningfully change when the game was introduced by a teacher interventionist and integrated within a school-based behavior intervention. We were concerned that, once delivered in schools with clear educational goals, players would lose interest. Players who struggle in a videogame can become discouraged, but our results suggest that this did not occur for youth with ADHD playing ATHEMOS. It was also conceivable that players would become critical of the game when asked to play it repeatedly over several weeks, while teachers monitored progress and linked it to real-world goals. But the results suggest that, to the contrary, players found the game just as enjoyable when embedded within a broader CABI approach as they did when encountering the game for the first time in a low-stakes setting. We believe that positive user responses to ATHEMOS might be partly attributable to the balance between the educational minigames and the recreational elements of the game. The three educational minigames are not time-limited but are likely to consume a small fraction of overall playtime. The space battles are limited to 3 minutes each, but time spent exploring the Silent Canopy base and the in-game upgrades system is unlimited. As a result, players might spend much more time on recreational elements as compared to educational elements. In our view, this imbalance is acceptable because adolescents with ADHD generally have low motivation for intervention. Players who spend disproportionate time on recreational elements can be identified by teacher mentors and then helped to “beat” the minigames, prior to shifting to real-world interventions. We are hopeful that teacher mentors can take advantage of the enjoyable and rewarding nature of ATHEMOS to make this initial game-based feedback pleasant and strengthen the working alliance early in the helping relationship. Limitations There are several limitations to the present study that readers must consider when interpreting our results. First, the GUESS videogame satisfaction scale was validated with adult gamers and has not been extensively tested with young adolescents. Still, we chose this scale because it is one of the few psychometrically validated satisfaction scales available. The internal reliability estimates of the GUESS subscales in the present study were similar to the reference sample [26], but youth may be more or less severe in their game satisfaction ratings than adults. Second, our results are complicated by the fact that boys and girls clearly responded differently to ATHEMOS. We believe this fits a broader pattern of receptivity to videogames in general [19, 20], but clearly, our overall estimates of acceptability and satisfaction mask these differences. Third, the focus of this study was on playability, acceptability, and satisfaction with ATHEMOS, which might predict user willingness to play the game and perhaps engage with a teacher mentor, but not necessarily behavior change. More research is needed to examine real-world behavioral outcomes, and whether those changes can be attributed to improvements in the intermediate skills targeted by the game [24]. Conclusion In general, we believe this development project shows that games based on psychosocial TIs offer a promising new direction in ADHD games. Research is needed to demonstrate that ATHEMOS leads to meaningful real-world outcomes (e.g., positive outcomes on parent and teacher behavior ratings), but we can conclude from the present study that the target population finds the game concept enjoyable and the intervention approach satisfactory. We believe a game-assisted TI approach offers a research-informed alternative to the cognitive training games that currently dominate this genre of serious games, but that have yet to produce convincing evidence of clinical benefit.

[END]
---
[1] Url: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000374

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/

You are viewing proxied material from magical.fish. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.