Organizations invest enormous sums of money in leadership training (Sørensen, 2017; Westfall, 2019). Studies have shown that leadership behaviors can have a large impact on workplace productivity and health (Inceoglu et al., 2018; Schyns & Schilling, 2013; Zwingmann et al., 2014). In order to reap the potential benefits of training (Lacerenza et al., 2017) it is important that skills learned during training are transferred to daily practice at work. Individual coaching can improve transfer of training (Joyce & Showers, 2002; Lacerenza et al., 2017), but demands considerable resources and costs.
Using a smartphone app can be helpful in affecting behavior change (Heron & Smyth, 2010; Marcolino et al., 2018; McDevitt-Murphy et al., 2018), and function as a form of coaching. Research on ecological momentary assessment (EMA; Engelen et al., 2016; Shiffman et al., 2008), which utilizes frequent in-the-moment assessments, has shown EMA to provide reliable assessment data compared to retrospective questionnaires, while also providing an opportunity to intervene. This is often referred to as ecological momentary intervention (EMI). However, the research on behavior change using smartphone apps, EMI, and EMA has mostly been focused on clinical and health care settings, and their use in non-clinical settings is less well studied. There is one example of app-based workplace stress reduction (Ly et al., 2014) with promising outcomes, and some examples of health-related workplace interventions (Engelen et al., 2016; Engelen & Held, 2019), but we have not been able to find any peer-reviewed studies on leadership training using smartphone apps.
Similar to many clinical behavior change interventions, homework assignments are a useful tool in leadership training, increasing the likelihood that skills learned during training sessions are practiced in work settings (Lacerenza et al., 2017). While individual coaching during homework practice periods is likely to help facilitate transfer of training, it is also resource-demanding and expensive, and not easily scalable to training large numbers of managers. A smartphone app can help support the compliance of homework (Reger et al., 2013; Tang & Kreindler, 2017), and thus improve outcomes. Seeking new ways to address measurement of transfer of training outcomes and using a consumer-centric model was called for in a review paper by Baldwin et al. (2017), and we believe that this study is a step in that direction. If a smartphone app can be efficient in increasing transfer of training, it is a scalable and cost-effective method to increase the value of management training, which could help improve work environments in almost any setting.
This was a feasibility field study, with an initial plan to randomize half of the participating managers to not use the app or start using it during the latter half of the training. However, the company we collaborated with was so enticed by the idea of app support that they insisted on everyone using it. Thus, it was not possible to compare outcomes to a control group, which severely limits the possibilities to draw conclusions about the effects of app use. Gathering data on participant experiences of using the app over a relatively long time period potentially provides relevant information that could inform better designed future studies. One of the key components of a smartphone app is the timing and frequency of the app prompts, to strike a balance in prompting often enough to support behavior change, but not so often as to become annoying for the user over time. The primary focus of this study was on smartphone app acceptability and the self-report of use of skills learned during training.
A Swedish business organization in the forest industry was planning a leadership training program with the second author and another consultant and was approached about participating in this study. The organization had 30 managers and 200 employees. All managers were taking part in the training and were offered to take part in the study and use the smartphone app by verbal information and through signing a consent form. Six managers quit their jobs early in the training period, and four started late, missing initial training sessions, and had very few measurements. Another two managers had almost no responses recorded, and one manager responded the same to all questions throughout all their responses. These 13 participants were removed, resulting in 17 participants (7 women, 41%) in the remaining dataset, with a mean age of 47.5 years (SD = 7.7, range 35–64).
A smartphone app provided by LifeData LLC was used for data collection and intervention. Users downloaded the app to their smartphones, and through the app, they could download the packages constructed by the first author in collaboration with the second author. Multiple figures describing the app design in detail are available in Appendix A.
This was an intensive longitudinal within-subject design (Bolger & Laurenceau, 2013) without a control group or randomization. Primary outcome variables were the reported use of skills learned in training, perceived app support, preference of more or less notifications from the app, and report of skill components used for two specific skills supported by the app. Secondary outcomes measured were work stress, meaningful work, workload, and work enjoyment.
The leadership training consisted of six separate days with homework assignments between each day. Training days were spaced about one month apart from each other. Homework consisted of practicing one of the skills taught during training and was supported by one individual phone call to each participant before the next training session. Training and homework support were performed by two experienced organizational consultants (one being the second author), both also licensed psychologists. Except for the initial training session, each session started with an hour of coaching in smaller groups. In addition to the training, participants were offered voluntary use of a smartphone app to support training. Since the study is focused on the smartphone app, we only provide a brief outline of the training content.
The theoretical background for leadership training was primarily based on organizational behavior management (OBM; Bucklin et al., 2000). In OBM, the basic model for understanding and affecting behavior is the “three-term contingency,” which describes antecedents to behavior and consequences of behavior as key aspects of the environment that shapes and selects behavior over time (Geller, 2003). The training encompassed skills needed to implement OBM tools, such as behavior-specific instructions and feedback, as well as general communication skills. Role-playing and feedback sessions were key components of the skills training. The two consultants constructed the training in collaboration with the HR manager of the company participating, and a feedback process involving the training participants was ongoing during the training to adjust content and structure along the way.
The smartphone app was the main intervention component, aiming to increase and improve the use of tools learned during training by providing antecedents to behavior in the form of prompts/reminders, and consequences by the use of checklists at the end of the day. Extra attention was given to the skills Active Listening and Feedback. These skills were chosen based on years of providing management training using similar structure and content, with participants most often voluntarily choosing Active Listening and Feedback as homework assignments and also describing them as the most helpful of the skills learned. The smartphone app used three separate “packages,” downloadable by participants within the app. Each package consisted of its own set of prompts and questions and/or checklists. Package 1 was used continuously throughout the training—on Tuesdays and Thursdays—and provided a prompt at a random time between 8 and 9 AM (“remember to use the skills from leadership training”), and a checklist with all the skills included in the training program (“check the boxes for the skills you used today”) in the afternoon at a random time between 4 and 5 PM. The latter also included single-item measures of daily work stress, workload, meaningful work, and work enjoyment; see Appendix A for details. Every Friday, acceptability was assessed by asking if participants experienced too many or too few notifications by the app (scale 1–7, where 4 is neutral, and lower score meaning fewer notifications desired) and to which extent the app was helpful as a support for learning skills (scale 1–7, “Not at all helpful–Very helpful”). See Figure 1 for a visual description of the smartphone app packages used over time.
Notifications with checklists and scale-rated items had a response window of two hours from the first time of notification, with one reminder 30 minutes after the first notification. See Figure 2 and Appendix A for a more detailed description of the app design. Due to smartphone app malfunction, we were only able to collect data for the first 11 weeks of the training period. The app had been pilot tested without errors with a different group of managers during three shorter time periods (2–3 weeks), to test and adapt the number of notifications that could likely be useful over a longer period.
The second and third packages used were training support for specific skills, with the two separate packages using the same structure. Each package was used for four weeks between two training sessions. The skills targeted, one at a time, were Active Listening and Feedback, where the latter included both Positive and Constructive Feedback. These two packages were introduced at the same training session as when the skill was introduced, which occurred at time point 5 for Active Listening and time point 13 for feedback, based on the twice-weekly assessment over 11 weeks (22 time-points in all).
The app package structure was almost identical to the first package described in the previous paragraph but occurred on Mondays and Wednesdays. There was a morning prompt, specifically reminding to practice the skill (“remember to listen actively today”) and an afternoon checklist of the components included in the skill. For Active Listening, examples of components were “let the other person speak until done; ask the other person to tell you more; summarize what the other person said.” All checklist options are documented in Appendix A.
Three research questions were related to the transfer of skills part, and two to the smartphone app:
All statistics were conducted using Stata/SE version 16.0. Linear mixed models were used for continuous outcome variables (checked for normality), while generalized linear mixed models were used for binary outcome variables. In both types of models, random intercepts were used with data nested under each individual ID. The cutoff for statistical significance was set at p = 0.01 to reduce the risk of type-2 errors since there was a small sample with attrition. All participants with data points were included in the models, using robust maximum likelihood estimation. Missing data were assumed to be missing completely at random. Checklists of skills and components (for Active Listening and Feedback) used were summarized into totals for each timepoint and participant. A dummy variable was created to indicate response or non-response to assess the probability of responding over time.
Data is available in an open access repository (DOI: 10.6084/m9.figshare.12789026) and Appendix A contains the commands needed to reproduce the statistics presented in the result section, using Stata or R.
The four primary outcomes are summarized in Table 1. Self-reported skill use, as indicated by checklists twice weekly over 11 weeks, saw an increase over time. The perception of the smartphone app as supportive in practicing skills also improved, based on 11 weekly assessments. Neither of the two specific skills targeted by separate smartphone app packages resulted in a statistically significant increase of the report of number of components used over the four weeks each skill received extra app support. Participants’ preferences of frequency of notifications from the app were neutral over the 11 weekly assessments.
|OUTCOME||INTERCEPT, 95% CI [LL UL]||FIXED EFFECT OF TIME: β, 95% CI [LL UL]||SE||P||ICC||N||OBSERVATIONS (RANGE, MEAN, RESPONSE RATE)|
|Number of skills used||2.79 [2.19 3.38]||.052 [.021 .083]||.016||.001||.32||17||206 (5–20, 12.1, 55%)|
|Active Listening components||2.89 [2.20 3.57]||.104 [–.011 .219]||.059||.075||.37||15||68 (2–7, 4.5, 56.7%)|
|Feedback components||2.05 [1.40 2.71]||.133 [.018 .247]||.058||.023||.18||10||42 (1–8, 4.2, 52.5%)|
|Perceived support||4.35 [3.88 4.81]||.071 [.033 .110]||.020||.000||.59||17||127 (3–11, 7.5, 67.9%)|
|More/less notifications||4.07 [3.71 4.43]||–.016 [–.048 .016]||.016||.336||.54||17||127 (3–11, 7.5, 67.9%)|
The twice-weekly self-reported use of Active Listening became more likely over the 11-week time period (Odd’s Ratio (OR) = 1.100 (95% CI [1.039 1.159], SE = 0.030 p = 0.001), and Constructive Feedback showed a similar result (OR = 1.116 (95% CI [1.054 1.182], SE = 0.032 p < 0.001), while Positive Feedback showed no change (p = 0.711). The increase in reporting of Active Listening and Constructive Feedback did not coincide with the training sessions or introduction of app packages targeting these skills, based on the visual inspection of graphs of these variables as shown in Figure 3.
Odd’s Ratio for responding to any of the twice-weekly checklists and questions over the 11-week time period was 0.902 (95% CI [0.868 0.940], SE = 0.018, p < 0.000), which means that the probability of responding decreased gradually over time (see Figure 4). None of the twice-weekly single-item questions about levels of work experiences changed significantly (work stress p = 0.09, meaningful work p = 0.42, work enjoyment p = 0.63, and workload p = 0.91).
There were no statistically significant effects of sex or age in any analysis.
The purpose of this study was to investigate the possible benefits of using a smartphone app to improve transfer of training of leadership skills. Statistical analysis of data collected through the smartphone app showed that the first research question about participants’ self-reported number of skills used increased over time. Based on the intercept and point estimate for the fixed effect of time, the change was on average from 2.79 skills reported at the beginning to 3.93 at the end. Or, more pragmatically summarized, an increase from approximately 3 to 4 skills reported during the 11 weeks. As a side note, the skills checklist included all of the skills taught during the training program, making it unlikely to see skills not yet trained to be indicated by participants in the early measurements. Research question two was about the reported use of Active Listening and Positive/Constructive Feedback, where Active Listening and Constructive Feedback showed an increase over time, although there was no discernible connection between the increase and the introduction of the skills in training and added app support. This could indicate that the app supporting specific skills did not contribute to behavior change beyond the basic app package with reminders and a checklist. However, the lack of a control group and no observational or objective data on actual skills and skill subcomponents used makes this mere speculation. Since participant data on the frequency of notifications did not change over time, the added app support for specific skills does not seem to have been perceived as burdensome. The third research question related to how many components of Active Listening and Feedback were reported in checklists during the extra app support periods. Neither of the two changed significantly over time.
Both research question four and five concerned participants’ experiences of the smartphone app, which arguably is the most important outcome in this study. The app was seemingly perceived as helpful in supporting the training of skills in daily work settings. On the rating scale from 1 to 7, the average rating of support rose from 4.35 at the start to 5.12 at the end, based on point estimates. Regarding whether participants desired more or fewer notifications from the app, there was no significant difference over time from the initial average of 4.07. This rating scale also used a range of 1–7, with 4 being neutral and lower scores indicating a desire for less notifications and a higher score indicating more notifications wanted.
A high workload can be a barrier to engage in practicing new skills. The training in itself is likely to increase the workload due to attending training sessions and engaging with homework assignments, which could affect stress levels (Hobson & Beach, 2000; Labrague et al., 2018). Managers were asked about their perceived stress and workload twice every week, and this did not indicate any statistically significant change over time. If the smartphone app itself had been perceived as stress-inducing, particularly during the periods of more frequent notifications for specific skills, this should have been visible as a change in the desired number of notifications, or a decrease in perceived support. On the other hand, actual responses recorded in the app did decrease over time, and the second skill package saw only 10 users, compared to 15 for the first skill package. While the perceived support score increased over time, as app responses also decreased, this effect could to some degree be due to those not perceiving the app as supportive also used it less. The decrease in app responses over time might be connected to what several participants described during coaching sessions as “internalizing” the skills after weeks of practice and becoming less reliant on the app for reminders.
At the telephone coaching sessions, participants were asked an open question about how they experienced the app, which provided some additional insight. While most of the participants appreciated the app, some indicated problems responding within the notification time frame because they spent a lot of time in meetings or in their car. It seems important that the participants experience the app as a useful complement to the training and not a burden. This could be better achieved if the app design can be adapted throughout the training period, based on feedback from the users. In this instance, we piloted the app with a different management group in a different industry, and they may have had different work conditions compared to the study participants. If the app leads to increased use of skills and these skills are helpful for the managers, this could in turn increase their engagement in the coming training sessions. The opposite might occur if users dislike the app and feel like they are forced to use it, which could influence the whole training negatively. Such feedback loops would be interesting to study in more detail. In this study, the data collected through the app was not used for continuous feedback or evaluation during the ongoing training. This is a major shortcoming, particularly from a behavior change perspective. The notifications used are most likely to have functioned as antecedent prompts and negative reinforcers. Since the app was generically constructed for data collection rather than intervention, it had no built-in functions for feedback. Adding feedback in a way that enables positive reinforcement to be utilized would likely lead to improved outcomes from using the app. This could be either focused on the individual only, receiving feedback on past performance (and perhaps prompting commitment for future performance), or it could entail comparisons with a group mean performance. Both possibilities are interesting areas of future study.
Since the study design did not include a control group, we cannot say whether the app itself had any effects. Outcomes could be entirely due to the training program itself or other factors not controlled for. The small number of participants combined with sizeable attrition in responses contributed to limited statistical power. The study also could have been strengthened by other means of related data collection, such as asking employees about how they perceived their managers’ use of leadership skills during the training period. There are also issues of individual preferences that have not been addressed or controlled for in any way, which could have been surveyed (Bregenzer et al., 2019; Dunkl & Jiménez, 2017). Even though all participants volunteered, managers may have had different levels of interest in using a smartphone app or would have preferred a more personalized prompting system.
The design of the app packages was limited by the functions available in the app platform itself, which is intended for data collection purposes rather than being used as a behavior change tool. While consequences of behavior are key to affecting future behavior, the app lacked the functionality to provide stronger consequence-focused intervention components, such as user-specific feedback with adaptive timing in relation to participant behavior.
While the ecological momentary assessments used in this study utilized daily retrospective ratings, rather than multiple momentary ratings every day, which is commonly used in briefer clinical studies, it still provides reliability benefits compared to ratings that demand retrospection over longer periods. It would probably have been helpful if we had discussed the design of our study with the supplier of the smartphone app, and the malfunctioning that occurred after 11 weeks might have been avoided. When communicating about the malfunctioning with LifeData LLC, they stated that nobody had used the app continuously for such long periods before.
Previous research in clinical settings has shown that smartphone apps can contribute to behavior change and skills acquisition (Heron & Smyth, 2010; Levin et al., 2019; McDevitt-Murphy et al., 2018). This study transfers this type of intervention to the field of leadership skill development and shows that app support can be perceived as useful, and possibly beneficial for transfer of training into the daily work context. The prospect of finding utility in smartphone app support for transfer of training of leadership skills seems viable and worthwhile to investigate further. There is a lot of room for improvement, both in terms of study design and app design, which has been elaborated in the discussion, and we hope that others can build on our findings in furthering this field.
Ethics approval was sought and received from the Norwegian Center for Research data (NSD) reference 252044, and Regional Committees for Medical and Health Research Ethics (REK), reference 2019/98. Participants received verbal information by the first author at the first training session and had the opportunity to ask questions. Upon downloading the smartphone package, participants received written information and indicated consent to participate. Each participant was assigned a code number, assuring anonymous data analysis.
The authors have no competing interests to declare.
The first author designed the study, collected and analyzed the data, and wrote the results section and large portions of the introduction, method section, and the discussion.
The second author designed the leadership training together with a third party, contributed feedback on the research and app package design, and wrote and edited large portions of the discussion, introduction, and method sections.
Baldwin, T. T., Ford, J. K., & Blume, B. D. (2017). The state of transfer of training research: Moving toward more consumer-centric inquiry. Human Resource Development Quarterly, 28(1), 17–28. DOI: https://doi.org/10.1002/hrdq.21278
Bregenzer, A., Wagner-Hartl, V., & Jiménez, P. (2019). Who uses apps in health promotion? A target group analysis of leaders. Health Informatics Journal, 25(3), 1038–1052. DOI: https://doi.org/10.1177/1460458217738121
Bucklin, B. R., Alvero, A. M., Dickinson, A. M., Austin, J., & Jackson, A. K. (2000). Industrial-organizational psychology and organizational behavior management. Journal of Organizational Behavior Management, 20(2), 27–75. DOI: https://doi.org/10.1300/J075v20n02_03
Dunkl, A., & Jiménez, P. (2017). Using smartphone-based applications (apps) in workplace health promotion: The opinion of German and Austrian leaders. Health Informatics Journal, 23(1), 44–55. DOI: https://doi.org/10.1177/1460458215623077
Engelen, L., Chau, J. Y., Burks-Young, S., & Bauman, A. (2016). Application of ecological momentary assessment in workplace health evaluation. Health Promotion Journal of Australia, 27(3), 259–263. DOI: https://doi.org/10.1071/HE16043
Engelen, L., & Held, F. (2019). Understanding the office: Using ecological momentary assessment to measure activities, posture, social interactions, mood, and work performance at the workplace. Buildings, 9(2), 54. DOI: https://doi.org/10.3390/buildings9020054
Geller, E. S. (2003). Organizational behavior management and industrial/organizational psychology. Journal of Organizational Behavior Management, 22(2), 111–130. DOI: https://doi.org/10.1300/J075v22n02_10
Heron, K. E., & Smyth, J. M. (2010). Ecological momentary interventions: Incorporating mobile technology into psychosocial and health behavior treatments. British Journal of Health Psychology, 15(Pt 1), 1–39. DOI: https://doi.org/10.1348/135910709X466063
Hobson, J., & Beach, J. R. (2000). An investigation of the relationship between psychological health and workload among managers. Occupational Medicine, 50(7), 518–522. DOI: https://doi.org/10.1093/occmed/50.7.518
Inceoglu, I., Thomas, G., Chu, C., Plans, D., & Gerbasi, A. (2018). Leadership behavior and employee well-being: An integrated review and a future research agenda. The Leadership Quarterly, 29(1), 179–202. DOI: https://doi.org/10.1016/j.leaqua.2017.12.006
Labrague, L. J., McEnroe-Petitte, D. M., Leocadio, M. C., Bogaert, P. V., & Cummings, G. G. (2018). Stress and ways of coping among nurse managers: An integrative review. Journal of Clinical Nursing, 27(7–8), 1346–1359. DOI: https://doi.org/10.1111/jocn.14165
Lacerenza, C. N., Reyes, D. L., Marlow, S. L., Joseph, D. L., & Salas, E. (2017). Leadership training design, delivery, and implementation: A meta-analysis. Journal of Applied Psychology. DOI: https://doi.org/10.1037/apl0000241
Levin, M. E., Haeger, J., & Cruz, R. A. (2019). Tailoring acceptance and commitment therapy skill coaching in the moment through smartphones: Results from a randomized controlled trial. Mindfulness, 10(4), 689–699. DOI: https://doi.org/10.1007/s12671-018-1004-2
Ly, K. H., Asplund, K., & Andersson, G. (2014). Stress management for middle managers via an acceptance and commitment-based smartphone application: A randomized controlled trial. Internet Interventions, 1(3), 95–101. DOI: https://doi.org/10.1016/j.invent.2014.06.003
Marcolino, M. S., Oliveira, J. A. Q., D’Agostino, M., Ribeiro, A. L., Alkmim, M. B. M., & Novillo-Ortiz, D. (2018). The Impact of mHealth Interventions: Systematic Review of Systematic Reviews. JMIR MHealth and UHealth, 6(1), e23. DOI: https://doi.org/10.2196/mhealth.8873
McDevitt-Murphy, M. E., Luciano, M. T., & Zakarian, R. J. (2018). Use of ecological momentary assessment and intervention in treatment with adults. FOCUS, 16(4), 370–375. DOI: https://doi.org/10.1176/appi.focus.20180017
Reger, G. M., Hoffman, J., Riggs, D., Rothbaum, B. O., Ruzek, J., Holloway, K. M., & Kuhn, E. (2013). The “PE coach” smartphone application: An innovative approach to improving implementation, fidelity, and homework adherence during prolonged exposure. Psychological Services, 10(3), 342–349. DOI: https://doi.org/10.1037/a0032774
Schyns, B., & Schilling, J. (2013). How bad are the effects of bad leaders? A meta-analysis of destructive leadership and its outcomes. The Leadership Quarterly, 24(1), 138–158. DOI: https://doi.org/10.1016/j.leaqua.2012.09.001
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4(1), 1–32. DOI: https://doi.org/10.1146/annurev.clinpsy.3.022806.091415
Sørensen, P. (2017). What research on learning transfer can teach about improving the impact of leadership-development initiatives. Consulting Psychology Journal: Practice and Research, 69(1), 47–62. DOI: https://doi.org/10.1037/cpb0000072
Tang, W., & Kreindler, D. (2017). Supporting homework compliance in cognitive behavioural therapy: Essential features of mobile apps. JMIR Mental Health, 4(2), e20. DOI: https://doi.org/10.2196/mental.5283
Westfall, C. (2019). Leadership development is a $366 billion industry: Here’s why most programs don’t work. Forbes. https://www.forbes.com/sites/chriswestfall/2019/06/20/leadership-development-why-most-programs-dont-work/
Zwingmann, I., Wegge, J., Wolf, S., Rudolf, M., Schmidt, M., & Richter, P. (2014). Is transformational leadership healthy for employees? A multilevel analysis in 16 nations. German Journal of Human Resource Management, 28(1–2), 24–51. DOI: https://doi.org/10.1177/239700221402800103