Coaching is far from being a fully regulated industry. I feel that there is still a big portion of people that look at coaches with contempt, while the industry is thriving in others.
Official certifications, like the ones offered by the ICF and ICC, are a form of governance based on core skills development, peer review, and mentoring. These institutions manage to unite coaches under a sphere of credibility that lays its very foundation on evidence-based practice.
That being said: not all coaches are part of these organizations, and even less will align with the ethics they propose. The industry remains a wild west festered with sellers and marketers reinvented as coaches.
There are two serious problems deriving from this fragmentariness of the coaching world. One touches the ethics (and will not be treated here); the other is that measuring the effectiveness, thus the validity of coaching, is difficult.
Measuring the unmeasurable
Let’s begin with the inability to offer concrete measurements of success in EUR (or USD) value. It has to do with the idea of Return On Investment (ROI), a concept dear to capital investors worldwide. It remains a prominent feature in coaching performance metrics. More on the matter in this ICF blog written by Nader Bechini; and this Edge International entry, by Jonathan Middleburgh.
ROI was firstly considered a possible metric of coaching performance in 1997 when J.J. Phillips addressed it as an additional dimension of success for training programs. Phillips specifically targeted coaching a decade later (2007). Data was collected from a coaching program in which executives volunteered and integrated from different resources, and then isolating the effects of coaching from the rest. According to the same study, the ROI was about 221% (to quote: “for every dollar invested, $3.21 was returned). This data was supported by his previous study (Phillips, 2005).
This application of the ROI is built on the top of Donald Kirkpatrick’s model of training evaluation (1977) and therefore it isn’t a lone metric. Kirkpatrick’s taxonomy for evaluating training programs relies on four levels of evaluation:
- L1: evaluating trainee’s reaction
- L2: evaluating learning
- L3: assessing behavioral change
- L4: Results following training
The popularity of ROI may be coming from the financial expectations of companies that invest in coaching programs for their employees, in particular executives. Then it should not be a surprise that even life coaches use ROI as part of their promotional strategy.
There is just one caveat. ROI is not a reliable measurement, and more recent studies argue that coaching success cannot be quantified in currency.
How reliable is ROI to measure the impact of coaching?
This is what has been argued by Grant (2012), who aims to demonstrate why ROI is a flawed metric. Grant’s arguments can be wrapped up in the following points:
- It is hard to distinguish the success attributed to the work of the coach (i.e. the coaching performance) against the one of the coachee (i.e. self-efficacy).
- The assessment of success may be highly subjective, and thus difficult to quantify.
- Results can also be altered if the coachee’s goal relates to managing other people, due to several causal relationships.
- Other externalities may impact the effectiveness of coaching too.
- The presence of a third party involved (for example, sponsors, stakeholders, and direct reports) can put pressure on the coaching relationship and encourage the coachee to report false results.
Many tools that evaluate coaching effectiveness use a holistic approach and recognize the importance of the so-called intangible benefits of coaching conversations. A literature review on coaching effectiveness done by De Meuse et al (2009), two years after the already cited paper by Phillips, already hints that a single measurement such as the infamous ROI can be misleading, and hence it is better to adopt a broader set of measurements. Coaching affects positively the coachee, however, the latter tends to see more progress than the coach (or a third party) would do. Moreover, with ROI used as evaluation criteria, the coaching impact return did not seem consistent.
The alternative
Currently, the most popular approach to measure effectiveness is surveying, with the best representative being the 360 feedback survey. These tools better capture the impact of coaching on people who undergoes it, beyond executive coaching and regardless of whether they are enrolled in a corporate coaching program or not.
An example is the Coaching Effectiveness Survey (CES) developed by IECL, which measures the reliability of coaching for:
- Evaluating the quality of the coaching performance
- Identifying key areas of benefits from coaching
- Overall satisfaction with the experience by the coachee
- Providing insights on the overall coaching process
In this matter, research of 2013 led by Tooth, Nielsen, and Armstrong focused on the improvement of motivation, communication, and interpersonal skills after coaching conversations, measured with CES-like tools. The results highlighted sustained learning and behavioral changes regarding:
- Emotional competence
- Impact on others
- Effective career management
And on top of that, 86.3% of the participants wanted to continue with coaching, and 78.5% wanted to continue with the same coach. Can we at least infer from this alone that, as a coach, fostering the relationship with the client is important for successful coaching?
The same study shows a highly relevant relationship between the benefits of coaching and their relevance. Positive change in mindset, self-reflection, and situational awareness are indeed all necessary when giving/receiving feedback effectively, communicating with impact, and managing conflict.
Implications
It is fair to mention that surveys are built and done upon assumptions and cannot be entirely objective. Nevertheless, they can give a bigger picture compared to a fictitious ROI percentage.
Shall we shift from a quantitative measurement of the ROI itself to a qualitative one, when it comes to coaching conversation?
We cannot consider it a pure financial factor, because it does not come from numbers alone. And it does not work like inferential statistics either.
It would be easier to address the ROI in form of improved relationships with peers and reports, job satisfaction, and teamwork rather than a numerical value. That is possible with 360 feedback models that evaluate the implicit benefits and, as Boysen et al (2018) remark, an estimated numerical ROI to be taken with reserve (as per their research, in a range from 1 to 15+ times the investment, the average is about 4-8).
Conclusion
Coaches have a hard time establishing measurements of coaching performances because there is no such thing as “one-size-fits-all”. The coaching environment, the personality of the coachee, social and causal relationships all play a role in the success of coaching. While the coach might be a master of his field, in the end, is always the coachee who must act and lead the journey to his own success and personal development.
On top of this, it is worth mentioning that the role of emotional intelligence in performance is now recognized worldwide (soon I will post a review of “Emotional Intelligence” by Daniel Goleman in my Inside The Book series).
The coaching community is well aware of that. As Betska K-Burr (2021) wraps up in a column of Choice Magazine, both executive and life coaches have now an infinity of tools to use: many are psychometrics (DISC, Whole-Brain Thinking) whereas others are targeted assessments (WBEF, Ikigai).
All cannot be reduced to a single numerical coefficient. We would simply miss the complexity of the world, and misjudge the part of reality we observe.
And this comes from a coach who happens to work in the field of data science.
References
- Boysen, Sheila, et al. “Organisational Coaching Outcomes: A Comparison of a Practitioner Survey and Key Findings from the Literature.” International Journal of Evidence Based Coaching and Mentoring, vol. 16, no. 1, 2018, pp. 159–166., doi:10.24384/000475.
- De Meuse, Kenneth P., et al. “Evaluating the Effectiveness of Executive Coaching: Beyond Roi?” Coaching: An International Journal of Theory, Research and Practice, vol. 2, no. 2, 2009, pp. 117–134., doi:10.1080/17521880902882413.
- Grant, Anthony M. “Roi Is a Poor Measure of Coaching Success: Towards a More Holistic Approach Using a Well-Being and Engagement Framework.” Coaching: An International Journal of Theory, Research and Practice, vol. 5, no. 2, 2012, pp. 74–85., doi:10.1080/17521882.2012.672438.
- K-Burr, Betska. “Targeting Success: Measuring the Impact of the Coaching Experience.” Choice, vol. 17, no. 1, 2019, pp. 21–22., doi:www.choice-online.com.
- Kirkpatrick, Donald L. “Evaluating Training Programs: Evidence vs. Proof.” Training and Development Journal, vol. 31, no. 11, 1977, pp. 9–12.
- Phillips, Jack J. “Measuring the Roi of a Coaching Intervention, Part 2.” Performance Improvement, vol. 46, no. 10, 2007, pp. 10–23., doi:10.1002/pfi.167.
- Phillips, Jack J. Return on Investment in Training and Performance Improvement Programs, 1st ed., Woburn, MA: Butterworth-Heinemann, 1997.
- Phillips, Jack J., and Patricia P. Phillips. “Measuring ROI in Executive Coaching.” International Journal of Coaching in Organizations, vol. 3, no. 1, 2005, pp. 53–62., doi:https://researchportal.coachfederation.org/Document/Pdf/2866.pdf.
- Tooth, Julie-Anne, et al. “Coaching Effectiveness Survey Instruments: Taking Stock of Measuring the Immeasurable.” Coaching: An International Journal of Theory, Research and Practice, vol. 6, no. 2, 2013, pp. 137–151., doi:10.1080/17521882.2013.802365.
The Key to Think 