Internal Controls Design website by Matthew Leitch (tutor, researcher, author, & consultant)
New website, new perspective: www.WorkingInUncertainty.co.uk - Related articles - All articles - The author - Services

Risk Uncertainty Management results Graphic


Individual differences in risk and uncertainty management
Results of an online survey

by Matthew Leitch, 8 May 2006


Summary
What did respondents agree on?
Which preferences were linked and why?
Were risk and uncertainty responded to rationally?
Differences between groups of respondents
Respondent profile
Scenarios and actions used in the survey
Potential future research and developments
Related reading

Thank you

First, thank you to everyone who responded to this survey. The response was at least twice that to any previous survey in this series, with nearly everyone interested to know how their results compared with others. Several respondents wrote saying how interesting they had found the experience and the evidence is that, despite the time needed to do the survey, many more people than usual persisted until the end.

Summary

The survey probed differences between people in how they think about risk and uncertainty in business situations.

The survey presented respondents with four different hypothetical scenarios designed to probe beliefs about how to manage risk and uncertainty. In each scenario they were offered five potential actions and asked to rate each action individually as either Awful, Poor, Neutral, Good, or Great. They also had to give a certainty rating between 0% and 100% for each rating of an action.

On some actions respondents agreed quite closely on the rating. For example, almost everyone thought it a good idea to see the evidence for a disputed technical problem oneself. On other actions there was a great deal of disagreement. For example, respondents were almost equally divided on whether it was a good idea to start a creative project by stating specific financial targets.

Statistical analysis of the action ratings showed that certain ratings correlated. For example, people who thought setting targets for the creative project was a good idea also tended to think that saying nothing about a serious potential technical problem on a building project was a good idea, or at least not as bad as others thought. The target setters were also more likely to favour pushing harder on products that were selling less well than expected and were correspondingly less inclined to shift focus towards products that were doing well, or at least better than expected.

Looking at the full correlation matrix it was possible to perform a manual factor analysis and this revealed the likely effect of a number of beliefs. The main factor was a tendency to prefer target setting, hold to original targets, and not disclose information about risks and uncertainties. Other tendencies were a tendency to persuade by giving extreme descriptions, a tendency to be influenced by emerging data (the opposite of the target setting style), and a desire to investigate and explain.

There were no right answers to this survey. However, some actions were more preferred than others and some seem to me more rational, honest, and objective than others. In previous publications I have argued that a number of cognitive and social factors tend to push us towards a blinkered view of the future, suppression of our uncertainties, and a resulting failure to manage risk and uncertainty effectively. Major objectives of risk management programmes should be to help people think more widely about the future and deal effectively with this more realistic outlook.

For the purposes of analysing the survey data a model profile of answers was selected based on compliance with the a small set of principles. These principles are that (1) objectivity and rationality are desirable, (2) we are frequently uncertain and facing risks and need to acknowledge this and act in accordance with it, (3) focus on single future outcomes is usually at the expense of an objective view of the future, which requires a recognition of uncertainty, (4) it is better to be honest, and (5) these considerations are very important and usually outweigh others. These principles imply a rating for each action.

Collectively, respondents behaved in accordance with the principles stated above in that the most highly rated actions were always the ones most consistent with the principles.

However, very few respondents consistently rated actions in accordance with the most preferred ratings or in accordance with the risk and uncertainty principles. This individual inconsistency may be because the underlying issues had not been thought about previously by many respondents, or perhaps because other considerations over-rode uncertainty in some scenarios.

However, I suspect the main reason for departing from the theoretical ideal of the principles was usually a failure to see the dangers in certain actions. Many people thought it was fine to say nothing about a worrying risk that had developed until they had found a solution, at which point they could explain it all. In the real case of the Holyrood building project, on which this scenario was loosely based, civil servants took exactly this approach. Unfortunately, they could not think of a solution and consequently kept quiet far too long. They did not see the dangers in their approach and neither did many respondents to the survey. Happily, the most highly rated approach to this problem was to give a full briefing of risks and uncertainties right away, but the results suggest that many people would not actually do so.

There were slight differences between different groups of respondents. The auditors tended to give answers further away from the principles than others and were particularly prone to concealing risks and exaggerating both good and bad news in pursuit of a reaction. People who were students or sole traders tended to score closest to the principles, while people in an organizational hierarchy scored further away, with "middle" managers being furthest from the principles as well as spending the least amount of time on the survey, on average. However, these differences are slight and it is likely that the wide variations between individuals are driven mainly by other things.

Taken as a whole, this exploratory study suggests that, although the most popular actions are generally those that follow the principles of risk and uncertainty management listed above, there will often be a high risk of people acting otherwise, with potentially damaging consequences. Organizations should seek to understand better what their people think and promote the practices they believe are appropriate, rather than leaving things to chance. The detailed findings of the survey suggest some unexpected but powerful tactics for doing that.

What did respondents agree on?

Two measures of (dis)agreement were calculated for each of the 20 actions to be rated. One was the percentage of ratings that were the most popular rating for that action. (This is called "Most popular %" in the table below, and higher numbers mean greater agreement.) The other was the percentage of ratings that were in the minority, where this was calculated as the sum of Good and Great ratings, or the sum of Poor and Awful ratings, whichever was lower. (This is called "Minority %" in the table below, and lower numbers mean greater agreement.)

No.Action summaryMost popular %Minority %Ave. certainty
1Set targets beforehand314278
2Sell the heights of success461876
3Give best estimate344075
4Describe full range471178
5Avoid discussing outcomes551384
6Investigate under performance55681
7Push under performers422074
8Investigate over performance54184
9Shift to surprisingly successful lines372471
10Shift to highest contributors491175
11Say nothing361181
12Visit roof company45481
13Tell the worst362477
14Find a solution, then disclose492281
15Full briefing54488
16Out-boast opponents313377
17Explain common uncertainties54682
18Explain your flexibility60283
19Short and positive presentation491577
20Show knowledge of past surprises55978

We can think of the minority percentage as an indication of how hard it would be to get a group of managers to agree to do the right thing. Clearly some actions caused more disagreement than others. Seven of the 20 items left less than 10% of people in the minority - so would be agreed relatively easily - while three items had a minority of more than 30% and could easily go the wrong way (especially if many respondents were Neutral on the item).

The minority percentage is also an indication of how many managers typically hold views on these points that are out of line with the majority and might be, perhaps routinely, making decisions that others would regard as poor.

You can also see that the ratings that people disagreed on most tended to be the ones they felt the least certain when giving the ratings. Here is a scatter plot showing both measures of agreement against average certainty for each of the 20 actions.

Which preferences were linked and why?

The ratings were turned into numbers (Awful = 0, Poor = 1, Neutral = 2, Good = 3, and Great = 4) and the answers by all 89 respondents were correlated taking every pair of actions in turn. This created a correlation matrix and revealed some interesting links between ratings, often across different scenarios. Here are the highest 7 correlations, with suggestions as to what belief might lie behind them.

ActionCorrelationActionPossible belief
Shift to highest contributors0.58Shift to surprisingly good performersData from experience should drive actions.
Set targets beforehand0.48Push under-performersControl works by negative feedback loops
Short and positive presentation0.43Out-boast the competitionTo persuade it is good to be 'confident' and 'positive.'
Investigate under-performance-0.35Shift to surprisingly successful linesNegative feedback control thinking versus learning from experience.
Find a solution, then disclose risk0.34Short and positive presentationGiving bad news is career damaging.
Find a solution, then disclose risk0.33Say nothingGiving bad news is career damaging.
Visit roof company0.33Shift to the highest contributorIt's good to get and respond to facts.

The actions were rearranged in the correlation matrix to group together the ones that tended to have correlated answers, giving a manual factor analysis. (There was not enough data for a formal analysis.) Even though the groups are faint they are suggestive and help make some sense of the data.

Action summaryNo.2051591012321316191411716817184
show past knowledge and surprises201-0.08-0.180.110.050.10.130.02-0.06-0.05-0.09-0.01-0.050.080.06-0.1-0.10.10.070.11
avoid discussion of outcomes5-0.0810.210.16-0.07-0.130.2-0.050.020.20.220.120.170.230.07-0.11-0.1600.06-0.11
full briefing15-0.180.211-0.17000.140.080.050.240.02-0.23-0.190.180.130.1400.110.030
shift to surprisingly successful lines90.110.16-0.1710.580.220.0600.10.170.20.16-0.05-0.030.15-0.35-0.05-0.09-0.070.06
shift to highest contributors100.05-0.0700.5810.330.260.0500.210.090.120-0.090.19-0.18-0.04-0.010.03-0.13
visit roof company120.1-0.1300.220.3310.1-0.03-0.110.130.150.21-0.03-0.020.30.010.060.090.05-0.01
give best estimate30.130.20.140.060.260.110.270.090.260.190.010.030.160.140.120.130.050.01-0.1
sell the heights of success20.02-0.050.0800.05-0.030.2710.220.240.22-0.03-0.080.040.130.070.06-0.060.07-0.26
tell the worst13-0.060.020.050.10-0.110.090.2210.290.16-0.18-0.040.080.13-0.06-0.18-0.32-0.050.01
out-boast opponents16-0.050.20.240.170.210.130.260.240.2910.430.130.10.120.280.04-0.06-0.290.01-0.14
short and positive19-0.090.220.020.20.090.150.190.220.160.4310.340.040.120.27-0.06-0.04-0.19-0.12-0.2
find a solution, then disclose14-0.010.12-0.230.160.120.210.01-0.03-0.180.130.3410.330.060.160.1-0.010.160.110
say nothing11-0.050.17-0.19-0.050-0.030.03-0.08-0.040.10.040.3310.270.250.21-0.140.02-0.04-0.02
push under performers70.080.230.18-0.03-0.09-0.020.160.040.080.120.120.060.2710.480.090.01-0.210.240
set targets beforehand10.060.070.130.150.190.30.140.130.130.280.270.160.250.4810.110.03-0.060.22-0.14
investigate under performance6-0.1-0.110.14-0.35-0.180.010.120.07-0.060.04-0.060.10.210.090.1110.260.140.150.03
investigate over performance8-0.1-0.160-0.05-0.040.060.130.06-0.18-0.06-0.04-0.01-0.140.010.030.2610.250.270.08
explain common uncertainties170.100.11-0.09-0.010.090.05-0.06-0.32-0.29-0.190.160.02-0.21-0.060.140.2510.230.08
explain your flexibility180.070.060.03-0.070.030.050.010.07-0.050.01-0.120.11-0.040.240.220.150.270.231-0.01
describe full range40.11-0.1100.06-0.13-0.01-0.1-0.260.01-0.14-0.20-0.020-0.140.030.080.08-0.011

Were risk and uncertainty responded to rationally?

There are no right answers to this survey. However, a person acting according to the principles below would give a certain set of answers to the survey and these were used as a profile against which to compare respondents' answers.

The principles are that:

  1. Objectivity and rationality are desirable.

  2. We usually face risk and uncertainty and need to acknowledge this and act in accordance with it.

  3. Focus on single future outcomes is usually at the expense of an objective view of the future, which requires a recognition of uncertainty.

  4. It is better to be honest.

  5. These considerations are very important and usually outweigh others.

These principles imply a rating for each action, which will be called the Principles Model for the purposes of this study.

Collective wisdom

The main observation is that, collectively, respondents rated actions similarly to the Principles Model. If you are a respondent there were probably several things in the preceding analysis that you disagreed with, but, overall, the level of agreement is impressive. The following table sets out where the collective wisdom of respondents agreed and disagreed with the Principles Model.

Action summaryAgreement with PrinciplesDisagreement with PrinciplesComments
Set targets beforehandSlightly more respondents against this than for it, and more Awful ratings than Great ratings.  
Sell the heights of successNot rated as highly as Describe full range.Large majority in favour of this action. 
Give best estimateNot rated as highly as Describe full range.Slight majority in favour of this action. 
Describe full rangeHeavily preferred.  
Avoid discussing outcomes Heavy majority against this.Not relevant to Principles
Investigate under performanceHeavy majority in favour, but not as much as for investigating over-performance.  
Push under performersMajority against this action.  
Investigate over performanceHeavy majority in favour, and more so than for investigating under-performance.  
Shift to surprisingly successful linesMajority in favour of this.  
Shift to highest contributorsHeavy majority in favour of this.  
Say nothingVery heavy majority against this, with more than a third rating it Awful.  
Visit roof companyHeavy majority in favour of this action.  
Tell the worstMajority against this.  
Find a solution, then discloseRated much lower than giving a full briefing.Heavy majority in favour of this action. 
Full briefingEnormous majority in favour of this, with 54% rating it Great.  
Out-boast opponentsSlight majority against this action.  
Explain common uncertaintiesHeavy majority against this.  
Explain your flexibilityHeavy majority in favour.  
Short and positive presentationMajority against.  
Show knowledge of past surprisesHeavy majority in favour.  

In the above analysis I have largely ignored disagreement between respondents and the Principles Model as to the exact rating i.e. Good vs Great, and Poor vs Awful. The extreme ratings "Awful" and "Great" were relatively rare among respondents, compared with the Principles.

On three items the majority view disagreed with the Principles Model:

As you can see from the Respondent Profile later in this report, the respondents were volunteers who heard about the survey through one of a number of Internet discussion lists for performance managers, auditors, risk managers, and risk analysts. Could their high degree of agreement with the Principles Model be because of their special knowledge and roles? Perhaps, but if this were so the Performance Managers (for whom risk and uncertainty is not such a focus as for auditors and risk specialists) should have scored further away from the Principles Model. In fact they were closer to the model than the Auditors.

Individual variations

Having said that the collective wisdom was close to the Principles Model there were considerable variations between individuals and no respondent made all their choices in agreement with the For/Against recommendations of the Principles Model, let alone agreeing on all the ratings.

Firstly, here is the analysis, scenario by scenario, reducing the answers to just "Yes" (if Good or Great) or "No" (if Neutral, Poor, or Awful):

Looking across all scenarios, no respondent chose the Principles Model profile in more than two scenarios. The model profile was chosen in two scenarios by 18 respondents, in one scenario by 39 respondents, and 32 respondents didn't choose the model profile in any scenario.

Potential remediation strategies

Suppose the leaders of an organization decided that they wanted more of their managers to prefer actions in accordance with the Principles Model or something similar. What strategies are suggested by this research?

Differences between groups of respondents

In the survey, respondents were asked which best described their role and background: Performance Manager, Risk Manager, Internal or External Auditor, or Other. They were also asked a question to assess their position in an organizational hierarchy.

To study possible differences between groups of respondents their ratings were turned into a more exact measure of difference from the Principles Model. The Principles Model was turned into a profile of ratings (Awful = 0 to Great = 4 as before) and each respondent was given a score that was the sum of the absolute difference between their ratings and the Principles Model. The lower the respondent's score the closer their match with the model. This score was analysed in various ways.

Lower scores are "better" with respect to the Principles Model.

Overall the differences tended to be slight and probably not very reliable. The number of respondents in each group was not equal of course and some combinations of role and pecking order had many more respondents than others.

Role/background \ Pecking orderSoleTopMiddleBottomTotal
Performance Manager25.7523.7534.3328.5727.09
Risk Managern/a24.52825.7525.91
Auditor3033.862928.7530.33
Other2526.4327.523.626.19
Total25.7327.3128.6727.3627.49

The strongest difference is the high score by auditors. Looking across the individual actions it seems that auditors score further from the model on most actions but dramatically so for the two major factors in the factor analysis linked to target setting, hiding risks, and exaggerating good and bad news. As someone who has been an external auditor for most of his career I find this hard to understand but perhaps people who have been on the receiving end of an audit report would have less difficulty.

It is also noticeable that people outside a corporate hierarchy score closer to the model, while the much maligned middle managers score furthest from the model (as well as being the group that spent least time on the survey on average).

[The time spent on the survey was measured by a timer within the web page, but obviously respondents working on the survey would be interrupted. Looking at the outliers it was decided to assume that if a respondent spent more than 400 seconds on one scenario then they had been interrupted. In these cases the recorded time was replaced by the average time across other respondents for that scenario.]

Respondent profile

The survey was open during late April and early May 2006.

Respondents were volunteers who heard about the survey through any of four Internet discussion lists:

Almost half the respondents were from the United States of America. The countries of respondents are shown on this graph:

Respondents were asked which most closely matched their background and role.

Respondents were asked about their role in their organization. "Sole" indicates someone who works alone or is a student or not employed, and therefore outside an organizational pecking order.

Scenarios and actions used in the survey

Here are the scenarios and actions used in the survey, verbatim, in the order they were presented.

Scenario 1

Imagine you are working in a large company and have been asked to lead a small marketing project to develop a concept for a new product to be used in commercial kitchens. Eight others are in the team you will lead during this initial work and you have called your first meeting for the team. Some of the people in it you know, and some you do not. You anticipate that the project will be tough to do, with internal resistance as well as competition from other companies to worry about. You want to get the project team off to a good start.

Scenario 2

Imagine you have recently started a business of your own, manufacturing wooden toys in a large barn in the country. Your sales force of three people have been selling to shops. To raise money you had to produce a business plan and the plan included estimates for sales of each of the six product lines you currently sell. After several months it is apparent that one of the product lines is doing much better than estimated, two are doing about as expected, and the remaining three are doing worse than expected. You are reviewing performance so far and have to decide what changes, if any, to make in support of each product line.

Scenario 3

Imagine you are a senior government official who has been put in charge of a high profile building project. So far the building is going according to schedule and the Minister (i.e. the senior politician involved) has taken almost no interest in the project apart from smiling press conference appearances. However, you have recently become aware that the company that is pre-fabricating sections of an elaborate glass roof have hit potentially serious technical difficulties with the design and are in dispute with the architects and construction managers. Angry letters are being exchanged between these three parties and you are being copied in as each positions itself as an innocent party. The true position is unclear. According to your plan there should still be enough time to complete the roof sections and install them on schedule. You have to decide what to tell the Minister, if anything, about the roof situation.

Scenario 4

Imagine you work in a large bank and have been designing a new form of savings account that you believe could make a big impact in the market and a lot of money for the bank - not to mention giving your career a huge boost. You and your team have developed the idea but you need senior approval to go further. Two days before the crucial meeting at which you must present your idea you discover that another, similar idea has been developed in another department and you see a copy of the presentation about it. You are horrified to see that if the other product is accepted yours will not be so this has become competitive. Also, their presentation makes bold claims about future sales and profits. You know these can be little more than guesswork but somehow you will have to compete with this exciting pitch. You consider your own presentation again.

Potential future research and developments

This survey was popular with respondents, who volunteered (and persisted through to the end of the survey) with no incentive other than their desire to help in research and an interest in their own self development. On top of that the survey results have suggested useful insights into how people think about the risk and uncertainty management issues involved and how they might be influenced in favourable directions (though 'favourable' remains a matter of choice).

Could surveys like this be used to assess the control environment of an organization? Perhaps. Certainly the scenarios would be more penetrating and convincing than asking senior executives if they think having accurate financial statements is a good idea. An enhanced version of the survey has already been developed with additional scenarios and improved usability.

Could the survey tool be used for educational purposes? I think so and a version of the survey that offers suggested answers and reasons for them is under development.

All these new developments need to be tested to find out how people respond and to study more varied groups of respondents. If you would like to comment or find out more about any of these future developments please contact me at matthew@internalcontrolsdesign.co.uk.

Related reading

"Open and honest about risk and uncertainty" discusses the problem of uncertainty suppression, its importance for risk management, and some potential strategies for tackling it.

"Designing intelligent internal control systems" covers more of the skills and knowledge that help achieve excellent management of risk and uncertainty.



© 2006 Matthew Leitch
New website, new perspective: www.WorkingInUncertainty.co.uk - Related articles - All articles - The author - Services

If you found any of these points relevant to you or your organisation please feel free to contact me to talk about them, pass links or extracts on to colleagues, or just let me know what you think. I can sometimes respond immediately, but usually respond within a few days. Contact details

Matthew Leitch - Author

About the author: Matthew Leitch is a tutor, researcher, author, and independent consultant who helps people to a better understanding and use of integral management of risk within core management activities, such as planning and design. He is also the author of the new website, www.WorkingInUncertainty.co.uk, and has written two breakthrough books. Intelligent internal control and risk management is a powerful and original approach including 60 controls that most organizations should use more. A pocket guide to risk mathematics: Key concepts every auditor should know is the first to provide a strong conceptual understanding of mathematics to auditors who are not mathematicians, without the need to wade through mathematical symbols. Matthew is a Chartered Accountant with a degree in psychology whose past career includes software development, marketing, auditing, accounting, and consulting. He spent 7 years as a controls specialist with PricewaterhouseCoopers, where he pioneered new methods for designing internal control systems for large scale business and financial processes, through projects for internationally known clients. Today he is well known as an expert in uncertainty and how to deal with it, and an increasingly sought after tutor (i.e. one-to-one teacher). more

Please share:            Share on Tumblr