|New website, new perspective: www.WorkingInUncertainty.co.uk - Related articles - All articles - The author - Services|
by Matthew Leitch, 25 August 2004
First, thank you to everyone who responded to this survey. It was a long and tough one that needed thought. The payoff is that we have some interesting results with profound implications for many, especially people in internal audit roles.
The results of this research strongly suggest that auditors, risk managers, and others who make recommendations for improving internal control and risk management can benefit from giving more attention to recommendations beyond the usual repertoire of sign offs, documentation, segregation of duties, and reconciliations. This implies that there could be great value in changes to audit approach and audit training and education.
In this online internet survey respondents were asked to consider eight hypothetical reviews of business activities and consider for each review a list of five potential recommendations for improvement. Respondents were asked:
if, in their experience of organisations, the recommendation would probably already be in place;
if the recommendation was probably a good one, assuming it wasn't already in place; and
if they thought they were probably expected to make such recommendations. (If the respondent did not make recommendations in their job they were asked to consider if they would expect such recommendations to be made.)
This combination of questions revealed that certain types of recommendation were as likely to be good recommendations as others but were much less likely to be in place already. They were also less likely to be "expected."
Making these types of recommendation requires greater knowledge of risk and uncertainty but someone who has that knowledge would be able to use it often because the controls in question are rarely in place.
The survey showed no differences between the public and private sectors, except that public sector respondents thought a much wider range of recommendations was expected of them. This finding contrasts with opinions expressed to me by some, who speculated that the public sector was not interested in risk taking or risk management. In fact, the public sector respondents felt they were expected to make more wide reaching and sophisticated recommendations than did the private sector respondents.
The survey form presented respondents with eight imaginary reviews (i.e. the scenarios) in the order shown below:
|prod dev||The review looked at the way product ideas are developed and approved in a particular business unit.|
|confer||The review looked at the way potential conferences were chosen and, in particular, how estimates of likely attendances were made. These are vital to the decision of whether to go ahead or not.|
|backbill||The review looked at a project that is trying to identify past billing errors and, where possible, raise back charges with customers who have been under-charged.|
|project||The review looked at a project plan being developed for a large project that is vital to the future of the organisation and expected to last over 2 years.|
|service||The review looked at plans to improve services to customers by introducing several innovations.|
|prod mgmt||The review looked at how a set of products have been managed.|
|oil||The review looked at a business case for exploring a region for oil. The case includes extensive financial projections.|
|overall||The review looked at the management of risk and uncertainty throughout the organisation.|
Each scenario was followed by five potential recommendations for improvement, displayed in a random order that differed for each respondent.
Respondents were asked three questions about each recommendation using the following instructions:
"Imagine that internal auditors or other risk management or internal control specialists have been doing some reviews of activities in an imaginary organisation. Each review found problems and some recommendations are under consideration."
"For each recommendation consider the following:"
"Probably already in place": In your experience of organisations would you expect the recommended action to have been taken already? Click the first checkbox on each line if you think that most organisations would not need the recommendation because they would already have done what is being recommended.
"Probably a good recommendation": Click the second checkbox if you think the idea would probably be a good one, assuming the action has not already been taken. Don't get picky about the details of the recommendation or its wording. The basic idea being suggested is what matters. Assume there are appropriate details that go along with this simple summary.
"Probably expected": For the third checkbox it depends if you make recommendations on risk management / internal control in your job.
If you do make recommendations about risk management/internal control in your job click the checkbox if you think you are expected to suggest ideas like this.
If you do not make recommendations on these topics in your job click the box if you would expect auditors and other risk management specialists to suggest the idea.
"All the recommendations should be taken individually. They are not intended to be linked. Also don't worry about the fact that the imaginary organisation seems to do a lot of unrelated things! The examples are drawn from life but not in the same organisation."
In all there were 40 recommendations across 8 scenarios for respondents to consider. The recommendations were designed to be realistic and were not generated according to any particular scheme. However, for the purposes of analysis and summary they have been put into groups.
The most interesting recommendations are those that (1) are good ones, and (2) are least likely to be in place already. A simple indicator of how interesting each recommendation is can be calculated as the number of respondents who thought the recommendation probably a good one (assuming it was not in place already) minus the number of respondents who thought it would probably be in place already. On this basis the most recommendations, in descending order of interest, were as shown in the table below. (See below for an explanation of the "Concept group" column.)
|Rank||Recommendation||Good - In place||Good||In place||Expected||Scenario||Concept group|
|2||Include in each iteration of the product development process a step where the current areas of uncertainty/risk relevant to the idea under development are quickly listed and their impact is considered before actions to reduce the uncertainties or manage the risks are decided.||38||45||7||19||prod dev||Uncertainty|
|19||The project management team should consider some form of training to develop their ability to talk openly about risk and uncertainty on the project and encourage others to report progress and risks honestly and completely.||37||44||7||24||project||Educate|
|4||Provide education/training for product developers on how risk and uncertainty affect their work and how identifying uncertainties that matter can guide their research efficiently.||36||43||7||26||prod dev||Educate|
|38||Risk management procedures should be revised to encourage people to revisit risks and responses much more often, to stay up to date, and to focus on things that are more specific and topical.||36||45||9||25||overall||Uncertainty|
|39||More effort should be made to incorporate risk/uncertainty awareness into strategic decision making and not just routine clerical procedures.||35||46||11||30||overall||Uncertainty|
|37||The way risk and uncertainty are quantified should be improved so that more numerical modelling and empirical support are used where appropriate.||34||38||4||23||overall||Quantify|
|32||The computer model underlying the financial projections should be independently reviewed to ensure that it is correctly programmed.||33||44||11||25||oil||Sign.|
|35||The risk management approach should also address the personal risk/uncertainty awareness, skills, and attitudes of staff, particularly managers at all levels.||33||41||8||26||overall||Educate|
|36||Documentary evidence of internal controls / risk management should be enhanced so that any failure to carry out agreed controls is highlighted promptly.||33||44||11||31||overall||Doc.|
|14||Projections about how much money the back-billing project will eventual raise should be expressed as ranges with probabilities rather than as a spuriously accurate 'best guess' number. For example, say the range of recoveries that is now 80% probable.||32||40||8||20||backbill||Quantify|
|0||Write a policy on risk management for product development in the business unit.||31||40||9||31||prod dev||Doc.|
|11||Predictions about how much money the back-billing project will eventually raise should be reviewed independently before being used in revenue forecasts.||30||39||9||22||backbill||Sign.|
|20||Risks to improving service should be identified, documented, and assigned owners.||30||43||13||28||service||Risk mgmt|
|7||The spreadsheet model estimating the financial results of a proposed conference should treat attendance as an uncertain variable with a probability distribution, and show the projected financial result as a distribution. From that it would be possible to see the estimated risk of, for example, making a loss. The number crunching can be done easily using widely available Excel add-ins.||29||42||13||20||confer||Quantify|
|9||The conferences team should take time out to consider the range of outcomes from past conferences, how predictable they really are, and to think of ways they can manage conferences more flexibly and gain more information about visitors that will help in selecting conference topics, venues, and dates.||28||46||18||27||confer||Learning|
|33||The source of all evidence used in making estimates should be stated, even if it is just to point out the name of the person whose gut feel it is.||28||40||12||23||oil||Uncertainty|
|18||As far as possible without creating inefficiency the project should be divided into short term deliveries to stakeholders, not just internal deliveries within the project. This would accelerate business benefits, reduce committed resource before benefit delivery, and increase learning from experience.||27||43||16||22||project||Structure|
|21||More short term indicators of progress should be sought, as the existing indicators are too long term to be used alone.||27||41||14||24||service||Learning|
|22||Since the results from the new ideas are not certain the ideas should be trialled rapidly and revised as necessary as they are rolled out more widely. It is very important to learn as much as possible from experience.||27||41||14||23||service||Learning|
|13||A small set of back charges should be taken through to bills and attempted recovery of money from customers as soon as possible to learn more about what it will take to do this on a larger scale.||26||38||12||23||backbill||Learning|
|16||The project steering committee and project management team should set a good example by being open about uncertainties, communicating them, and showing that they expect others to be open with them.||26||45||19||25||project||Uncertainty|
|17||The project plan should be reviewed to see if the dependencies can be reduced to improve the risk profile of the project.||26||45||19||24||project||Structure|
|31||There are various uncertain variables in the projection and these should be modelled using probability distributions to explicitly represent the uncertainty and avoid the flaw of averages.||26||41||15||23||oil||Quantify|
|30||The financial model needs to reflect the fact that decisions about whether to proceed further and how will be taken at various points in the proposed exploration. These options should be valued.||25||44||19||24||oil||Quantify|
|3||Formal proposals for new products to be approved should include a section listing risks and how, if at all, they can be managed.||24||45||21||27||prod dev||Risk mgmt|
|23||Priorities should be revised regularly - probably more often while ideas are still relatively untried.||22||38||16||19||service||Adaptability|
|5||Evidence relating to likely attendance, revenues, and costs for proposed conferences should be documented in a written business case.||18||44||26||22||confer||Doc.|
|29||Changes to resource allocations between products should be authorised appropriately and in writing.||18||43||25||26||prod mgmt||Sign.|
|8||More should be found out about interest in potential conferences, for example by using surveys and looking at the readership of related magazines and journals.||17||42||25||22||confer||Learning|
|15||A risk management process should be put in place to identify significant risks to the project, plan responses, and track progress.||16||45||29||32||project||Risk mgmt|
|28||Alternative promotional strategies should be tried to find out which work best in each category.||16||38||22||19||prod mgmt||Learning|
|6||Approval to proceed with a conference should be given by a committee of suitable managers and their approval should be evidenced in writing, for example in the minutes of the committee's meetings.||15||42||27||27||confer||Sign.|
|12||The project should have a formally agreed scope and definition document, and a project plan.||14||41||27||27||backbill||Doc.|
|26||The products should be managed as a portfolio during the year, with new products that go well being given extra resources to develop, while disappointing products get less.||14||39||25||19||prod mgmt||Adaptability|
|10||Where it is discovered that past bills to a customer have been incomplete but the back charges are to be waived this should be authorised appropriately and documented.||13||44||31||29||backbill||Sign.|
|34||A formal risk assessment exercise should be carried out.||13||45||32||28||oil||Risk mgmt|
|1||Ensure that product development approvals are given in writing and signed off by suitable officials in the organisation.||11||44||33||29||prod dev||Sign.|
|24||The service improvement plan should be authorised in writing at a high level.||11||42||31||24||service||Sign.|
|25||Clear revenue, growth, and profit targets should be agreed for each product annually.||7||41||34||24||prod mgmt||Targets|
|27||Products should be managed tightly to ensure that each product meets its annual targets.||0||31||31||22||prod mgmt||Targets|
Here is a scatter graph showing how the concept groups are distributed:
The most interesting types of recommendation are towards the bottom right of the graph.
The recommendations in the Targets group were brought down by the context used. The two recommendations on using targets were the least popular of all, though still thought to be good by most respondents. The recommendation is to set targets for individual products and ensure that each product meets its target.
These were the only items in the survey I considered bad recommendations and it is comforting to know that one marketing specialist who responded to the survey (and identified himself to me) agreed.
In other contexts the idea of setting fixed targets and managing towards them would have got higher approval, but in the product portfolio case more people are aware of the problems.
The choice of this context for the Targets recommendations probably brought down the Targets group more than most would have done.
In the table above the concept groups have been classified as "New" or "Old" according to my perception of which ones are traditional audit mainstays, based on over 10 years of audit experience. The final summary is of the figures for new and old recommendations:
Public and private sector responses did not differ noticeably in any way except one. The proportion of recommendations public sector respondents felt they were expected to make was dramatically higher than for the private sector respondents, and this difference was greatest for recommendations in the "new" concept groups.
Only 16 of the 46 respondents were from the public sector, so to give an idea of how statistically significant the difference is I have calculated that if there was in fact no difference between the public and private sectors the probability of getting the result I got or something more extreme is only 12.4%, using a 1 tailed T test. In other words, it is very likely that this is a real difference, despite the sample size.
The results seemed pretty much the same regardless of whether respondents were auditors, risk managers, performance managers or held some other role. This was a bit of a surprise as I had thought risk managers would be expected to make more varied recommendations than auditors.
Different scenarios produced different responses, but I could see no meaning in the pattern. The low figure for "In place" for the overall review is perhaps because respondents may have had difficulty seeing a specific control within the recommendation.
The survey asked if the respondent had any other comments they would like to make about the survey or about recommendations. Excluding comments purely about the survey, the respondent comments were:
"Wow, these questions made my head hurt. Any time you wandered into recommendations regarding modelling and statistics, I backed off. I haven't thought about those matters since college Econometric courses some 20 years ago.
As internal auditors are primarily accountants first, and operational observers secondly, we would not be expected by my current organization to make comments on statistical models or product marketing recommendations. Because of the detailed nature of these comments, I'm much more comfortable suggesting a topic for strategic direction than a specific management model. The point is to get management to clarify their aims and directions. Internal audit attempts to help them formalize this and then audit to their stated objectives. We are not subject matter experts. But we can comment on the effectiveness of processes based on outcomes."
"There is a lot of work to be done, especially in the government sector in incorporating risk management (information/operations risk). I would like to see more concrete work on how to integrate or rather align specific business goals/missions with the IT part of the business. To date, it still seems to me there is a big gap. The push from C-level executives to figure out ROI/ROSI (Return on Investment/Return on Security Investment)are still rather "fuzzy" numbers. How do we really get from here to a fully integrated risk management organization where risk is just business as usual and as much as possible, fully automated and dynamic?"
"The "probably expected" box I interpreted to mean "probably expected within my current role". Working on a Sarbox project, I am planning to raise business issues but am not generally expected to do so. Some of the survey's suggested recommendations are also somewhat outside management's own expectations of a process or touching on areas that management are not expecting me to review - e.g. strategic marketing decisions and are not included on those grounds."
"Risk management itself is a risk process - the degree of risk evaluation/ management depends on many factors, and the answers to the questions above, of necessity, cannot assess all the factors that would apply, especially between e.g. SMEs and multinationals."
"Quantification doesn't help if there is no suitable information to quantify - which is often the case."
"The oil case encourages the use of real options methodologies. However, virtually all the cases would benefit from this perspective."
"Much of what's 'recommended' above is (or should be) standard public sector practice, given the push to formal project management (It's all in PRINCE). Risk management (Per HM Treasury) and OGC Gateway reviews."
Respondents were invited to participate using professional discussion lists on the Internet and by some personal e-mails, but only where I was confident the person would not be predisposed to answer in a particular way. For example, if someone wrote to me about how they were interested in evolutionary project management I would not invite them to participate.
Most respondents were from the USA and UK. The countries of respondents are shown on this graph:
Most were internal auditors and risk managers. The roles of respondents are shown on this graphic:
Besides the obvious limitations of a study where respondents represent a tiny section of a large population and are self-selected, and where the number of respondents is fairly small, there are some other imperfections in the design that should be borne in mind.
The recommendations were nearly all intended to be good ones, so that some respondents may have been led into a pattern of answering "Good" to all of them. If there had been more bad recommendations to consider then the proportion of Good responses to the good recommendations might have been lower.
Some respondents had technical difficulties with their browser's rendition of the survey and one reported that he had been unable to amend answers to some questions. This was not part of the design. The survey worked perfectly when tested on Internet Explorer.
Although the evidence suggests that the new types of recommendation are likely to be very useful to auditors and others who make recommendations on controls there is a missing step in the chain of reasoning. The survey does not show how often in practice the newer types of control are appropriate. It only shows that there are good recommendations for controls that are unlikely to be in place already.
Finally, the simple binary responses asked for do not tell us how confident respondents were that particular controls would already be in place or how good they thought the recommendations were. We only know what proportion of respondents thought the controls would probably be in place and what proportion of respondents thought the recommendations were probably good. There is a difference, though one would expect strength of feeling and volume of support to be correlated to some extent.
A good explanation of the rationale for this study is given in one of my articles for IRMI. The article is "Embedded risk management: the auditor's role".
If you would like to analyse the original data yourself I can provide a matrix of the ratings given. The information will not allow you to identify respondents or their organisations.
Copies of the original survey are also available. Please contact me at firstname.lastname@example.org.
|New website, new perspective: www.WorkingInUncertainty.co.uk - Related articles - All articles - The author - Services|
|If you found any of these points relevant to you or your organisation please feel free to contact me to talk about them, pass links or extracts on to colleagues, or just let me know what you think. I can sometimes respond immediately, but usually respond within a few days. Contact details|
About the author: Matthew Leitch is a tutor, researcher, author, and independent consultant who helps people to a better understanding and use of integral management of risk within core management activities, such as planning and design. He is also the author of the new website, www.WorkingInUncertainty.co.uk, and has written two breakthrough books. Intelligent internal control and risk management is a powerful and original approach including 60 controls that most organizations should use more. A pocket guide to risk mathematics: Key concepts every auditor should know is the first to provide a strong conceptual understanding of mathematics to auditors who are not mathematicians, without the need to wade through mathematical symbols. Matthew is a Chartered Accountant with a degree in psychology whose past career includes software development, marketing, auditing, accounting, and consulting. He spent 7 years as a controls specialist with PricewaterhouseCoopers, where he pioneered new methods for designing internal control systems for large scale business and financial processes, through projects for internationally known clients. Today he is well known as an expert in uncertainty and how to deal with it, and an increasingly sought after tutor (i.e. one-to-one teacher). more
Please share: Tweet