A practical look at whether file checking is doing what advisers think it is doing.

Open a handful of completed files at most advice firms and you’ll find evidence the check happened: a sign-off, a tick, a reviewer’s initials. Some cases are checked before they go out, some are sampled afterwards, and some get extra review depending on the type of advice or who worked on the case.

What’s harder to find is evidence that the check would have caught anything. Knowing a file went through checking doesn’t tell you whether the checking works.

The issues that cause the most friction on files are rarely dramatic. More often it’s a figure that differs between two documents, an older version sitting alongside the current one, a missing piece of evidence, or a question that was answered somewhere but not recorded clearly enough for the next person to follow. Those things make a case harder to construct properly even when nothing looks wrong at first glance.

Tony explores this in the latest paper in our Advice, Evidence & Judgement series. Rather than measuring a control by whether it was completed, the paper suggests occasionally testing the control itself.

 

Plant something on purpose

Take a completed case that’s safe to use for testing, make a working copy and deliberately change one important thing. You might give it two conflicting versions of a client objective, strip out the evidence behind an assertion, leave an older risk assessment next to the current one, take out a sign-off that should be mandatory, or add an unexplained cost difference between two possible recommendations. Then put the case through the normal review process.

You’re testing the process, so it’s worth telling reviewers nobody is being marked. What you want to know is whether it reacts the way you assumed it would, which tells you whether the checking picks up the sort of issues that make technical planning harder. It also separates a process that adds value from one that just confirms a checklist was completed.

Not every difference should produce a failure. Sometimes the right response is a question, a request for more evidence or no challenge at all, and a reviewer who challenges everything isn’t doing the job well either.

 

Look at what happened after the issue was found

Recording “caught” or “missed” doesn’t tell you much about the quality of the process. Say a reviewer spots two different withdrawal figures. You’d want to know whether the problem was described correctly, whether the challenge was linked to the right document, and whether anyone resolved it. If the adviser explained why one figure was right, did that explanation stay visible in the file? And if the system raised something that wasn’t a problem, that deserves a note too, because a control that flags everything creates more work without necessarily creating better advice.

 

Keep it small

You don’t need a big testing programme. Start with three past cases representing different advice situations, introduce one known issue into each and put them through the normal route. On a retirement income case, that might be a withdrawal figure in the report that no longer matches the cashflow model. On an investment switch, it could be last year’s risk questionnaire left in the file next to the current one with a different score. On a tax-efficient investment case, it could be a mandatory sign-off taken out. Each one is a quick edit to a working copy of a file you already have.

Ideally one of the three contains something that looks unusual but shouldn’t result in a challenge, such as a portfolio carrying more risk than the client’s attitude to risk would suggest, where the file clearly evidences their capacity for loss and their reasons.

That’s a manageable exercise for most firms, and it gives you a better question than “how many files did we review?”: what have we learnt about what our process can actually detect? This gets more pressing as firms bring technology into review and quality assurance, because a dashboard can show every case was processed without showing whether it would spot the missing evidence or conflicting objective you assumed it was looking for. The FCA makes a similar point in its work on outcomes monitoring: collecting data and producing reports isn’t enough on its own, and firms need information that helps them identify poor outcomes, understand the causes and act on them.

 

One question to take back to the team

Next time file checking comes up, try asking:

If we deliberately put something wrong into a test case, how confident are we that our normal process would find it?

Tony’s full paper, Would your file checks catch a mistake you planted?, includes practical examples and a simple first exercise firms can run.