Automated application controls and the benchmarking strategy

• Distinguish automated, IT-dependent manual and manual controls
• Design a test of an automated control using a test of one per scenario
• Apply the AS 2201 benchmarking conditions and risk factors
• Determine when to re-establish a baseline

Three kinds of control

• Automated application controls run inside the system without human action: a three-way match that blocks payment, an edit check that rejects a duplicate invoice number, a calculation of depreciation.
• IT-dependent manual controls are performed by a person using system output: a manager reviews an exception report.
• Manual controls do not depend on system output.

AS 2201.B28 notes that entirely automated application controls are generally not subject to breakdowns due to human failure. That is why testing differs.

Testing an automated control

Because an automated control performs the same way every time for the same inputs, you do not need a sample of 25 to show it works. Test each relevant scenario once, provided the ITGCs that protect it are effective.

Approach:

1. Identify the configuration or logic (for example, the match tolerance is 2 percent).
2. Inspect the configuration setting in the system.
3. Test each scenario: an invoice within tolerance passes, an invoice outside tolerance is blocked, a missing receipt blocks payment.
4. Confirm that users cannot override the block, or that overrides are themselves controlled.

Common methods are inspection of configuration, reperformance with test transactions in a test environment that mirrors production, or reperformance using real transactions selected from the period.

Dependency on ITGCs

The test of one shows the control worked at the moment of the test. Effective change management and access controls let you conclude it worked for the whole period. If ITGCs are deficient, you need another way to cover the period: test the control at multiple points, inspect change logs for that specific object, or rely more on substantive procedures.

Benchmarking

AS 2201.B28 through .B33 describe benchmarking for automated application controls in later years. Under AS 2201.B29, if general controls over program changes, access to programs and computer operations are effective and continue to be tested, and if the auditor verifies that the automated application control has not changed since the auditor established a baseline, the auditor may conclude the control continues to be effective without repeating the prior year's specific tests of its operation.

AS 2201.B30 adds that automated controls may depend on related files, tables, data and parameters, for example an interest calculation that depends on a rate table. Those dependencies also need to be unchanged or controlled.

AS 2201.B31 lists risk factors for deciding whether benchmarking fits, including:

• the extent to which the application control can be matched to a defined program within an application;
• the extent to which the application is stable, with few changes from period to period;
• the availability and reliability of a report of the compilation dates of the programs placed in production.

Heavily customized software with frequent releases is a poor fit.

Re-establishing the baseline

AS 2201.B33 says that after a period of time, the length of which depends on the circumstances, the baseline should be reestablished. The factors include the effectiveness of the IT control environment and the auditor's understanding of the nature of any changes to the programs that contain the control. Firms set a policy for the maximum interval. AS 2201.B32 notes benchmarking can be especially effective for purchased software when the possibility of program changes is remote, for example when the vendor does not allow access to the source code. Document the baseline date, the evidence that the control has not changed (for example, comparison of program compilation dates or configuration hash values), and the date of the next full test under firm policy.

Nonissuer audits

AU-C 330 has no benchmarking section, but AU-C 330.13 and .14 cover using evidence about controls from previous audits. The factors in AU-C 330.13 include whether the control is manual or automated and the effectiveness of general IT controls. Under AU-C 330.14, you establish continuing relevance through inquiry combined with observation or inspection. If the control changed, you test it in the current audit; if not, you test it at least once in every third audit and test some controls in each audit. If you rely on controls over a significant risk, you test them in the current period (AU-C 330.15).

Benchmarking the three-way match at Juniper Valley Hardware