Testing design and operating effectiveness of controls

• Distinguish a test of design effectiveness from a test of operating effectiveness
• Scale the nature, timing and extent of testing to the risk of the control
• Plan interim testing and a roll-forward to the as-of date
• Apply a firm sample size table and explain why it is policy, not a rule

Design effectiveness

AS 2201.42 says the auditor tests design effectiveness by determining whether the controls, if operated as prescribed by people with the necessary authority and competence, satisfy the control objectives and can prevent or detect errors or fraud that could result in material misstatement. Procedures include inquiry, observation and inspection, and a walkthrough that includes them ordinarily is sufficient to evaluate design.

Design questions to answer:

• Does the control address the specific risk and assertion in the RCM?
• Is it precise enough: would it catch a misstatement that could be material?
• Is the performer competent and independent of the activity checked?
• Does it rely on reports or system functions, and are those reliable?

Operating effectiveness

AS 2201.44 says the auditor tests operating effectiveness by determining whether the control operates as designed and whether the person performing it has the necessary authority and competence. AS 2201.45 lists the procedures: a mix of inquiry, observation, inspection of documentation and reperformance. AS 2201.50 ranks them in that order, from least to most evidence, and says inquiry alone does not support a conclusion that a control is effective, so pair it with inspection or reperformance. Reperformance gives the strongest evidence for review controls: redo the reconciliation, recompute the accrual, or rerun the analytic with the reviewer’s thresholds.

Risk drives the evidence

AS 2201.46 links evidence to the risk associated with the control. AS 2201.47 lists factors that raise that risk, including:

• the nature and materiality of misstatements the control addresses;
• inherent risk of the related accounts and assertions;
• changes in volume or nature of transactions;
• a history of errors in the account;
• the effectiveness of entity-level controls, especially those that monitor other controls;
• the nature of the control and how often it operates;
• the degree to which the control relies on other controls, such as ITGCs;
• the competence of the performer and changes in key personnel;
• whether the control is manual or automated;
• the complexity of the control and the judgment involved.

Release 33-8810 applies the same logic to management through ICFR risk.

Timing and roll-forward

AS 2201.52: testing over a longer period gives more evidence than a shorter period, and testing closer to the as-of date gives more than earlier testing. The auditor balances the two. AS 2201.55 requires the auditor, when testing at an interim date, to determine what additional evidence is needed for the remaining period. AS 2201.56 lists the factors: the specific control and its risk, the evidence obtained at interim, the length of the remaining period and any significant changes in ICFR after the interim date. When those factors show low risk, inquiry alone might be sufficient; for higher-risk controls, test a second sample covering the last months.

AS 2201.53 covers redesigned controls: if new controls achieve the objectives and have operated long enough to test, the superseded controls need not be tested for ICFR purposes.

AS 2201.B2 requires evidence that controls operated effectively for a sufficient period, which may be less than the full year.

Sample sizes are firm policy

Neither AS 2201 nor Release 33-8810 sets sample sizes. Each firm and each company sets them in its methodology, based on frequency and risk. An illustrative policy:

Use your own methodology’s table. Whatever it says, document the population, how the sample was selected, and how deviations are handled. If a deviation appears, the methodology usually allows a sample expansion only when the deviation is isolated and explainable; otherwise the control is ineffective.

Automated controls and ITGCs

In practice, an automated control is tested by inspecting the configuration and testing one instance of each relevant condition, provided ITGCs over access and change are effective. AS 2201.B28 to B31 permit a benchmarking strategy in later years when ITGCs are effective and the control has not changed (see the ITGC overview).

Year-to-year variation

AS 2201.61 requires the auditor to vary the nature, timing and extent of testing from year to year to introduce unpredictability. Internal audit should do the same for management testing.

Testing the monthly accrual review at Fairmount Hospitality Group