1.5.4 Section 1204/Regulation 801 Guidance for Appeals
1.5.4.1 (10-01-2000)
Section Overview
- Section 1201 of the IRS Restructuring and Reform Act of 1988 (RRA) mandated that IRS establish a performance management system that establishes goals or objectives for individual, group, or organizational performance. Section 1204 of the RRA addresses the use of tax enforcement results.
- Section 1204 codified Policy Statement P-1-20, and replaced a narrow prohibition on the use of enforcement results that impacted personnel directly involved in collection activity, and their immediate supervisors. The new provision provides a prohibition against using such “records of tax enforcement results” (ROTERs) to evaluate or to impose or suggest production quotas or goals for an IRS “employee.”
- The Commissioner adopted a regulation that establishes an overall performance measurement system and sets forth the rules governing the use of ROTERs. The regulation is found at 26 CFR Part 801 (the regulation).
- This section describes the use of the Balanced Measurement System in Appeals. Also see sections 1 and 2 in IRM 1.5, Managing Statistics in a Balanced Measurement System.
- The three strategic goals of the Service are:
- Service to each taxpayer,
- Service to all taxpayers, and
- Productivity through a quality work environment.
- The Balanced Measurement System focuses on what is important to achieve our strategic goals. The aim is to improve those processes that will make a difference. Dollar measures of performance are not part of Balanced Measures and there will be no ranking of offices. The new organizational review process will be more forward-looking and focuses on actions taken to improve performance; not just measure numeric results.
- Measures fall into three categories:
- Customer Satisfaction
- Employee Satisfaction
- Business Results
- The Business Results measure will consist of two elements: Quantity and Quality.
- Each of the measures of the Balanced Measurement System in (2) above is of equal importance in carrying out the Service’s programs and functions. Managers are expected to make use of data from each category to improve the performance of their organizational unit.
- A “tax enforcement result” is: “the outcome produced by an IRS employee’s exercise of judgement in recommending or determining whether or how the Internal Revenue Service should pursue enforcement of the tax laws.” See IRM 1.5.2.
- The exercise of this judgement is referred to as “Section 1204 judgement.”
- An example of an Appeals tax enforcement result is “the amount assessed.”
- Examples of data that are not tax enforcement results include, but are not limited to:
- Cycle time
- Case closures
- Time per case
- AO direct time
- Number or percentage of overage cases
- Inventory information
- Data derived from a quality review of a case
- An IRS employee who exercises “Section 1204 judgement” is considered a “Section 1204 employee.” Whether an employee is covered by Section 1204 for this purpose is determined by the task performed and not the title, position description or location of the employee. In Appeals, covered employees may include, but are not limited to:
- Appeals Officers
- Settlement Officers
- Technical Analysts
- Art and Real Estate Appraisers
- Training Analysts
- First line managers of all covered employees.
- The manager of any manager who exercises Section 1204 judgement.
- National Office and Operating Unit analysts and managers who are in a position to influence or direct field activities involving Section 1204 judgement.
- If a manager determines that an employee who does not usually exercise Section 1204 judgement did so in a particular quarter, that manager should follow the self-certification process described in IRM 1.5.3 for the quarter.
- Records of Tax Enforcement Results (ROTERs) may not be used to evaluate employees or to impose or suggest production quotas or goals. These prohibitions are reflected in Section 1204 and regulation 801.
- A ROTER is defined as “data, compilations of information or other numerical or quantitative recordations of the tax enforcement results reached in one or more cases.” See IRM 1.5.2.9.
- Note that a single case can be a ROTER. This could occur if a manager referred to the outcome in a single case as the standard for similar cases.
- ROTERs do not include the tax enforcement results of individual cases that are reviewed in determining the quality of an employee’s work on that case. See Reg.801.6(d)(iii).
- Examples of Appeals ROTERs are:
- Agreement rate
- Recovery rate
- Sustention rate
- No change rate
- The Balanced Measures categories of Customer Satisfaction, Employee Satisfaction, and Business Results may each have one or more measures. Those measures may be used to set goals for future performance. Therefore, since ROTERs may not be used to set or suggest goals for improvement, they may not be used as balanced measures.
- Diagnostic Tools are statistics and other information, which, when used properly, are an effective and appropriate management tool. Diagnostic tools may not be used as measures, but can be used to help explain the outcome of the process; i.e., to help analyze the measure. Improvement goals or targets may be established for diagnostic tools, but only in direct support of the overlying balanced measure. Such goals or targets will not be used in individual performance evaluations. See IRM 1.5.1.9.
- Although ROTERs may not be used as balanced measures, they may be used as diagnostic tools. No goals or targets can be set for ROTERs.
- A Reference Guide for Diagnostic Tools has been developed for Appeals and is included in this section. See Exhibit 1.5.4–5.
- While goals may be set for balanced measures, the Regulation stipulates special rules for the use of Business Results-Quantity measures.
- Regulation 801.6(c) defines Quantity measures as consisting of outcome-neutral production and resource data, such as:
- the number of cases closed,
- work items completed,
- customer education,
- assistance and outreach efforts undertaken,
- hours expended and similar inventory,
- workload and staffing information.
that does not contain information regarding the tax enforcement result reached in any case involving particular taxpayers.
- Quantity measures will not be used to impose or suggest production goals for any organizational unit with employees who are responsible for exercising judgment with respect to tax enforcement results except in conjunction with goals based also upon the other measures (Customer Satisfaction, Employee Satisfaction. and Business Results-Quality). See Reg. 801.2(b).
- For example, a goal to improve quantity by 10% in an Appeal Area may not be set or suggested unless goals are set in consideration of and in conjunction with the other balanced measures (Customer Satisfaction, Employee Satisfaction, and Business Results-Quality).
- An example of an Appeals Quantity measure is “total disposals.”
- Goals for organizational performance measures will be set under the Balanced Measures framework in the areas of Customer Satisfaction, Employee Satisfaction and Business Results. See IRM 1.5.
- There are two (2) types of goals an organization can use to communicate priorities and guide performance – qualitative and quantitative.
- Qualitative Goals:
- Qualitative goals are general in nature and suggest a desired direction but do not establish a numeric target, e.g., “Improve Customer Satisfaction.”
- Qualitative goals can be established down to the team level for use in Action Plans established during the formal plan development process.
- At this time, Appeals is not developing Action Plans at the team level.
- Therefore, qualitative goals could be established at the Appeals Area level and above.
- Quantitative Goals:
- Quantitative goals (numeric goals) are specific and do establish a specific numeric target, e.g., “Improve Customer Satisfaction from 70% to 80%.”
- Quantitative and numeric goals will be set for the balanced measures at the Servicewide area level (Appeals National level) to satisfy requirements of the Government Performance and Results Act and other legislation.
- Numerical goals will be set only to the levels for which statistically valid results are available for all components of the Balanced Measurement System - currently to the Appeals Area level. The numeric goals set at the levels below National Office will be used primarily for planning purposes.
- Quantitative goals will not be used to directly determine the evaluations of either organizational units or individuals.
- Appeals National numeric and qualitative goals will be included in organizational documents such as the budget submission, the annual performance plan, and the Business plan that will be distributed both within and outside the organization.
- The qualitative goals can be shared and discussed at all levels of the organization with both managerial and non-managerial employees.
- The quantitative (numeric) goals should only be shared and discussed with managerial and non-managerial employees when there is a legitimate business purpose for sharing such data, such as the organizational planning process. See IRM 1.5.1.11.3.
- When numeric organizational goals are shared, caution must be exercised to ensure that any such discussion does not imply or suggest numeric goals for an individual or organization.
- Lower level organization (Operating Units and Appeals Areas) numeric and qualitative goals will be included in annual business plans and action plans.
- The qualitative goals can be shared at all levels of the organization.
- The Quantitative (numeric) goals should only be shared and discussed with managerial employees at the levels for which there is a legitimate business purpose for sharing such data, such as at the team manager level for planning purposes.
- When numeric goals are shared at these levels, caution must be exercised in how the numeric goals are shared in order to avoid numbers focused pressures and not to imply or suggest numeric goals.
- Caution must also be exercised to ensure that any discussion of numeric goals does not imply or suggest numeric goals for an individual.
- Under the Balanced Measurement System, an organizational unit may share balanced measures and other non-ROTER data with other units at the same level.
- The performance of any one unit at any level of the organization should not be used as a standard by which the performance of any other unit would be evaluated.
- The appropriate purpose for sharing balanced measures results and diagnostic tool data among organizational units is to:
- conduct analysis
- identify potential areas for improvement
- explore best practices
- Organizational Unit:
- All components of the Balanced Measurement System must be considered and addressed.
- The numeric results achieved with any of the balanced measures can be communicated orally and in writing in an organizational review only to the level for which numeric goals were established for all components of the Balanced Measurement System, currently the Appeals Area level.
- The inclusion of numeric results is only to provide a point of reference for a more detailed discussion of the impact of the actions that were taken to help achieve the IRS’ mission and strategic goals as translated through the balanced measures.
- Individual - Manager:
- The balanced measures are organizational measures and the numeric results achieved may not be used to directly evaluate any employee or manager.
- A managerial evaluation must focus on the actions that were taken to improve performance in each area of the balanced measures.
- Numeric results may be used indirectly as a resource to help assess the impact of the actions on the balanced measures.
- Individual - Non-supervisory:
- Non-supervisory employees may not be evaluated on any measures, either directly or indirectly.
- Non-supervisory employees will continue to be evaluated on the basis of elements and standards.
- Performance criteria for employees must be developed to support the new measures.
- Performance criteria must include a method of evaluation that addresses how well the employee provided fair and equitable treatment to taxpayers. See Reg. 801.3(b)
- Current measures are:
- Customer Satisfaction
- Employee Satisfaction
- Business Results (Quality - AQMS, and Cycle Time: Quantity - Disposals)
- Using balanced measures, or “getting behind the numbers,” is a seven step problem solving process. This process is outlined in section 1 of this IRM. Throughout this process, keep in mind that numbers are only a starting point and that, through analysis, implementation of a revised course of action is the objective. See IRM 1.1.12.
- Appeals is required to comply with the review and reporting procedures for the quarterly certification required by RRA Section 1204 and the independent review process. section 3 of this IRM provides guidance and instruction on these processes. See IRM 1.5.3.
The following analysis refers to “steps” . These references relate to the Section 1204 Decision Table in section 2, Exhibit 1.5.2-2, following the regulation.
| Q1. Will ROTERs continue to be used? |
| Yes, ROTERs will be used, but are restricted to diagnostic tools. They will not be used as balanced measures. ROTERs may not be used to evaluate employees or to impose or suggest production quotas or goals at any level. |
| Q2. Can any organizational unit receive its own results? |
| Yes, an organizational unit may have its results for the current year and for prior years. This applies to both balanced measures and diagnostic tools.
The purpose for sharing these results is to assist managers in the planning process, determine the impact actions are having on performance, and establish revised plans based on what is learned from analysis of the numbers. |
| Q3. Can any organizational unit receive its own ROTER results? |
| Yes, an organizational unit may receive performance results for any ROTER if the ROTER results are used as a diagnostic tool and the unit has a business purpose for using the tool. The unit may receive its ROTER performance for the current year and for prior years. |
| Q4. Can organizational units receive comparative data on other organizational units? |
| The answer to this question is YES under some circumstances and NO under others. There are restrictions in both the law and in Service policy on sharing comparative data. The law prohibits sharing comparative ROTER information where the sharing would set or suggest goals. Non-ROTER data may be shared. (See IRM 1.5.1.11.3.) |
| Prohibitions against sharing comparative data do not apply to data that is shared for process analysis purposes. Therefore, process analysis teams, at any level, may receive comparative data for non-ROTERs and ROTERs for any other organizational unit, at any level, if there is a process analysis purpose for receiving the information. |
| Q5. Is Appeals using Process Analysis to set numeric goals in FY 2000? |
| No, in FY 2000 Appeals is not establishing numeric goals. We will use Process Analysis as a tool to plan and direct actions, not to establish goals. |
| Q6. May a Regional Director of Appeals (RDA) express his or her concern to an Appeals Office Chief about a particular statistic? |
| Yes. The RDA may ask about one of the Balanced Measures. If the measure is a quantity measure, then there must be similar measures for all of the other Balanced Measure categories. The discussion that must follow, however, needs to focus on the specific actions, the extent to which it seems they are working and whether there is a need to revise or recommend additional actions. |
| If the RDA’s inquiry is about a diagnostic tool, the purpose of the discussion should be to analyze, understand and plan actions related to that measure. Any conversations that set or implied a goal for a ROTER would be a violation of Section 1204. |
| Q7. May an office chief tell his or her subordinate managers what the office’s goal is for a quantity measure? |
| Yes. However, there must be a legitimate business purpose for sharing such goals, e.g., for planning purposes. Furthermore, caution must be exercised in how to communicate these goals. |
| The communication:
• must not suggest group or employee goals • must cover all of the balanced measures components, and • must focus on actions |
| Q8a. May any non-supervisory employees be told about goals for the group? |
| Yes, if there is a legitimate business purpose for sharing such goals, e.g., for planning purposes. At the group level, goals will center around action plans and related accomplishments. At this time, Appeals is not developing action plans at the group level. |
| Q8b. May any non-supervisory employees be told what the office’s goal is for a quantity measure? |
| Yes, if there is a legitimate business purpose for sharing such goals, e.g., for planning purposes. |
| Furthermore, caution must be exercised in how to communicate these goals. The communication:
• must not suggest group or organizational goals • must cover all of the balanced measures components, and • must focus on actions |
| Q9. Can an associate chief compile results of individual case reviews in preparing for an employee’s evaluation? |
| Yes, individual case reviews are a quality process. Summarizing or compiling results of individual quality reviews is not prohibited under Section 1204. A manager may focus on any aspect of a case review, consistent with the employee’s elements and standards. |
| The manager may also comment, orally and in writing, on such compilations of work from the evaluation period. For example, the manager may note that 12 cases were reviewed during the year and that 7 of them had reasonable time applied, while 5 of them were evaluated as having too much time applied. |
| Q10. Can a manager tell an employee that the employee spent too much time on a particular case? |
| Yes, the manager may comment on the length of time it took to work a case and/or the hours applied to it. The manager must comment with respect to the difficulty of the case and problems encountered in it. The manager may not comment with regard to some average or standard, as that may set or suggest a goal. |
| Measures Matrix |
| The following matrix was developed to assist in determining what data can be made available to various levels in the organization. It is useful in applying the criteria of IRM 1.5.2.15, IRM 1.5.2.18, IRM 1.5.2.18.1 and section 1 to decide whether making a statistic available is likely to suggest a quota or goal (step 2 of the Decision Table). The matrix does not require that information be shared. The specific business purpose and risks and benefits should still be considered in making decisions about each use of Records of Tax Enforcement Results (ROTER) data. |
| This matrix applies to the sharing of information for various business purposes. In no event may a ROTER be used to set a goal or evaluate an enforcement officer or their manager. Data which is not a ROTER may be communicated to non-managers. ROTER data should not be communicated to non-managers. |
| The matrix describes the process for sharing data and the business purpose of data. Incidental access to ROTER data by employees or managers who are not authorized to regularly receive the data does not violate Section 1204 or the Regulation. For example, there is no violation of Section 1204 or the Regulation if managers or employees see published reports containing national compliance data. Similarly, managers or employees may collect data needed at higher levels, or receive ROTER data in the course of an acting assignment. |
| The matrix cannot cover every conceivable ROTER; if a specific ROTER is not shown, the decision as to how it may be shared must be made by applying the criteria in paragraph 1.9.3 and referring to similar ROTER in the matrix. The matrix applies to statistics (e.g., dollars, units, time) regardless of the source of data (e.g., AIMS, ERCS, Entity, CIMIS, manual reports, etc.) |
| Where a ROTER may be shared, the level of management receiving the data may also have the equivalent prior period or baseline data if the receipt of such ROTERs serves a necessary business purpose. This related ROTER information is provided for purposes of analysis and does not imply that current performance may be evaluated with reference to prior periods or baseline statistics. Comparative ROTER data about other offices will not be provided. For example, if a division chief has access to a ROTER for their district, they will not receive data about other districts; a branch chief with access to their own ROTER data will not receive comparable data about other branches; first line managers with access to their own ROTER data will not receive comparable data about other first line managers. |
| Business Purposes | |
|---|---|
| 1 | Long range planning |
| 2 | Forward planning and control |
| 3 | Financial planning |
| 4 | Allocation of Resources |
| 5 | Work planning and control |
| 6 | Obtain appropriate enforcement results |
| 7 | Effective functional management (in conjunction with other data) |
| 8 | Preparation for operational reviews |
| 9 | Identify areas for program review |
| 10 | Identify trends and best practices |
| 11 | Identify areas for process analysis review |
| 12 | Public relations/outreach |
| 13 | General knowledge of business operations |
| 14 | Quality improvement projects/special projects or tests |
| 15 | Inventory management |
| 16 | Data available as public information |
| 17 | Identify training needs |
| 18 | Identify need for management counseling |
| 19 | Identify systemic trends |
| 20 | Ensure customer satisfaction |
| 21 | Cross functional quality indicator |
| Level of Data Available | |||
|---|---|---|---|
| N | =National | OOU | =Own Operating Unit |
| OA | =Own Area | OT | =Own Team |
| Note : For Process Analysis purposes all statistical measures are available. | |||
| APPEALS STATISTICS MATRIX FOR GENERAL PURPOSES | ||||||
|---|---|---|---|---|---|---|
| Statistic Measure | Rover (Y or N) | Operating Unit Director (A) | Area Manager (B) | 1st Line Mgr (C) | Business Purpose (See attached list) | |
| 1. | Recovery Rate | Yes | OOU,OA,OT | OA,OT | OT | A,B: 7,9,10,11,13,14,17,21 C: 9,11,13,14,17,21 |
| 2. | Sustention Rate | Yes | N | - - - - - | - - - - | A,B,C: 9,11,12,14,15,17,19,21,22 |
| 3. | No Change Rate | Yes | OOU,OA,OT | OA,OT | - - - - | A,B,C: 7,9,10,11,13,14,16,17,19,21 |
| 4. | Revenue Yield Report | Yes | OOU,OA,OT | OA,OT | OT | A,B,C: 11,13,14,15,20 |
| ACDS – Selected Statistics and Table 11 Reports | ||||||
| 5. | Receipts | No | ||||
| 6. | Disposals | No | ||||
| 7. | Inventory | No | ||||
| 8. | Agreement Rate | Yes | OOU,OA,OT | OA,OT | OT | A,B,C: 7,9,10,11,13,14,16,17,19,21 |
| 9. | AQMS | No | ||||
| Rationale for Appeals Balanced Measures | ||
|---|---|---|
| MEASURE | DEFINITION | RATIONALE FOR SELECTION |
| CUSTOMER SATISFACTION | CUSTOMER SATISFACTION Appeals Customer Satisfaction transaction survey score - ( “keystone” question in transaction survey): “Regardless of whether you agree or disagree with the final outcome, how would you rate your overall experience with the audit process?” |
CUSTOMER SATISFACTION Reflects important aspect of IRS mission (e.g., provide American taxpayers “top quality service” Balances pressure to achieve Employee Satisfaction and Business Results Helps in comparing actual performance against customer expectations and identifying areas needing improvement Encourages candid dialogue with customers to understand what they need Helps the organization unit to examine its processes from the customers’ perspective and identifies areas for improvement |
| EMPLOYEE SATISFACTION | EMPLOYEE SATISFACTION Appeals Employee Satisfaction survey score - The Employee Satisfaction Measure is an average of scores from work group, management and summary sections of the survey |
EMPLOYEE SATISFACTION Measures the level of overall satisfaction that Service employees have with their jobs Promotes a work environment that permits employees to do quality, productive work Balances pressure to achieve Customer Satisfaction and Quality and Quantity measures Encourages candid dialogue with employees and helps management identify areas for improvement Serves as a guide for workgroups or managers to develop action planning strategies Helps improve employee morale, and reduces turnover of experienced employees |
| QUALITY AQMS Cycle Time |
AQMS Case quality review system for AQMS (0 - 100) |
AQMS Will allow Appeals to better communicate expectations regarding what constitutes quality work at all levels of the organization Will help to identify specific areas for improvement in quality and better understand what causes poor quality Indicates relative importance of quality elements Provides independent perspective on Appeals work external to the operating unit Permits reviewer judgment to address complexities of each case Balances pressure to increase Quantity, Customer Satisfaction, and Employee Satisfaction Replaces process measures (how work is done) that may have had unintended negative effects |
| CYCLE TIME | CYCLE TIME Helps evaluate timeliness with which cases are resolved, consistent with easing taxpayer burden and ensuring Appeals efficiency |
|
| QUANTITY Total Disposals Outreach |
TOTAL DISPOSALS | TOTAL DISPOSALS Outcome neutral measure of the primary output of the Appeals function A critical element of productivity/efficiency Can be considered in comparison to planned quantity Balances pressure to improve Quality, Customer Satisfaction, and Employee Satisfaction |
| OUTREACH | OUTREACH | |
| Appeals - Customer Satisfaction | ||||
|---|---|---|---|---|
| CATEGORY | DIAGNOSTIC TOOL | REASON FOR NOT USING AS PERFORMANCE MEASURE | HOW TO USE/NOT TO USE | BALANCED MEASURE IMPACTED |
| Customer Satisfaction survey | Appeals Transaction Survey Items Explanation of taxpayer rights Explanation of the Appeals process Courtesy of the Appeals Officer Professionalism of the Appeals Officer Showing the right attitude Time it took to hear from Appeals after you notified the IRS you wanted an appeals conference Time it took Appeals to schedule your initial conference after they first contacted you Explanation of what was expected of you Time given to respond to the IRS Consideration given to the information you presented Application of the law to the facts in your case Listening to your concerns Length of time it took you to get through the Appeals process Making you aware of your payment options Fairness and impartiality of Appeals in resolving your case |
Individual items only address specific aspects of Customer Satisfaction Difficult to apply a weighting scheme to convey overall level of performance |
Help identify specific areas for improvement in service to customers Results should be compared to historical levels for the organization unit No ranking relative to other organization units |
Customer Satisfaction score ( “keystone” question in transaction survey): Regardless of whether you agree or disagree with the final outcome, how would you rate your overall experience with the audit process? |
| Other | External factors Tax law Macreconomic cycles Public perception etc. |
Not a measure of customer satisfaction with Appeals service Appeals has no control over the external factors |
Help isolate factors that are beyond the IRS’ control Use to understand effects of external factors on Customer Satisfaction |
Primary Impact: Customer Satisfaction survey score Secondary Impact: Employee Satisfaction survey score Quality: - Cycle Time - AQMS score Quantity: - Total Disposals |
| Open minded-ness/ Independence | Although this is an Appeals strategy (Independent Review), there seems no objective way to measure this other than the single survey source | Appeals needs to “listen” to both the taxpayer and operating divisions for feedback on this critical area. | ||
| Customer Feedback System (CFS) Current system uses Form 10004 Mandated by RRA 98 Section 1203(b) Feedback from Appeals Customer Customer Service Reps |
Only partially represents Customer Satisfaction | Appeals needs to “listen” to available customer satisfaction feedback | ||
| Appeals – Employee Satisfaction | ||||
|---|---|---|---|---|
| Category | Diagnostic Tool | Reasons For Not Using As Performance Measure | How To Use/Not To Use | Balanced Measure Impacted |
| Employee Satisfaction survey | Work Group Survey Sample Workgroup/manager focus - My manager spends enough time with me to fairly evaluate my performance - My manager asks my work group about ways to improve the work produced Management focus - Employees have the job knowledge and skills needed to deliver quality work and service - Management lets employees know how their work contributes to IRS’ objectives and goals Employee Satisfaction Survey questions #3, 4, 5 & 40 Q3: Management provides sufficient time for job training Q4: I receive the training I need to perform my job effectively Q5: I have the job knowledge and skills needed to deliver quality work and service Employee Satisfaction Survey question #40: My manager spends enough time with me to fairly evaluate my performance All other questions The remaining questions will also be analyzed |
Individual items only address specific aspects of Employee Satisfaction | Help identify specific areas for improvement in Employee Satisfaction Seek to identify and resolve root causes not just symptoms of problems Identify business factors that cause employee satisfaction to change (e.g., changes in workload or work policies) Help to distinguish what can be easily influenced by a manager Results should be compared to historical levels for the organization unit No ranking relative to other organization units |
Primary Impact: Employee Satisfaction survey score (willingness and capability to do a good job): Weighted average of scores from work group and management sections of the survey Secondary Impact: Customer Satisfaction Quality: - Cycle Time - AQMS score |
| Work Volume | Average Inventory per AO (measures the number of returns open at a given point in time in an AO’s hands) | N/A - not a potential measure of Business Results in Appeals Inventory level can be subject to manipulation The ideal average inventory per AO has not been determined - Further, it will differ based on mix. |
Review what actions were taken to keep inventory within a reasonable range for a type of work (based on comparison with plan or historical level) Potential comparison to AO hours per disposal Needs appropriate stratification No ranking relative to other organization units |
Primary Impact: Employee Satisfaction Quantity: Total Disposals Secondary impact: Customer Satisfaction Quality - Cycle Time - AQMS |
| Appeals – Employee Satisfaction | ||||
|---|---|---|---|---|
| CATEGORY | DIAGNOSTIC TOOL | REASONS FOR NOT USING AS PERFORMANCE MEASURE | HOW TO USE / NOT TO USE | BALANCED MEASURE IMPACTED |
| Employee attitude/ behavior | Employee absenteeism (measures how often employees are absent from work | Not a comprehensive measure of employee working environment | Understand reasons for change in employee attitude/behavior Identify management actions that can impact attitude/behavior |
Primary Impact: Employee Satisfaction survey score Secondary Impact: Customer Satisfaction Quality: - AQMS - Cycle Time |
| Employee turnover (level of worker attrition in the organization) | ||||
| Other Indirect Feedback Level of participation in the survey Complaints and grievances |
||||
| Other | Employee experience/training/skill levels Hours of training per employee: Manager/analyst; Technical; Support/admin Training $ per employee: Manager/ analyst; Technical; Support/admin Training delivery: Classroom; CBT; Video Type of training per employee: Technical/ procedural; Automation; Mandatory (non-technical) |
Not potential measures of Customer Satisfaction - only partially represent a measure of employee satisfaction | Understand changes in level of experience and/or training received by staff and its impact on Customer Satisfaction | Primary Impact: Employee Satisfaction survey score Secondary Impact: Customer Satisfaction Quality: - AQMS - Cycle Time |
| Physical resources (availability of proper supplies and equipment to help IRS staff do their work) | Not a comprehensive measure of employee working environment Not a comprehensive measure of employee working environment |
Evaluate whether available physical resources are adequate | ||
| Work policies (e.g., policies concerning Flexiplace and Flexitime) | Evaluate impact on Employee Satisfaction | |||
| Appeals - Quality and Quantity | ||||
|---|---|---|---|---|
| CATEGORY | DIAGNOSTIC TOOL | REASONS FOR NOT USING AS PERFORMANCE MEASURE | HOW TO USE / NOT TO USE | BALANCED MEASURE IMPACTED |
| Quality Product | AQMS Standards Fair & Equitable Treatment of Taxpayers Quality of Decision Accuracy of Liability Computation Appeals Case Memorandum Timeliness Procedural Compliance |
Individual AQMS items only address specific aspects of Quality Process measures do not identify measure or encourage desired outcomes of Appeals activities |
Communicate expectations for quality Appeals work at all levels Help identify specific areas for improvement in Quality Seek to identify and resolve root causes not just symptoms of problems Help to distinguish what can be easily influenced by a manager Results should be compared to historical levels for the organization unit No ranking relative to other organization units Do not use to evaluate front-line staff |
Primary Impact: Quality: AQMS Score Secondary Impact: Customer Satisfaction Employee Satisfaction Quality: Cycle Time |
| Cycle Time | Cycle Time stratified by docketed/non-docketed/rework, divisions, level, etc. | An overall organizational measure of service quality is best described by the Cycle Time measure - not a portion of or stratification of that overall number. | May indicate areas where additional training, review or management involvement are warranted No ranking relative to other organization units Do not use to evaluate front-line staff |
Primary Impact: Quality: Cycle Time Secondary Impact: Customer Satisfaction Employee Satisfaction |
| 24-month Rolling Average for LMSB | The 24 month average is only meaningful for one Operating Unit within Appeals (L&MSB)and is, therefore, not suitable as an overall organizational measure of Appeals performance. | Give managers in L&MSB a truer indication of their closed case Cycle Time - Use to identify need for additional resources, training, review or management involvement No ranking relative to other organization units Do not use to evaluate front-line staff |
||
| Cycle Time | Open lapse time | What is most important to the organization and its customers is how long it actually takes to work a case - which is measured organizationally by Cycle Time, not Open Cycle Time | Gives us a clearer, more complete picture of Appeals production Closed case cycle times when considered alone may be misleading if the current process is bogged down in some way (i.e. too many new receipts) which may dramatically affect future, short-term results No ranking relative to other organization units Do not use to evaluate front-line staff |
Quality: Cycle Time Secondary Impact: Customer Satisfaction Employee Satisfaction |
| Appeals – Quality and Quantity | ||||
|---|---|---|---|---|
| Category | Diagnostic Tool | Reasons For Not Using As Performance Measure | How To Use/Not To Use | Balanced Measure Impacted |
| Work mix | Source of work (Income tax, Employment tax, Offer-in-Compromise, etc.) | N/A-not a potential measure of Business Results in Appeals | Understand if sources of work have changed from historical mix or plan Determine if mix has shifted to more time-consuming sources Understand relationship with compliance efforts |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality |
| Resources | Staffing resources: FTE appropriated FTE realized Resource utilization (direct Appeals officers time as a percentage of realized FTE) Hours and DET by business unit |
Undue pressure to apply direct time may have adverse effects on other performance measures - Quality - Customer Satisfaction - Employee Satisfaction Undue emphasis on time applied may cause manipulation of numbers - Open/closed cases/returns - Appeals plan |
Help identify business factors that impact other diagnostic tools (e.g., shifting resources to customer service, training or research projects may affect utilization, realization rate) Results should be compared to historical levels for the organization No ranking relative to other organization units |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality - Cycle Time - AQMS |
| Staffing resources (includes front-line workers and managers): Experience/skill levels (e.g., % new hires on staff, significant % retirement) Training (level of experience and amount of training of the staff) |
N/A-not potential measures of Business Results in Appeals | Analyze changes in level of experience and/or training received by staff to understand impact on other measures (e.g., number of returns closed) Anticipate needs for skill levels and training depending upon projected work mix/plan |
||
| Speed/Efficiency | Return closures per unit of effort (number of returns closed relative to direct time charged to returns closed by appeals officers and technical analysts) | Undue emphasis on return closures per unit of effort may also cause manipulation of numbers - Open/closed cases/returns | Help identify business factors that cause return closures per unit of effort to decrease (e.g., changes in work source, case complexity, work mix) Consider on a situation-by-situation basis Results should be compared to historical levels for the organization unit No ranking relative to other organization units |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction |
| Other (e.g., staff days per disposal-total direct Appeals hours applied divided by the number of returns closed) |
Undue pressure on average hours per return may cause early closure of cases, and thus have adverse effects on other performance measures Undue emphasis on average hours per return may also cause manipulation of numbers - Open/closed cases/returns |
Help as a guide to develop the Appeals planning Use as a tool along with other key diagnostic tools (not in isolation) Identify processing errors Results should be compared to historical levels for the organization No ranking relative to other organization units |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction |
|
| Appeals - Quality and Quantity | ||||
|---|---|---|---|---|
| Category | Diagnostic Tool | Reasons For Not Using As Performance Measure | How To Use/Not To Use | Balanced Measure Impacted |
| Internal feedback | Internal customer satisfaction (feedback loops) | Primary focus is on external (taxpayer) customer satisfaction | Use in conjunction with the Cycle Time measure and other relevant diagnostics to identify geographic areas or customer groups that require more attention, training or management involvement than others Better understand the needs of our internal customers and identify actions aimed at meeting those needs within the confines of a balanced measures environment No ranking relative to other organization units Do not use to evaluate front-line staff |
Primary Impact: Quality: Cycle Time Secondary Impact: Customer Satisfaction Employee Satisfaction |
| Direct Time Applied as a % of total time | While Direct time does lend insight into the current measures, standing alone it says very little about quality or quantity. It must be combined with other information - e.g., staffing levels - to be of maximum benefit. |
As a indicator of resources applied, Direct Time Applied should be considered when analyzing Cycle Time and Cycle Time trends. For example, if Cycle Time is trending up it may be an indication that resources are no longer being applied at the same rate - Analysis should lead to the identification of specific actions to improve results focused on the areas that reflect the most potential impact. No ranking relative to other organization units Do not use to evaluate front-line staff |
||
| Accuracy | Primary Impact: Quality Secondary Impact: Customer Satisfaction Employee Satisfaction |
|||
| No Change Rate (measures the proportion of returns fully conceded by Appeals) |
It is unclear whether high or low survey rates will lead to improved Business Results Agreement rates can be subject to manipulation Undue pressure at calendar time may cause AO’s to concede issues/cases they might not otherwise concede and thus have adverse effects on other performance measures |
No ranking relative to other organization units | Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality - Cycle Time - AQMS |
|
| Appeals return disposition mix (measures the number or proportion of returns by source and type; e.g., Income tax, Employment tax, Offer in Compromise, etc.) | Undue pressure to achieve a certain type of dispositions may improperly influence outcomes, and thus have adverse effects on other performance measures This is really a subcategory of total disposals |
Determine necessary staffing Determine to what extent training may be necessary Identify areas of continuous conflict and areas in need of legislative change Understand the impact of management involvement on disposition mix Results should be compared to historical levels for the organization unit No ranking relative to other organization units |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality - Cycle Time - AQMS |
|
| Average Disposals per AO | Consider 12 month rolling average Needs appropriate stratification |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality |
||
| Payroll Costs per Disposal | Can not easily be determined or even described at office level |
Use only at the National level Will need overhead data Consider 12 month rolling average Need to stratify by business unit |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality - Cycle Time - AQMS |
|
| Appeals agreement rate Docketed/non-docketed Operating Unit |
Agreement Rate is a ROTER | Use to evaluate the results of the full dispute resolution system and its principle processes: primary and re-work | Primary Impact: Quality: Cycle Time Secondary Impact: Customer Satisfaction Employee Satisfaction Quality: AQMS |
|
| Disposals as a % of Beginning Inventory | Consider 12 month rolling average Needs appropriate stratification |
Primary Impact: Quantity: Total Disposals Secondary impact: Employee Satisfaction Customer Satisfaction Quality - Cycle Time - AQMS |
||