Introduction and Context

Wilson Workforce and Rehabilitation Center (WWRC) is a comprehensive rehabilitation center within a federal Vocational Rehabilitation (VR) system.  As such, WWRC shares complex systemic processes in Program Evaluation (PE) and Quality Assurance (QA) common to all VR organizations.  WWRC's renewed commitment to continuous improvement in the delivery of VR services, performance, and accountability form the basis of its comprehensive Performance Management System. All components of this system support WWRC's vision, mission, and values. The variety of activities that comprise the Performance Management System complement each other, are inter-related and ultimately are used to develop strategic and tactical plans and to document the organization's performance metric.

 

The evolution to a formalized organizational culture and infrastructure supportive of a purposeful, holistic, and integrated approach to the measurement and use of data for strategic planning, program decision-making, and process improvements began in 2007 with the creation of WWRC's Organizational Development & Quality Assurance (OD&QA) Division. Prior to that date, WWRC had historically garnered a variety of strategies and tools to measure performance against defined program goals, objectives, and indicators across administrations; however theses strategies and tools were not documented as a consistent written strategy with a clear vision and scope.  For the first time beginning in 2007, WWRC gave intentional and simultaneous attention to both Program Evaluation and Quality Assurance, recognizing them as distinct, yet critically interrelated functions necessary to growth and development of the Center.

 

In 2012, the OD&QA Team published a baseline report to the WWRC Executive Team to document the Center's then-current level of performance vis-à-vis its revised vision, with further opportunities for consideration.  One of the recommended opportunities was to adopt a formal, systematic Performance Management System for WWRC, based on the 36th Institute on Rehabilitation Issues (IRI) study titled "Performance Management:  Program Evaluation & Quality Assurance in Vocational Rehabilitation".   This recommendation was accepted and fully integrated as a major initiative launched under WWRC's Blueprint for Direction. 

 

What is a Performance Management System?

Performance Management is defined in the IRI monograph as a "system that a VR Agency uses to establish its mission, set goals, implement service programs and develop measures to ensure continuous quality improvement" (page 9).  Program Evaluation (PE) and Quality Assurance (QA) are operationally defined as distinct, yet critically interrelated components.  While PE "focuses specifically on acquiring the data associated with measures, analyzing the data, and reporting results so that appropriate decisions can be made about policy and practice directions", QA "occurs when an agency systematically takes corrective action based on the ongoing flow of information that provides signals as to how well it is performing" (pages 9-10).    The two elements represent a balance between an examination of the effectiveness of a rehabilitation program and intentional actions taken to improve and strengthen it.    

 

An effective Performance Management System answers questions such as:

  • Are we serving our customers in the best way possible?
  • Are we doing a good job in providing services to people with disabilities?
  • To what extent have we assisted our customers in reaching their goals?
  • Are we meeting standards or going beyond?
  • How can we encourage innovation while simultaneously developing measures of accountability?
  • How well is our vision translated to action?
  • Are we using our resources efficiently to achieve our mission?
  • Are we continuously soliciting feedback from our customers that helps us strengthen our programs and improve outcomes for our clients?

 

A summary of key system assumptions, components, and anticipated outcomes/impact is contained in a Logic Model for WWRC's Performance Management System.  Details are provided in the rest of this Reference Document. 

  

What are the Guiding Principles and Components of WWRC's Performance Management System?

WWRC is committed to being a high performance, data-driven organization reflective of its mission.  WWRC's Performance Management System is the organized, purposeful structure and corresponding set of interacting and interdependent PE/QA components through which this vision is realized.  WWRC's Performance Management System is based on the following guiding principles: 

  • Organizational Leadership and Program Accountability set the standard.
  • There is transparent stewardship in use of public resources with emphasis on quality client services, resulting in quality client outcomes and efficient organization operations.
  • Voices of stakeholders, including clients, enhance performance.
  • Performance metrics are easily understood and meaningful; targets are reasonable, but require effort to meet.
  • Relevant data reports are available to WWRC managers and supervisors, and their designees, on demand.
  • There is frequent communication with all WWRC staff relative to progress and successes.
  • There is clear integration of state and federal legal and reporting requirements, as well as data that documents compliance with relevant accreditation and licensure standards.
  • At all times, PE and QA initiatives are proactive and solution-focused.
  • Professional development enriches work practices within WWRC's Performance Management System and cultivates competent personnel.

 

WWRC's OD&QA Division houses two staff dedicated to WWRC's Performance Management System, one as a shared resource through the Virginia Department for Aging and Rehabilitative Services (DARS) Policy and Planning Unit.  The Lead Analyst for Program Evaluation and the Lead Analyst for Quality Assurance work as a team to deliver staff training, provide consultation and technical support, and facilitate an organizational culture that supports an effective Performance Management System.  DARS's Policy and Planning Unit and DARS/WWRC Information Technology Services provide consultative and collaborative staff expertise and resource support, as indicated.  Related role clarifications and business functions are contained in Section 2 of this Administrative Governance Manual.

 

Operating together, PE and QA sub-systems provide a consistent framework for monitoring WWRC's performance across Center operations relative to Agency (DARS/WWRC) policies, the federal VR State Plan, WWRC Strategic Plans, and the annual Blueprint for Direction.  Sanctioned teams and committees, to include the WWRC Executive Team, WWRC Manager/Supervisor Team, Pegboard Team, and Performance Management Steering Committee, provide routine leadership forums for review of PE/QA findings and recommendations, management level decision-making, and organizational accountability (Appendix 5.2.2).   The WWRC Administrative Governance Manual provides detailed protocols, standards, and expectations for effective implementation and maintenance of WWRC's overall Performance Management System.  Employee Work Profiles specify standards and expectations specific to individual position responsibilities.  A thorough Staff Education, Training, and Development Plan ensures that staff at all levels of the organization understand what a Performance Management System is, what it means in performing daily job responsibilities, and how it benefits our clients and VR sponsors.  

In addition to these essential components of WWRC's Performance Management System, the OD&QA Team meets monthly with the IS Director to review joint operational issues related to organizational PE and QA functions and to mutually address any identified issues.  Similarly, the OD&QA Team meets routinely with the WWRC Director for organizational strategy sessions related to performance accountability and data integrity.  Program Evaluation and Quality Assurance, as the core elements of WWRC's overall PM System, are described in more detail below.  

Program Evaluation

Program Evaluation is a systemic method for collecting, analyzing, and using information for purposes of program planning, service delivery, staff training, and public accountability.  Program Evaluation activities allow managers and supervisors within an organization to make reasonable judgments about the effectiveness, adequacy, efficiency, and comparative value of program options.  Program Evaluation presents an opportunity to review organizational operations to ensure they are appropriately focused.  Program Evaluation also ensures compliance with federal and state requirements governing WWRC operations.   Timing is a critical factor when using information derived from a system of PE for management activities.  Data must retain enough currency to be useful in deliberations.

Accountable for these functions and outcomes, macro level position responsibilities for WWRC's Lead Analyst for PE include: 

  • Plan, design, manage and implement studies of client outcomes, program effectiveness, agency organization, and managerial efficiency and effectiveness to meet accreditation requirements
  • Provide executive management detailed analysis to support programmatic decisions. 
  • Analyze study results and recommend to management methods to enhance programs, improve service quality, and increase efficient management services for WWRC clients. 
  • Formulate alternatives and solutions to address agency problems and identify policies and to generate procedures to improve service delivery, client progress and satisfaction. 
  • Direct WWRC's consumer satisfaction system. 
  • Lead and direct staff in program and service evaluation to produce information for planning and budgeting.
  • Represent WWRC on Agency wide teams and in Agency wide meetings pertaining to PE functions, return-on-investment data, and longitudinal data systems. 
  • Provide ongoing training, technical expertise, and resource consultation for WWRC Program Managers, Supervisors, and the Executive Staff relative to PE functions, systems, and tools.

 

The Lead Analyst for PE follows standardized Agency (DARS/WWRC) business processes to extract data from AWARE (the official recognized source for VR client information), import and validate data from offline systems, and create routine and customized reports/dashboards. These standardized Agency business processes, along with integrated satisfaction data, form what is known as the Agency PE Data System.  This PE Data System ensures that analyses are based on reliable and valid data, with active management of data anomalies. 

 

A major strategy within WWRC's Program Evaluation sub-system is the formally adopted Business Intelligence (BI) Model.  The BI model transforms raw data from AWARE and linked offline databases into meaningful information that is easily accessible and can be used in organization-wide and program level decision-making.  BI technologies provide historical, current, and predictive views of business operations and can help determine when current business practices do and do not readily support the reporting of information/data.

Quality Assurance

Quality Assurance refers to the systematic monitoring, evaluation, and correction of various aspects of a project or service to ensure that standards of quality are being met.  Standards are critical indicators for the achievement of excellence in VR practice and can be used as a benchmark against which to evaluate their practice and performance.  Quality Assurance activities present opportunities to drive changes in business practices and policies that result in strengthened program operations and improved VR client outcomes.

 

Accountable for these functions and outcomes, primary position responsibilities for WWRC's Lead Analyst for QA include:

  • Facilitate the development and usage of standardized program development/modification business processes that are consistently applied across Center operations; 
  • Provide ongoing training, technical expertise and resource consultation for WWRC Program Managers, Supervisors and the Executive Staff in effective use of data, cross-functional teams, and business models/processes/strategies/tools to achieve continuous quality improvement goals;
  • Conduct specialized QA studies (including accreditation reviews and preparations) in response to legislative or Agency mandates, WWRC strategic planning initiatives, and other defined management level priorities, as requested;
  • Partner with Information Systems staff and assist the Lead PE Analyst with identifying and designing customized AWARE reports and determine their frequency to support center service operational needs;
  • Represent WWRC on Agency wide teams or at agency wide meetings pertaining to QA functions;
  • Assist PE with identifying and determining how off-line data sources are integrated within the Business Intelligence Model; and,
  • Facilitate the development and documentation of business processes that support the integration of off-line data sources' integration with the Business Intelligence Model.

 

A major strategy within WWRC's Quality Assurance sub-system is the formally adopted industry-recognized Rummler-Brache Performance Improvement Model.  This model is used in combination with Business Intelligence to review critical operational processes that drive WWRC's performance metric.  The Lead Analyst for QA also assists with accreditation reviews and preparations, internal case audits, and AWARE support, education, and training.

 

 

What Are the Anticipated Benefits of an Effective Performance Management System?

Excerpted from the previously referenced 2011 IRI Study:   "Managing the overall performance of your agency is the mainstay of survival in this environment of government accountability, outcomes, and evaluation of cost benefit ratios.  Developing sound PE processes gives timely information about your program which can be used over and above established standards and indicators. It can inform you as to whether your vision of what your agency should be and do in terms of outcomes for consumers is being translated into action…… Well implemented QA processes assure the effectiveness of your service delivery system and provide information that can be incorporated into strategic planning… to drive decisions about priorities and the allocation of resources.  Information from PE efforts can also be used to advertise success, prove the economic value of the program, and justify requests for increased funding…." (pages 10-11).

 

If all components of WWRC's Performance Management System are functioning properly the following benefits will be realized:

Employee Level 

  • WWRC staff, at all levels, understand their own role and related standards, and expectations relative to WWRC's Performance Management System as well as how their own performance affects the goals, vision, and mission of the Center. 
  • Individual employee contributions to WWRC's Performance Management System are consistently and fairly recognized, acknowledged, and celebrated.
  • WWRC Supervisors and Managers understand the value of and access PE and QA resources and tools available to them. 
  • WWRC staff has ready access to reliable data reports that are meaningfully displayed and assist them in managing their own area(s) of responsibility. 
  • WWRC staff has routine opportunities to voice feedback regarding their perceptions of the effectiveness of WWRC's Performance Management System through an annual cultural assessment survey.

 

Organization/Systems Level 

  • There are clearly documented policies, procedures, standards, and expectations that are readily accessible, understood, and followed by staff at all levels of the organization. 
  • Performance Metrics meet management, statutory, and other organization business needs in a manner that is automated, reliable, and transparent.
  • Administrative and program level business processes fully support real-time processing of data and information for report and communication strategies. 
  • Agency dashboards create dialogue for ongoing program and process improvements. 
  • Data is used annually in strategic planning processes. 
  • Data reporting is proactive, anticipated, and routine, rarely based on ad hoc decision-making. 
  • There is consistency in access and use of PE and QA functions across Center operations. 
  • Offline databases are routinely and consistently linked with AWARE for analysis, reporting, and information dissemination. 
  • Monitoring systems are in place to determine if/when adjustments and refinements are needed. 
  • There are documented program and process improvements that positively impact VR client outcomes.
  • There is evidence of effective and efficient Team/Committee operations that strategically support WWRC's vision, mission, and values.