It is easy for procurement professionals to understand that the best indicator of how a contractor will do in the future is how they have performed in the past, but very few are working with a credible, objective and fact-based system for collecting and utilizing this information on a regular basis. And given the specific reference to ‘significant or persistent deficiencies in performance’ as grounds for excluding vendors under the trade agreements, the issue is taking on heightened importance.
This article explores how extensive consultation with industry coupled with internal re-education and internal change management processes have enabled the City of Ottawa to establish a defensible system for building a history of vendor performance over time, allowing future vendor selection decisions to include a historical performance perspective.
In April 2011, Council directed staff to implement a procedure for reporting supplier performance to Supply Services. In response to this, as well as a recommendation from a 2012 Audit of Construction Supervision, the City of Ottawa has created a robust Vendor Performance Management (VPM) program. Since 2015 the City has been monitoring, evaluating and recording contractor and consultant performance on the delivery of construction related contracts for design, construction and contract administration services. They are now beginning to use vendor scores in bid evaluation and contact award decisions, to ensure the best value contractors are selected for future contracts.
So how did the City get to this point, and what have been some of the critical learnings along the way?
Program Objectives and Scope
Recognizing the importance of collaboration with vendors, key objectives for the VPM program include:
o Improve communication between staff that manage contracts and vendors
o Improve overall vendor (consultants and contractors) performance on City contracts, in a measurable way
o Provide feedback to vendors with the goal of performance excellence
o Build a history of vendor performance over time, driving improvements and allowing future vendor selection decisions to include historical performance perspective
o Support the contract administration process used to address non-performance
Prior to embarking on this initiative, the City was in the position of many Canadian organizations – only paper based evaluations were being conducted, it was inconsistent, there were no formal rules around the process, the results were not being analyzed nor was there a centralized database for capturing the evaluation information. Engaging with multiple user groups across the City enabled the development of standard processes and templates, configuration of a centralized database and a ‘phase-in’ strategy to speed up the implementation.
Given the time and resources involved, the City elected to focus the first phase of the Program on construction related contracts, including engineering and design consultant contracts over $15,000, as well as general construction contracts over $100,000.
Consultation with Industry
Preliminary research on other organizations embarking on similar initiatives helped to shape the way forward for the City of Ottawa. Prior to implementation, the City reviewed other public sector vendor management systems and explored lessons learned by those organizations. They also consulted with numerous industry associations on the contractors’ side of the equation, including Ottawa and Canadian Construction Associations, General Contractors Association of Ottawa and the National Capital Heavy Construction Association. They also consulted with the Ontario Association of Architects and Ottawa Regional Association of Architects, the Consulting Engineers of Ontario and the Ontario Association of Landscape Architects. These key stakeholders provided detailed feedback, and identified five overarching requirements from their perspective:
1. Clearly defined scoring criteria
2. Multiple controls to ensure consistency and fairness
3. A chance to review and discuss the evaluation prior to it being entered into the system
4. The evaluation to reflect certain extra costs and delays outside their control
5. An appeal process.
The City adopted these suggestions, ensuring specific processes were in place to deal with each of these areas of concern.
In close consultation with industry, the City developed 8 Key Performance indicators (KPIs) for General Contractors, 6 for Consultants, and 8 for Contract Administration, each with associated point weightings. Individual vendor performance evaluation scores are tracked, and all low scores must have supporting comments and documentation. Each of the three areas (Contractor, Consultant, Contract Admin) has an associated VPM Expectations Document which clearly outlines the requirements, how they will be scored and describes the five performance levels ranging from ‘outstanding’ to ‘not satisfactory’. Those scoring below ‘satisfactory’ are monitored, and various remedial steps may be required depending on the severity of the performance issues encountered. It is the Overall Vendor Score (OVS), a weighted-average over the previous 3 years, that is used in future vendor selection decisions.
The ‘No Surprises’ Strategy
Embedded in the new VPM process is the underlying philosophy of ongoing communication and consultation with vendors as the project unfolds. In addition to sending and tracking emails and letters as soon as issues arise, formal performance discussions have been made mandatory as part of the program, and include the Project Kick-Off Meeting, Progress Meetings, Interim Evaluations, and the Project Close Out Meeting.
The Project Kick-Off Meeting in particular ensures sharing of the City’s performance expectations with the vendor by providing them with the VPM Expectations Overview document at the very beginning of the project. This provides the basis for a discussion with the vendor about how their performance will be evaluated and at what intervals. The vendor is required to sign off on the Expectations Document and the evaluation strategy. The approved Expectations Document, along with any significant documentation related to performance issues identified, are to be posted to the MERX system and visible to the vendor being evaluated. The final evaluation is saved in PDF format from MERX and stored in the vendor performance folder of the project file on the City’s records management system.
The normal process flow for project managers is depicted in the following chart.*
*from City of Ottawa – VPM Business Process Guide, Feb 2018
Where Are They Now?
At the end of Q1 2018 the City has completed 1116 final evaluations. Of these, only 4% have resulted in scores that fell below the ‘satisfactory level’. As of the end of Q1 2018 there have been only 12 appeals, and vendors had an average of 5.24 projects evaluated and a median of 2.
Overall, vendors are very satisfied with the objective and transparent VPM process, which has enabled the City to bring meaningful past performance data into the RFX evaluation process. In early 2018 the City began implementing this historical information as part of its procurement processes, enabling them to reward the best performing vendors and putting others on notice that improvements are required if they wish to remain competitive. The less visible beneficiaries of the VPM process are, of course, the citizens of Ottawa.
The City continues to consult with industry on an ongoing basis, to enhance the VPM program moving forward. In particular, they are working on improvements to the KPI wording for clarity, and maintaining the delicate balance between transparency and confidentiality when using the scores in contract award decisions. They have also prepared a ‘VPM Business Process Guide’ as a companion reference to the VPM program, which is available to City staff, vendors and other stakeholders.
The Procurement School extends many thanks to Will McDonald, Chief Procurement Officer, City of Ottawa for his help with this article.