California payroll: Another failure waiting to happen?

The State Controller’s Office is launching an assessment to recover its failed payroll project. This new project seems doomed even before it starts.

The State of California is soliciting bids for contractors to conduct an “independent assessment” for the $371 million failed payroll project known as MyCalPays and today called 21st Century Project.

Project history. The MyCalPays project was started in 2004 and suffered a series of reversals, including the firing of Bearingpoint, the original vendor. In 2010, California hired software company, SAP, to continue the project; by 2013, SAP was also fired, amid a storm of accusations and counter-accusations

A failure analysis (PDF download) conducted by the state concluded:

It will be difficult to determine, if anyone ever does, exactly what doomed the project, the largest payroll modernization in the nation. The system was designed to take over human resources and payroll responsibilities for 240,000 state employees. At the end, the project had nearly tripled in cost, to $373 million, and was years behind schedule.

In March, 2014, the state’s Legislative Analyst’s Office released an informative update (PDF download) describing the project’s history.

Current situation. Today, the state seeks an independent vendor to examine the current situation and make recommendations on how to proceed. The budget for this assessment is $1.275 million.

The Request for Offer (PDF download) document describes the deliverable goals such as the following:

Determine which system requirements are met and which ones remain unmet by the MyCalPAYS system.

identify what deliverables are useable given the Contractor’s understanding of the requirements that have been met, what requirements remain unmet, and an analysis of the scalability of the existing design to the statewide population. The Contractor shall identify all areas that must be redesigned in order for the system to serve the 240,000 statewide workforce population.

Determine the cost for a vendor to complete (as needed) modification and implementation of the MyCalPAYS system

The Contractor will identify a single price that a prospective integrator is likely to charge to complete the project

In short, the findings and recommendation deliverable(s) must answer the following questions:

  1. Which requirements are met by the MyCalPAYS solution, as currently implemented?
  2. What requirements have not been met with the MyCalPAYS solution?
  3. What portion of the current MyCalPAYS design and system may be usable?
  4. What is the estimated cost for a vendor to complete (as appropriate) modifications and implementation of the system ensuring that it is compliant with the stated system requirements?

The state specifies the following requirements for team members assigned to this assessment project:

  1. Two full implementation cycles of SAP payroll
  2. Five years SAP payroll experience
  3. System assessment

In addition, the contracted firm must have performed at least two successful IT project assessments similar to this one and have five years' relevant experience.

One staffing firm, b2b workforce technologies, offers a job listing corresponding to these requirements. Since this business appears to be a recruiting company, rather a system integrator, presumably a vendor hoping to bid on this assessment asked b2b to find qualified people. I left a message for their Vice President of Recruiting, but he did not return my call.

The RFO document describes the state’s responsibilities on this assessment project being rather minimal:

  • Designate a contact point
  • Provide contractor personnel with a “standard cubicle”
  • Ensure access to state staff
  • Review and approve (or reject) deliverables


In my opinion, as someone who regularly advises CIOs on strategies for IT success, I believe this assessment is doomed for the following reasons:

Complex requirements. In effect, the RFO document asks the contractor to figure out what went wrong, what works or does not work, how to fix these problems, and what is necessary to make the system scale.

In addition, the contractor must determine the cost for a vendor to complete the project. Also, the contractor must identify the “single price a prospective integrator is likely to charge to complete” the integrations required.

The requirements also seem predicated on the assumption that existing technology is salvageable. The state should be open to the possibility that it may be necessary to discard previous work and start over.

In summary, the project underlying this assessment is sufficiently complicated that even the auditors could not untangle all the threads.

Minimal state involvement. The state’s role seems defined to push full responsibility for completing the project onto the vendor. Given the intertwined nature of vendor and client in these situations and the negative history of this particular project, establishing such boundaries seems unrealistic and naive.

The state should recognize that collaboration is key to success on this kind of initiative; silos don’t work.

Politics is baked in. Rather than start from a fresh slate, looking for the best solution, the RFO includes a one-sided project history casting blame on SAP without accepting any responsibility itself. For example:

SAP’s failure to meet its contractual obligation to cure, left the State with no choice but to terminate the contract with SAP

Given the track record of this project, it is time for the state to acknowledge its shortcomings rather than blame everyone else. As the old saying goes: “Accepting responsibility is the first step to learning and improving.”

By casting blame from the start, the state begins this project without a critical examination of its crucial role in this situation. If history is a guide, the state will likely continue to blame external vendors despite its own shortcomings.

Inexperience. Great people can overcome almost any hurdle, making the relatively minimal personnel requirements on this project so surprising. I believe that personnel with only five years’ experience are not qualified to undertake this project successfully, given the size, scope, history, and politics. I asked two HR technology experts to comment on this issue.

Naomi Bloom is one of the most respected HR technology advisors in the world. She raises concerns about the level of experience needed to execute this assessment successfully:

This kind of assessment requires someone who has worked on and reviewed similar projects for many years. The right person should possess very strong SAP payroll expertise, so they can sort out the issues in a reasonable period. In addition, they must have sufficient confidence to stand up to the politics of the situation. If I were recruiting for this, I’d want the most experienced person I could find.

Jarret Pazahanick is a top SAP and SuccessFactors consultant, with deep experience in SAP payroll:

The sad truth is these public sector projects are often low bid or have several consulting “layers” so they rarely get the highest caliber consultants; seeing the job posting, it appears that history could repeat itself. I would advise the State to find an extremely senior consultant to help lead them in this important discovery exercise.

I raised concerns about this qualifications issue with a spokesperson for the California Department of Technology. He told me that five years is the "minimum" level of experience and that bids will come from vendors who are “pre-qualified.” He added this contract does not have to be awarded to the lowest bidder.

To address my concerns, the spokesperson said, “The people who drafted the RFO document are very experienced in project management.” When I asked whether these folks are the same ones who managed the original failed project, he stumbled a bit and then said, “no,” because there were personnel changes. However, the RFO says there are “seven (7) project team members (both staff and consultants) in the Project Management Office remaining.” Hmmm....

I’m skeptical about this whole endeavor, but you can draw your own conclusions.

Update Aug 30: A further thought and point of advice for the Department of Technology and the State Controller's Office... Is there a single person responsible for this project? I don't mean the project manager, but a trusted executive sponsor whose credibility and integrity are beyond question; someone able and willing to roll up their sleeves to bring the various interest groups together around project definition, goals, and execution. If the answer is no, then you must start there.

Frankly, I don't think you are ready to move forward right now and I wonder about any firm willing to take on the project at this time, given the circumstances.

Also read

Did California’s CIO mislead public on IT success?

California abandons $2 billion court management system

Understanding Marin County’s $30 million ERP failure

Turnaround strategy for Los Angeles school district payroll failure

Bizarre ERP implementation experiment at Arizona State

The IT failures blame game (part 2)

Who’s accountable for IT failure? (part one)

Exploring the Devil’s Triangle