It begins when management realizes that a systemic problem exists in the software they are developing, usually security, performance or reliability (SPR) related. Management gets worried and decides to bring in an expert, a guru who is to fix the problem.
<>The design, programming and test teams see the new guru and are glad that somebody has been called in to deal with the issue because that means it's not their problem anymore. One starts to hear refrains throughout the hallways of "Oh yeah that looks like a SPR problem, call in X."
The flaw in the strategy is the assumption that systemic problems like SPR can be dealt with by one expert or even a team of experts. If a developer puts in a function call that doesn't check for buffer overruns then maybe the security guru will catch it. But if the same developer puts in two publicly callable interfaces deep down in the API that by themselves are harmless but when put together spell trouble it is likely that the security flaw will go unnoticed.
To deal with systematic problems one must have an educated work force. Each programmer, designer and tester must understand the SPR issues that apply to their area. They must be trained in how to handle routine problems and to know when to call in the experts.
Thus in dealing with SPR the person to hire is not a domain expert but an area manager who owns SPR issues. Someone whose job it is to create an education program for each team that provides the members of the team, on an on going basis, with the information they need to meet the company's SPR needs. The skills to look for in this person are not so much an expertise in the particular area of SPR they are to manage as much as the ability to effectively work with teams, identify their educational needs and then provide for those needs.
The area manager's communication and organizational skills are paramount. They must be able to take information from journals, books, etc. and digest it for the team. Throwing a book at the team or holding a 1/2 day seminar is unlikely to be effective. Concise e-mails, top 10 lists and the occasional brown bag are likely to make the information more palatable. This puts a heavy burden on the area manager to be able to translate important content into concise formats that are appropriate to the team's members.
When a team does call for help it is not necessarily the area manager's job to provide consultations. After all, if the area manager is a domain expert then that person is probably better off being a domain expert than an area manager. Rather, the area manager is responsible for identifying what sort of experts the company is likely to need and provide an approved list of experts with whom compensation (if they are outside of the company) has already been arranged. Therefore, in hiring area managers look for people who can evaluate the expertise of domain experts and determine if they are appropriate for the company.
It is inevitable that the SPR management team will be viewed as a cost center. This means that every year around budget time cost conscious CFOs will be looking to cut the SPR budget. I have yet to even hear rumor of a company that on an organizational level has found a way around this problem. This is why so many SPR personnel are dissatisfied with their jobs. Ask any security manager what its like to get people or equipment when everyone views you as a cost center.
Until a means is found to properly evaluate contributions SPR area managers make to the bottom line even the largest of companies may want to consider outsourcing their SPR area managers. Outsourcing provides many benefits including SPR experts who have:
- Experience working with many companies,
- Existing educational materials that can be relatively easily adapted for particular needs,
- The reassurance of multiple customers that provides a buffer against the moral destroying effects of SPR budget cuts.
Hence an outsourced SPR team is likely to be better educated, better prepared and more motivated than an internal team. Of course all the usual caveats of outsourcing apply including problems getting the outsourcer's time and ensuring that they provide quality people.