A CEO is a Leader: The Recent Mozilla CEO Kerfuffle

Experience comes for free. Learning takes effort.

On March 24th, Mozilla announced that they had named Brendan Eich as CEO.

On April 3rd, Brendan Eich, after much internet uproar over his $1000 donation to disallow same-sex marriage, stepped down as CEO of Mozilla.

So.

WTF happened here?

First of all, some facts…

  • FACT: Brendan Eich is well within his rights to support his personal beliefs through financial donations. Furthermore, that he did not back down from his position shows a strong sense of integrity. He was correct to do what he did. (ed. note: I very strongly disagree with Eich’s position on this matter and share Rarebit’s disappointment, but defend to the death Eich’s right to express his opinion.)
  • FACT: Team Rarebit and others were well within their rights to express outrage at the appointment. They were correct to say what they did.
  • FACT: Yet others were well within their rights to defend the appointment. They were correct to say what they did.
  • FACT: The Mozilla board of directors appointed Eich because they truly believed they were doing what was best for the organization. They, however, erred.

What appears to have happened is that the Mozilla board of directors misunderstood the role of CEO in an organization, particularly in a non-profit and mission-driven organization.

A CEO isn’t merely a leader; a CEO is a Leader. In a non-profit and mission-driven organization, a CEO is a LEADER. LEADERship extends far beyond merely setting technical and strategic direction for an organization. It includes becoming the very public face of everything for which the organization stands.

As an organization with a brilliant track record for supporting diversity and freedom of all sorts, Mozilla could not have a CEO who was on public record as opposing what a great many people see as a civil right.

Undoubtedly, as CEO, Eich would have continued his admirable track record of not allowing his personal political and religious beliefs to impact his professional performance. He had, after all, been CTO of Mozilla almost since the beginning and from all appearances his personal beliefs did not affect that performance. But being CTO is not being CEO. A CTO is responsible for setting and delivering the technical vision for an organization. One’s opinion on social issues is unlikely to come up in such a role.

A CEO, on the other hand, is responsible for setting and delivering the overall vision for an organization. This vision includes not only business strategy but also culture and mission. As such, a CEO’s beliefs must completely track with those of the organization which they lead. This is part of what makes the hunt for a new CEO so challenging. While a CEO’s personal beliefs may not affect the easily quantifiable parts of their job, (as we have now seen) they can have untold effects on responsibilities which, while more fungible, are no less important.

If there is any blame to be placed here, it is with the Mozilla board for not realizing this fact. They were aware that Eich had taken a public stance on an issue which could be seen as contrary to the Mozilla culture, yet they apparently did not see this as a potential problem for acceptance of him as a CEO. It was this ignorance and lack of awareness which led to the recent drama. Thankfully ignorance is curable. And undoubtedly Mozilla has had an effective dose of its medicine for this ailment.

There is no doubt in my mind that Brendan Eich has the experience, the vision, and the competence to lead Mozilla for years to come. He has proved that with his many years of experience as CTO. But from a cultural point of view he was, unfortunately, damaged goods. As such, the Mozilla board of directors should have refused delivery of the CEO package. If they had, they could have retained him, his passion for the open web, and his talents to help further the Mozilla mission.

Instead they are left with a learning opportunity and without the contributions of a founding father. Thankfully, the organization is populated by thousands of brilliant and insightful human beings. Mozilla will recover from this mistake, and even while recovering it will continue its work toward the open web. Those of us who agree with and believe in this mission should, now more than ever, support Mozilla. Let’s turn this experience into learning and continue to move forward.

The Rising Costs of Aging Perlers: Part 2, The Business

At this point you’re entirely justified in asking, “So what?”

The fact of the matter is, the lack of junior programmers is a very big deal from a business point of view, both to companies which already use Perl and those who might otherwise consider it. This directly and strongly impacts the bottom line of any Perl-using development shop.

According to Payscale.com, in San Francisco the median pay for a junior software developer is $66,000. For a senior software developer it’s $111,000. To be honest, we’re in a boom here right now. It’s my experience as a software development manager that each of these numbers is at least $20-$30K too low, but let’s run with the Payscale numbers. It’s obvious that a senior software developer is going to cost a shop a minimum of nearly $50,000 more than a junior developer.

Now consider the structure and needs of your typical software development department. There are a number of tasks which absolutely require the skills and expertise only a senior developer can bring to bear, but the majority of the work most likely can be performed by more junior developers with appropriate supervision. You do not need every member of your team to be senior and, at an extra $50K a head, you can’t afford to do it anyway. That sort of staff expenditure is an irresponsible business practice. However, as unreasonable as it is to expect a company to pay so much more for experience they don’t need, it’s equally unreasonable to expect a senior developer to accept less pay than (s)he has earned through years of training and experience.

Companies in this position, in order to continue using Perl, will have few options available to them. One of those options is to start training all incoming developers to use the language. While this is possible (undoubtedly some companies already do this), it is itself a very expensive proposition, often costing tens of thousands of dollars in training costs, time and productivity. A number of companies may run the numbers on the investment they’d be required to make in training and decide that it provides a much lower return than undertaking the arduous task of rearchitecting the software to use a language where they are able to hire staff (and at a more reasonable rate of pay).

Alternatively, companies which do not yet use Perl may be turned away from the language for similar reasons. The harsh financial realities of running a business—rather than technological merit—will end up dictating which programming language a company will use for their product. Selecting a language for which you can only hire senior developers is a very bad business practice, which leaves Perl in a poor position from a business point of view.

From where I’m standing, it does not appear as though the Perl community is doing much to correct this issue. As I detailed in my earlier post, in many cases Perl’s new programmer outreach appears fairly crummy if not virtually non-existent. This needs to change before Perl starts to face a cultural extinction. Perl needs to start creating fledgling Perlers to help sustain and grow the language through it’s next twenty-five years. As well, this will add new blood to the community and help diversify the gene pool. The more diverse a community, the better it’s able to adapt to the changing conditions which might otherwise overwhelm it.

A diverse or deep gene pool gives a population a higher chance of surviving an adverse change in conditions. Effects that cause or reward a loss in genetic diversity can increase the chances of extinction of a species. Population bottlenecks can dramatically reduce genetic diversity by severely limiting the number of reproducing individuals and make inbreeding more frequent. Wikipedia page on Extinction

In the final post of this series, I’ll detail some suggestions for avoiding this cultural extinction. Read Part 3.

Putting Away My Pitchfork: The Yahoo Remote Worker Announcement

On February 22nd, 2013, Yahoo! CEO Marissa Mayer announced that all remote employees will be required to work from their local Yahoo! office by June.

The internet grabbed its pitchforks and lit its torches then exploded with anger and derision.

Does this announcement place a hardship on remote Yahoo! employees? Probably.

Is this a draconian move by Mayer? It seems that way.

Would I have done the same in her place? I don’t know. And neither do you.

The fact of the matter is that none of us are in her place. As a company, Yahoo! has been foundering for years and Mayer is now at the wheel trying to right the ship. I’m not their CEO. Neither are you (unless you actually are Marissa, in which case, “Hi!”).

To be entirely clear and open: My initial reaction to the announcement was horror and anger. I am a supporter of remote working arrangements, and not only because I have one myself. I realize that there are good people everywhere and setting up an infrastructure to support remote workers allows companies to find the best people for the job, no matter where they’re located. However I also realize that without the appropriate infrastructure a remote working arrangement can be disastrous to a product and a company. It’s not for everyone. It’s not for every company.

Mayer is many things, but one thing she is not is a fool. Before this decision was announced she knew it wouldn’t play in Peoria. She knew there would be PR fallout. She knew the internet would cast her in the role of the wicked witch. She also knew a hell of a lot of other details to which none of us are privy.

This could not have been a swift decision, made in a moment of passion. Undoubtedly it was researched, calculated and discussed ad nauseam. Without access to that research and the discussions around it, none of us can know whether this is a good decision for Yahoo!.

I feel for the Yahooligans who now must uproot their working lives, I truly do. But I’m going to resist my impulse to pillory Marissa Mayer for her decision until I know more about what led to it.

‘Fail’ doesn’t have to be a four-letter word

Projects fail. Companies crash and burn. Screws fall out all the time; the world is an imperfect place. Just because it happens doesn’t mean we can’t do our best to prevent it or—at the very least—to minimize the damage when it does. As a matter of fact, embracing failure can be one of the best things you do for your organization.

First, let’s start with a few extreme examples of how things can go wrong…

A grim roll call

  • Despite warnings from many engineers, the space shuttle Challenger was allowed to launch with a faulty design on its oxygen tanks. The thinking, paraphrased, was “it will only be dangerous in these specific and unlikely circumstances…” Those circumstances, unfortunately, occurred.
  • A stuck valve, a signal light known to be ambiguous and lack of training led to the near meltdown of the TMI-2 cooling tower at Three Mile Island, to date still one of the worst nuclear disasters in history.
  • The Deepwater Horizon oil rig, off the coast of Louisiana, exploded in 2010 due to a number of factors: A cementing procedure was not followed correctly, allowing gas to escape from the well…on a windless day when a man was welding nearby.

Each of these tragedies has something in common: it could have been avoided. In each case there were known problems or procedures which were ignored and in each lives were unnecessarily lost. Each was the result of a compounding of small issues and oversights which unfortunately combined in just the right proportion at just the right (or wrong) time, small issues which were insignificant…until they weren’t.

Most likely lives are not on the line in your business but that doesn’t make your project failures feel any less personally and professionally devastating at times. Like these three tragedies, it’s likely that your failures can also be avoided.

Before suggesting how to avoid project failures, we should take some time to understand some of the factors which lead to them in the first place. There are many reasons why a project might fail, but two of the biggies are Culture and Cognitive Biases.

Cause #1: Culture

Do you like failing? No, nor do I. In general, humans do not like to fail. If things don’t go as expected we tend to get very cranky and sometimes irrational. This is unfortunate as failure is one of the most effective teachers we have.

Many project cultures are openly hostile to failure in any degree. In these cultures, backing or working on a project which doesn’t succeed as forecast can be very unhealthy for one’s career. Mistakes—even honest ones—are punished. This frequently leads to errors and oversights being swept under the carpet rather than being brought to light and addressed promptly. As we’ve seen in the examples above, ignoring the small things can have devastating results.

People in these cultures go to great lengths to avoid association with failure. Frequently, they’ll shelve a new and potentially profitable endeavor rather than take the chance it may fail. This risk aversion stymies innovation and company growth. As well, when a new project is pursued and is not going as planned the people associated with it will not report this underperformance. Doing so is often perceived (at best) as rocking the boat or (at worst) hitching yourself to the failing project. Therefore the project chugs along, devouring company resources, rather than being cut as it should be. All because of a culture which is hostile to failure.

Cause #2: Cognitive Biases

We all come with mental software packages which help us cope with life, allowing us to form judgments and take actions in situations both mundane and exceptional. Usually these work as expected and everything is just fine. But then there is the edge case, where the software doesn’t work quite as it should. This is a cognitive bias, “…a pattern of deviation in judgment that occurs in particular situations, which may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.

While many cognitive biases have been identified, there are four which most commonly contribute to instances of failure. You will all recognize them:

  • Normalization of deviance: The tendency to accept anomalies as normal. “Yeah, we get that error message all the time. Nothing’s ever blown up so we just ignore it now.”
  • Outcome bias: The tendency to focus on the outcome rather than the complex process leading up to it. “Sure, some things went wrong during the code deploy but the product shipped and isn’t that what really matters?”
  • Confirmation bias: The tendency to favor evidence which will support existing beliefs. “No, see, I know Windows would suck for this purpose because I read this article by RMS…”
  • Fundamental attribution error: The tendency to put too much blame for failure on personal rather than situational factors. “We fired the out-sourcers because they were hacks who didn’t know anything.” (Actual: the out-sourcers weren’t given the information and support required for success)

Suggestions

The possible causes for project failure are too varied and unpredictable to be able to avoid them all. However, given what we know it’s possible to minimize the impact and occurence of many potential failures.

First and foremost, it’s necessary to change the project and/or company culture so that it encourages people to report problems, no matter the size. The saying of this is, of course, much easier than the doing. Changing a culture is difficult but, at least in this case, very worthwhile. You can only solve those problems you know. The ones which go unreported end up costing dearly in money, time, effort and team morale.

Any cultural change must have honest and complete support at the highest levels of the organization. An effort like this can only succeed with commitment and leadership. Management must understand why the change is needed and lead by example. Only then will other members of the team feel comfortable enough to pull back the curtain and expose the problems which are holding the organization back.

Leadership should also learn to celebrate rather than blame those who bring errors, problems and failures into the light. Such actions should be seen for what they are: successfully preventing the waste of vital resources. Most people in tech have heard the phrase, “Fail early, fail often.” Few live by those words or reward those who do.

Many organizations will perform a postmortem on a project which has failed. While useful, these processes don’t even show half of the picture. The best way to identify issues which may put future projects at risk is to postmortem all projects, successful or not. Thanks to outcome bias, a near-miss will look like a success and typically will then escape a postmortem scrutiny. This leaves a raft of issues unidentified, ready to appear and compound in future efforts.

But even this won’t help you identify potential problems when you need to. While locating and anaylzing them after they’ve occured is, of course, valuable, wouldn’t it be better to spot them before they become real problems in the first place? Part of the cultural changes which must occur is an organization-wide commitment to spot and report potential problems as quickly as possible. This is more likely to happen if people are asked to use simple and familiar tools to make the reports. Rather than re-invent the wheel or roll out yet another tool for team members to learn, I suggest that the organization’s existing issue tracking system be used for this purpose. After all, while these problems may not be software bugs they are issues which need resolving. Simply create a new database/project/what have you in your issue tracker then open it up to everyone in the organization. It does little good to report issues if people are not able to see them, after all.

Reporting the issues isn’t enough. If people believe that the reports go into a black hole, where they are never viewed let alone addressed, they will stop creating them and you’ll be right back where you started. These issues should be analyzed to determine the root causes for them. When resolving a problem, all too often people will scratch the surface and leave it at that. “This build failed because $team_member didn’t follow the procedure.” That’s not a reason for the problem; that’s a symptom. The true reason is buried somewhere in there. Why did $team_member not follow the procedure? Is the procedure wrong? Was there some exception? Was $team_member getting undue pressure to work faster, causing corners to be cut? Did $team_member not know that the procedure exists or that it should be used in this situation?

The statement, “This build failed because $team_member didn’t follow the procedure” is a presumptive and possibly incorrect placement of blame. As you can see, it is much more likely the case that the system within which $team_member is working is flawed than that this person is incompetent. The urge to point fingers is strong, but a vital part of the cultural change required is to discourage laying blame without doing analysis. Take an hour to speak with everyone involved to figure out the true underlying reasons for the error. Never punish but in the cases of gross negligence or repeated incompetence. Let people know it’s not only OK to make mistakes, it’s entirely natural to the human condition. Not only will you find that people start reporting and addressing issues before they become real problems, you’ll also find that team morale improves as people realize that they’re respected and trusted.

Once you’ve identified the underlying reason(s), make sure to correct and communicate. It does no good to analyze a problem if you’re not going to do something with what you discover. Fix the procedure. Provide additional training. Add a new Nagios warning. Modify the schedule to allow time to do things right. Bring in a specialist. Do whatever needs to be done to be sure the issue never happens again. And, of course, it’s vital that you communicate not only the necessary fixes but why they were necessary in the first place. Team meetings are a good way to do this but don’t forget that issue tracker. Any resolution should be captured here for posterity. Just like adding comments to a complicated piece of code, Future You will be grateful if you document both the problem and its solution.

Other Resources

As I mentioned above, there is a myriad of ways in which a project can fail. This article only starts to address that and some of the things you can do to prevent or minimize damage. If you’re interested in learning more, here are some of the resources I’ve stored up on the subject. Most of them are from HBR, so you may need to pay to access them. You can also probably get them from your local public or university library.