i, quaid

My man Donnie Berkholz wrote a good analysis (and great blog post) about the new relationship between Red Hat and CentOS, with a delicious analogy of myself appearing as the Jabberwocky. (Does that make Donnie the young man with his vorpal blade?) For many reasons, I’m honored at the comparison. Yet, I detect some underlying concern in Donnie’s post that I want to address … line by line, in true open source fashion 🙂 … as well as a few items worth commenting/supporting. In the below quotes, Donnie is the first indent, and whatever he is quoting is the deeper indent.

LWN made some excellent points in its writeup (emphasis and links mine):

The ownership of the CentOS trademarks, along with the requirement that the board have a majority of Red Hat employees makes it clear that, for all the talk of partnership and joining forces, this is really an acquisition by Red Hat. The CentOS project will live on, but as a subsidiary of Red Hat—much as Fedora is today. Some will disagree, but most would agree that Red Hat’s stewardship of Fedora has been quite good over the years; one expects its treatment of CentOS will be similar. Like with Fedora, though, some (perhaps large) part of the development of the distribution will be directed by Red Hat, possibly in directions others in the CentOS community are not particularly interested in.

I’m not sure acquisition is the right way to describe this, as for a company it’s usually used in the context of a takeover of one company by another. It’s not really a takeover, it’s more like a marriage where each part comes in and remains on equal footing.

Red Hat has simply followed our long-term practice of hiring smart people to work full-time on the independent community they already care about.

The CentOS Project has set its own practices and procedures since creation, but Red Hat has provided influence on the project simply by being the upstream.Whatever influence I might have as a Board member with my Liaison hat on, it’s going to be squish-and-squat compared to the influence of the many hundreds of people already working upstream in RHEL Engineering and Fedora.

An explicit goal of this new relationship is to grow the variants — something CentOS has been doing already, so it’s a shared interest that brought both groups together — and that itself will create an expanded roadmap for the Project that is outside of the monolothic influence of the RHEL roadmap. By getting more involved, we’re helping the project go beyond just being a downstream of RHEL.

The playing field may never be perfectly level due to the influence of RHEL on the overall CentOS roadmap, but considering the Liaison’s limited-and-not-unique powers by comparison to RHEL Engineering controlling the code, it’s much more playable than before. And only going to get more so, as the community and leadership work out the kinks.

I’ll also point out that the idea of CentOS variants came equally from the community, with it’s desire for and consumption of Xen4CentOS.

I’m not convinced it had to go nearly as far as it did to realize those benefits, though — formalizing a partnership would have sufficed. However, giving three of the existing lead developers the opportunity to dedicate full-time effort to CentOS will be a huge win, as well the other resources Red Hat is providing around infra, legal, etc. But the handover of the trademark and the governance structure are a bit unusual for the benefits as explained, although entirely unsurprising for an acquisition and company ownership of an open-source project.

Yes, Red Hat has taken on the overall management of the CentOS trademark. Having a defendable mark and someone who will defend it is valuable to the future of the project.

In a free and open source software project, the trademark is often the only way the community can assert that such-and-such a combination of software is the community’s work and a similar-yet-different combination of software is not the same, so cannot be called the same name (under the trademark.)

Regarding the governance structure, I cover that below.

Red Hat still needs a breeding ground for innovation of the Linux OS, so I don’t see anything significant changing here. What I would hope to see over time is a stronger integration of developers between Fedora and CentOS such that it’s easy to maintain packages in both places if you desire.

While I can’t promise how the CentOS and Fedora communities will interact, this exact topic came up amongst the group of Fedora and CentOS folks discussing this behind the scenes pre-announcement. We concluded that i) sure enough, there are lots of ways we might be able to collaborate, and ii) we needed to get the new story out to the public first and see where the actual communities want to go. The idea of making it easier to maintain packages in both places was definitely discussed as a probable goal.

Perhaps the largest concern for Fedora is a lessening of Red Hat employees contributing to it on paid time, in the longer term. As the company pivots more toward cloud infrastructure (see its recent appointment of Tim Yeaton and Craig Muzilla to lead groups that own cloud software at Red Hat) with a clear hope of increasing its cloud revenue share, Red Hat’s need to differentiate at the OS level may shrink and thus its need to contribute as many resources to Fedora. However, Robyn duly points out that Fedora’s role as upstream for RHEL isn’t going anywhere, so neither is the project.

One might conclude that more resources in area A might have to mean fewer in area B, but this isn’t the case. In fact, if the cloud makes the OS less popular so fewer people are using it directly, that makes it harder to innovate and integrate — the brunt of the work ends up on the paid OS developers to integrate, test, fix, etc. As that work starts in Fedora makes its way to CentOS (not the reverse), the investment in Fedora is primary and core to all the missions.

Red Hat’s Karsten Wade seems to have become the closest thing there is to a CentOS BDFL (or at least an avatar of Red Hat as BDFL) by virtue of being the “Liaison” on the newly created governing board. The other named board role is Chair, who is a coordinator and “lead voice” but cannot take decisions for the board as the liaison can.

What’s missed here is the way the role works with the consensus decision making model, as the Liaison doesn’t have more power than the Chair. The Chair has traditional agenda and lead-voice responsibilities that give the role the clearest equivalence to project leader. All of us have the same voting powers to approve (+1) or block (-1).

In case you didn’t see the fine print, here’s the reason I say that:

The Liaison may, in exceptional circumstances, make a decision on behalf of the Board if a consensus has not been reached on an issue that is deemed time or business critical by Red Hat if: (1) a board quorum (i.e., a majority) is present or a quorum of Board members has cast their votes; or (2) after 3 working days if a Board quorum is not present at a meeting or a quorum has not cast their votes (list votes); provided that the Chair may (or at the request of the Liaison, will) call a meeting and demand that a quorum be present. Unless the Liaison specifically indicates on a specific issue that he/she is acting in his/her official capacity as Liaison, either prior to a vote or later (e.g., after an issue has been deemed time or business critical), the Liaison’s voice and vote is treated the same as any other member of the Board. Decisions indicated as Liaison decisions made on behalf of the Board by the Liaison may not be overturned.

Translation? If the board (the majority of which is Red Hat employees) can’t come to a consensus or can’t meet/vote within 3 days, the Red-Hat–appointed liaison can make an irrevocable, unilateral decision on behalf of Red Hat.

Note that the governance clause specifically limits this ability to when the Liaison puts on the Shadowman hat and says, “Such-and-so is time or business critical for Red Hat” and the Board has been unable to meet or come to consensus in a reasonable time period on that specific topic. This is clearly a limited set of potential situations, and not the same as if any vote, at any time, can be forced or overruled by the Liaison.

Also worth noting is that Karsten will be the direct manager of the three CentOS employees joining Red Hat, giving him further influence in both formal and informal forms. Although whoever’s in the liaison role theoretically steps down in power when not acting as liaison, this is much like temporarily removing “operator” status on IRC. Everyone knows you’ve got it and could put it back on at any point in time, so every word you say carries much more weight. It is therefore of great interest to understand Karsten more deeply.

Let me unpack this a bit by starting with a quick lesson in the consensus decision model that the CentOS Governing Board and many other projects follow. People don’t usually view it in light of the formal model, but the classic “-1, 0, +1” voting model is basically a consensus model with blocking.

In a consensus decision making process, one or more persons puts forth a proposal. In responding, others need to decide if they like the proposal, have some reservations about it that need to be resolved, or are solidly against the proposal as written/proposed:

Consensus decision making flowchart (Image by Grant Horwood (frymaster) from Wikipedia under CC BY 3.0 license)

1. If you like the proposal, you vote +1.
2. If you have some reservations, you let others know and work toward resolution. In the this case you are saying, “I am not actually blocking with a -1 vote, but if my reservations are not resolved, I may block with a -1 vote.”
3. If you can’t accept the proposal, you can block it with a -1 vote. This vote can appear to be a veto … but isn’t a veto.
4. Based on the above, you drive toward consensus by working through reservations and blocks until you have a proposal that all can agree upon.

Importantly, in the consensus model, the -1 vote must be accompanied with a sufficient reason (usually technical or community oriented in free and open source software projects) as to why the proposal doesn’t work. You can’t block a proposal without a sufficient reason.

Also, the blocking vote is an opportunity to work toward consensus to get the Board member(s) to unblock. It is not a permanent vote per se the way a veto is in a democracy. The blocking reason can be resolved, and then the blocker removes the -1 vote (changed to a +1 or 0), consensus is achieved, and things go on. Not at all like a majority rules with veto.

In this model, every single CentOS Board member can cast a -1 vote and block a proposal.

What the new governance provides is that a set of reasons that are normally not considered in an open source project such as, “Will doing this action cause a ‘foo bar baaz’ risk to Red Hat?” are now considered to be sufficient reasons for casting a -1 vote … but only if the Liaison says so.

To summarize, the Liaison role doesn’t provide special veto powers that no other Board member can use. Rather, it makes it clear that items “deemed time or business critical to Red Hat” can be brought up by the Liaison as sufficient reason for a block (-1). It adds a set of sufficient reasons that are otherwise not normally present in such situations.

That post makes it perfectly clear where Karsten’s interests lie, so it, along with his background in community management is what drives my initial expectations of Red Hat’s influence upon CentOS. It remains to be seen how often Karsten will need to step up to liaison mode, and to what extent his actions in that role will be handed down from higher up in Red Hat vs independent, so I’m looking forward to seeing how these changes play out.

You and me both. 🙂

As to what my interests in the project are, I suppose it’s fair to figure I’ll use my influence via the Board to get there, but I don’t think that’s related to my role as Liaison.I will definitely agree that the focus of our team at Red Hat is to work to enable variants, but only after all the regular work of the project is safely covered.