DistroWatch Weekly

KS: Hi, my name is Karanbir Singh. I am 35 years old, love Belgian beers, enjoy travelling and a good steak. Born and brought up in India, I moved to the UK in May 2001. I am currently based in South East London working as the Infrastructure Lead for a major on-line services company.

KS: When Red Hat moved from the Red Hat Linux to Red Hat Enterprise Linux (RHEL) model it left my work place, of the time, in a dilemma. We had a lot of in-house experience on the "Red Hat" platform which also meant that we were very capable of self supporting our internal requirements. The RHEL platform, on the other hand, was and still is targeted primarily at people who need both business and technical support - neither of which was high on our priority list. The options at the time were to rebase to an alternative like Debian or SUSE or to hedge our bets with the faster moving target that was to become the Fedora distro.

However, none of those options seemed natural and would all require a fair bit of effort. Plus, introducing a drastic change in our work process is something that I didn't want. Luckily, it was at that point that I came across the White Box Enterprise Linux (WBEL) distro and we saw the third option - to continue to use a Red Hat platform, while being able to self-support our own requirements. So after a bit of testing, we started moving out production base over to WBEL, and then actually started doing the package rebuilds in-house.

It was a few months later, while talking with Johnny Hughes Jr, that I realised that WBEL was being too specifically targeted at an internal requirement that the only developer had. So we came up with a grand plan of starting our own project with a more open set of goals, better defined and easier for people to identify with. Our driving force was not the implementation goals that people had. Our driving goals were more along the lines of making sure we get a stable and well-managed platform together and the technical depth that people who already used this platform could bring into a common pool. But it was Donavan Nelson who pointed us at CentOS, which had already got a release for CentOS 3 out of the door. And after a bit of deliberation, Johnny and I decided that rather than start our own project we would adopt CentOS 4, and the two of us became the "CentOS 4" team.

Now almost six years later, we are still involved with the project and doing a lot of the work within the CentOS core team. Of course, personal life needs to take a priority and we all need to take breaks every now and again from CentOS "efforts".

KS: I think it's a bit of both, official role and my own desire. Helped along with a case of implied visibility. There are quite a lot of people contributing in their own ways. And many of them, including me, put in dozens of hours every week towards CentOS and the various efforts in and around the project. It just so happens that things I'm working with at the moment has been that much more visible publicly, so plenty of people see and hear quite a lot from me.

There is every effort to make CentOS more popular and we all like to get the word out there about the platform. If you say that the small things that I do have been making an impact, it makes me feel good. Along the same line, one thing I would love to do more of is be able to travel to various open-source and business-centric shows to be able to talk to people about CentOS. But the day job gets in the way! The necessity to earn a living means that I can only do two or three such events every year.

KS: There are a couple of things that contribute towards this. Firstly, there is the whole issue of trust. We have demonstrated over a long period of time that the userbase can expect a consistent, reliable and stable platform. And a large part of that comes from the extensive efforts we make in being and staying true to our upstream, Red Hat. As I mentioned previously, the goals within CentOS are not to target implementation, but to target this platform. What this means for the user is that they can trust the product that comes through CentOS and know they get the accrued benefit of the applications in the distro having been through the Red Hat testing process as well.

Another factor that contributes towards our popularity is the idea of peer recommendations. It has been a long and hard road, but we have always encouraged new industry segments to evaluate and grow with us. For example in early 2005, with CentOS-4/x86_64 we were able to reach out to the VoIP industry in a big way. We tried to make sure that the opinion leaders in that industry were aware of CentOS, knew what our goals were, had a route to provide direct feedback and know that their feedback would matter. End result was that most of them went out recommending CentOS. I've heard rumours recently that CentOS might be the largest installed platform for VoIP providers worldwide now! The hosting industry is another place where we had similar success. Both of these market segments are areas that I worked on personally; let's not forget that there are many other people who have had similar success with CentOS in their own respective areas of interest.

A third factor that has helped spread CentOS is availability. We have always tried to make sure that we have plenty of mirrors, plenty of bandwidth to those mirrors and spread them out all over the world. Today there are more than a hundred machines that are managed by the CentOS Infrastructure team and more than 500 external mirrors. Pretty much every major open-source mirror network in the world carries CentOS. It was a lot of effort getting there; in many cases it was one of us having to repeatedly badger the administrators in order to get them to carry our content.

Another major issue, and perhaps the most underrated aspect of CentOS, is that it makes for a fantastic platform to build emerging technologies and to host rapidly changing targets. I know many people will be surprised about that. The general impression is that CentOS is a stable platform, with a strongly reliable ABI tracking application base that does not divert from its core functionality during the course of a release. And they are all right. But because it does all this, it also makes for a fantastic platform to change a couple of specific applications that are important to you, and be able to rely on the fact that the other programs in the distro will not change drastically. And to be able to do this, yet receive security and bug-fix updates for the platform is of immense value to developers.

For example, if you want to build and host a Ruby on Rails application, CentOS is a great choice. It allows you to use whatever mechanism you want to build, test and deploy your application, while assuring the user that the environment and platform are secure, dependable and stable. And it is this feature that contributes in large portions to the fact that almost every emerging technology today has some level of roots in CentOS. Cloud computing, messaging, high-performance clustering, virtualisation, appliances, VoIP, commodity level hosting, etc are all areas that CentOS has been involved with.

Then there is the huge community around CentOS - an extremely capable and knowledgeable pool of people who all work to achieve different things using the same platform. So if you run into issues, there are usually quite a few people who would be very ready to help out over the lists, IRC, forums or even direct email. This does come with its caveat of community level support so one does need some level of awareness and an interest to learn about things in order to best use such a resource pool. If you need a phone number to call when things break, and prefer to pay someone to answer that phone and know what they are talking about, you should be talking to Red Hat!

Of course, many of these points are not specific to CentOS and would apply to any rebuild, but I feel we have had the right level of juice and focus to make all of them applicable to us.

KS: There are really two questions here. Firstly the idea of numbers. I have no idea how many people use CentOS, but we know it's lots! Six figures for sure. Seven figures? Maybe. Tracking IPs works for Fedora due to the demographic and the install roles they target. With the CentOS userbase, it's not that easy. For example, many organisations and universities run their own local mirrors, and CentOS installs from inside there will never hit the YUM repositories that we run. Similarly we know there are large installs in China and Japan, and we rely entirely on external mirror hosts to serve those areas.

The second part of your question is about how our numbers stack up against RHEL's. In my opinion, that comparison isn't really worth doing. We are not competing with or trying to hurt the business that Red Hat is in. I know there are edge cases where people who could potentially be using RHEL are using CentOS, and similarly people who might be better off with CentOS are using RHEL instead. But in the grand scheme of things what RHEL targets and what we are doing are quite different. If a user needs business and technical support, along with a SLA and an assurance that they can get that support at any time for their application, they really should be talking to Red Hat. On the other hand, if there is enough expertise in-house to self support your application and be able to have the flexibility to rely on community for support then CentOS is a real option.

Then there are areas where people just need a stable platform to host an alien application or role, which will need a minimal level of involvement in administration, like HPC and appliance machines. Again, CentOS makes for a good fit here, since that would allow you to change as much or as little as you like with no implications on support availability. It's also not uncommon to hear about hybrid CentOS / RHEL setups where people use CentOS in development roles, with RHEL hosting their production role. So I don't think the number comparisons with RHEL and CentOS matter. What matters is that people have a choice, and the ability to pick something that works best for them on a platform that they are familiar with.

KS: It was great to see CentOS specific books out on the shelves; made sweeter by the fact that some of that content come from people who are deeply involved with the community. With respect to certification and training if there is a process that can be put into place which can guarantee, within reason, the level of competence of a "graduate", I don't see any reason why we could not do that. On the other hand, given that the product base is so similar, I imagine the best value in a CentOS certification would be as a compliment to a Red Hat certification. Based on what I've seen in the last few years, most people who ask for an RHCE expect the candidate to be working with RHEL. So one needs to keep that in mind as well.

KS: Because the target audience is so different, RHEL and CentOS interests don't overlap too much. Think of it like this: RHEL is a service and support contract you get a Linux distribution with. CentOS on the other hand is a Linux distribution that you can download and install, and expect to put in the hard work yourself in order to support. The CentOS community has a very good and functional support mechanism in the mailing lists, IRC and forums. But no insurance company is going to give you business liability indemnity against that. Similarly, if you need to run the latest kernel for some specific function, doing that on RHEL will invalidate your support contract with them.

So when one uses the "support" word, exactly what we support in CentOS and what support one gets from Red Hat are quite different. Although, don't be surprised if a CentOS mailing list can help you get the solution to the problem faster than Red Hat tech support can :)

KS: The big difference between Red Hat and Novell is their attitude towards the sources. Unlike Red Hat, Novell does not publish all their sources. Till very recently they also didn't publish a complete set of tools that would be needed to make a community SLES possible. And as far as I can tell, they seem to prefer working through the openSUSE effort rather than accepting an open source community rebuild effort around SLES. Further highlighted by the fact that sources for their updates in SLES are not published publicly at all. I accept that it has been about 18 months since I looked last, so they might have changed their attitude now.

Then there is also the lack of interest. I am aware of at least two groups of people who have considered going down the route of building up a community SLES-based distro, and abandoned the idea when they found little or no interest. Most people who want to use the SUSE platform seem happy with openSUSE. I can only imagine that this hurts Novell overall, since there is very little openSUSE penetration into either the business or academic setups.

KS: Our process is fairly automated at this time, but still relies on people doing the actual approvals. It would be nice to get to a stage where the process could run end-to-end completely automated, but we are not quite there yet! Hopefully soon we can start looking at that, at the moment it's just not a major target.

When a new release is announced at Red Hat, we would download all applicable sources. Step one is to rebuild all the sources against the same targets used upstream and try to use the same build order as much as possible. Once this is done, we do update / upgrade testing against these packages. For example, for the 5.5 release, we did extensive tests for upgrades from the various 5.x releases and also from 4.x releases. Once we are satisfied with the results, and are reasonably sure that we produce the same results with our packages as upstream would - we move onto the building ISO stage. Which are then handed over to our QA team.

Our QA team is small, but focused and works in a private group. This allows us to have a very high rate of churn and rapidly fix issues. During the 5.5 release, it was not uncommon to have four sets of ISO builds on the same day and yet have a relatively good level of QA feedback. At the moment our QA effort is limited to installation scenarios, upgrade and repository testing. What we would really love to do is expand this to include application functionality testing, but lack the manpower to do so. If anyone wants to adopt some applications and can devote some chunks of time a few times every year, please come and talk to us.

Our target is to get a new release out of the door within 4 weeks of upstream.

KS: Yes, CentOS makes for a great desktop and workstation distro. Like I said before, since its a stable platform you can freely add and remove components from the system using either your own builds or third party repositories, yet feel secure in the knowledge that the base operating stack isn't going to break or change on you in a long time. This really does transfer the upgrade cycle over from what is enforced by faster moving distros like Fedora and Ubuntu, over to the user where you can then choose when and how you want to upgrade what. It has been over three years since CentOS 5 came out, but I still find users running CentOS 4 happily with the latest versions of Firefox, MPlayer etc. Also, one needs to keep in mind that Red Hat do a lot of kernel backports, so while the main kernel version number does not change, new and better hardware support is always going in.

I know the existing repositories spread for CentOS isn't ideal at the moment. Users still need to make some efforts to locate the right ones for them, then might still need to contend with conflicting packages, etc. Putting in place a better system is quite high on the agenda, if anyone has ideas on what and how we could do that, please get in touch!