It's no secret that ATI Technologies has had a rough time in the past delivering display drivers that met the expectations of their customers. When ATI started out producing a FireGL and Radeon Linux driver they for some time were greatly behind NVIDIA's feature-rich driver. The early ATI Linux drivers had lacked essential functionality such as PCI Express and x86_64 architecture support and was also affected by stability and performance problems along with a great deal of bugs. However, over time many of these issues have been worked out and ATI has dedicated more resources to the advancement of their Linux software, but that's not to say they have been sailing smoothly. It had taken ATI approximately six months to deliver Radeon X1000 (R500) Linux product support and it is going on a year now that they have been lacking AIGLX support for Compiz and Beryl. Their current generation fglrx Linux driver is also handicapped when it comes to the performance capabilities with the fglrx driver performing much slower than the Windows ATI Catalyst driver and that of NVIDIA's driver. While ATI/AMD is working steadfast in addressing all of these issues and further enhancing their level of Linux support, many of their customers do not realize all of the work that goes into these drivers. You don't need to go far to find Linux users flaming about the lack of a particular feature or those bashing the driver over an experience they had years ago, but these consumers often do not realize the time invested into these drivers and that ATI/AMD is doing something about improving these drivers.
Last year when AMD announced their acquisition of ATI it led many to wonder how this would impact the quality of their Linux support and driver. Some had even speculated that AMD would be opening the code to at least a subset of their graphics drivers, and while this issue has come up again more recently, we will cover this particular topic in a different article. In this article we will be exposing what truly consists of the ATI/AMD driver development cycle and ultimately what they are really doing to improve their image in the Linux community. We have been granted unprecedented access to share with you their once unknown driver development model.
What's The Problem?
If you're new to the Linux scene or just have been out of the hardware loop for a while, you may want to check out some of our past Phoronix articles such as An Outcry For Improved ATI Linux Drivers, ATI Has Open-Source Drivers Too, and our ATI 2006 Year in Review to see where the ATI/AMD fglrx driver is coming from and where it's going. If you've been an ATI Linux customer for years, you may recall the times when just installing the driver was difficult and for many people their installation just turned into a frustrating failure at the hands of ATI. If you were one of the lucky ones to get the driver loaded, you then had a basically useless graphical control panel and were forced to modify all of the driver settings yourself through the xorg.conf. After all of the hardship to get the drivers installed and configured you then usually found out that the ATI Radeon graphics card you spent several hundred dollars on would perform about the same speed of a sub-$100 NVIDIA graphics card.
Today when it comes to binary display drivers in Linux, the ATI/AMD Linux installer surpasses that of NVIDIA's and the text-based aticonfig utility and the graphically pleasing AMD Catalyst Control Center make it a breeze for configuring your system whether it be a notebook or a multi-headed workstation. However, as was mentioned earlier, the ATI/AMD driver still has a number of issues that need to be (and will be) addressed when it comes to the driver's performance, AIGLX support, and a number of other different "gotchas". In the past year to two years ATI/AMD has made great progress in improving the quality of their Linux software through pushing out monthly driver updates and with what we have seen from AMD proves their dedication to the Linux customer base.
Before we begin we would like to first thank Advanced Micro Devices especially their Linux developers and Matthew Tippett (Linux Core Engineering) for these slides, their support, and allowing this material to become public information. This article has been in the works for several months now and these slides covering their Linux development overview for the AMD Graphics Products Group were finalized in April of 2007.
Timing a Release
Whether you know it or not, ATI Technologies had made the strategic decision to make the Linux driver part of their unified release cycle, but what does this mean? Well, these slides we are sharing with you today is the simplified version of what this unified development cycle means along with some other information to share with the public for the first time.
AMD provides updated display Catalyst drivers on a monthly basis for both their Windows and Linux customers, but these monthly drivers actually take about 11 weeks, or close to three months, to put together. Each release consists of pure development time for roughly one month, creating a branch for that specific release, and then carrying out quality assurance. After development has ended there is three weeks of validation, three weeks of beta testing, and then one week to bake. Any and all new features or fixes that AMD engineers would like in the release must be completed during the development stage. Granted, however, there are exceptions to that development rule for fixing any serious regressions as well as for the inclusion of distribution-specific packaging scripts. Community members with AMD's internal beta testing program maintain these packaging scripts and thus the scripts for each release are generally all updated during the beta testing stage. The bake stage in the development of the Catalyst driver is merely a buffer of time to ensure that the Windows and Linux drivers are completed and launched on the same day.
This schedule does also explain why new kernel and X.Org support isn't generally added the same month as its release. If a new kernel at the start of the month breaks fglrx support, that month's driver is already far into the validation and beta stages, which prevents engineers from appending support to the branched driver. Adding support for newer versions of the Linux kernel and X.Org in some cases require extensive modifications, more so than end-users may sometimes realize. However, with the distribution-specific packaging scripts not having set deadlines, and in some cases being updated a few days prior to the driver's public release, often times they will include non-official patches.
It is also important to keep in mind that while the AMD release notes in every driver may not be as long as an end-user would like, the developers are actually working on things all of the time. Like we had saw last year with the Dynamic Display Management Options and other features, there are times during the year when the official change-log may be short for a period of releases but that is when they are actually working on tackling some of the long-term objectives. Right now we seem to be going through a similar period with the Linux driver.
The Release Train
With each AMD graphics driver taking about 11 weeks from start to finish, in order to provide driver updates on a monthly basis we have the AMD release train. At any moment there are at least two driver releases under development. When one driver leaves development and steams forward into the validation process, the next driver enters development, and the process repeats itself. For instance, in May we saw the release of the 8.37.6 fglrx driver and development for next month's 8.38 driver has already ended and is in the validation stage currently but will be entering beta shortly. Meanwhile, development on the July 8.39 driver will be underway shortly.
If you're a FireGL owner, you may have noticed that the unified Linux driver on the FireGL series page is not updated on a monthly basis. According to Matthew Tippett as for the reason why it's not updated monthly, "Releases with a non-monthly cadence typically have considerably higher levels of targeted testing. In particular for the Workstation driver, there is more focus on the Workstation ISVs and certifications." For features that take longer than a month to implement, Tippett had went on to explain the following: "As expected high risk or experimental changes would be developed in separate trees, but long term features, architectural changes all happen while the release train is continuing. The analogy I like to use is that we are on the train, and to add a new carriage or update the carriage, we have to do it while the train is running, without stopping the train, or letting anything fall off."
It's a Continuous Build
In order to shorten the time to market for each driver almost every code check-in triggers a complete driver build in its tree. With AMD having switched to a unified driver package for their Linux x86 and x86_64 architectures, the 32-bit, 64-bit, and multiple X version support can be found all within the triggered build. Not only does AMD's unified packaging make it easier for the end-user who has multiple systems with different platforms, but it also provides similar benefits to AMD engineers when testing the driver. A regular automated build process is able to shorten the testing time since it provides a history for managing regressions and providing greater consistency between builds while removing developer tweaks from any build. Installing the driver no matter the platform is also carried out in just one step as opposed to manually installing everything.
Driver Testing
In addition to the beta testers, AMD relies upon a series of manual and automated tests for each driver. This testing begins in the validation cycle and continues with the beta, but AMD only tests each driver on their supported Linux distributions. Currently Novell's SuSE, Red Hat Enterprise Linux, and Red Flag are the only AMD supported Linux distributions. With the AMD beta program are testers that verify the support and provide packaging scripts for Debian, Fedora, Mandriva, Slackware, and Ubuntu. Outside of those that provide packaging scripts there are approximately thirteen distribution vendors that are involved with the closed beta program. If you are a distribution vendor and would like to join the closed beta program, please contact us and we can forward the information on to the appropriate representatives at AMD.
During the testing stage, AMD primarily tests the driver on the market requirements for their direct consumers that have made requests for Linux support, which are generally workstations and consumer desktops. While the number of actual developers working on the Linux driver is confidential, there are over 100 people involved with the closed beta program.
We had asked what makes up the automated testing process that the driver goes through during the development cycle and validation, and the response we had received by Matthew Tippett was: "We try to bring automation as much as possible into the testing, if it can be automated, it eventually will. At different stages through our test cycle, we focus testing differently to manage particular risks of that part of the development cycle. Likewise we balance the increased effort of manual testing with the automated testing based on the return of investment at any particular point in the development cycle."
What's The Competition Got?
While one may think that in this extremely competitive discrete graphics duopoly both NVIDIA and AMD would have relatively similar software development cycles, but from what we have been exposed to speaks quite to the contrary. The entire year we have seen only one formal Linux display driver release from NVIDIA for the GeForce series, along with two beta releases (100.14.03 and 100.14.06) and finally another legacy release (1.0-7185) for their older graphics cards. On the AMD side, however, there have been five Linux drivers this year with another seven expected by year's end. NVIDIA Corporation has no set in stone release cycle other than pushing out a new release when bug fixes or new features warrant an upgrade. We have come to expect a new Linux driver from NVIDIA every few months and it wasn't until last year that they had introduced public beta drivers. NVIDIA's FreeBSD and Solaris drivers follow a similar release schedule.
As far as the NVIDIA Linux driver itself goes, it is separated into independent x86 and x86_64 packages and no distribution-specific scripts are included with the mainstream driver. NVIDIA Corporation does not rely upon distribution vendors and community maintainers for allowing the end-user to generate distribution-specific packages, which AMD relies upon heavily.
So What Else? Random Remarks.
The ATI/AMD Linux community also has several assets, which include an unofficial BugZilla, an unofficial Wiki, a section on the Rage3D forums, and the Phoronix Forums. While the BugZilla and Wiki are unofficial, they are linked to from the AMD website and developers are known to browse the bug list from time to time.
On the NVIDIA side, there is an official section setup on the NvNews forums and an unofficial support and discussion area on the Phoronix Forums. At this time there is no public BugZilla for NVIDIA users.
While R200 product support still does exist in a driver branch after being discontinued in the fglrx 8.28 release, AMD has no plans to release an updated R200 class driver. AMD's belief behind this is the open-source community was given the needed specifications a number of years ago and the X.Org Radeon driver is reasonably well supported (except for TV-Out due to Macrovision).
If you're curious as to what Linux distributions the AMD developers use when working on these drivers, there is no single distribution other than matching the common choices of the Linux community. Matthew Tippett happens to be an obsessive-compulsive daily Ubuntu dist-upgrader tracking Ubuntu Gutsy Gibbon.
While the AMD beta program is closed for the public, deserving individuals are allowed to join the program. AMD is always looking for new distribution vendors to join this program with vendors being more than welcome to provide packaging scripts for their respective distribution. If you can provide a particular value to the beta testing cycle, let us know in the Phoronix Forums and we would be glad to pass along prospective names to AMD.
While AMD's Catalyst development cycle for both the Windows and Linux drivers is a bit more complicated than what we were allowed to share with you today, each AMD Linux "fglrx" driver release takes usually about eleven to twelve weeks from start to finish. With the development, validation, beta, and bake phases, there are always at least two releases being prepared. This rigid development cycle allows AMD to release updated drivers on a monthly basis while ensuring that each driver has been tested and contains more changes than just a simple version bump. It is also due to this structured development cycle why AMD generally cannot deliver same-month support for new kernels and X.Org releases.
If you have any additional questions on the AMD development cycle or would like to share your ATI/AMD experiences, check out the Phoronix Forums. Discuss this article in this thread.
Stay tuned to Phoronix as AMD should have some interesting things in the pipeline for later this year...
Discuss this article in our forums, IRC channel, or email the author. You can also follow our content via RSS and on social networks like Facebook, Identi.ca, and Twitter. Subscribe to Phoronix Premium to view our content without advertisements, view entire articles on a single page, and experience other benefits.
