Hint: Get the entire U.S. government onboard, and don’t test first
By the time the federal government buys an IT security product, it is several generations obsolete.It takes the Pentagon an average of 81 months to buy commercial-off-the-shelf (COTS) information technology security products, according to estimates by the Office of the Secretary of Defense and the deputy secretary of defense. Federal civilian agencies generally move no faster.
The Defense Department has hired several prestigious think tanks to study the problem at considerable expense — one report took several years to produce — and they have offered a plethora of recommendations about speeding up the acquisition of COTS cyberspace IT products.
But a solution already exists, at least in some instances. Over a span of just six months in 2006 and 2007, the DoD-led, intergovernmental Data At Rest Tiger Team (DARTT) created competitive blanket-purchase agreements for 10 commercial-off-the-shelf encryption products and licenses, and developed a process for technology upgrades for the five-year life of the contract.
The agreements created a menu of interoperable products for DoD; other federal, state and local government agencies; and even NATO’s C3 Agency to mix and match to fill their technical and security needs. Using the purchasing power of the entire U.S. government (plus post-award competition, huge buying pool and large-scale purchasing and provisioning), we drove prices to rock-bottom levels.
This article does not propose a pilot project to explore and validate a potential process. It describes an actual procurement and shows how the process could be replicated for other critical, time-sensitive procurements of relatively mature IT COTS security products. (By “relatively mature” I mean: characterized by neither immature cutting-edge still-in-development IT products made by numerous small companies nor commodity IT products generally identified by a technology segment featuring very few manufacturers with large, stable market shares.)
Horror stories
The years before DARTT was formed saw several high-profile incidents in which sensitive government data were lost or stolen. Numerous horror stories appeared in the press concerning lost or stolen laptops, smartphones, PDAs, CDs and removable hard drives. U.S. military officers purchased unencrypted U.S. military thumb drives in Afghan bazaars, lost or stolen from military bases. In summer 2006, the Veterans Affairs Department lost a laptop that contained personally identifiable information on more than 26 million veterans.
In late 2006, senior Pentagon leaders and the White House’s Office of Management and Budget (OMB) were preparing to issue policy requirements for sensitive government information to be encrypted wherever it was stored — so-called data-at-rest encryption — and to do so quickly. Agencies would have no time to allocate funds via multiyear planning; they would have to comply using currently available funds. Therefore it was critical that a supporting acquisition process was developed in conjunction with policy development that would allow government agencies to quickly purchase inexpensive data-at-rest encryption products at a very low cost.
Moreover, a search would have to be performed at lightning speed for a range of products that would fulfill the diverse requirements of the entire spectrum of government agencies yet remain interoperable and upgradable. Overall, the situation placed an unusual premium on speed of acquisition, cost containment, coordination and consensus, and a flexible technology refreshment/upgrade process.
So the Pentagon created DARTT, a tiger team that operated in conjunction with the DoD Enterprise Software Initiative (DoD ESI). The group was later joined by the General Services Administration’s SMARTBuy office, then, as mandated by OMB, the rest of the civilian agencies. Eventually, state and local governments would voluntarily join the acquisition program management and customer base.
Ultimately, the DARTT program management team and Source Selection Board included representatives from 20 DoD components; 18 federal civilian agencies; the New York State Chief Information Security Office, which represented state, tribal and local government agencies; and NATO’s C3 Agency. This breadth of intergovernmental management and customer base, unprecedented for a cyberspace/IT acquisition effort, was key to the program’s success.
Many agencies had already bought data-at-rest encryption products, and they worried that DARTT might pick a different one. Faced with steep replacement costs, agencies will fight each other in a cage death match to ensure that their favored product will be the awardee. But if more than one product is to be picked, it is possible — if still difficult — to get diverse agencies to cooperate and achieve consensus on a common set of minimal yet stringent technical standards. It requires product interoperability, which is not as difficult as commonly imagined but which does demand a careful ranking of technology priorities and equally careful construction of technical requirements.
Solving the dilemma
Supported by the Air Force’s 754th Electronics Systems Group (754th ELSG) at Maxwell-Gunter Air Force Base, Ala., DARTT’s OSD program manager and Air Force contracting officer chose a little-used approach to solve their acquisition dilemma: They would pursue it under the Part 8 of the Federal Acquisition Regulations instead of the customary Part 16.
This would allow the team to set up technical requirements, then award blanket-purchase agreements to any product that met them. Any agency — whether U.S. military and NATO via DoD ESI, or federal, state and local customers via GSA SMARTBuy — could buy one or more products from the list, either sole-source or through its own secondary acquisition process. DARTT decided to aim for nine products, plus or minus three, in order to provide a menu as diverse in technology, security, cost and maintenance characteristics as its customer base.
In December 2006, DARTT held an industry day conference in Reston, Va. Officials provided a threat briefing, a technology briefing and a draft of the technical requirements to representatives from more than 60 companies. In return, they asked the vendors to comment on the requirements. When comments came in, DARTT refined the requirements and sent them out for comment again. Then they repeated the process a third time. By February 2007, nearly 250 comments from industry had helped shape a final list of 106 technical requirements.
The formal request for bids went out in March. Vendors replied by April, and the DARTT Source Selection Board met for three weeks in May. The final decision was released in June, six months after the industry day.
Ten products made the grade and were made available for purchase. (The list included one manufacturer and nine resellers; several products were carried by multiple vendors and several vendors carried multiple products.)
No prior testing
Among the keys to the swift action: DARTT performed no formal testing of the products before they were picked. Instead, vendors were notified several months ahead of the formal request for bids that they would have to assess their own products’ ability to meet each of the technical requirements.
When DARTT officials described that aspect of the process to various U.S. government audiences, the wave of skepticism was almost overwhelming. Government officials were absolutely certain that the vendor community would not honestly evaluate their products’ ability to meet the technicial requirements.
Yet when the 10 products were evaluated after the award (each was tested twice; all 10 by the Air Force, five by the Army, and five by the Navy), only one product missed a technical requirement. That product failed only in very unusual circumstances, and when the manufacturer was notified, its officials were quite embarrassed and fixed the shortcoming within 48 hours.
How could we be confident in buying before testing? There were several reasons:
• We accepted only products that had already passed the government’s FIPS 140-2 encryption-validation process.
• We involved vendors early in the process, sought their help in shaping requirements and therefore issued no surprises when our formal request for bids went out.
• We required each prospective vendor to detail in writing in its proposal exactly how a product met our technical requirements — simple yes-or-no answers were insufficient. We also warned the vendors that failure to be honest could bar them from future federal contracts. This threat was particularly effective with resellers, who simply would not risk catastrophe by allowing a manufacturer to be less than truthful.
• The source selection board that evaluated the bids and self-assessments included 45 people from across the U.S. government, including numerous subject-matter experts who collectively brought hands-on experience with virtually all the proposed products. We also contacted other associated off-site government experts when we needed even more help.
The board ultimately evaluated 30 products, probably more than could have been individually tested by DoD. But the Pentagon could test 10 products and did so twice, and every single one passed.
We found that prospective vendors either met all of the requirements or they missed by a wide margin. The best of the nonselectees missed nine requirements, a gulf that underlined the quality and accuracy of the list and of the process.
Conclusions
By creating a limited menu of interoperable products meeting minimal yet stringent standards that the entire U.S. government would buy, then inducing competition both before and after awarding blanket-purchase agreements, we drove pricing to unprecedented depths.
To emphasize the huge quantity of sales that were coming, we told prospective vendors to provide pricing for purchases of 10,000, 33,000 and 100,000 user licenses. Indeed, the Agriculture Department soon bought 180,000 licenses for one product, while the Army later bought 150,000 licenses for another product with an option for up to 800,000 more. This was a drastic change to a technology niche that previously saw purchases of hundreds of licenses, not hundreds of thousands.
Prices responded accordingly. One higher-end product was priced at $256 per user license on the GSA schedule before the DARTT award, and $23 per user license under the new DARTT blanket-purchase agreement.
Another vendor’s product cost $100 per user license before 2007, and $1.20 afterward. Frankly, you can’t get a decent-size cup of coffee at a 7-Eleven for $1.20. But you can get a user license to encrypt sensitive government data on a laptop hard drive from the DARTT blanket-purchase agreement.
By December 2009, the OSD Program Management Office, using monthly reports from the 754th ELSG, calculated that about 3.5 million licenses had been purchased via the DARTT blanket-purchase agreement at a cost of $52 million by multiple customers across the federal, state and local spectrum of government agencies — saving more than $160 million from the pre-award GSA schedule pricing. The DARTT program received numerous awards for its savings, innovation, speed of acquisition and synchronization with policy development. Among them was DoD’s Excellence in Information Assurance Award for 2008, awarded by DoD’s chief information officer.
This process isn’t a panacea, but given the right circumstances, it can save tremendous amounts of money by using the purchasing power of the entire U.S. government, and drastically increase the speed of government cyberspace/IT acquisition.
DAVID HOLLIS was recently named chief of the J51 Strategy Division, U.S. Cyber Command. He was a senior policy analyst/planner with the Office of the Undersecretary of Defense for Intelligence’s Cyberspace, Warfighter Integration and Strategic Engagement Division. A lieutenant colonel in the Army Reserve, Hollis also serves as the senior Army Reserve officer/officer in charge for the Joint U.S. CYBERCOM USAR element.
Recent Comments