<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Times;
panose-1:2 2 6 3 5 4 5 2 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoFooter, li.MsoFooter, div.MsoFooter
{mso-style-link:"Footer Char";
margin:0in;
margin-bottom:.0001pt;
mso-para-margin-top:.01gd;
mso-para-margin-right:0in;
mso-para-margin-bottom:0in;
mso-para-margin-left:0in;
mso-para-margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{margin:0in;
margin-bottom:.0001pt;
mso-para-margin-top:.01gd;
mso-para-margin-right:0in;
mso-para-margin-bottom:0in;
mso-para-margin-left:0in;
mso-para-margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Times","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Tahoma","sans-serif";
color:black;}
span.FooterChar
{mso-style-name:"Footer Char";
mso-style-link:Footer;
font-family:"Times New Roman","serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:274750596;
mso-list-type:hybrid;
mso-list-template-ids:28476678 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:387068864;
mso-list-type:hybrid;
mso-list-template-ids:-1395098964 -2061076510 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Times New Roman","serif";
mso-fareast-font-family:"Times New Roman";}
@list l1:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2
{mso-list-id:580868256;
mso-list-type:hybrid;
mso-list-template-ids:-1973113918 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3
{mso-list-id:690690499;
mso-list-type:hybrid;
mso-list-template-ids:63764962 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l3:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4
{mso-list-id:836919849;
mso-list-template-ids:-963489924;}
@list l4:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l4:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5
{mso-list-id:1217425459;
mso-list-type:hybrid;
mso-list-template-ids:912584422 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l5:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l5:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6
{mso-list-id:1675062137;
mso-list-type:hybrid;
mso-list-template-ids:-1951754800 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l6:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l6:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l6:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7
{mso-list-id:1907523572;
mso-list-type:hybrid;
mso-list-template-ids:-242080564 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l7:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l7:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8
{mso-list-id:1955745684;
mso-list-type:hybrid;
mso-list-template-ids:-1604696014 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l8:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l8:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l8:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9
{mso-list-id:2080982687;
mso-list-type:hybrid;
mso-list-template-ids:-152139010 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l9:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l9:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><font size="3" color="black" face="Tahoma"><span style="font-size:12.0pt;font-family:"Tahoma","sans-serif";color:black">Capers Jones is compiling a catalog of software benchmark data sources as a public service for the software community.
If you publish or know of benchmarks that you would like included, please send them to Mr. Jones. Details in his note below …
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" color="black" face="Tahoma"><span style="font-size:12.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" color="black" face="Tahoma"><span style="font-size:12.0pt;font-family:"Tahoma","sans-serif";color:black">Hope all is good with you!<br>
<br>
Best wishes,<br>
Leon<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="1" color="black" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="2" color="black" face="Tahoma"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black;font-weight:bold">“Anything less than a comprehensive economic plan that addresses not only the budget deficit but
the underlying industrial, education, energy, tax and regulatory policies that have contributed to this crisis will leave us chasing our tails.” – Michael Lewitt, 1-August-2011.<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="1" color="black" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="1" color="black" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black">----------------------------------------------------------------------------------------------------------------------------------------------<br>
</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black">Leon A. Kappelman, Ph.D.</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black">
<br>
Professor of Information Systems <br>
Director Emeritus, Information Systems Research Center<br>
Fellow, Texas Center for Digital Knowledge<br>
College of Business, University of North Texas<br>
Voice: 940-565-4698 Fax: 940-565-4935 Email: </span></font><font size="2" color="black" face="Tahoma"><span style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="mailto:kapp@unt.edu"><font size="1" color="blue"><span style="font-size:8.0pt;color:blue">kapp@unt.edu</span></font></a></span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black"><br>
Website: </span></font><font size="2" color="black" face="Tahoma"><span style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://courses.unt.edu/kappelman/"><font size="1" color="blue"><span style="font-size:8.0pt;color:blue">http://courses.unt.edu/kappelman/</span></font></a></span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black"><br>
</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black">Founding Chair, Society for Information Management's Enterprise Architecture Working Group</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black"><br>
</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:black">----------------------------------------------------------------------------------------------------------------------------------------------</span></font><font size="1" color="black" face="Tahoma"><span style="font-size:8.0pt;font-family:"Tahoma","sans-serif";color:black"><br>
</span></font><font size="1" color="#1c7c00" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:#1C7C00">Please consider the environment before printing this e-mail</span></font><font size="1" color="#006666" face="Tahoma"><span style="font-size:9.0pt;font-family:"Tahoma","sans-serif";color:#006666"> </span></font><font size="2" color="black" face="Tahoma"><span style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="2" color="black" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" color="black" face="Tahoma"><span style="font-size:12.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></font></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";font-weight:bold">From:</span></font></b><font size="2" face="Tahoma"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
CJonesIII@cs.com [mailto:CJonesIII@cs.com] <br>
<b><span style="font-weight:bold">Subject:</span></b> Benchmark sources from Capers Jones<o:p></o:p></span></font></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:.5in"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Leon,<br>
<br>
I'm planning to publish a catalog of benchmark sources. If you or any colleague publish benchmarks and want to be included, please write up what is available using a format similar to the
<font color="black"><span style="color:black">ones</span></font> already here.<br>
<br>
This work is just starting out<font color="black"><span style="color:black">. </span>
</font><br>
<br>
Thanks,<br>
Capers Jones<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" color="black" face="Tahoma"><span style="font-size:12.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></font></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">SOURCES OF SOFTWARE BENCHMARKS<o:p></o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Version 5.0 August 6, 2011<o:p></o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Capers Jones & Associates LLC.<o:p></o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">INTRODUCTION<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Number of benchmark sources currently: 8<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Quantitative software benchmark data is valuable for measuring process improvement programs, for calibrating software estimating tools, and
for improving software quality levels.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">There are a number of organizations that gather and report quantitative benchmark information. However these organizations are independent
and in fact some of them are competitors.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">This catalog of software benchmark data sources is produced as a public service for the software community.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">There are many different kinds of benchmarks including productivity and quality levels for specific projects; portfolio benchmarks for large
numbers of projects, operational benchmarks for data center performance; security benchmarks, compensation and staffing benchmarks for human resource purposes; and software customer satisfaction benchmarks.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">This catalog is intended to grow over time and to include all major sources of software benchmark information.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The information in this catalog is provided by the benchmark groups themselves and includes topics that the benchmark groups wish to be made
available. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In this version the benchmark groups are listed in alphabetical order. In later versions the groups will be listed alphabetically for reach
type of benchmark. <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p></o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">TABLE OF CONTENTS<o:p></o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Capers Jones & Associates LLC 3<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="2" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Galorath Incorporated 8<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="3" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">International Software Benchmark Standards Group (ISBSG) 9<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="4" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Price Systems LLC 12<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="5" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Process Fusion 15<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="6" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Quantimetrics 16<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="7" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">RCBS, Inc. 18<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="8" type="1">
<li class="MsoNormal" style="mso-list:l5 level1 lfo1"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Benchmarking Organization 21<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Appendix A: Survey of Software Benchmark Usage and Interest 22<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Appendix B: A New Form of Software Benchmark 23<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Capers Jones & Associates LLC<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL: Under construction<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:Capers.Jones3@gmail.com">Capers.Jones3@gmail.com</a><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> Primarily on-site interviews of software projects. Much of the data<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> is collected under non-disclosure agreements. Some self-reported
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> data is included from Capers Jones studies while working at IBM<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> and ITT corporations. Additional self-reported data from clients<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> taught by Capers Jones and permitted to use assessment and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> benchmark questionnaires.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> Productivity data is expressed in terms of function point<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> metrics as defined by the International Function Point<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> User’s Group (IFPUG). Quality data is expressed in<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> terms of defects per function point.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Also collected is data on defect potentials, defect removal efficiency, delivered
defects, and customer defect reports at 90<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> day and 12 month intervals.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Long-range data over a period of years is collected from a small group of clients to
study total cost of ownership (TCO) and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> cost of quality (COQ). Internal data from IBM also used for<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> long-range studies due to author’s 12 year period at IBM..<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> At the request of specific clients some data is converted<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> into COSMIC function points, use-case points, story points,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> or other metrics.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Data is used to create software estimating tools and predictive<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> models for risk analysis. Data is also published in a number of<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> books including
<u>The Economics of Software Quality</u>, <u>Software<o:p></o:p></u></span></font></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:1.0in"><u><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Engineering Best Practices</span></font></u>,
<u>Applied Software Measurement</u>,<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<u>Estimating Software Costs </u> and 12 others. Data has also<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> been published in about 200 journal articles and monographs.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is provided to specific clients of assessment, baseline, and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> benchmark studies. These studies compare clients against similar<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> companies in the same industry.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data from Capers Jones is frequently cited in software litigation<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> for breach of contract lawsuits or for suits alleging poor quality.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some data is also used in tax litigation dealing with the value of<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> software assets.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Data is provided to clients of assessment and benchmark studies.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> General data is published in books and journal articles.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Samples of data and some reports are available upon request.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Software productivity levels and software quality levels<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">for projects ranging from 10 to 200,000 function points.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is primarily for individual software projects, but some<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">portfolio data is also collected. Data also supports activity-based<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">costing down to the level of 40 activities for development<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">and 25 activities for maintenance. Agile data is collected<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">for individual sprints. Unlike most Agile data collections<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">function points are used for both productivity and quality.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Some data comes from commissioned studies such as an<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Air Force contract to evaluate the effectiveness of the CMMI<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">and from an AT&T study to identify occupations employed
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">within large software labs and development groups.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> About 13,500 projects from 1978 through today.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> New data is added monthly. Old data is retained,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> which allows long-range studies at 5 and 10-year<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> intervals. New data is received at between 5 and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> 10 projects per month from client interviews.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from systems and embedded software, military<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> software, commercial software, IT projects, civilian<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> government projects, and outsourced projects.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Industries include banking, insurance, manufacturing,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> telecommunications, medical equipment, aerospace,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> defense, and government at both state and national levels.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is collected primarily from large organizations with<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> more than 500 software personnel. Little data from small<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> companies due to the fact that data collection is on-site and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> fee based.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Little or no data from the computer game industry or<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> the entertainment industry. Little data from open-source<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> organizations.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Data is collected for a variety of methodologies including<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Agile, waterfall, Rational Unified Process (RUP), Team<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Software Process, (TSP), Extreme Programming (XP),<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> and hybrid methods that combine features of several methods.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some data is collected on the impact of Six Sigma, Quality<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Function Deployment (QFD), formal inspections, Joint<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Application Design (JAD), static analysis, and 40 kinds of
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> testing.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is also collected for the five levels of the Capability<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Maturity Model Integrated (CMMI™) of the Software<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Engineering Institute.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: As is usual with large collections of data a variety of<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> programming languages are included. The number of<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> languages per application ranges from 1 to 15, with an<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Average of about 2.5. Most common combinations<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Include COBOL and SQL and Java and HTML.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Specific languages include Ada. Algol, APL,ASP Net, BLISS,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> C, C++, C#, CHILL. CORAL, Jovial, PL/I and many<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Derivatives, Objective-C. Jovial, and Visual Basic..<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> More than 150 languages out of a world total of 2,500<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> are included.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> About 80% of the data is from the U.S. Substantial data<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> From Japan, United Kingdom, Germany, France, Norway,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Denmark, Belgium, and other major European countries.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some data from Australia, South Korea, Thailand, Spain, and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Malaysia.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Little or no data from Russia, South America, Central America,<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">China, India, South East Asia, or the Middle East.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Unique data</span></font></b>: Due to special studies Capers Jones data includes information<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> on more than 90 software occupation groups and more than 100<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> kinds of documents produced for large software projects. Also,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> the data supports activity-based cost studies down to the levels
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> of 40 development activities and 25 maintenance tasks. Also<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> included are data on the defect removal efficiency levels of<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> 65 kinds of inspection. static analysis, and test stages.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some of the test data on unit testing and desk checking came<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> from volunteers who agreed to record information that is<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> normally invisible and unreported. When working as a
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> programmer Capers Jones was such a volunteer.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> From longitudinal studies during development and after release<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> the Jones data also shows the rate at which software requirements<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> grow and change during development and after release. Monthly<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> change rates exceed 1% per calendar month during<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> development and more than 8% per year after release.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> From working as an expert witness in 15 lawsuits, some special<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> data is available on litigation costs for plaintiffs and defendants.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> From on-site data collection and carrying out interviews<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> With project teams and then comparing the results to
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Corporate resource tracking systems, it has been noted<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> that “leakage” or missing data is endemic and approximates<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> 50% of actual software effort. Unpaid overtime and the<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> work of managers and part-time specialists are most common.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Quality data also leaks and omits more than 70% of internal<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> defects. Most common omissions are those of desk checking,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> unit testing, static analysis, and all defect removal activities<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> prior to release.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Leakage from both productivity and quality data bases inside<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Corporations makes it difficult to calibrate estimating tools and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> also causes alarm to higher executives when the gaps are revealed.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> The best solution for leakage is activity-based cost collection.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Future data</span></font></b>: There are several critical areas which lack good sources of<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> quantitative data. These include studies of data quality,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> studies of intangible value, and studies of multi-national<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> projects with geographically distributed development
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> locations.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:
</span></font></b>Capers Jones has been collecting software data since working<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> for IBM in 1978. In 1984 he founded Software Productivity<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Research and continued to collect data via SPR until 2000.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Capers Jones & Associates LLC was formed in 2001.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> He owns several proprietary data collection questionnaires
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> that include both qualitative assessment information and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> quantitative data on productivity and quality. The majority<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> of data comes from on-site interviews with software project<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> teams but self-reported data is also included, especially from<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> clients who have been trained and authorized to use the<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Jones questionnaires.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> More recently remote data collection has been carried<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> out via Skype and telephone conference calls using shorter forms
of the data collection questionnaires. <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some self-reported or client-reported benchmark data<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> is included from companies taught by Capers Jones and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> from consortium members.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Some self-reported data is also included from internal<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> studies carried out while at IBM and ITT, and also<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> from clients such as AT&T, Siemens, NASA, the Navy, and<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> the like.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Galorath Incorporated<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL: www.galorath.com<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:info@galorath.com">info@galorath.com</a><o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> Repositories obtained through product customers, industry sources,
and public domain and consulting. Galorath also maintains a partnership with ISBSG and offers the ISBSG data in SEER Historical Database (SEER-HD) format to subscribers.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> Productivity expressed in function points and source lines
of code. Most data has language, platform and application descriptors. Some of the data also contains, dates, defects delivered, time phased staffing, documentation pages, detailed SEER parameter settings.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Data is used to calibrate SEER knowledge bases, determine
language factors, generate trend lines, benchmark organizations, and validate model relationships.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Data is delivered in the form of knowledge bases, language
factors, trend lines and estimating relationships to SEER customers. Where permitted data is provided in source form so SEER users can see their estimates compared with individual data points. SEER customers who submit data to Galorath receive their data
in the SEER Historical Database format that can be used directly by the application.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Records of completed projects at the program and project level.
Size ranges from extremely small (less than one month duration) to very large (hundreds of staff, many years’ duration.) Additionally metadata is available from some sources, showing items such as productivity ranges, size ranges, etc. where management is
not willing to release raw data.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> Over 20,000 records with nearly 13,000 total records containing
at least size, in addition to other metrics of which approximately 8500 records contain completed effort and a lesser number, project duration. Volume increases regularly.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from IT/business systems, industrial systems and embedded
software, military, and commercial enterprises. <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Project processes in collected data range from traditional waterfall
through incremental and agile development.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> Military and aerospace data is US and European. Commercial
data is worldwide.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:
</span></font></b>Galorath maintains an active data collection program and has done so for the last 20 years. It makes data available to users of SEER for Software in several forms such as its ProjectMiner data mining tool and Metrics and Benchmarking data
visualization tool.<o:p></o:p></p>
<p class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">ISBSG Limited<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL: www.isbsg.org<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:Capers.Jones3@gmail.com">peter.r.hill@isbsg.org</a><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> Voluntary submission of software projects data. The
data is collected under an agreement of anonymity. Data is submitted either directly or by consultants with the permission of their customers.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> Productivity data is expressed in terms of function
point metrics as defined by the International Function Point User’s Group (IFPUG); NESMA; COSMIC; and FiSMA. Quality data is expressed in terms of defects per function point or functional unit.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">LOC sizes are stored if provided but is not validated and not used for any analysis.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Also collected is data on the report of defect occurrence during the project and within the first 30 days of operation.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data has come from more 25 countries and 20 major organization types<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Data is used to create some basic ISBSG software estimating
tools; used by commercial estimation tool vendors; and for benchmarking services. The data is also analysed and published in a number of analysis reports and books including
<i><span style="font-style:italic">Practical Software Project Estimation.</span></i><o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is provided to academics for use in approved research work.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> The ISBSG data is ‘open’ and can be licensed by anyone
in its ‘raw’ form at a moderate cost.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">General data and analysis results are published in books and journal articles. Samples of data and some reports are available upon request.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Data on both Development and Enhancement projects.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Projects ranging from 10 to 10,000 function points in size, but with an average size around 300FP.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is primarily for individual software projects, but some<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">portfolio data is also collected.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> About 6,000 projects from 1989 through today. New data
is added monthly. Old data is retained, allowing long-range studies in four-year groups..<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from MIS commercial software; real-time systems;
IT projects; government projects; and outsourced projects; and Off-shore projects.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Industries include banking; insurance; legal; manufacturing;<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Telecommunications; Accounting; Sales; Transport; Government; and public sector.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is submitted from a wide range of organizations of differing sizes. There is no data from the military.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Data is collected for a variety of methodologies including
Agile, Waterfall, Joint Application Design (JAD), Rational Unified Process (RUP), Team Software Process, (TSP), and hybrid methods that combine features of several methods.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Some data is collected on the impact of being compliant with the CMM and CMMI five levels and relevant ISO standards.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> About 30% of the data is from the U.S. 16% from Japan;
16% Australia; 10% Finland; 8% Netherlands; 6% India; 5% Canada; 2% China. Also data from 12 other countries.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:0in;margin-left:106.35pt;margin-bottom:.0001pt;text-indent:-106.35pt">
<b><font size="3" face="Times"><span lang="EN-AU" style="font-size:12.0pt;font-weight:bold">Summary:</span></font><span lang="EN-AU">
</span></b><font size="3" face="Times New Roman"><span lang="EN-AU" style="font-size:12.0pt;font-family:"Times New Roman","serif"">The ISBSG's formation in 1997 was built upon several years of previous cooperation by a group of national software metrics associations
with the common aim of developing and promoting the use of IT industry history data to improve software processes and products, for the benefit of both businesses and governments worldwide.<o:p></o:p></span></font></p>
<p style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:0in;margin-left:106.35pt;margin-bottom:.0001pt;text-indent:-106.35pt">
<font size="3" face="Times New Roman"><span lang="EN-AU" style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></font></p>
<p style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:0in;margin-left:106.35pt;margin-bottom:.0001pt">
<font size="3" face="Times New Roman"><span lang="EN-AU" style="font-size:12.0pt;font-family:"Times New Roman","serif"">Current members of the ISBSG represent IT and Metrics Associations based in the following countries: Australia, China, Finland, Germany,
India, Italy, Japan, Korea, Netherlands, Spain, Switzerland, UK, USA.<o:p></o:p></span></font></p>
<p style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:0in;margin-left:212.7pt;margin-bottom:.0001pt;text-indent:-106.35pt">
<font size="3" face="Times New Roman"><span lang="EN-AU" style="font-size:12.0pt;font-family:"Times New Roman","serif"">Two independent repositories of IT industry data:</span></font><font size="3" face="Times New Roman"><span lang="EN-AU" style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p></o:p></span></font></p>
<p class="MsoNormal" style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:.1pt;margin-left:212.7pt;mso-margin-top-alt:.01gd;mso-para-margin-right:0in;mso-para-margin-bottom:.01gd;mso-para-margin-left:212.7pt;text-indent:-106.35pt;mso-list:l4 level1 lfo2">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">1.<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Software Development and Enhancement – over 5,700 projects
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:.1pt;margin-right:0in;margin-bottom:.1pt;margin-left:212.7pt;mso-margin-top-alt:.01gd;mso-para-margin-right:0in;mso-para-margin-bottom:.01gd;mso-para-margin-left:212.7pt;text-indent:-106.35pt;mso-list:l4 level1 lfo2">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">2.<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Software Maintenance and Support – ~500 applications
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p></o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">PRICE Systems, LLC<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">www.pricesystems.com<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email: arlene.minkiewicz@pricesystems.com<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data comes from many sources. Proprietary data is collected from clients through consulting engagements, research engagements and training.
Publically available data is purchased from external sources. Public data is also mined from sources published technical reports and articles as well as reports in the media.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data collected include software size (in Source Lines of Code (SLOC), Function Points or Use Cases), project effort, project cost, various technical
parameters about the nature of the product being developed, the development team, and the developing organization.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Schedule duration for projects is collected as well.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>:
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is used to for creation, verification and validation of estimating relationships for cost, effort and schedule in the TruePlanning for Software
product. Data is used to update knowledge bases which provide user guidance through various means within the product.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data collected is also used to support research efforts, internally and externally funded, focused on specific areas of software development,
technology, or process.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Raw data is generally unavailable due to its proprietary nature, summary data is available upon request<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software project data for projects (or project components) ranging from 1000 – 3,000,000 SLOC and projects (or project components) ranging from
25 – 1,000,000 Function Points. Data is primarily for individual software projects – some data is provided down to the component level, others are higher level project data.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data collected from more than 3500 projects from 1977 through today. New data is added monthly. Old data is retained.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data from systems and embedded software, military software, real time software, commercial software, IT projects, MIS systems, civilian government
projects, and outsourced projects. <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Industries include banking, insurance, manufacturing, accounting, transportation, telecommunications, aerospace, defense, and government at
both state and national levels.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is collected primarily from large organizations or agencies.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>:
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is collected for a variety of methodologies and development process models.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: Data collected covers a wide range of programming languages.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b>
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">About 85% of the data comes from sources in the US. Additional data is provided from mostly European sources.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Unique data:
<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In addition to data that we have collected from our clients and other sources, PRICE also has been collecting data from our own software development
efforts over the last 30+ years we have been developing software applications. Having been an agile shop for much of the last decade, this data includes many of the common productivity metrics associated with agile development<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:
<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">PRICE Systems has been collecting software data since the mid 70’s during the research and development of the PRICE Software model, which has
evolved into the TruePlanning for Software model. Data collection continues to support regular upgrades to the software cost, schedule and effort estimating ability in this model.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">PRICE Systems is a global leader of cost management solution in Aerospace, Defense, Space and Information Systems, serving over 250 customers
worldwide.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Process-Fusion.net<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">URL:
<a href="http://www.Process-fusion.net">www.Process-fusion.net</a><o:p></o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Email: ggack@process-fusion.net</span></font></b><o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> Approximately 75% of this data set comes from formal inspections
training event workshops in which participants inspect actual work products prepared prior to the workshop. Approximately 25% of the data are from post-training inspections of work products.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> This data set includes approximately 1,600 individual inspections
conducted by team of 3 or 4 in most cases. All of these inspections are consistent with IEEE Std. 1028-2008. Data elements collected for each inspection include work product type (requirements, design, code), size in lines, orientation hours, preparation hours,
team meeting hours, number of major and minor defects identified, and post-correction review hours.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Summaries of these data may be used as a basis for estimating
the number of defects likely to be found in the several work product types identified, the number of hours per defect (or defects per hour) likely to be required when using formal inspections consistent with the process defined by IEEE Std. 1028-2008 (a.k.a.
“Fagan-style Inspections”). Standard deviation and distribution data are also available.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Summary data are available from the author upon request.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> Approximately 1,600 inspections.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from systems and embedded software, military<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">software, commercial software, IT projects and outsourced projects.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is collected primarily from large organizations with<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> more than 500 software personnel.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: A variety of programming languages are included.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> About 80% of the data is from the U.S.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">QuantiMetrics<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL:
<a href="http://www.quantimetrics.net">www.quantimetrics.net</a> <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:bram@quantimetrics.net">bram@quantimetrics.net</a> <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> Organisations (public and private) who subscribe to the
software benchmarking services of QuantiMetrics. Data submitted is subject to rigorous consistency, validity and reasonableness checks; measurement practices of subscribers are pre-audited to avoid potential data problems. In many instances QuantiMetrics
carry out the size measurements using certified staff; where size measures are provided by subscribers, these are audited by QuantiMetrics.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> Project data: size measures; resource and time use by
development phase, resource use by activity type; costs; quality measures by quality step; methods, languages/tools and platforms used; project team staffing and experience; qualitative factors deemed to have affected performance. Measures reflecting plans
at significant mile-stones, are also collected.<br>
<br>
Applications data: size measures; annual resource used by activity type; volumes of changes, change and support requests; costs; application support team experience and ratings on maintainability and other factors; languages/tools and platforms used; application
age; application instances/versions; user population; support window<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Applications project and support performance benchmarking
and estimating; at least a third of instances this data has been used in the context of outsourcing contracts<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Data is deemed to be proprietary – participants are provided
with like-for-like benchmarks for all measures they provide for their projects and applications. Anonymity is preserved at all times.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Project and Application support for business applications<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> Data accumulated goes back to mid 1980s; early data largely
based on line-of-code measures of size, while use of IFPUG functional size measurement dominant from 1990s and onward. The number of projects accumulated in total is about 10,000, yet benchmarks are only based on the most recent 3,000 or so projects, of
which a large proportion are post Y2K. Applications measured number about 1,000.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Industry participants are largely from the private sector,
with financial institutions the most dominant, then telecoms businesses although the applicability of the benchmarks is appropriate for all business type projects.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Project data includes information on approaches and methodology
(proprietary or generic type). In the case of Agile-based methods ratings of the degree of adoption of practices also obtained.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: No restriction is placed on the development languages
(tools) used; where statistically reliable data is available benchmarks are language specific, otherwise they are based on type of language/tool. Early submissions were dominated by third generation languages, followed by a period when a variety 4<sup>th</sup>
generation tools was the norm, while now object oriented languages and “visual” tools are more the norm. Projects also include those that configure and roll out business applications software.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:
</span></font></b>Largest and dominant contributions are from UK, Germany, South Africa and India<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Unique data</span></font></b>: QuantiMetrics use statistical techniques to model and
normalize performance, particularly for the effects of: size and time in the case of projects; size, age and user population for applications support data.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Future data</span></font></b>: Expect to see more use of COSMIC as a size measure<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:113.4pt;text-indent:-113.4pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:</span></font></b> QuantiMetrics have worked to ensure the database contains
good-quality data and that the most statistically reliable like-for-like benchmarks are provided. Reports and presentations are aimed to identify systemic cause-effect relationships and, in particular, to suggest best practices and opportunities for improvement.<o:p></o:p></p>
<p class="MsoNormal" style="page-break-before:always"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">RBCS, Inc.<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL: www.rbcs-us.com<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:info@rbcs-us.com">info@rbcs-us.com</a><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> On-site interviews of software testing and quality assurance
teams and groups within larger organizations that develop software and/or systems for internal use or for sale to customers, or which provide testing and quality assurance services. Discussions with clients in public and private training sessions. The data
is collected under non-disclosure agreements, and thus only anonymized and collective data is available for comparative purposes.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> The metrics are some two dozen testing and quality related
process, product, and project metrics. Examples include: defect detection effectiveness, cost per defect found, percentage coverage of risks and/or requirements, percentage of invalid defect reports, percentage of re-opened defect reports, consistency of test
case documentation, accuracy of test estimates, percentage overhead of test environment setup/maintenance, and various others.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Data is used to assess the capability and process improvement
potential of organizations involved in creation of software or systems for their own use or for use by clients/customers, and of organizations engaged in providing testing services. Data and metrics are mentioned in some of Rex Black’s book, such as
<i><span style="font-style:italic">Managing the Testing Process</span></i>, <i><span style="font-style:italic">Advanced Software Testing: Volume 2</span></i>, and
<i><span style="font-style:italic">The Expert Test Manager</span></i> (scheduled for publication in 2012). Data is also used in many of Rex Black’s presentations and articles.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is provided to specific clients of assessment, baseline, and benchmark studies. These studies compare clients against similar companies.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is discussed in various RBCS training courses, especially test management courses.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is also discussed in various RBCS free webinars on topics related to test management.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Data is provided to clients of assessment and benchmark studies.
General data is published in books and journal articles.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Software and system quality and testing capability measurements
focused on project, process, and product. People-focused metrics are discussed in consultancy but not provided for general use, due to the potential for problems with such metrics.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data comes from work with clients on assessments of their projects, processes, and products, as well as work with training clients.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> RBCS has over 200 clients and has been in the software testing
and quality assurance training, consulting, and outsource business since 1994.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from systems and software projects spanning almost all
applications of software and hardware/software systems, with no particular emphasis in terms of industry. Both private and government clients are included.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data is collected from organizations ranging in size from small companies with less than a dozen employees to some of the largest organizations
in the world (e.g., Sony, CA, etc.) <o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Due to the fact that assessments are paid consulting services, open source data is not included.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Data is collected for a variety of methodologies including Agile,
waterfall, Rational Unified Process (RUP), Scrum, Extreme Programming (XP), and hybrid methods that combine features of several methods.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data covers all types of testing, including functional testing, performance testing, usability testing, internationalization testing, manual
testing, automated testing, regression testing, and so forth. <o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: Many programming languages are included, but some of the older
languages such as COBOL and FORTRAN are under-represented. <o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> RBCS clients are distributed widely around the world, and
include many multinationals. Emerging software testing countries are under-represented in some cases, but RBCS is heavily involved in software testing and quality training and consulting with emerging economies such as India, Bangladesh, Kenya, Nigeria, South
Africa, Malaysia, and China. <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Unique data</span></font></b>: RBCS data is unique in the range of test-related metrics
and data available to clients. While other consultancies provide proprietary test assessment and/or “maturity” services, many of these are almost entirely questionnaire-based and/or interview based, and thus suffer from the Rashomon Effect. RBCS assessments,
being evidence-focused and data-focused, provide deeper insights to clients. In addition, RBCS assessments are based on an open-source framework published in Rex Black’s book
<i><span style="font-style:italic">Critical Testing Processes</span></i>.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Future data</span></font></b>: RBCS intends to continue to provide quality and testing
assessment services to clients around the world, and is working to expand the depth of the data through more assessments. RBCS is also considering developing an online assessment process using its proven e-learning platform.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:
</span></font></b>RBCS is unique in test consultancies in being focused on an evidence-based and metrics-based approach to assessment, which provides clients with reliability benchmarks and assessment results. RBCS has been in the quality and testing consulting
business since 1994, and founder Rex Black’s background in software engineering extends back to 1983.<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:-1.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Software Benchmarking Organization<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web site URL: www.sw-benchmarking.org<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Email:
<a href="mailto:hans.sassenburg@sw-benchmarking.org">hans.sassenburg@sw-benchmarking.org</a><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Sources of data:</span></font></b> On-site benchmark studies and assessments of software
projects. Data is always collected under non-disclosure agreements. <o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> <o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data metrics:</span></font></b> Data is collected for the following metrics:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Schedule (calendar months)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Effort (person months)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Productivity (function points per hour using IFPUG definition)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Cost of Quality (effort distribution over 4 areas)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Deferral rate (ratio of deferred baselined features)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Feature size (function points)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Technical size (KLOC)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Re-use level (ratio)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Complexity (using McCabe definition)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Test coverage (unit, integration, system testing)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Defect density (defects per function point at release time)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>Defect removal efficiency (ratio of defects removed before releasing)<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data usage</span></font></b>: Data is used:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>to benchmark “engineering capability” against industry averages and best-in-class figures in same industry, using data from SBO assessments as well as published data from other sources (like Capers Jones)<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:127.6pt;text-indent:-.25in;mso-list:l1 level1 lfo3">
<![if !supportLists]><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><span style="mso-list:Ignore">-<font size="1" face="Times New Roman"><span style="font:7.0pt "Times New Roman"">
</span></font></span></span></font><![endif]>to assess the feasibility of initial and running projects<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Data availability:</span></font></b> Data is provided to clients of studies and assessments<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Kinds of data:</span></font></b> Data is primarily for individual software projects<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Volume of data:</span></font></b> About 150 projects from 2001 through today.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> New data is added frequently.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Industry data:</span></font></b> Data from systems and embedded software, military<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> software, commercial software, IT projects, civilian<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> government projects, and outsourced projects.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Industries include banking, insurance, manufacturing,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> telecommunications, medical equipment, aerospace,<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> defense, and government at both state and national levels.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is collected primarily from projects with 10 – 75 software engineers
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Methodology data</span></font></b>: Data is collected for a variety of methodologies including<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Agile, waterfall, Rational Unified Process (RUP), and hybrid methods that combine
features of several methods.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Data is also collected for the five levels of the Capability<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Maturity Model Integrated (CMMI™) of the Software<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Engineering Institute.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Language data</span></font></b>: As is usual with large collections of data a variety of<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> programming languages are included. The number of<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> languages per application ranges from 1 to 5, with an<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> average of about 2. Most common combinations<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> include Assembler, C, C++, C# and Java.
<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Country data:</span></font></b> Most of the data is from Western Europe (including The
Netherlands, Germany, France, Switzerland, United Kingdom) with a strong focus on the embedded software industry.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> Limited data from other countries/regions.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:1.0in;text-indent:.5in"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Unique data</span></font></b>: The collected data is used to compute 16 KPIs, arranged
in four different categories. This enables an organization to identify root causes of underperformance and estimate the effect of corrective measures.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Future data</span></font></b>: Currently, limited data is available regarding complexity
and test coverage. This will receive more attention.<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary:
</span></font></b>SBO has been collecting software data since working 2001 using benchmarking and assessment studies. SBO uses a network of accredited partners.
<o:p></o:p></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="margin-left:106.35pt;text-indent:-106.35pt"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> In addition, SBO delivers metrics workshops as well as supporting technology for
metrics collection and analysis.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">APPENDIX A: SURVEY OF BENCHMARK USAGE AND INTERESTS<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="margin-left:4.65pt;border-collapse:collapse">
<tbody>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><b><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";font-weight:bold">SOFTWARE BENCHMARK USAGE SURVEY<o:p></o:p></span></font></b></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold"><o:p> </o:p></span></font></b></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Version 2.0 3/7/2010<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">No<o:p></o:p></span></font></b></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Used<o:p></o:p></span></font></b></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Would<o:p></o:p></span></font></b></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Interest<o:p></o:p></span></font></b></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Today<o:p></o:p></span></font></b></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Use if<o:p></o:p></span></font></b></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold"><o:p> </o:p></span></font></b></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">Available<o:p></o:p></span></font></b></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">"0"<o:p></o:p></span></font></b></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">"1"<o:p></o:p></span></font></b></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="right" style="text-align:right;layout-grid-mode:char">
<b><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";font-weight:bold">"2"<o:p></o:p></span></font></b></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="center" style="text-align:center;layout-grid-mode:char">
<font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" align="center" style="text-align:center;layout-grid-mode:char">
<font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">1. Portfolio benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">2. Industry benchmarks (banks, insurance, defense, etc.)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">3. International benchmarks (US, UK, Japan, China, etc.)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">4. Application class bench marks (embedded, systems, IT, etc.)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">5. Application size benchmarks (1, 10, 100, 1000, function points etc.)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">6. Requirements creep benchmarks (monthly rates of change)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">7. Data center and operations benchmarks (availability, MTTF, etc.)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">8. Data quality benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">9. Data base volume benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">10. Staffing and specialization benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">11. Staff turnover and attrition benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">12. Staff compensation benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">13. Organization structure benchmarks (matrix, small team, Agile, etc.)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">14. Development productivity benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">15. Software quality benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">16. Software security benchmarks (cost of prevention, recovery, etc.)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">17. Maintenance and support benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">18. Legacy renovation benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">19. Total cost of ownership (TCO) benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">20. Cost of quality (COQ) benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">21. Customer satisfaction benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">22. Methodology benchmarks (Agile, RUP, TSP, etc.)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">23. Tool usage benchmarks (Project management, static analysis, etc.)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">24. Reusability benchmarks (volumes of various reusable deliverables)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">25. Software usage benchmarks (by occupation, by function)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">26. Outsource results benchmarks (domestic)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">27. Outsource results benchmarks (international)<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">28. Schedule slip benchmarks
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">29. Cost overrun benchmarks<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">30. Project failure benchmarks (from litigation records)
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">31. Litigation cost benchmarks - breach of contract
<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">32. Litigation cost benchmarks - taxable value of software<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">33. Litigation cost benchmarks - non competition violations<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">34. Litigation cost benchmarks - damages due to poor quality<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt"><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
<tr height="17" style="height:12.75pt">
<td width="372" height="17" valign="bottom" style="width:279.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="text-indent:9.0pt;layout-grid-mode:char"><font size="1" face="Times New Roman"><span style="font-size:9.0pt">35. Litigation cost benchmarks - intellectual property<o:p></o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="64" height="17" valign="bottom" style="width:48.0pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="1" face="Arial"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
<td width="73" height="17" valign="bottom" style="width:54.75pt;padding:0in 5.4pt 0in 5.4pt;height:12.75pt">
<p class="MsoNormal" style="layout-grid-mode:char"><font size="2" face="Arial"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></font></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold"><o:p> </o:p></span></font></b></p>
<font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br clear="all" style="page-break-before:always">
</span></font>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">APPENDIX B: A NEW FORM OF SOFTWARE BENCHMARK<o:p></o:p></span></font></b></p>
<p class="MsoNormal" align="center" style="text-align:center"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Introduction<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Normally software assessments and benchmarks are provided to specific companies and compare selected applications against similar applications
from other companies in the same industry. This is useful information, but it does not provide any context or any information about the industry itself.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">What would be both useful and salable would be a new kind of benchmark that would consolidate information about specific industries, the major
companies within the industries, the software used by those companies, and also productivity and quality ranges derived from assessment and benchmark studies.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Using “banking” as an industry example here are some 55 kinds of assessment and benchmark information that would be provided:<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 1: Fifty Five Data Topics for Industry-Specific Software Benchmarks<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Number of large companies (source = Hoover’s Guides to business)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Number of medium companies (source = Hoover’s Guides to business)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Number of small companies (source = Hoover’s Guides to business)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Regulatory agencies that control business sectors<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Standards and safety regulations for the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Supply chains and related industries<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Current government investigations involving the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Mergers and acquisitions within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Start-up companies within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Business failures, government takeovers, or bankruptcies within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Recent patents and intellectual property filed by industry members<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Ranges of industry profitability and economic health<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Current litigation involving the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Domestic competitive situation of the major players within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Global competitive situation of the major players within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Professional associations that serve the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major kinds of hardware platforms used within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major kinds of data utilized within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major kinds of software applications utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major kinds of ERP applications utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major COTS vendors that provide packages to the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major open-source applications utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major outsource vendors that service the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major sources of reusable components serving the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Ranges of portfolio and application sizes within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Ranges of data base and data warehouse sizes within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Numbers of software users within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Number of customer organizations served by software<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Number of actual clients or users served by software<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Numbers of software developers within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Numbers of maintenance personnel within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Numbers of technical specialists within the industry (quality, testing, etc.)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Rates of change for software personnel (expanding, shrinking, stable)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software security issues within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Statistics on data theft, denial of service, and other security breaches<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Security policies, standards, and best practices for the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software development productivity benchmarks with function points<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software maintenance/enhancement productivity benchmarks (function points)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software total cost of ownership (TCO) benchmarks with function points<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software quality benchmarks with function points<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software cancelled project benchmarks by size in function points and type<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software costs and schedule overruns within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Legacy application replacement strategy within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Distribution of CMMI levels within the industry
<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Distribution of TickIt scores or maintainability scores within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major development methods used within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major maintenance methods used within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Typical tool suites used within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Best practices utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Average practices utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Worst practices utilized by the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major quality control methods used within the industry<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Future industry technology trends (cloud computing, SOA, etc.)<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Future industry software trends<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l2 level1 lfo4"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Major sources of industry data (web sites; periodicals, etc.)<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">For a bank to assemble all of this information by itself it would be necessary to gather data from about a dozen industry and government sources
plus probably commissioning benchmarks on a sample of 10 to more than 25 applications. Competitive information from other banks would not be accessible. Essentially this kind of information would not be gathered by individual banks because of a lack of organizational
focus, plus the rather high costs involved. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Probable Clients for Software Benchmarks by Industry<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">As of 2011 the software benchmark business sector is divided into two subsets. One form of benchmark uses fairly simple questionnaires with
the data being self-reported by clients. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Because self-reported benchmarks have no fees for providing information and consolidated benchmark reports are available for low cost, this
form of benchmarking is widespread and popular. The International Software Benchmark Standards Group (ISBSG) is the major provider of self-reported benchmarks. The ISBSG clients consist mainly of project managers and some CIO’s.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The second form of software benchmark uses more complicated questionnaires and also includes on-site data collection in order to gather and
validate quantitative and qualitative information from samples of 10 to more than 25 projects at the same time.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">These on-site benchmarks usually include non-disclosure agreements for data collection and distribution so the data is delivered to specific
companies. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Because the costs for collecting and analyzing the data ranges from $25,000 to more than $50,000 these benchmarks require approval and funding
from the level of a CIO or a CTO. The reports back to clients are of course used by first-line and project managers, but the funding is usually provided by a higher level of management.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software assessments are also on-site consulting studies. For assessments using the model of the Software Engineering Institute (SEI) certified
assessors are used. For other forms of assessment such as the author’s, trained consultants are used. Here too the costs are fairly expensive and in the $25,000 to $75,000 range.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The data that would be assembled for the new kind of benchmarks discussed in this report would include a combination of self-reported data,
on-site data collection, and subscriptions to a number of industry information providers such as Hoover Business Guides, Gartner Group, the Department of Commerce, plus access to other government statistics as well. At a nominal charge of $10,000 for such
a benchmark report, funding approval would probably be at the CIO and CTO level.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">While the costs of these benchmarks are less than the cost of today’s on-site benchmarks and assessments for individual clients, these new kinds
of benchmarks could be offered to dozens or even hundreds of clients so they would generate much greater revenues and profits than conventional single-client benchmarks.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In order to be useful, the benchmark reports would consolidate data from at least 10 companies and 100 projects within an industry, and then
use extrapolation to cover other companies within the same industry. Of course larger samples would be desirable. Remote data might ge gathered from 100 banks or more, while on-site data might be gathered from 20 banks or more.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The on-site data collection would probably be derived from conventional fee-based studies that provide information to specific clients. However
once the data is sanitized and aggregated, it would be marketed to multiple clients.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Because of the richness of the data provided, these new benchmarks would attract a much wider and more diverse set of clients than normal self-reported
or on-site software benchmarks. Obviously the information would be of use to CIO’s and CTO’s, but because of the in-depth industry coverage the information would also be of use to CEO’s and to client executives as well.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">For example these new benchmarks would be of use to VP’s of marketing, sales, manufacturing, human resources, and research and development.
In addition the information would no doubt be acquired by major consulting companies, by law firms that specialize in software litigation, by outsource vendors, and by other kinds of information providers such as journals and web sites.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In addition the new form of benchmark would also be useful to many related companies that provide services or products to banking clients:
outsource vendors, software vendors, consulting companies, equipment manufacturers, personnel companies, major law firms, and government regulators. In fact sales of the new kind of benchmark to these ancillary companies would probably exceed sales to the
banking community itself. For each benchmark study acquired by a bank, probably at least three studies would be acquired by banking service and product providers. This is a new and previously untapped market for benchmark studies.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The basic idea of the new form of benchmark is to elevate the value of benchmark information from data that is “useful but not essential” to
the level of “we must have this information to stay competitive.” A second goal is to elevate the target audience of the benchmark information from project managers, CIO’s, and CTO’s up the level of CEO’s and senior operating unit executives.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Once this new form of benchmark is launched, it will probably lead to a significant increase in other forms of benchmarking.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">It is obvious that the initial launch within an industry such as banking needs to attract a fairly significant number of clients. Therefore
the new form of benchmark should start with and industry where such information is already perceived as valuable; i.e. banks, defense, insurance, health care, medical equipment, and the like.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Once launched in the United States, these new benchmarks would certainly lead to an increase in overseas benchmarks using the same formats and
data collection methods. However to facilitate overseas benchmark data collection, local subcontractors would probably be a desirable method of proceeding.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In addition, some overseas data might be gathered via on-line methods such as Skype, remote surveys, and perhaps Wiki sites. In fact a virtual
benchmark environment using the same technology as Second Life is technically possible. In such an environment avatars of consultants and clients might have conversations and discuss data gathering methods without actual travel.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">It is apparent that once the expanded benchmarks start being created, continuous collection of data and continuous updates will become part
of the benchmark and assessment process.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Expanding the Sources of Benchmark Data<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Currently the actual personnel who provide data for both assessments and benchmarks are primarily software engineers and technical workers,
project managers, some higher-level managers, and occasionally executives at the level of CIO or CTO.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Once the value of the expanded benchmarks becomes apparent, it can be anticipated that additional information might be collected from a much
wider variety of stakeholders, executives, software personnel, and actual users of software:
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 2: Executive Sources of Benchmark Information<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Corporate CEO’s<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Boards of directors or advisory boards<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Government executives (state and Federal CIO’s, agency chiefs, etc.)<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Operating unit VP’s (manufacturing, finance, etc.)<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Agile embedded stakeholders<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">CIO’s for information systems<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">CTO’s for systems software<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">CTO’s for embedded applications and hybrid systems<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Customer executives who approve and acquire software<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Outsource executives<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Corporate attorneys<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l7 level1 lfo5"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">User association executives
<o:p></o:p></span></font></li></ol>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In addition, technical information might be acquired from a larger spectrum of software technical personnel than is normal for today’s assessments
and benchmarks:<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 3: Technical Sources of Benchmark Information<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Architects<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Business analysts<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data base analysts<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software quality assurance (SQA)<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Project Office personnel<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Test personnel<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Scrum Masters<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Integration and configuration control personnel<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Scrum masters<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Embedded stakeholders<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software users and clients<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l0 level1 lfo6"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Customer support personnel<o:p></o:p></span></font></li></ol>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">A combination of remote interviews using simple questionnaire, more detailed questionnaires for on-site use, and conference calls would be used
to gather the expanded forms of information from an expanded set of software stakeholders and software production personnel.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The basic idea is to consolidate information on what stakeholders, top executives, development personnel, maintenance personnel, quality assurance
personnel, and actual clients or users think about software applications and the methods used to create them.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The expansion of information sources will obviously cause some additional effort in gathering information. But after a few months of trials
and tuning, hopefully the additional effort will not cause more than about a 25% increase in total data collection effort.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">International Software Benchmarks by Industry<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">As of 2011 there are benchmark studies and assessments performed overseas, and there are a few reports that compare performance by both country
and industry. Of the ones that attempt to do so, sample sizes are small and the results are somewhat marginal in terms of economic breadth and reliability.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">There is a very strong demand for reliable international benchmarks that would show the comparative performance of the countries where software
production is a major economic topic. The probable revenues for international benchmarks would be proportional to the size of the software industries within the countries. Due to the logistical issues of carrying out on-site international benchmarks, the
global market for such benchmarks would probably be three to five years behind the U.S. market. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">That being said, the revenues from international software benchmarks by industry would probably be sufficient to fund expansion throughout the
world. The more countries that provide data, the more valuable the overall collection of data will become.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Following are a few hypothetical examples of potential annual benchmark revenues by about 2020, assuming U.S. benchmarks as described in this
report begin in 2012:<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 4: Probable Software Benchmark Revenues by Country Circa 2020<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><a name="OLE_LINK1"></a><a name="OLE_LINK2"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">China $45,000,000<o:p></o:p></span></font></a></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">India $45,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Japan $35,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Russia $30,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Brazil $25,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">U.K. $20,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Germany $20,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">France $20,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Italy $15,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Ukraine $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Spain $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Scandinavia $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Australia $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Mexico $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">South Korea $10,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Canada $7,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Taiwan $7,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Israel $5,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Netherlands $3,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Belgium $3,000,000<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">TOTAL $340,000,000
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Of course all of these countries could not be studied at the same time, but eventually the value of expanded global and industry software benchmarks
has the potential to create a significant body of knowledge about software in every industrialized country and every major industry.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Over and above the countries shown in table 2 benchmarks might also be provided for many other countries in Asia, Central and South America,
Africa, and the Pacific regions such as Singapore and Malaysia.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Effort Needed to Create Benchmarks by Industry<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Assembling the initial information needed to produce this example of an industry benchmark for banking would probably require about 90 days
of full-time effort by at least one capable researcher. Two would be better in order to provide backup in case of illness.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The initial data collection would also require fee-based subscriptions to a variety of data sources such as Hoover, Gartner Group, the ISBSG,
and other industrial information sources. However some of these sources of data provide short-term trials, so the initial expense would be fairly low for commercial data sources.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The probable initial cost of such an industry benchmark in a single industry such as banking would probably be in the range of $150,000. This
includes data collection from clients, commercial information providers, analysis of the data, and production of an initial report.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Later expenses would include development of a web site, marketing materials, and other collateral materials that are not absolutely needed for
the initial benchmark report.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Because the data in such a benchmark is dynamic and changes rapidly, continuous updates would be needed to keep the information current. Probably
36 days per year would be needed to refresh the information (i.e. 3 days per month per industry). Monthly or quarterly updates would be provided to clients.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The value of this kind of benchmark compendium would be high enough so that the benchmark report might be marketed at perhaps $10,000 for the
initial report and annual subscription fees for updates of perhaps $2,500. Extracts and subsets from the report could be marketed individually for costs in the range of $500. These would appeal to smaller companies within the industry.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Information would also be made available via the web, with some samples of the data being provided for free, but extended data being fee-based.
The fees could either be on a per-use basis or an annual subscription basis.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Another possible deliverable would be a daily web site entitled “Software Daily News” that resembles the very useful and popular current website
entitled “Science Daily News.” The science web site covers a wide range of scientific disciplines ranging from archeology through astrophysics and includes both news summaries and full-length articles.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">For a major industry such as banking, a benchmark report of this nature might attract about 250 domestic banks and 75 overseas banks. It would
also go to government agencies and major software vendors. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Each industry benchmark report would probably generate about $2,500,000 in the U.S. and about $750,000 abroad, or $2,750,000. Recurring revenues
would amount to perhaps $400,000 per year per industry per year. If there were 10 industries supported the revenues would ascend to more than $27,500,000 for initial subscriptions and more than $4,000,000 per year in recurring revenues.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Eventually on a global level these new benchmarks might have more than 5,000 major corporate clients, more than 1,000 government clients, and
a large but unpredictable set of ancillary clients such as law firms, other consulting groups, universities, and the like.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Obviously such benchmarks would be most useful for industries that have a large amount of effort in the area of software and data processing.
These industries include but are not limited to:<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 5: Industry Candidates for Software Benchmarks<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Aerospace<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Agriculture<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Airlines<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Automotive<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Banking<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Chemicals<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Computers and peripheral equipment<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Cruise lines<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Defense<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Education – university<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Education – primary and secondary<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Entertainment<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Energy and oil<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Governments – state<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Governments – Federal<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Government - municipal<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Health care<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Hotels<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Insurance<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Manufacturing<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Open-source<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Process control<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Pharmaceuticals<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Public Utilities<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Publishing<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Retail trade<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Telecommunications<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Transportation<o:p></o:p></span></font></li><li class="MsoNormal" style="mso-list:l3 level1 lfo7"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Wholesale trade<o:p></o:p></span></font></li></ol>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">A full suite of such benchmarks for major industries would probably generate in the range of $30,000,000 to $50,000,000 per year from U.S. clients.
A team of perhaps 10 researchers and 5 logistical support personnel, plus licenses and subscriptions for external sources of data.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">If the company producing the benchmark reports also collected benchmark and assessment data itself, probably another 10 consultants would be
needed. The data collection would probably generate about $5,000,000 to $10,000,000 per year.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data collection could also be subcontracted to existing benchmark groups such as ISBSG, Software Productivity Research (SPR), the Davids Consulting
Group, and the like. Marketing and sales personnel plus a small executive contingent would be needed. The total size would probably be close to 35 total personnel. However subcontracts for collecting benchmark data might be issued to more than 25 companies
in more than 25 countries. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">It is conceivable that within 10 years of the initial launch, the new form of benchmark might involve more than 300 subcontract personnel in
more than 25 countries.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The annual cost for operating such the core group would probably be in the range of $4,000,000 per year. However except for the $150,000 investment
for the initial report, the organization should be self-sustaining and profitable because no heavy capital investments are needed.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The organization might generate revenues for benchmark subscriptions from eventually more than 500 domestic companies and perhaps 300 overseas
companies and perhaps 100 government agencies. Assuming three to five subscriptions per company revenues from benchmark subscriptions might be in the range of $10,000,000 to $15,000,000 per year, plus the initial cost for each subscription.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The consulting work of collecting data on a fee basis would probably bring in revenues of perhaps $8,000,000 per year. Total revenues from
all sources might total $30,000,000 to $50,000,000 per year.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">By contrast as of 2011 software benchmarks are a niche industry with perhaps 20 U.S. companies collecting data, with combined annual revenues
that probably are below $20,000,000. This is because benchmarks are perceived by clients as useful but not essential.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The business idea behind the new form of benchmark is to elevate the importance from being perceived as useful to being perceived as essential.
To achieve this goal, more and better information needs to be provided to clients than is currently made available.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">As useful as today’s benchmarks are for productivity and quality studies, the lack of context information limits the utility of benchmarks and
restricts the potential audience.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Additional Tools and Services in Addition to Benchmarks and Assessments<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">In addition to marketing benchmarks and performing assessment and benchmark consulting studies, the organization would also be positioned to
market and perhaps develop several kinds of tools and ancillary products and services:<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Benchmark tools for data collection that clients might use to facilitate their own data collection and analysis.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Predictive tools for estimating software schedules, efforts, costs, quality, reliability, maintenance, and enhancements.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Special studies that evaluate important topics. For example large-scale studies that compared a number of development
methods such as waterfall, iterative, object-oriented, Agile, RUP, TSP, XP, and others would be both useful to the industry and highly salable.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Training in skills which the benchmarks demonstrate need improvement within an industry such as quality control, test
case design, change control, legacy migration, and a number of others.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Portfolio analyses are seldom performed because of the difficulty of sizing and analyzing as many as 3,000 applications
that might total more than 7,500,000 function points. Further, between 25% and more than 50% of typical portfolios are in the form of COTS packages or open-source software which cannot be sized using standard methods because the vendors do not provide the
needed inputs. It is possible to size a portfolio using one or more of the new high-speed function point sizing methods. Portfolio studies would no doubt be covered by non-disclosure agreements and hence marketed only to specific companies.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Litigation support depends upon accurate data to determine industry averages for quality, productivity, and other benchmarks.
The kinds of data discussed herein would probably be widely used as background information in breach of contract litigation between clients and outsource vendors. It would also be used in litigation against software companies for poor quality or damages.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Outsource contracts frequently include clauses dealing with quality, schedule, productivity, reliability, change control,
and the like. Without accurate benchmark data some contracts contain clauses that are probably impossible to achieve. Accurate benchmarks would be very useful for developing outsource contracts that are mutually agreeable.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Mergers and acquisitions occur almost daily in many industries.. The data in this new form of benchmark would be of
significant interest to business brokers and companies considering either acquisition or being acquired. It would also be of interest to companies that are merely seeking partnerships, distributors, subcontractors, or potential clients.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l8 level1 lfo8"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Venture-backed start-up businesses have been in decline due to the recession, but are starting to ramp up again. The
data contained in this new form of benchmark report should be of some interest to both entrepreneurs and venture capitalists considering starting new businesses.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Tool and special study revenues would be over and above the revenues already discussed. They are currently unpredictable because the suite
of tools is not fully defined. However annual revenues in excess of $10,000,000 per tool would not be uncommon. Special studies could easily top $ 35,000,000 per year.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Benchmark Company Exit Strategies<o:p></o:p></span></font></b></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Normally companies have four possible end games: 1) they go public; 2) they are acquired by larger companies; 3) they go out of business 4)
they continue indefinitely with only marginal growth..<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Options 1 and 2 are the most likely end games for such a benchmark organization as the one described here. While such benchmarks might be created
by a non-profit group or perhaps by a university, it is more likely that a for-profit organization would be the best choice.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">A for-profit company is most likely because if the idea of the new form of benchmark expands and becomes successful, the benchmark production
group would be an attractive acquisition candidate for large data providers such as Gartner Group, Accenture, Google, or similar large corporations where information is a valuable commodity..<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Software Benchmarks Circa 2011<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">As of early 2011 none of the normal benchmark sources or consulting companies provide this kind of information so there is no real competition.
Gartner Group and the International Data Group (IDG) provide subsets of the kinds of information discussed, but not actual benchmarks.
<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The International Software Benchmark Standards Group (ISBSG) provides useful benchmarks, but does not cover embedded
software, systems software, or have any data on large applications > 10,000 function points in size.
<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The Software Engineering Institute (SEI) provides assessments, but is sparse with benchmark data and provides little
or no information about companies, industries, data, and other key topics. <o:p>
</o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The Information Technology Metrics and Productivity Institute (ITMPI) provides many useful reports and webinars on
specific topics, but does not provide assessment and benchmark information or any context information about various industry segments. Some of the CAI tools might well be useful for collecting benchmark data.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The government-sponsored Data Analysis Center for Software (DACS) provides useful information in a government and defense
context, but no benchmarks or assessments. <o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The Standish Group publishes interesting statistics on software failures, but does not provide conventional benchmarks
and assessments.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">A number of privately held consulting companies such as the Davids’ Consulting Group, Quantitative Software Management
(QSM), Software Productivity Research (SPR), and the Software Improvement Group (SIG) in Amsterdam and several others provide benchmarks and assessments for individual clients. These groups occasionally publish studies using data from multiple clients, but
sample sizes are fairly small.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Universities tend to provide small-scale studies on specific topics but are not funded or equipped to produce large-scale
industry-wide studies. <o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="text-align:justify;mso-list:l6 level1 lfo9"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The software metrics associations such as IFPUG and COSMIC provide the current rules for counting functional metrics,
but seldom produce benchmarks and they don’t do assessments at all.<o:p></o:p></span></font></li></ul>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The bottom line is that the new kind of benchmark discussed in this report has little competition circa 2011. Gartner Group is the best positioned
to compete, but to date has not attempted specific software benchmarks or assessments. Gartner Group aims at CEO’s and top executives, but does not get down to the level of assessments and benchmarks.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">It is apparent that the overall benchmark that contains all forms of data shown in this report might be subset into special purpose reports
that might be marketed separately or offered via subscription to specialized communities. Examples of such special reports might include, but are not limited to:<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Table 6: Special Benchmark Reports<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Quality Assurance (SQA)<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Estimation Tool Analysis<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Development<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Maintenance and Enhancement<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Security and Safety<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Failure Rates by Industry<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Project Management<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Project Offices<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data Quality by Industry<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Data Base Development<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Web Site Development<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Process Improvement<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Software Education and Training<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Analysis of Cloud Computing<o:p></o:p></span></font></li><li class="MsoNormal" style="text-align:justify;mso-list:l9 level1 lfo10"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Best Practice Analysis Based on Empirical Results<o:p></o:p></span></font></li></ol>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">These separate subsets would clearly generate additional revenues over and above those discussed for the entire assessment and benchmark report.
However it is premature to attempt to quantify the numbers of subscribers and revenues for these subsets of information.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><b><font size="3" face="Times New Roman"><span style="font-size:12.0pt;font-weight:bold">Summary and Conclusions about Software Benchmarks<o:p></o:p></span></font></b></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The software industry circa 2011 is characterized by many project failures, by frequent cost and schedule overruns, by poor quality, and by
lack of reliable information as to what the phrase “best practices” really means in terms of results.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The software industry is also subject to frequent fads and fallacies as new development methods surface, are touted as panaceas, and then gradually
demonstrate only marginal improvements if any over alternate development methods. Poor measurement practices and inadequate benchmarks are what makes these fads and fallacies endemic problems for the software community.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Benchmarks are very useful methods for minimizing these common problems, but less than 30% of large U.S. software companies have either commissioned
benchmarks or use benchmark data. For small and mid-sized U.S. companies, less than 10% utilize benchmark data. In fact many corporations not only fail to use benchmarks and assessments, they have never even heard of the SEI, ISBSG, ITMPI, and the other
major sources of assessment and benchmark information.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The new form of benchmark discussed in this paper is aimed at expanding the information contained in software benchmarks from basic productivity
and quality levels up to a complete condensation of critical industry topics where software is part of the equation. The additional kinds of data and information will hopefully elevate benchmarks and assessments from useful but optional studies to mandatory
business practices that are demanded by a majority of CEO’s and top operational executives.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Once such information is published for a specific industry such as banking, it is expected that demands from other industries will drive the
business growth for similar benchmarks in other industries. <o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">The goal for the new form of benchmark is to reach close to 100% of major corporations, more than 50% of medium corporations, and perhaps 25%
of smaller corporations. It is difficult to predict government penetration at all levels, but no doubt all 50 states would want to subscribe to this new form of benchmark if state government data were included. A number of Federal agencies would also want
to have access to the kinds of data provided by the new benchmarks.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
<p class="MsoNormal" style="text-align:justify"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">It is hoped that the kinds of information included in these new benchmarks will not only lead to a profitable business, but will assist the
software industry in overcoming its traditional problems of cost and schedule overruns combined with poor quality.<o:p></o:p></span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><o:p> </o:p></span></font></p>
</div>
</body>
</html>