sss ssss rrrrrrrrrr
ssss ss rrrr rrrr
sssss s rrrr rrrr
ssssss rrrr rrrr
ssssssss rrrr rrrr
ssssss rrrrrrrr
s ssssss rrrr rrrr
ss sssss rrrr rrrr
sss sssss rrrr rrrr
s sssssss rrrrr rrrrr
+===================================================+
+======= Testing Techniques Newsletter (TTN) =======+
+======= ON-LINE EDITION =======+
+======= February 1996 =======+
+===================================================+
TESTING TECHNIQUES NEWSLETTER (TTN), On-Line Edition, is E-Mailed
monthly to support the Software Research, Inc. (SR) user community and
provide information of general use to the world software testing commun-
ity.
(c) Copyright 1995 by Software Research, Inc. Permission to copy and/or
re-distribute is granted to recipients of the TTN On-Line Edition pro-
vided that the entire document/file is kept intact and this copyright
notice appears with it.
TRADEMARKS: STW, Software TestWorks, CAPBAK/X, SMARTS, EXDIFF,
CAPBAK/UNIX, Xdemo, Xvirtual, Xflight, STW/Regression, STW/Coverage,
STW/Advisor and the SR logo are trademarks or registered trademarks of
Software Research, Inc. All other systems are either trademarks or
registered trademarks of their respective companies.
========================================================================
INSIDE THIS ISSUE:
o Quality Week 1996: An Early Sneak Preview
o Empirical Software Engineering: An International Journal
o Software Negligence and Testing Coverage (Part 2 of 3), by Cem
Kaner
o TTN Online Back Issues Now On WWW
o Frequently Asked Questions about the Space Shuttle Computers (Part
1 of 2)
o SR Introduces New STW/Web Tool Bundle
o Is "Software Quality" Intrinsic, Subjective, or Relational?, by
Dick Carey
o Correction Noted
o TTN SUBMITTAL POLICY
o TTN SUBSCRIPTION INFORMATION
========================================================================
QUALITY WEEK 1996: AN EARLY SNEAK PREVIEW
The final program for QW'96 is currently being assembled. Look for your
copy of the conference announcement in a few weeks, or consult URL:
http://www.soft.com/QualWeek/index.html for up to date details.
Here are a few features of the QW'96 program that are sure to be of
interest...
o KEYNOTERS FOR THE CONFERENCE...
* Dr. Boris Beizer will keynote the QW'96 Conference with "Software
*is* Different!"
* Prof. Lee Osterweil will present a Keynote on the breakthrough con-
cept of "Perpetually Testing Software"
* Dr. Guenter Koch will describe the European Software Institute's
role as a change agent.
* Mr. Clark S. Turner will discuss the issue of legal sufficiency of
safety-critical testing processes.
o Quick-Start Track
There is a new "Quick-Start Track" with six focused mini-Tutorials by
acknowledged experts on basic technological aspects of applying soft-
ware quality methods.
o Debate by the Experts: Testing vs. Inspection
Mr. Tom Gilb and Dr. Boris Beizer, two speakers internationally known
for the contributions in the software quality arena, will debate:
"Testing Vs. Inspection" Each is an expert in one of the areas. Will
they debate their own side of the issue, or take the opposing posi-
tion? Will their sides be decided by a coin toss?
o Panel Session on Model Validation
Mr. Bob Binder will moderate a panel of industry experts on "Automated
Model Validation For Object Oriented Development"
o OTHER FEATURES
QW'96 will also feature nearly 40 regular papers, 10 half-day tutori-
als, and a full 2-day Vendor Exhibit program.
========================================================================
Empirical Software Engineering
An International Journal
Editors-in-Chief: Victor Basili (basili@cs.umd.edu), University of Mary-
land;
Warren Harrison (warren@cs.pdx.edu), Portland State University.
Associate Editors: Dieter H. Rombach (rombach@informatik.uni-kl.de)
University of Kaiserslautern, Germany (Regional Editor for Europe);
Ross Jeffrey (r.jeffrey@unsw.edu.au), University of New South Wales,
Australia (Regional Editor for Australia);
Koji Torii (torii@is.aist-nara.ac.jp), Nara Institute of Science and
Technology, Japan (Regional Editor for Asia)
Empirical Software Engineering, An International Journal provides a
forum for researchers and practitioners to report both original and
replicated studies. The studies can vary from controlled experiments to
field studies, from data intensive to qualitative. Preference will be
given to studies that can be replicated or expanded upon. The aim of
the studies should be to expose, in an experimental setting, the primi-
tives of software engineering. Papers on the supporting infrastructure
for experimentation would also be appropriate.
Empirical Software Engineering home page: http://www.cs.pdx.edu/emp-se/
========================================================================
SOFTWARE NEGLIGENCE AND TESTING COVERAGE (Part 2)
by
CEM KANER, J.D., PH.D., ASQC-C.Q.E.
Editors Note: This article was first published in the Software QA Quar-
terly, Volume 2, No. 2, 1995, pp. 18-26. Copyright (c) Cem Kaner, 1995.
All rights reserved. It is reprinted in TTN-Online, in four parts, by
permission of the author.
(Part 2 of 4)
* A Windows program might fail when printing to any printer. You
achieve complete coverage of printer compatibility tests (across
printers) if you use the set of all Windows-supported printers, using
all Windows printer drivers available for each printer. These drivers
are part of the operating system, not part of your program, but your
program can fail or cause a system failure when working with them. The
critical test case is not whether a particular line of code is tested,
but whether it is tested in conjunction with a specific driver.
* Suppose that you test a desktop publishing program. One effective
way to find bugs and usability failures is to use the program to create
interesting documents. This approach is particularly effective if you
use a stack of existing documents (Footnote 4) and try to make exact
copies of them with your program. To create your stack, perhaps you'll
use all the sample files and examples that come with PageMaker, Quark,
FrameMaker, and one or two other desktop publishers. In this case, you
achieve complete coverage if you recreate all of the samples provided by
all available desktop publishers.
------------
Footnote 4 I was the project manager for a desktop publishing program. I
speak from experience when I say that this strategy is a remarkably
effective way of finding bugs and convincingly demonstrating their sig-
nificance.
------------
The Appendix to this article lists 101 measures of testing coverage.
Line coverage is just one of many. There are too many possible tests for
you to achieve complete coverage for every type of coverage in the list.
I hope that the list helps you make priority decisions consciously and
communicate them explicitly. The tradeoffs will differ across applica-
tions -- in one case you might set an objective of 85% for line coverage
(Footnote 5), 100% for data coverage, but only 5% for printer / driver
compatibility coverage. For a different program whose primary benefit is
beautiful output, you would assign printer coverage a much higher
weight.
------------
Footnote 5 It can be very expensive and difficult to achieve 100% line
or branch coverage. Grady reports that values of 80-85% are reasonably
achievable. R.B. Grady, Practical Software Metrics for Project Manage-
ment and Process Improvement, PTR Prentice Hall (Hewlett-Packard Profes-
sional Books), 1992, p. 171.
------------
If you had an extra $50,000 to spend, would you focus your efforts on
increasing line coverage or increasing some of the others? Surely, the
answer should depend on the nature of your application, the types of
risks involved in your application, and the probable effectiveness of
the different types of tests. The most desirable strategy will be the
one that is most likely to find the most bugs, or to find the most seri-
ous bugs.
The legal (negligence) test for the coverage tradeoffs that you make is
reasonability. No matter what tradeoffs you make, and no matter how much
money you spend on testing, you will miss some bugs. (Footnote 6)
Whether or not those bugs are products of negligence in the testing pro-
cess depends on your reasonability, not on your luck in selecting just
the right tests.
------------
Footnote 6 See Chapter 2 of C. Kaner, J. Falk, and H.Q. Nguyen, Testing
Computer Software (2nd. Ed.), Van Nostrand Reinhold, 1993.
------------
Your task is to set priorities among tests in the way that a reasonably
careful company would -- and to me that means to select the test stra-
tegy that you rationally believe is the most likely to find the most
bugs or the most serious bugs.
There is no magic talisman in coverage that you can use blindly and be
free of negligence liability. Being reasonable in your efforts to safe-
guard your customer requires careful thought and analysis. Achieving
complete (line, branch, whatever) coverage will not insulate you. The
plaintiff's attorney will just ask you why you spent all that money on
line coverage, at the expense of, say, interrupt coverage. Try to assign
your weights sensibly, in a way that you can explain and justify.
The same reasoning applies to customer satisfaction in general. If your
approach will control the risks, you've done your job. But if you can
identify gaps that leave an unreasonable degree of risk to customer
safety or satisfaction, there is no reasonable alternative to addressing
those risks. As a final note, I hope that you'll take a moment to
appreciate the richness, multidimensionality, and complexity of what we
do as testers. Sometimes we hear that only programmers should be tes-
ters, or that all testing should be driven from a knowledge of the work-
ings of the code. This list highlights the degree to which that view is
mistaken. Programming skills and code knowledge are certainly valuable
for glass box testing tasks, but as we explore the full range of black
box testing approaches, we find the need for skills and knowledge in:
* the application itself (subject matter experts)
* safety and hazard analysis
* hardware (modems, printers, etc.)
* customer relations.
APPENDIX: THE MANY TYPES OF TESTING COVERAGE
This appendix lists 101 coverage measures. Coverage measures the amount
of testing done of a certain type. Because testing is done to find bugs,
coverage is also a measure of your effort to detect a certain class of
potential errors. For example, 100% line coverage doesn't just mean that
you've executed every line of code; it also means that you've tested for
every bug that can be revealed by simple execution of a line of code.
Please note that this list is far from complete. For example, it doesn't
adequately cover safety issues. (Footnote 7) Nor does it convey the
richness of the tests and test strategies that you can derive from cus-
tomer complaints and surveys and from tests involving controlled custo-
mer observation. And you will add measures as you analyze the applica-
tion that you're testing.
------------
Footnote 7 N. Leveson, Safeware: System Safety and Computers, Addison-
Wesley, 1995.
------------
1. Line coverage. Test every line of code (Or Statement coverage: test
every statement).
2. Branch coverage. Test every line, and every branch on multi-branch
lines.
3. N-length sub-path coverage. Test every sub-path through the program
of length N. For example, in a 10,000 line program, test every possible
10-line sequence of execution.
4. Path coverage. Test every path through the program, from entry to
exit. The number of paths is impossibly large to test. (Footnote 8)
------------
Footnote 8 See G. Myers, The Art of Software Testing, Wiley, 1979, and
Chapter 2 of C. Kaner, J. Falk, and H.Q. Nguyen, Testing Computer Soft-
ware (2nd. Ed.), Van Nostrand Reinhold, 1993.
------------
5. Multicondition or predicate coverage. (Footnote 9) Force every logi-
cal operand to take every possible value. Two different conditions
within the same test may result in the same branch, and so branch cover-
age would only require the testing of one of them.
------------
Footnote 9 G. Myers, The Art of Software Testing, Wiley, 1979 (multicon-
dition coverage) and B. Beizer, Software Testing Techniques (2nd Ed.),
Van Nostrand Reinhold, 1990.
------------
6. Trigger every assertion check in the program. Use impossible data if
necessary.
7. Loop coverage. "Detect bugs that exhibit themselves only when a loop
is executed more than once." (Footnote 10)
------------
Footnote 10 B. Marick, The Craft of Software Testing, Prentice Hall,
1995, p. 146.
------------
8. Every module, object, component, tool, subsystem, etc. This seems
obvious until you realize that many programs rely on off-the-shelf com-
ponents. The programming staff doesn't have the source code to these
components, so measuring line coverage is impossible. At a minimum
(which is what is measured here), you need a list of all these com-
ponents and test cases that exercise each one at least once.
9. Fuzzy decision coverage. If the program makes heuristically-based or
similarity-based decisions, and uses comparison rules or data sets that
evolve over time, check every rule several times over the course of
training.
( T O B E C O N T I N U E D )
========================================================================
TTN-ONLINE BACK ISSUES NOW ON WWW
Many readers Email us asking for copies of back issues. Because of the
wide interest in the content we're providing we have put the prior
YEAR's worth of back issues up at our WWW site for everyone to read.
Eventually we'll put up complete set of issues.
Please bring up the URL: http://www.soft.com and click on News for com-
plete copies of the prior 12 issues of TTN-Online Edition.
If you can't reach this WWW page please request back issues of ttn at
"info@soft.com".
========================================================================
FREQUENTLY ASKED QUESTIONS ABOUT THE SPACE SHUTTLE COMPUTERS
Part 1 of 2
This is the FAQ list just for the Space Shuttle computer systems. The
information here was collected by Brad Mears during several years of
working in the shuttle flight software arena, then expanded by Ken Jenks
with major assistance from Kaylene Kindt of the NASA/Johnson Space
Center's Engineering Directorate. If you believe any part of this docu-
ment is in error, contact me and I'll try to resolve the issue thru
further research. My email address is kjenks@gothamcity.jsc.nasa.gov.
The latest version of this document is available via anonymous FTP at
ftp://explorer.arc.nasa.gov/pub/SPACE/FAQ/shuttle-GPC-FAQ.txt
(1) What kind of computer does the shuttle use?
All of the flight-critical software is executed on five IBM AP-101 Gen-
eral Purpose Computers (GPCs). The crew may use commercially available
laptops and programmable calculators for non-critical applications.
The AP-101 is a descendant of the IBM 360/370 family that was specially
redesigned for use on the shuttle. By today's standards, the AP-101 is
incredibly slow. The original shuttle computers, AP-101B, were replaced
in 1991 by an upgraded version, the AP-101S. The AP-101S is three times
faster than the AP101B and is half the size and weight. The AP101S also
decreased power consumption by about 100 Watts per computer. (The
AP101B and its associated electronics consumed 650 Watts, and the AP101S
consumes 560 Watts. Reduced power consumption gives increased reliabil-
ity, reduced consumption of consumables like Hydrogen and Oxygen, and
reduced requirements for heat rejection.)
The AP101S first flew on STS-37, April 5-11, 1991. The computers were
manufactured by IBM in Owego, New York, IBM has since sold its Federal
Systems Company to Loral Space Information Systems.
(2) Tell me more about the GPCs
The AP-101 is a very different computer than what most people are fami-
liar with. Like most computers used in aerospace applications, it does
not have a keyboard or CRT. It is a black box with chips and a power
supply.
Relax - there *are* keyboards and CRTs available for the crew, but they
aren't considered part of the GPC. In fact, there isn't even a one-to-
one correspondence between computers and keyboards. Any CRT or keyboard
can be addressed by any GPC, depending on the state of cockpit switches.
BTW, the keyboards aren't your typical QWERTY keyboard. They are a
funky hexadecimal keypad with several special-purpose function keys.
Besides the CPU, the GPC has twenty-five (25) additional processors.
All of the processors besides the CPU are dedicated to I/O. There is
one processor for each of the twenty-four serial data busses the AP-101
uses to control and monitor the vehicle.
Each of the I/O processors can be programmed independently to handle
incoming/outgoing data transfers for its respective data bus. The
twenty-fifth processor is a master I/O scheduler. It serves as an
interface between the CPU and the other I/O processors.
The use of truly independent I/O processors allows the CPU to concen-
trate on number-crunching and not get bogged down with I/O. This is
very important considering the hard real-time requirements the software
must meet.
(3) How much memory do the GPCs have?
The amount of memory available in the AP-101B is 212,992 16-bit words.
The AP-101S holds up to 524,288 words. Yes, this is ludicrously small
by today's standards. These numbers do not include error detection
bits.
The memory in the original AP-101B was honest-to-gosh ferrite core.
Since core memory is non-volatile (meaning it retains its state when it
loses power), a GPC can be turned off and turned on later without losing
its mind. The AP-101S uses normal random access memory (RAM) with bat-
tery backup to achieve the same result.
Ferrite core memory was originally selected for the GPC's because little
was known about the effects of radiation on the high-density solid state
RAM which is now used in almost all digital computer hardware.
The fact that the AP-101B used non-volatile memory was put to use after
the Challenger accident. Once the GPCs were recovered from the ocean
floor, NASA was able to clean them up and recover a significant percen-
tage of the memory contents. This data was used to help confirm what
happened.
The AP101S uses a solid state CMOS memory with battery backup. Because
the CMOS memory can be altered by radiation which the Shuttle encounters
in orbit, the memory was designed with error detection and correction
(EDAC) logic. Whenever the GPC is powered, a hardware process called
"scrub" continually checks the GPC's memory for alterations. Scrub is
capable of detecting and correcting single bit errors. Multi-bit errors
can be detected, but not corrected. Scrub checks all of the GPC's
memory in 1.7 seconds. Each memory location is also checked by EDAC
prior to being executed by the GPC.
With the non-volatile memory of the old AP101B, the crew would configure
one GPC for De-orbit and Entry, then shut off its power. This is called
"freeze-drying" and will be discussed later. With the AP101S, the crew
configures one GPC with software for the De-orbit and Entry phases of
Shuttle flight then puts the GPC to "Sleep." Sleep is a reduced power
mode in the GPC which uses only 10% of its rated power. This reduced
power mode reduces the amount of electricity which must be generated by
the Orbiter's Fuel Cell system, but allows the EDAC logic to continually
scrub the GPC's memory.
(4) How much disk space do the computers have?
None. All secondary storage for the flight software is on tape. There
are two Mass Memory Units (MMU) that hold all of the software and data
that the GPCs require during flight.
At the time the shuttle was designed and built, disk drive technology
was pretty young. No disk drive at the time had sufficient reliability
for flight critical equipment in a manned spacecraft.
(5) How big are the GPCs?
The GPC's are housed in air transport racks (ATR), whose dimensions are
10.2 inches wide, 7.625 inches high and 19.52 inches long. The original
APS101B was packaged in two such boxes, one for the CPU and another for
the Input/Output Processor (IOP), but the newer AP101S combines both
processors into one ATR.
(6) OK, so now I know what a GPC *is*. What does it *do*?
The shuttle is a fly-by-wire vehicle (like the F-16). This means that
there are no direct connections between the cockpit controls and the
vehicle. Every switch and every button is connected to the computers.
The engines, SRBs, flaps, payload bay doors, etc are all controlled by
the software.
The entire flight, from pre-launch to roll-out, is under computer con-
trol. This isn't to say that the crew is unnecessary. They are needed,
but the software is between them and everything they do.
When the software detects a change of state in a switch, it determines
what that switch is used for, what the new state means, and if the com-
mand is legal at this time. Once it has done all that, then the soft-
ware will issue the appropriate command via the serial data busses.
An example - If the commander wants to push the nose down during flight,
he pushes forward on the joystick in front of him. The GPC reads this
input, determines what it means, and then issues commands to the proper
control surfaces. These commands are electronic signals that go to an
actuator. The actuator is what actually moves the aero-surface. The
actuator uses hydraulics to move the aero-surface.
It is very important to realize that the shuttle is completely fly-by-
wire. If the flight software fails to operate, the vehicle will crash.
There are two versions of the software on-board: the Primary Avionics
Software System (PASS) and the Backup Flight Software (BFS). If the
PASS should fail to operate correctly, the crew can engage the BFS.
More about that later.
(7) Can the astronauts override the software?
Yes and no. Yes the astronauts can override decisions made by the soft-
ware. But the mechanism used to perform the override is in SOFTWARE!
Do you see the catch? If the software is fouled up badly enough, the
override won't work.
(End of Part 1; Look for Part 2 in Next Month's Issue)
========================================================================
SR INTRODUCES NEW STW/WEB TOOL BUNDLE
STW/Web Helps IS Organizations Improve the Quality and Reliability of
Corporate Web Sites
San Francisco, California, 12 February 1996 -- Software Research, Inc.,
the leading provider of integrated automated software testing tools and
technology, today announces the sixth TestWorks bundle: Software
TestWorks/Web , applying the proven TestWorks product line specifically
for the support of World Wide Web (WWW) development and maintenance.
The STW/Web suite includes CAPBAK/X for capture/replay of complete tests
of your web pages; XVirtual for realistic loading to assure WWW response
times; and TCAT/Java for thorough test of Java applets before public Web
site releases.
The new STW/Web tool bundle will help organizations improve the quality
and reliability of their WWW-based information systems in the following
way: by fully exercising them assuring completeness and consistency of
WWW pages; by allowing Web developers to investigate response time to
browser requests; by permitting them to experiment with the maximum
number of hits their WWW server can support, and by validating their
system's ability to function correctly under varying conditions.
To test a Web-based system using STW/Web, developers first create test
scripts using a special Web-adapted version of Software Research's GUI
testing product CABPAK/X. Test scripts are created by recording actual
user interactions with the browser, a step made possible with minimal
effort through use of CAPBAK/X's Automated Object Mode Recording and
Automated Output Synchronization features. The tester/recorder only has
to select validation screens and checkpoints. Every time the WWW pages
are modified, CAPBAK/X can run unattended tests to detect functionality
errors, such as missing links or incorrect or changed content.
To test performance and capacity, Web page developers apply our XVirtual
tool, a background-only version of an X11 display server. XVirtual can
simulate thousands of hits against the Web display by simulating hun-
dreds of interacting users assuring 100% load accuracy. Using XVirtual,
developers can assess the performance delivered by typical WWW-surfers
of their site.
The third STW/Web component, TCAT/Java (available in 96Q2) is a test
coverage analysis tool configured specifically for Java applications and
for use on Java-enabled browsers. Animated WWW site users can now
determine that their Java applets are fully exercised by their test
suites. This is a critical quality verification function when a Java
applet is used in support of financial transactions on the WWW.
"STW/Web is the application of our extensive software test experience
accumulated in our 15+ years in the software testing business and
represents our three-part entry in the WWW development and testing
market,'' said Dr. Edward Miller, President of Software Research, Inc.
"Testing of Web sites has previously been a tedious, manual process. By
automating the Web testing process with STW/Web, Web site developers can
now build automated processes that ensure that corporate Web sites have
the quality, availability, reliability, and capacity to handle the anti-
cipated user loads with the expected quality level. As for information
systems organizations, STW/Web permits management of the risk of the
system's exposure to millions of consumers accessing corporate informa-
tion through the Internet; it improves the quality and performance of
their Web-based systems.''
The World Wide Web offers millions of potential consumers access to a
great variety of corporate information today. By the year 2000, 10 mil-
lion users will be surfing the net and an astronomical $5 billion worth
of world business will be conducted electronically on the Internet. An
increasing number of business-critical applications will be designed
specifically for the Internet. Automated software testing will be the
only option for WWW developers to assure that these fast-changing appli-
cations operate reliably.
STW/Web with CAPBAK/X and XVirtual is available immediately in a special
short-time introductory offer. TCAT/Java becomes available in 96Q2.
Contact SR for complete pricing details. Or, visit SR's Web site
http://www.soft.com.
========================================================================
Is "Software Quality" Intrinsic, Subjective, or Relational?
Dick Carey
20 Riverside St. #22
Watertown, MA 02172-2670
+1 (617) 926-2556
The software community almost unanimously measures Software Quality (SQ)
as the ratio of Errors per Kilo Lines Of Code (E/KLOC). In spite of
this, the underlying assumptions for this concept and its application
have not been critically examined enough. In this paper, it is asserted
that E/KLOC is not a reasonable measure of SQ, and a different model is
introduced.
An important requirement for objectivity is that judgements be quantifi-
able. How we define SQ has consequences in determining a software
organization's goals. I classify three different interpretations of SQ
in terms of what Chris Sciabarra [1995] calls Ayn Rand's dialectic of
the Intrinsic, the Subjective, and the Relational.
INTRINSIC-SQ
Intrinsic-SQ says quality is only a measurement of the current code,
independent of past changes. A large program is going to have more
errors than a small program, so in order to judge the density of errors,
Intrinsic-SQ requires us to know the number of Lines of Code (LOC). The
E/KLOC formula makes errors and LOC dependent on each other. A big
fraction is bad, a small fraction is good. Program changes that
decrease the ratio are seen as improvements.
This satisfies our need to have an objective measure of SQ but is it
correct? The E/KLOC formula says that if we don't fix any errors but we
add code my SQ improves (because I've increased the denominator). If we
take 1000 lines of unstructured code and modularize it so it shrinks to
1/10th its original size with the same functionality, I've improved its
SQ 10 times. But E/KLOC says I've worsened its Intrinsic-SQ 10 times
from 10/1000 to 10/100 (because I've decreased the denominator).
E/KLOC says there's a trade-off between error count and code size.
There's a point where if we remove enough code in order to fix a error
it won't improve its Intrinsic-SQ at all:
PREVIOUS VERSION: CURRENT VERSION:
10 errors 9 errors
E/KLOC = ---------- = ---------
100 LOC x LOC
10 x = 900
x = 90
therefore
10 errors 9 errors
E/KLOC = ---------- = ---------
100 LOC 90 LOC
If we fix 1 error by removing 10 lines of code E/KLOC says the
Intrinsic-SQ hasn't changed. We could almost make the opposite case
that a large E/KLOC is good.
If we fix a error by adding code has SQ improved? Yes. If we fix a
error by removing code has SQ improved? Yes. If we fix a error without
adding or removing code has SQ improved? Yes. The use of E/KLOC is at
odds with the original objective of improving SQ. These weaknesses of
E/KLOC make it a very inaccurate model of true SQ. It's wrong to accuse
people of "fooling" the E/KLOC metric. The problem is that the E/KLOC
metric "mismeasures" true SQ. It is the underlying false assumption
that SQ is intrinsic that causes us to make LOC and errors dependent
variables.
SUBJECTIVE-SQ
Most software organizations apply externally-generated methods to their
software development. This top-down approach treats SQ subjectively,
not as the actual goal of producing error-free software, but with fol-
lowing some generally accepted standard process. How close they match
the standard indicates how good their SQ is. The assumption is that,
once the process changes to match a standard one, all products generated
will be improved. This methodology differs from the other two by
centering on activities rather than results. Subjective-SQ formalizes
SQ without understanding what SQ is.
If Subjective-SQ is based only on what we want to see, SQ can change
without touching the actual software.
Removing duplicate error reports and calling it one error
instead of two improves Subjective-SQ.
Another paper shuffling technique to improve Subjective-SQ is to
lower an error's priority.
Discovering a previous release's errors causes Subjective-SQ to
go down.
The staff demonstrates it's support of Subjective-SQ by championing and
supporting the approach. People who resist indoctrination are seen as
anti-SQ. This creates a culture where programmers are penalized for
discovering errors and rewarded for not reporting errors.
RELATIONAL-SQ
The failures of Intrinsic-SQ and Subjective-SQ has led this writer to
introduce a different model. Relational-SQ defines quality as the
difference in the changes between the previous and current versions.
Relational-SQ says quality cannot be judged instantaneously by just
looking at the current version of software. Relational-SQ makes LOC and
errors independent of each other.
Removing errors from the previous version improves Relational-SQ.
Introducing errors into the current version decreases Relational-SQ.
Discovering errors in the current version that were already there in the
previous version doesn't affect Relational-SQ.
When judging LOC there is an additional restriction that there be no
change in functionality. Decreasing LOC without changing functionality
improves Relational-SQ. Changing the functionality to add a new feature
or to fix an error, whether it increases or decreases LOC, doesn't
change Relational-SQ. Since the feature or error isn't in both ver-
sions, there's no comparison. Unlike the other two models, Relational-
SQ rewards programmers for structuring code.
Relational-SQ is the best representation of true SQ. Using this defini-
tion restores us to the original rationale of a quantifiable, objective
standard. Unlike Subjective-SQ, Relational-SQ can only be improved by
changing code. This creates a culture where programmers are not penal-
ized for reporting errors. Programmers can only improve SQ by fixing
errors, so the bigger the pool of known errors the better.
Chris Sciabarra, AYN RAND: THE RUSSIAN RADICAL, Penn State Press, 1995.
========================================================================
C O R R E C T I O N N O T E D !
Yes, as many sharp-eyed readers pointed out, we misspelled "January" in
the mast of last month's issue. Now the only question you have to
wonder about is this: was that an intentional goof to a test the reader-
ship?
========================================================================
------------>>> TTN SUBMITTAL POLICY <<<------------
========================================================================
The TTN On-Line Edition is forwarded on the 15th of each month to Email
subscribers via InterNet. To have your event listed in an upcoming
issue, please Email a description of your event or Call for Papers or
Participation to "ttn@soft.com". The TTN On-Line submittal policy is as
follows:
o Submission deadlines indicated in "Calls for Papers" should provide
at least a 1-month lead time from the TTN On-Line issue date. For
example, submission deadlines for "Calls for Papers" in the January
issue of TTN On-Line would be for February and beyond.
o Length of submitted non-calendar items should not exceed 350 lines
(about four pages).
o Length of submitted calendar items should not exceed 68 lines (one
page).
o Publication of submitted items is determined by Software Research,
Inc., and may be edited for style and content as necessary.
========================================================================
----------------->>> TTN SUBSCRIPTION INFORMATION <<<-----------------
========================================================================
To request your FREE subscription, to CANCEL your subscription, or to
submit or propose a calendar item or an article send E-mail to
"ttn@soft.com".
TO SUBSCRIBE: Use the keyword "subscribe" in front of your Email address
in the body of your Email message.
TO UNSUBSCRIBE: please use the keyword "unsubscribe" in front of your
Email address in the body of your message.
TESTING TECHNIQUES NEWSLETTER
Software Research, Inc.
901 Minnesota Street
San Francisco, CA 94107 USA
Phone: +1 (415) 550-3020
Toll Free: +1 (800) 942-SOFT (USA Only)
FAX: + (415) 550-3030
Email: ttn@soft.com
WWW URL: http://www.soft.com
## End ##