sss ssss rrrrrrrrrr
ssss ss rrrr rrrr
sssss s rrrr rrrr
ssssss rrrr rrrr
ssssssss rrrr rrrr
ssssss rrrrrrrr
s ssssss rrrr rrrr
ss sssss rrrr rrrr
sss sssss rrrr rrrr
s sssssss rrrrr rrrrr
+===================================================+
+======= Testing Techniques Newsletter (TTN) =======+
+======= ON-LINE EDITION =======+
+======= October 1995 =======+
+===================================================+
TESTING TECHNIQUES NEWSLETTER (TTN), On-Line Edition, is E-Mailed
monthly to support the Software Research, Inc. (SR) user community and
provide information of general use to the world software testing commun-
ity.
(c) Copyright 1995 by Software Research, Inc. Permission to copy and/or
re-distribute is granted to recipients of the TTN On-Line Edition pro-
vided that the entire document/file is kept intact and this copyright
notice appears with it.
TRADEMARKS: STW, Software TestWorks, CAPBAK/X, SMARTS, EXDIFF,
CAPBAK/UNIX, Xdemo, Xvirtual, Xflight, STW/Regression, STW/Coverage,
STW/Advisor and the SR logo are trademarks or registered trademarks of
Software Research, Inc. All other systems are either trademarks or
registered trademarks of their respective companies.
========================================================================
INSIDE THIS ISSUE:
o NETHERLANDS STW DISTRIBUTOR
Part of a new series, highlighting SR's international distributors.
o THE PENTIUM BUG: AN INDUSTRY WATERSHED (Part 2 of 4)
Dr. Boris Beizer's story on the hardware defect of the century!
o NINTH INTERNATIONAL SOFTWARE QUALITY WEEK (QW'96)
Call for participation in SR/Institute's 9th annual conference.
o SPECIAL STW PROMOTIONAL OFFER
Details on a special all-users on one LAN deal.
o CALENDAR OF EVENTS
o TTN SUBMITTAL POLICY
o TTN SUBSCRIPTION INFORMATION
========================================================================
NETHERLANDS STW DISTRIBUTOR
Precision Software B.V., Veenendaal, NETHERLANDS
Precision Software B.V. in Veenendaal, The Netherlands, is part of the
Precision Software Group with the main office in Frankfurt, and offices
in France, UK and The Netherlands.
Precision Software sells and supports software development tools for C
and C++ developers. The following products are part of PVI's product
line: Software Testing (Software TestWorks), Portable Graphical User
Interfaces (XVT), Client/Server Development Tools (Noblenet's EZ-RPC),
Database Connection (Visigenic's ODBC tools, RogueWave DBtools).
Herbert Weustenenk holds a degree in Electronics Engineering; he is in
charge of sales of Software TestWorks. Sander Dinsbach holds a degree
in Computer Sciences, and leads technical support of Software TestWorks.
Many of the large organizations in the Benelux area are customers of
Precision Software B.V., including Shell, Fokker, ESA, Dow, Philips,
Agfa, Barco, Generale Bank, etc.
Contact:
Mr. Herbert Weustenenk, PVI Precision Software B.V., Koningsschot
45 3905 PR Veenendaal NETHERLANDS.
Phone: +31 (318) 552.588; FAX: +31 (318) 551.506; Email:
100334.315@compuserve.COM
EDITORS' NOTE: Beginning this month we will be devoting a small part of
the TTN-Online space to features about SR's international distributors
and US Domestic sales agents. This item is the first in a series.
========================================================================
THE PENTIUM BUG: AN INDUSTRY WATERSHED (Part 2 of 2)
Dr. Boris Beizer
Editor's note: This article has been divided into four parts, which
will be published over the next four consecutive editions of the TTN.
------------------------------------------------------------------------
Copyright, Boris Beizer, 1995. Permission is granted to print this docu-
ment for personal use and to redistribute it for internal corporate use
subject to the provision that this entire copyright notice must appear.
Use of this document or parts thereof in any commercial publication or
document without the written permission of the author is strictly prohi-
bited.
------------------------------------------------------------------------
1. General
Two guys in the Australian Outback, a dozen polar Eskimos and a tribe in
the Amazon jungle have not heard about the Pentium bug. People oblivi-
ous to floating point arithmetic before INTEL's customer relations
fiasco now speak angrily about this bug and how it will affect their
lives. There's a half-dozen class action suits afoot claiming warrantee
breaches and false advertising as well as few stockholders' suits
[INTE95]. This bug has probably been blamed for every human and techni-
cal misery from anorexia to zygopteran infestations.
Was this bug serious? Yes and no. Technically, it wasn't the first, it
wasn't the worst and it won't be the last of its kind. But from the
point of view of its impact on the computer industry, it is probably the
most significant bug ever. It has forever transformed the industry. It
is a watershed for all of us.
Some of you who are in software might mistakenly believe that this was a
hardware bug and that it therefore doesn't affect you: if so, you're
wrong on both counts. It was a software bug, but one that happened to
be compiled into silicon rather than RAM. But it doesn't matter if it
was hardware or software because the users, as we have just learned, do
not make that distinction.
<>
2.4. Who's Affected and How They Determined That
INTEL's position on the impact of this bug was that heavy spreadsheet
users would be affected on an average of once in 27,000 years. IBM's
position, by contrast, was that the typical spreadsheet user would be
affected once every 24 days. Neither positions are truly believable
because the values depend critically on making correct assumptions about
the users. Not only must the frequency of spreadsheet use be known, but
also detailed technical things such as: the specific numbers used in
numerators and denominators of floating point divisions, the likelihood
that such numbers appear in real spreadsheets, the likelihood that both
an at-risk numerator and an at-risk denominator are created as a result
of a calculation, the size of the spreadsheet, the number of divisions
in it, how many digits of precision are needed by the user for the
potentially affected cells, how often the numbers that led to the bad
numerator and denominator are changed, and how often recalculations
occur. Most of these factors did not appear in either INTEL's or IBM's
analyses.
INTEL claims that spreadsheet users will be affected only once in 27,000
years: IBM contends that the figure is once in 24 days. While both com-
panies offered analyses of the probability of a user being affected by
this bug, having read both, both analyses could be shifted by orders of
magnitudes to suit the needs of the analyst. Of the two, I'm more
inclined to accept INTEL's analysis because:
1. INTEL did both a mathematical model and actual experiments. The
experiments confirmed the analysis and, in fact, showed that the
analysis was pessimistic. The experiments consisted of a mind bog-
gling run of over one trillion test cases.
2. The INTEL study was a carefully written, properly edited research
paper with references. The IBM white paper looked unprofessional
and hasty.
3. INTEL's conclusions were in part based on a user profile drawn from
their own internal spreadsheet users. While one might observe that
INTEL's internal spreadsheet users are not a disinterested group, IBM
did not offer comparable data based on its many spreadsheet users.
IBM's user profile, such as it is, is conjectural and does not appear
to be based on actual usages, disinterested or otherwise.
4. INTEL provided a point-by-point refutation of IBM's analysis and
showed that some IBM assumptions actually supported INTEL's position.
IBM has offered no counter-refutation.
IBM's response to the chip's problem strikes me as fraught with ulterior
motives. The INTERNET hysteria (and the problem, in the public's mind)
had almost died out when IBM fanned the flames to greater heights by
announcing that they would not ship systems that used the faulty Pentium
chip. A possible ulterior motive is IBM's stake in Power-PC based sys-
tems it wouldn't hurt to erode INTEL's credibility. Interestingly,
other companies with a vested interest in the Power-PC such as Apple and
Motorola have kept a low profile on this bug. Will we see a similar
announcement in August or September when IBM announces that they won't
ship systems with Windows 95 because of a real or suspected bug in an
attempt to boost lagging OS2-Warp sales?
Who might have real information about spreadsheet usage and other high-
powered users with which more believable analyses could be done? The
spreadsheet companies of which Lotus and Microsoft are dominant, and
technical software vendors such as SAS Institute. I called all three
for comments. They did not have such an analysis, or if they did, it
wasn't for public consumption. All three wisely decided to keep a low
profile and not get involved in the cross fire. All three had software
fixes available as downloads.
Is it an important bug from the point of view of possible risk to the
public? Probably not, but who knows? Could we find out? Probably,
but at a cost that would exceed all possible benefits. If we wanted to
know, we would have to learn more about real spreadsheet user profiles
than anybody knows today. And to what end? Such detailed numerical
specifics, even if obtained at a very high cost, would have little or no
substantive impact on the design of spreadsheets or on the way we use
them similarly for other intensive floating point users.
3. Why A Watershed?
3.1 Press and Other Relations
The most telling comment on the entire fiasco was provided by a Lotus
Development spokesman (also repeated in a press release) that Lotus had
many more inquiries from the press than they did from users concerned
about the bug [LOTU95]. INTEL's spokesman also said that calls from the
press outnumbered calls from concerned users [INTL95].
Watching this drama unfold reminded me of one of those feeding frenzies
you see on a National Geographics special on sharks. First, the bug's
discovery got circulated on the INTERNET. When INTEL confirmed it, it
sent out the usual press releases to the trade press and notices to
software developers and system designers: releases similar to hundreds
of comparable semi-public bug reports every week by many industry com-
panies. As I said earlier, it wasn't the first and hardly the worst bug
of its kind. The INTERNET was a new factor because there were so many
first-time computer users on it and because many of them were new or
prospective Pentium owners. The INTERNET alone raised this bug to a new
high in the public consciousness. As usual, except for the trade press,
most reporters threw the press releases in the wastebasket. Then CNN
(the other new factor) did a story on the bug, followed shortly
thereafter by pieces in The Wall Street Journal and The New York Times.
Local newspaper editors decided that if CNN thought there was a story in
it, then they had better cover it and the feeding frenzy was on. This
was followed in rapid succession by night show hosts and comics getting
into the act, Pentium jokes on the INTERNET, etc., etc., topped off by
IBM's stunning announcement. That was probably the most damaging act,
because it lent credibility to the hysteria: in the public's mind, facts
aside, the computer industry is Microsoft, IBM, and INTEL, in that
order.
INTEL didn't help matters. It took its usual approach to such bugs,
saying something like "We don't think that it's a problem for any but a
few users who do very specialized calculations. If you're one of those,
then we will fix it for you." However, the way the public heard it was
something like: "Idiots! If you're not a PhD mathematician or a rocket
scientist doing really sophisticated mathematics, you have no right to
be concerned. Tell us what use you make of our chip and we in our wis-
dom will tell you if you are or are not qualified to get the replace-
ment! The rest of you ignorant fools will have to be satisfied with the
buggy chip!"
I'm not a psychologist but it seems to me that there was a lot of latent
anger out there against our whole industry and that INTEL, fanned by
INTERNET and press flamings (and INTEL shooting off its own toes), took
the fall. Was the anger a reflection of the frustration they've been
feeling all along over buggy software? Impossible installation pro-
cedures? The economy? Or just the miasma of anxiety over a rapidly
changing world that forces them to use technology? Whatever the cause,
the anger, the rapidity with which it spread, the heights it reached,
are all new realities with which we will have to contend in the future.
3.2. User Expectations
The user's expectations were made painfully clear. They still expect
and want bug-free products. And they want it at a reasonable price.
That's an expectation we can neither theoretically nor practically ful-
fill. Assuming that they'd accept a statistical warrantee (they won't),
what would be the cost of attempting to satisfy their expectations? A
$275,000 chip? The industry can't be held to a higher quality standard
than all other human endeavors. And it can't be held to a standard that
is known to be theoretically impossible. That expectation must be
changed. This is especially important for the American industry because
in the U.S., more so than in any other country, there's a surfeit of
lawyers ready to file class- action and other suits on anything that
affects the consumer whether or not such suits have merit. Whether or
not the suits have merit, just fighting them and even if you do win,
diverts valuable human and financial resources that would be better
spent perfecting the product and its manufacture. Unrealistic consumer
expectations and a litigious society puts the American portion of the
industry at a disadvantage with its international competitors.
3.3. Consumerism
That's another change brought into focus by the bug. Buyers in the
past, especially of high-end products such as the Pentium, had been pro-
fessionals, many of them in the computer industry. That's changed. The
technically adept no longer dominate the market. The general consumer
does. And, as INTEL learned, you can't handle general consumers the way
you do professionals.
<>
5. References
ATKI68 Atkins, Daniel E. Higher Radix Division Using Estimates
of the Divisor and Partial Remainder. IEEE Transactions
on Computers, Vol. C-17, #10, Oct. 1968, pps. 925-934.
BEIZ90 Boris Beizer, Software Testing Techniques, 2nd Edition,
Van Nostrand Reinhold, 1990.
BEIZ95 Samuel Beizer. Private communication.
IBMS95 Pentium Study, IBM Research White Paper, December 12,
1994.
INTE95 Interview of Howard High, INTEL, February 2, 1995.
LOTU95 Interview of Peter Cohen, Lotus Development, February 2,
1995.
NADL56 Nadler, Morton. A High Speed Electronic Arithmetic Unit
for Automatic Computing Machines, Acta Tech (Prague), #6,
1956, pp. 464-478.
SASS95 McGrath, Sue. Private Correspondence, SAS Institute,
February 9, 1995.
SHAR94 Sharangpani, H.P., and Barton, M.L., Statistical Analysis
of Floating Point Flaw in the Pentium Processor. INTEL
Corporation white paper, November 30, 1994.
------------------------------------------------------------------------
Boris Beizer, PhD
ANALYSIS
1232 Glenbrook Road
Huntingdon Valley, PA 19006
PHONE: 215-572-5580
FAX : 215-886-0144
Email: BBEIZER@MCIMAIL.COM
========================================================================
NINTH INTERNATIONAL SOFTWARE QUALITY WEEK 1996 (QW`96)
Conference Theme: Quality Process Convergence
San Francisco, California -- 21-24 May 1996
QW`96 is the ninth in a continuing series of International Software
Quality Week Conferences focusing on advances in software test technol-
ogy, quality control, risk management, software safety, and test automa-
tion. Software analysis methodologies, supported by advanced automated
software test methods, promise major advances in system quality and
reliability, assuring continued competitiveness.
The mission of the QW`96 Conference is to increase awareness of the
importance of software quality and methods used to achieve it. It seeks
to promote software quality by providing technological education and
opportunities for information exchange within the software development
and testing community.
The QW`96 program consists of four days of mini-tutorials, panels,
technical papers and workshops that focus on software test automation
and new technology. QW`96 provides the Software Testing and QA/QC com-
munity with:
Quality Assurance and Test involvement in the development process.
Exchange of critical information among technologists.
State-of-the-art information on software test methods.
Analysis of method and process effectiveness through case studies.
Vendor Technical Presentations.
Two-Day Vendor Show.
QW`96 is soliciting 45 and 90 minute presentations, half-day standard
seminar/tutorial proposals, 90-minute mini-tutorial proposals, or propo-
sals participation in a panel and "hot topic" discussions on any area of
testing and automation, including:
Cost / Schedule Estimation
ISO-9000 Application and Methods
Test Automation
CASE/CAST Technology
Test Data Generation
Test Documentation Standards
Data Flow Testing
Load Generation and Analysis
SEI CMM Process Assessment
Risk Management
Test Management Automation
Test Planning Methods
Test Policies and Standards
Real-Time Software
Real-World Experience
Software Metrics in Test Planning
Automated Inspection
Reliability Studies
Productivity and Quality Issues
GUI Test Technology
Function Point Testing
New and Novel Test Methods
Testing Multi-Threaded Code
Integrated Environments
Software Re-Use
Process Assessment/Improvement
Object Oriented Testing
Defect Tracking / Monitoring
Client-Server Computing
IMPORTANT DATES:
Abstracts and Proposal Due: 15 December 1995
Notification of Participation: 15 February 1996
Camera Ready Materials Due: 15 March 1996
FINAL PAPER LENGTH:
Papers should be limited to 10 - 20 pages, including Text, Slides
and/or View Graphs
SUBMISSION INFORMATION:
Abstracts should be 2-4 pages long, with enough detail to give
reviewers an understanding of the final paper, including a rough
outline of its contents. Indicate if the most likely audience is
technical, managerial or application-oriented.
In addition, please include:
o A cover page with the paper title, complete mailing and e-mail
address(es), and telephone and FAX number(s) of each author.
o A list of keywords describing the paper.
o A brief biographical sketch of each author.
Send abstracts and proposals including complete contact information to:
Ms. Rita Bral
Quality Week '96 Director
Software Research Institute
901 Minnesota Street
San Francisco, CA 94107 USA.
For complete information on the QW'96 Conference, send E-mail to
qw@soft.com, phone SR Institute at +1 (415) 550-3020, or, send a FAX to
SR/Institute at +1 (415) 550-3030.
========================================================================
SPECIAL STW LAN-LICENSE PROMOTIONAL OFFER
Subject: WELCOME SUITE -- SUCH A DEAL!
Do you need help testing your comprehensive GUI or Client/Server appli-
cations with tools that work? Do you want an opportunity to enhance
your software quality commitment to your customers without spending a
fortune? Software Research believes in helping software development
organizations meet and exceed their drive for quality and excellence.
And we apply the same standards to our own products, so we continue to
improve our own testing tools.
We are very excited about the newest capabilities in our family of
integrated tools, Software TestWorks (tm): STW/Regression 3.0 has a new
breakthrough triple-mode capture/playback tool (CAPBAK/X 5.1); and
STW/Coverage 3.0 has a spectacular compiler based engine and multiple
ways of producing reports easily.
Now we want YOU to get the BENEFITS of better tested code AND save
costs!
From October 16, 1995 through December 15, 1995 Software Research will
sell you a special "Welcome Suite (tm)" license of Software TestWorks
for $50,000. You get:
o All the users you want on one LAN.
o The complete Software TestWorks suite of integrated testing tools
including:
STW/Regression 3.0, STW/Coverage 3.0, and STW/Advisor 2.6.
o Support for one language of your choice: C or C++.
o The choice for any ONE Unix platform of the many platforms that Soft-
ware TestWorks runs on: SPARC, SunOS; SPARC, Solaris: IBM RS-6000,
AIX; HP-9000/7xx, HPUX.
You will be delighted to find that:
o Three important, but independent, operating modes are now inter-
operable: TrueTime, Object Mode and Character Recognition.
o TrueTime plays back the exact recording with complete synchroniza-
tion.
o Optical Character Recognition will extend the life of your test
script by allowing any changes in window layouts or fonts.
o Cross platform file handling allows you re-use of the same script on
different platforms, across networks and across multiple releases.
o Switching between TrueTime Mode and Object Mode in a single test run
gives unique flexibility in testing! It gets results faster too!
o "XVirtual" supports client/server load generation from single or mul-
tiple workstations.
o The new "C" language interpreter allows creation of all tests in C.
o Integration of coverage analysis into your standard "build-test-edit"
process is as easy as a simple one-line change.
o Full support for all standard versions of C and C++ for most major
UNIX platforms is included.
o Multiple ways to look at coverage results through easily produced
reports:
- textual reports (easy to include in internal documents)
- graphical views of the code show coverage level through
numerical and color annotation
- quick-look interactive tabular report solving the need for
quick analysis of results
o Branch and call-pair coverage in a single test run is provided saving
testers valuable time and shortening production time to market
release.
In short: We want to welcome all of your testers to use our tools.
There has never been a more powerful, flexible, easy to use, and fun
automated testing suite on the market, for a better price! Why don't you
find out for yourselves. But the more important cost saver in the long
run is that you will be delivering code that is as close as possible to
flawless. Hurry while the offer lasts.
Certain restrictions and limitations to this offer apply. For a com-
plete description of this special LAN-license offer send return E-mail
to: sales@soft.com or contact SR at +1 (415) 550-3020; FAX +1 (415)
550-3030.
========================================================================
------------>>>CALENDAR OF EVENTS<<<---------------
========================================================================
Here is a list of upcoming events of interest.
"o" indicates that Software Research, Inc. will lead or participate in
these events.
- 16 Oct: Testing Smalltalk Applications; OOPSLA '95 Workshop,
Contact: Barbara Yates, Tel: 503-657-7703, E-mail:
barbara.bytesmiths@acm.org
- 16-20 Oct: International Conference on Software Maintenance '95,
Opio, France, Contact: Mari Georges, Tel:
+ 33 1 49 10 5398, Fax: + 33 1 49 10 0615, E-mail:
mari@capsogeti.fr
- 23-26 Oct: 5th International Conference on Software Quality '95,
Stouffer Renaissance Hotel, Austin, TX, Contact: Red
Sturdevant, Tel: 512-328-4264, 512-794-0033,
512-282-8362 Fax: 512-328-3260, 512-794-0001 E-mail:
wstur@aol.com
- Oct 24-26: Client/Server Application Packages, Hyatt Regency,
Orlando International Airport, Orlando, Florida,
Contact: Diane Munini or Jean Desmarais, Tel:
508-470-3870, Fax: 508-470-1992
- 30 Oct-1 Nov: ABUI Fall '95 Conference, San Jose Convention Center,
San Jose, CA, Contact: Tricia Laursen, Tel:
508-779-5520/2724
- 5-10 Nov: TRI-Ada '95, The Disneyland Hotel, Anaheim, CA Contact:
Mr. Paul Grazda, Tel: 508-443-3330 X 1227,
1-800-833-7751 Fax: 508-443-4715, Email:
dok.boston@applelink.apple.com,
triada.dok@notes.compuserve.com
o 27-30 Nov: EuroSTAR'95, Contact: EuroSTAR Conferences Limited,
4th Floor, 7 Hanover Square, London W1R 9HE, Tel:
44-171-4934229 Fax: 171-3553738, Email:
eurostar@evolutif.demon.co.uk
========================================================================
------------>>> TTN SUBMITTAL POLICY <<<------------
========================================================================
The TTN On-Line Edition is forwarded on the 15th of each month to sub-
scribers via InterNet. To have your event listed in an upcoming issue,
please e-mail a description of your event or Call for Papers or Partici-
pation to "ttn@soft.com". The TTN On-Line submittal policy is as fol-
lows:
o Submission deadlines indicated in "Calls for Papers" should provide
at least a 1-month lead time from the TTN On-Line issue date. For
example, submission deadlines for "Calls for Papers" in the January
issue of TTN On-Line would be for February and beyond.
o Length of submitted items should not exceed 68 lines (one page).
o Publication of submitted items is determined by Software Research,
Inc., and may be edited as necessary.
========================================================================
----------------->>> TTN SUBSCRIPTION INFORMATION <<<-----------------
------------------->>>NEW INSTRUCTIONS!!<<<-------------------
========================================================================
To request a FREE subscription or submit articles, please send E-mail to
"ttn@soft.com".
TO SUBSCRIBE: please use the keywords "Request-TTN" or "subscribe" **AND
INCLUDE YOUR EMAIL ADDRESS** in the Subject line of your E-mail header.
To have your name added to the subscription list for the biannual hard-
copy version of the TTN -- which contains additional information beyond
the monthly electronic version -- include your name, company, and postal
address in the body of the mail message.
TO CANCEL: include the phrase "unsubscribe" or "UNrequest-TTN" **AND
YOUR EMAIL ADDRESS** in the Subject line.
Note: To order back copies of the TTN On-Line (August 1993 onward),
please use the keywords "Back issue request" in the Subject line, and
please specify the month(s) and year(s) in the body of your message when
E-mailing requests to "ttn@soft.com".
TESTING TECHNIQUES NEWSLETTER
Software Research, Inc.
901 Minnesota Street
San Francisco, CA 94107 USA
Phone: (415) 550-3020
Toll Free: (800) 942-SOFT
FAX: (415) 550-3030
E-mail: ttn@soft.com