QW2000 Paper 4W2

Mr. Steven Rabin
(Interworld Corp.)

eCommerce Performance Management Lifecycle -- Benchmarking, Methodology and Criteria


Presentation Abstract

One of the key aspects of a successful eCommerce initiative is performance. While this is often overlooked or taken for granted very few customers will return to a site that exhibits sluggish performance. Features, functions, robust tools, etc. are all important to the success of an eCom site but in the long run don't mean much if customers don't get efficient responses from the site their shopping. This session discusses the eCommerce Performance Management Lifecycle. This includes the key elements in understanding customer/site dynamics and how to use this information to design an eCom based performance testing benchmark (including the methodology that goes along with it). This includes workload characteristics, transactional definitions and site assumptions. A framework for eCom performance benchmarking and the criteria that need be measured is also introduced. Real world scenarios from a variety of sites are analyzed from the lessons learned perspective.

The life cycle takes into account the 8 key elements of delivering and maintaining a robust eCommerce environment. This includes understanding customer behavior, setting performance goals, benchmarking, tuning and adjusting for peak periods. The eCommerce Benchmark must be designed to measure the key processes utilized during the customer shopping experience (B2B and/or B2C). Industry experience and customer input must be incorporated into the benchmark's design to represent the typical workload placed on a Commerce server. Based on the results organizations can predict how the site will perform under tough, real-world conditions. For example, the benchmark must consider and measure the following granular activities:

1. authenticating an existing user
2. registering a new user or changing the profile of an existing user
3. browsing a catalog by category and/or product
4. processing an order
5. accessing key site pages
6. searching for products, full text and parametric
7. adding, changing, deleting items from a shopping basket (or purchase order)
8. purchasing/processing a committed order.
These activities must be incorporated into a series of transactions that measure round trip performance, execution under load and scalability. The benchmark must be designed to measure transactional workloads that exercise the key components utilized during an eCom session. This includes:

* concurrent browser sessions
* dynamic page generation with database access and update
* processing multiple web objects
* on-line transaction execution scenarios
* multiple databases consisting of complex relationships
* efficiently resolving resource contention
Consideration must be given to capturing and interpreting the collected data. Translating these metrics into meaningful statistics provides the information required to make key decisions. The following are examples of what need be measured and reported:

1. orders per minute
2. registrations per minute
3. complex browsing per minute
4. simple browsing per minute
5. searches per minute
6. transactions per minute
7. dynamic vs. static pages served per minute
Finally, software and hardware environmental factors must be taken into account to insure the measured results are meaningful. Operating System, Web Server, Database Server, SSL (Secure Socket Layer) are software examples. RAM, CPU, disk access speeds and HTTP routing are hardware examples.

About the Author

Steven Rabin is Chief Technology Officer of InterWorld Corp., a New York firm offering a variety of electronic commerce application technology solutions.

He is a certified CDP and has published articles in Computer Language, Data Based Advisor, and Systems Development as well as authoring the "Mission Critical Views" column in Application Development Trends.