Why is my web app running slowly? -- Part 1.

This series of blog posts picks up on a topic I made mention of earlier, namely scalability models, where I wrote about how implicit models of application scalability often impact the kinds of performance tests that are devised to evaluate the performance of an application. As discussed in that earlier blog post, sometimes the influence of the underlying scalability model is subtle, often because the scalability model itself is implicit. In the context of performance testing, my experience is that it can be very useful to render the application’s performance and scalability model explicitly. At the very least, making your assumptions explicit opens them to scrutiny, allowing questions to be asked about their validity, for example.

The example I used in that earlier discussion was the scalability model implicit when employing stress test tools like HP LoadRunner and Soasta CloudTest against a web-based application. Load testing by successively increasing the arrival rate of customer requests assumes there is a relationship between the response time for web requests and the number of concurrent requests, namely

RT = f(λ)

where λ represents the arrival rate of requests (the standard notation for the term in queuing theory). In the context of a stress test this implicit scalability model is often correct – drive the request rate high enough and you are apt to drive some hardware or software component to saturation, at which point queuing delays will start to have an impact on web response time.

Of course, the rate customer requests arrive to be processed represents only one dimension of a web application’s scalability profile. Experienced performance testers understand that there are other factors that influence performance. For instance, in the crucial order processing portion of the application, the size of the order, the number of items in the customer’s shopping cart that need to be processed, etc., can also be very important scalability factors. The point I tried to make in the earlier blog entry was that all the relevant dimensions that impact the application’s scalability need to be addressed in performance testing to assess the quality of the release effectively. Another way to think about this is that the application scalability model you formulate is a hypothesis that the performance and acceptance testing process is designed to test. All of which reinforces the notion that significant benefits can be derived from making the scalability assumptions that are implicit in performance testing explicit.

In another variation on this theme, I will focus in this series of blog posts on a particular model of web application performance that has proved extremely influential. This is something I call the YSlow model of web application performance, named after the YSlow performance tool, originally developed at Yahoo, associated with the work of Steve Souders. To begin, I will strive to make the scalability model implicit in web performance tools like YSlow explicit.

I will also discuss how the YSlow scalability model influenced the development of other web application performance tooling, culminating in the W3C specification of a navigation and timing API that provides access from JavaScript to web application performance measurements. The W3C spec for the web client navigation and timing API is currently imbedded in all the major web clients, including Chrome, Mozilla, Foxfire and Internet Explorer. I will drill into the W3C navigation and timing APIs to demonstrate how to gather and utilize these performance measurements, or Real User Measurements (RUM), as they have become known. The navigation and timing API is a great help to anyone with a need to understand the end-to-end web application response time experience of actual, real-life web site customers. I expect the navigation and timing API to spawn a whole new generation of web application performance tools that will exploit this valuable measurement data.

In addition, I want to cast a critical eye on the YSlow model of web application performance and highlight some areas where the reality of web application performance can depart from expectations raised by the model. There are some areas where the YSlow model is just a little too simple for the burgeoning complexity of networked-enabled applications developed for the web, the cloud, or both. Using an example of a data-rich ASP.NET application that requires extensive processing at the web server and the back-end database to generate Response messages, I will try to show what additional measurements may be required to solve performance and scalability issues that transcend the diagnostic capabilities of YSlow and similar tools.

Why is this web app running slowly?

To understand what a web application performance tool like YSlow does, it will help to be able to refer to a concrete example. Accordingly, I will discuss running YSlow against a web application that was perceived as running slowly. The application in question is also one that I care about. Figure 1 shows a screen shot of that application in its “before” stage when it exhibited serious performance and scalability problems.

Figure 1. A screen shot of an ASP.NET web application that runs slowly.

The app is a graphical reporting application devoted to visualizing performance measurements which exist in the form of time series data. It is a web front end to a back-end SQL Server-based repository of performance data that my software company provides to its customers. For purposes of this discussion, its most salient characteristic is it is a data-rich query application, which then renders the results in high resolution charts using Microsoft’s ASP.NET server-side technology. It relies specifically on the web forms Chart facility in the .NET Framework to generate presentation-quality charts and graphs, creating two such graphic images per query, as illustrated. Relatively large jpeg images of charts are generated on the server based on the result set of the designated query. These jpeg files are then transmitted to the web form over the network. In an application development environment where I was seeking to understand why it was running so slowly at times, the performance issues that were evident were quite convincingly not due to networking performance, since the web client, IIS web server, and back-end database all resided on the same (physical) Windows machine.

Other relevant features of the example application that was the subject of a performance investigation proved to include the following:

·       

• The queries to generate the charts are defined using a set of additional web forms to create a re-usable template for the report. These chart definition templates are also stored in the same SQL Server database where the performance data lives, allowing the queries be re-executed in subsequent interactions and sessions.

• Dynamic elements of the database queries that are resolved at run-time include a calendar control for date selection and menus for selecting the machine or machines of interest.

Using YSlow.

Whenever you have reports of a web application that is running slowing, looking for answers from YSlow or similar performance tools is quick and easy. Tools like YSlow provide expert advice about why a web page such as this takes so long to load. It estimates the page load time from the various HTTP objects contained in the web page document that the browser constructs during page composition. This composition is performed within the web client in response to the specific instructions that determine page layout. These instructions are encoded in html, style sheets, cookies, image files and scripts. Web browsers perform this page composition and rendering based on the contents of the DOM, the Document Object Model, which is assembled from static elements identified in html and dynamic modifications to the DOM that occur when associated script code executes. Rather than worry about all HTTP elements in the DOM, YSlow is mainly concerned with analyzing the files referenced in the HTML, since each file referenced requires the web client to issue a GET Request to a web server and await the Response message in which the file’s contents are returned.

To illustrate HTML references to external files, see the following snippet of html that I pulled from an Amazon.com page devoted to customer reviews of one of the products that is for sale on the Amazon commercial site:

<!-- nav-config-asset-injection US::desktop::standard::31406:T1&33556:C::auiDebug=0::isSecure=0::jsOnEvent= navc-eXpxmA4nBPBlMbmWwZRHGJ1GDQQC6QoQsW7GbDEV+UO0nPvAKHhHpV3zhaAXSzyFWGYzZBN9pKc= rid-0HCNMQC3W4AQCDBMGZAJ seq-81 (Mon Jul 28 19:29:53 2014) **CACHED-BY-NCCC** -->

<link rel="stylesheet" href="http://z-ecx.images-amazon.com/images/G/01/AUIClients/NavAuiAssets-7a37ff9cf24ac9bd811dc5cbdec16d08155faabc.us.min._V2_.css" />

The first line in green is a comment, so it has no effect on the DOM. It was evidently inserted into the HTML markup for documentation and debugging purposes. The line that impacts the DOM is the link statement on the second line that references a .css style sheet file that the page needs. The web browser will attempt to resolve the link statement by issuing a GET Request for the URL indicated in the href (short for “HTML reference”) tag. The URL references an auxiliary Amazon web site named http://images-amazon.com where this particular style sheet file can be found. HTML references to external files are expensive. The web browser first has to locate the IP address of this web server using DNS. A GET Request is then issued for the object using the proper IP address referencing either TCP Port 80 or 441, which are the TCP Port address associated with the HTTP and HTTPS protocols, respectively. Prior to sending the GET Request, the web browser must first establish a TCP session with that web server. And, when the GET Request is finally sent, the browser must await the Response message. Once the URL is resolved and the .css style sheet file referenced is returned in a Response message, the web browser will use the style sheet tags to format the any elements in the DOM that the style sheet applies to when the page is ultimately rendered.

If that seems like a good deal of effort is involved in web page composition, it is because there is. 

The key insight baked into the YSlow performance rules is that the processing time inside the web browser to apply the style sheet is probably trivial compared to the time it takes to resolve the URL over the network and retrieve the file using the HTTP protocol.

YSlow itself was based on the work of Steve Souders, who was originally at Yahoo, but currently hangs his hat at Google. Souders is the author of a popular book on web application performance called High Performance Web Sites, which explains in some detail the rationale behind the YSlow tool. YSlow inspired the PageSpeed Insights tool that is currently available for Google’s Chrome web client, highly esteemed among web developers, and the performance-oriented Developer Tools in Microsoft’s Internet Explorer. YSlow also influenced the development of other, similar tools, including the WebPagetest.com performance testing site and Visual Round Trip Analyzer, for example, that was developed by a team responsible for web application performance for Microsoft web properties like HotMail.

In order to use YSlow, you have to first install the YSlow extensions into your browser. (YSlow supports Chrome, Safari, Firefox, and Opera, among others.) Then, on command, YSlow re-loads your web page and interrogates the DOM. It identifies each component of the DOM that was loaded by the page, determines its size, the contents of its headers, and other characteristics that can affect page load time performance. YSlow then generates a report that analyzes the page and provides guidance for reducing the amount of the time it would take to load the page.

Note that YSlow does not actually measure the time it takes to re-load the page it is analyzing. This is mainly due to the fact that caching of the page’s content by the browser and elsewhere on the network – caching is a ubiquitous feature of web technologies – improves the time to reload the page’s content significantly. This is a crucial point that we will revisit when we look at other web performance tools that do actually try to measure web application response time from the point of view of the web client application. It is in that context that I will also review the relatively, recent standardization effort backed by the W3C, the consortium that develops the standards web applications must adhere to, to incorporate performance-oriented timing data into the DOM where it can be gathered in a consistent fashion using JavaScript code.

But, meanwhile, back to YSlow. With the information in hand that it gathered about the components of the page that need to be loaded, YSlow then applies a number of performance Rules and calculates a grade for each rule, where “A” is excellent and “E” or “F” are failing grades. YSlow’s evaluation of the web page from our case study is shown in Figure 2. 

Figure 2. The report YSlow generates when the tool reloads the example web page shown in Figure 1.

We see in Figure 2 that the web page being analyzed receives a near failing grade of “E” for the first and foremost of the YSlow performance rules, which is to make fewer HTTP requests.

To understand why this performance rule is so important for web application performance, it will help to dive deeper into the HTML protocol that is used in web page composition. At this point in the discussion, it will also be helpful to derive the scalability model for web application performance that is implicit in YSlow and similar performance tools.

I will take up those topics in the next blog post in this series.