When working on a project, it's important that you have the assurance that your applications will perform well in production under various load scenarios. Don't overlook the need to performance-engineer Java Web applications.
Using input from a typical consulting assignment to implement a Web application in production, I'll review some observations on typical performance engineering practices, as well as basic performance concepts and some tools that can help.
The fantasy: Performance will meet expectations
One of the common fantasies in Web projects is that the timetable to build a production Web application should include time for requirements, design, development, packaging, user testing and deployment. In practice, those items are usually blocked out with real hours or days on a project plan. Looking at this list, one key item is missing: performance engineering. It is a task that project managers don't want to admit is required on most Web projects that go to production. Even architects like to pretend it's not that important a step in many cases.
A recent project for me was typical. The project manager had built a project plan with a committed budget and a fixed timeframe. Another group built the application and implemented it earlier for internal users. It was a heavyweight application with dynamic JSPs pulling together many resources, but it ran well with adequate response time for internal users.
After that, the project manager asked if I would help implement secure access to the Web application for a new group of external users. The company already had a network in place with a DMZ for Web servers, routers, firewalls with caching proxies, DNS servers and a T1 connection to an ISP.
He asked me for an estimate on the time it would take to implement a secure link for external users on their network. I reviewed several implementation options with him and gave him an estimate of three to four days of work. Based on experience, I explained any of the options would slow user response time, perhaps as much as 1 second or more typically.
He cut a PO for a fixed price for that deliverable. He wasn't interested in reviewing the scope of his project at this point. So he requested I setup an SSL reverse proxy server that would provide external users a secure link and not load the application server with the SSL overhead.
The reality: Performance IS an issue
Reality didn't match the fantasy in several ways. While my estimate of three to four days was adequate for most networks, it actually was a real challenge on this particular network. There were a variety of access issues to overcome specific to this network during my work. The bottom line: I delivered the SSL reverse proxy server connection to the Web application on budget.
What else happened? The major problem in the project was the failure to define a performance-engineering task in the project and budget for it. Several weeks after the link was completed, a frantic call came in that performance was unacceptable. And it was.
Reacting to poor performance
The project manager was hoping for a SIMPLE explanation with a SIMPLE solution. Sorry, no such luck. He asked if my setup was the cause of the 15-second or more response time. I knew it wasn't. He also explained they didn't want to spend anymore on implementing the application. No such luck again.
I defined some use cases and set up a load tester to measure the overhead of the SSL proxy server. Tests showed quickly that the SSL proxy server added between 1 and 1.5 seconds of overhead to an average transaction. Case closed on my work on the SSL proxy server.
The customer's performance-engineering work was just beginning. While we showed the proxy server had the expected impact, there was at least 10 or more seconds of delay on the network. Identifying the causes of this and resolving it quickly would be a real challenge for their network staff.
Despite the performance issues for this project, the commitment was made to role out the solution to external users within a month. This led to the expected flurry of concerned e-mails from the staff and a number of rapidly arranged conference calls with outside experts. No staff had been tasked with performance-engineering on the project to begin with. Not only was the in-house staff not fully prepared to analyze these problems, but they also lacked the tools needed. The project's rollout date now had considerable risks along with the expected user satisfaction with the application's performance.
Is this an unusual scenario for application deployment? No. It happens a lot.
A good performance engineering example
By comparison, I worked on WebSphere performance tuning for another company. The approach was to engineer for performance throughout the project. It started with the user requirements for performance. Here, 1,000 users would run the application concurrently and need a response time of 2 seconds or less. Then the application was designed with performance in mind throughout.
The test phase was more than user tests of the application function. They created a performance-engineering lab staffed with in-house and outside experts. They allowed time and money in the budget up front to model and engineer-performance in the expected environment. The rollout date for the application and the budget were second in priority to creating the right solution for the users.
They spent approximately two months working to engineer performance on this first large Web application for the company. They brought in outside experts and tools to help. They spread the Web application across multiple WebSphere servers with load-balancing and performance-monitoring tools. Initially they used the Network Deployment version of WebSphere for load balancing. Eventually they went with a simpler scheme to use the Apache HTTP server to do front-end load balancing.
Load testers were used to capture specific usage scenarios from actual users and then run them to stress-test the application. We ran application profilers to understand the workload characteristics of the application and identify potential areas of tuning. Loggers and monitors were used to observe performance times and record them in a database with the specific use-case scenarios. Distributed debug tools allowed us to isolate specific events in the application that accessed remote servers.
All the work paid off. Initially they could only get about 75 concurrent users running with 2-second response times. By the end, they had achieved their goal of 1,000 users consistently getting response times of 2 seconds or less.
What a nice feeling to know BEFORE you rollout your application that the users will be happy with the results and the business goals have a high probability of success.
How do you handle performance problems?
The FIRST question on performance is always WHY. Once you've identified the causes of poor performance, you can define your options to improve it.
Some tools for performance engineering
Several types of tools are useful for performance engineering. Each can make a great contribution to identifying and resolving problems with distributed, network applications.
There are many sources for performance tools. IBM offers a wide variety on many platforms. They also sell the Tivoli suite of management tools. Microsoft also offers tools to buy. Sometimes third-party tools are more popular for performance engineering than those from the big software vendors. Examples include JProbe and Mercury Interactive. Finally, yes, open-source software providers can be a big benefit too. Apache offers JMeter, a load test tool for Web applications that is excellent. It's now my preferred tool for performance-testing Web applications.
|Logger||Apache HTTP Logs||Selective logging of events in a server shows time request received and sent||Apache HTTP Server|
|Monitor||Tivoli||Interactive visibility into actual traffic at a given node in a network. Usually allows selective filtering based on source or target IP addresses.||Tivoli or other tool management suites. Microsoft offers MMC on servers. Linux vendors offer tools as well.|
|Debugger||IBM Distributed Debugger||Allows developers to remotely debug application components dynamically on remote servers. The Java debuggers are based on the JMX architecture for virtual machines allowing a JMX agent control of the application.|
|Application Profiler||JProbe or WDSC Advanced||Shows the performance and resource usage of an application internally -- which functions are slow, etc. Indicates where application redesign may pay off.||JProbe or WDSC Advanced|
|Load Tester||Apache JMeter||Captures user test-case scenarios and run them back to simulate expected workloads. Also provides reporting tools to view results.||Apache JMeter|
|Tracer||iSeries System COMM Trace tool||Similar to a monitor but set up in batch mode. Set and start the trace for a period or an event, then dump or report the trace results for analysis.||iSeries has a free built-in COMM trace tool that can monitor detail traffic on a line or controller.|
Performance focus points in Web applications
Web applications are complex. They integrate many types of resources from many servers in a network. They are accessed with complex routes from clients to the server they run in. Using the different types of tools above, you measure performance impacts across the different types of resources below in your Web application.
|WAN connection (to the Web)||The bandwidth (and related load usage profile) limit throughput for an application|
|LAN||The bandwidth (and related load usage profile) limit throughput for an application|
|Router||How well are requests routed between network domains?|
|Firewall||How efficiently are requests mapped and filtered across different networks?|
|DNS server||How fast does IP address resolution occur. This CAN be a BIG factor in some networks.|
|DHCP server||Dynamic assignment of IP addresses is usually not a factor, since it occurs only as address leases expire.|
|Security server||How are users authenticated to the network (local profile, LDAP, etc)? What is the encryption protocol, if any?|
|Proxy server||Connects an external user to your internal application server logically. Generally used to provide caching and secure connections. MAY be a bottleneck.|
|Web server||Standard HTTP server. MAY not be configured correctly for performance. Experts usually can do a great job with a server like the Apache HTTP server.|
|Web application server||WebSphere version 5 family (Express, Base, Network Deployment) can all perform well when configured correctly. So can Apache Tomcat. Don't use earlier versions of WebSphere! (You probably already knew that.)|
|Resource servers: database, images||DB2, Oracle, MySQL all can perform well in Web database applications.|
|Application server||What IS the application profile when run? Where are the hot spots? What can be changed?|
|Expected workload profile||What is the expected workload profile for any of the servers in the chain? Is there available capacity to meet your expected application workload?|
|Expected network load profile||What is the available network capacity to meet your application needs?|
Lessons learned on performance engineering
So you've seen the good and bad of performance engineering. You understand the types of tools you need, and you have some idea of the performance focus points for Web applications. So what have you learned from all of this? Hopefully, these concepts stand out:
- Plan ahead. Plan ahead. Plan ahead.
Performance engineering done at the requirements phase offers a lot more options than engineering that starts during the testing phase.
- Make the performance requirements a clear priority up front.
Negotiate with users what they want and what they can accept up front. It gives everyone a clear target to aim at during all phases of the project.
- Have the courage up front to properly resource a performance-engineering task.
If this is your first major Web application in your company, assume you'll need to make a considerable investment in performance engineering (skills, tools, experts, time) up front. Don't shortchange your budget here.
- Understand the performance-engineering skills your staff has.
Is your staff very experienced at end-to-end engineering for high-volume Web applications that integrate with your existing business applications? Certainly your staff has some good skills today, but where are they less skilled?
- Find good performance engineers to close the performance skills gap.
Yes, most companies already have some skills in-house, along with knowledge of your specific network Don't be satisfied with less than the best in this area. Bring in outside help where needed. Get second and third opinions just like you do with a doctor.
- Get the right performance tools for the job.
The test phase is NOT the time to discover you need better tools. That should be part of the performance engineering up front.
- Understand your leverage points to improve performance.
After testing, you will usually be able to identify specific factors that impact performance and know your options to improve it. In one network, I found WebSphere requests timed out. Tracing the problem, I discovered it was a DNS issue. The company had their DNS servers located 2,000 miles away and all requests were taking longer than 1 minute for DNS resolution.
- Set the right expectations with the users.
It always helps to let users know up front what range of performance may be possible. There are clearly associated budget costs for meeting different performance targets given workload and capacity. Users can supply the business case to validate the budget needed.
- Setup a performance audit program.
Yes, performance was good when you first turned on the order-entry application. Since then you've added batch data transfers to distributors, a physical inventory application over the Web at month's end, employee benefits over the Web and all your existing server applications. Yes, the additional new users added to order entry may now experience significant delays.
---------------------------------------About the author: Jim Mason, president of ebt-now, is an iSeries WebSphere engineer. ebt-now provides iSeries WebSphere, WebFacing project management, engineering, development and training services. You can reach Jim at firstname.lastname@example.org or call 508-888-0344.
- WDSC: How to build a Java Web application calling RPG
Jim Mason says IBM's WDSC tools offer the fastest, easiest way to Web-enable your existing applications for e-business. In this tip he shows you how to take an existing RPG and CL program you've created and Web-enable them with the Web Interaction wizard in WDSC.
- WDSC lets you tap into the power of XML
When it comes to Web development, XML is a powerful tool. But you don't need to be a skilled Java XML developer to make use of it. With WDSC you can easily create XML apps.
- Planning an iSeries Java education strategy
What's the "right" strategy to learn Java? The "right" strategy is whatever is RIGHT for your specific situation. Your first step is to define your e-business education strategy, looking at your Java application needs, your goals in learning Java, your background, resources available, and how you will use Java.