User Manual

1. Introduction

A typical website is supported by a multi-tier, multi-server backend. Such websites need performance analysis carried out to answer questions such as shown in Fig. 1. Capacity and performance analysis of such systems can be done by measurement as well as modeling. Load testing can be done to determine the capacity of a particular deployment and to determine its performance under the workload that is synthetically generated. For creating a (queuing) model of the back-end servers, more detailed server-side performance measurements are required.

AutoPerf is an automated load-generator and profiler which requires minimal input and configuration from the user, and produces a comprehensive capacity analysis of a Web application, as well as server-side resource usage profile of the requests that are served by this application.

The tool requires only the workload and deployment description of the distributed system, and automatically derives various parameters that typical load generator programs need, such as maximum number of users to be emulated, number of users for each experiment, warm-up time, etc. The tool also does all the co-ordination required to generate the resource usage per transaction for each server.

 

Figure 1: A typical multi-tier Web site back-end

2. Guiding Principle and High-Level Functionality

AutoPerf was developed with the following guiding principle in mind: a lot of tasks involved in running performance tests are tedious and routine, and thus should be automated.  

First, consider the problem of carrying out a traditional capacity analysis of an application. Typically, the following curves are required:

  • Response time vs number of users
  • Throughput vs number of users

The usual expectation is that these plots be generated from a low value of number of users, to the maximum supportable number of users. But how does one know what this maximum is? With existing load generators, one has to do this by trial and error. For running every load test, one usually has to specify a min and a max, and a step size for increase. With AutoPerf, none of these parameters need to be specified. They are determined automatically by AutoPerf using concepts from Queueing Theory. AutoPerf even minimizes the number of experiments required to generate a typical curve. It also detects server warm up and starts measurement only after that, so that the measurements are representive of the actual values of the response time, throughput, etc.

Second, for example, consider the problem of determining the CPU milliseconds used by a given transaction, on the Web Server. Such quantities are required by performance analysts who are creating queuing models of the systems under study, so that performance under different scenarios can be predicted. We call these the server resource usage profiles per transaction. Using existing load generators, one has to do fairly tedious and error-prone co-ordination between the client load generator and the server profiler to arrive at these measurements. AutoPerf handles this entire co-ordination and produces detailed resource usage profiles per transaction.

Figure 2: Correlating results of a load generator and profilers manually

3. AutoPerf: Architecture

AutoPerf has two parts, Master and Profiler. Master, written in Java is deployed on client machine from where load is generated. Profiler, written in C is set up on the server to be tested. It saves profile snapshots of the server system and sends them to master when instructed.

Figure 3: High Level Overview of AutoPerf Architecture

 

4. Configuration Files

The configuration files can be found in the following path: <path to autoperf folder>/AutoPerf-Master/config and <path to autoperf folder>/AutoPerf-Master/docs/

  • input.xml
  • cbmg.txt
  • manual instruction.txt
  • config properties file 

4.1 input.xml

This file contains the necessary information about the nodes to be profiled and also the information about various transactions.Following are the tags and their meaning

<Load>

This block contains information about following tags:

<transaction>

<transaction>
    <name>ViewEntryByDay</name>
    <sequencelist>
        <sequence name="y2">3,4</sequence>
        <sequence name="month">01,02,03,04,05,06,07,08,09,10,11,12</sequence
        <sequence name="d1">0,1,2</sequence>
        <sequence name="d2">1,2,3,4,5,6,7,8</sequence>
    </sequencelist>
    <url>http://10.129.41.154/webcal/day.php?date=201${y2}${month}${d1}${d2}</url>
</transaction>

Sample excerpt from input.xml

The <transaction> tag mainly consists mainly of <name> and <url> tags. The <sequencelist> tag is used to specify rules for generating different values for the variable used in the <url> tag which is explained later. The tags within the transaction block are as follows:

  • <name> - an identifier with which a transaction definition can be identified.
  • <url> - the URL which is to be contacted to execute this transaction.

The default method used is "GET". In case of "POST" method, the transaction defined as

<url method="post">http://a?b=c</url>

the value of form variable 'b' is set to 'c' and the URL 'http://a' is contacted (i.e. URL is interpreted as that part before the '?', the rest is used to POST the variable values).

In the <url> tag, '&' has to be encoded as '&amp;' .  That is, if we want to  connect to the following URL :

<url>http://a?b=c&d=e</url>

we need to specify the url definition as:

<url>http://a?b=c&amp;d=e</url>

 

Attributes of <url> tag

  1. method -

values: {get, post},  default value={get}

For e.g.,

<url method="post">http://a</url>

 

  1. inputype

  2. <url inputype="sequence">http://10.129.41.154/webcal/year.php?year=20${y1}${y2}</url>

This method randomly picks values from the sequences named y1, y2 respectively, and replaces ${y1}, ${y2} by them.  This descriptor is optional. If the inputype is undefined (as shown in the example), but variables are seen in the URL, then a "sequence" input type is assumed. Sequence is defined in sequence tag as explained later.  

  • <url inputype="file">http://10.129.41.154/webcal/year.php?year=20${filename1#columnNo1}${filename2#columnNo2}</url>

This method randomly picks a value from file "filename" and "columnNo" of that file and replaces ${filename1#columnNo1} to dynamically generate a url. The file specified in the ${filename1#columnNo1} should be kept in the folder where the "input.xml" file is kept or the file path can be relative to it. Columns of the file should be seperated by single tab space.

Column numbers start from 0. For e.g.,

<url inputype="file">http://10.129.41.154/webcal/year.php?year=20${file1.txt#0}${file2.txt#1}</url>

For this example AutoPerf will randomly pick values form the 0th column of file1.txt and 1st column of file2.txt respectively.

  • <url inputype="parse" >http://10.129.41.154/year.php?year=${0}</url>

The parse function extracts the value of variables from the last loaded page to generate URLs. This tag is only relevant in capacity analysis mode. The url with "parse" should not be the first url to fire because there would be response page which it will parse. This method finds all strings that match the given pattern in the loaded page. It then randomly picks one of the urls which matched the given pattern issues it. 

 

  1. unique

values:{unique}, default value: N.A.

For e.g.,

<sequencelist>
            <sequence name="y1" type="range">1,6</sequence>
</sequencelist>
<url inputype="sequence" unique="unique">http://10.129.41.154/dummy.php?id=${y1}</url>
 

unique attribute allows value of key to be unique for each user. Say, load is generated at level: 10 and file or range of sequence contains more than or equal to 10 values for key in url, then for each the 10 users the value would be different. If file or range of sequence contains less than 10 values then remaining users will get repeated values. Unique attribute can only be used with inputype="file" or "sequence". For inputype="sequence" all the sequence name used in url with type="range" only has effect of unique. Values specified in the "sequence" tag doesn't have the effect of being taken uniquely.

 

<sequencelist>

Can be used to generate URLs dynamically.

Sample excerpt from input.xml

<sequencelist>
    <sequence name="y1" type="range">0,9</sequence>
    <sequence name="y2">0,1,2,3,4,5,6,7,8,9</sequence>
</sequencelist>
Attributes:

e.g., in the following URL

<url inputype="sequence">http://10.129.41.154/year.php?year=20${y1}${y2}</url>

the values of "variable" - y1, y2 will be randomly determined from the corresponding <sequence name> tag. This generates requests for files at http://10.129.41.154/year.php?year=2000, http://10.129.41.154/year.php?year=2001, ... ,http://10.129.41.154/year.php?year=2099 randomly with uniform probability.

<timeout>

Optional tag. Transaction specific timeout (in milli seconds) can be defined here. A timeout value of -1 means no timeouts. Default timeout value is -1 (i.e., no timeouts)

<thinktime>

This specifies the amount of time (in milli seconds) that Autoperf will wait after receiving the previous response and before this transaction is issued.This value is used in capacity Analysis mode.

<warmup>

Takes boolean values: true or false.Warm up is set to true if you desire that autoperf automatically determine "server warm up". In this case, autoperf generates load but does not start collecting statistics unless "warm up" is detected. If set to false, statistics are collected right from the beginning. Default value is set to true if no value is specified.

<cbmgfile>

This tag is relevant only in capacity Analysis mode.This specifies the name of the cbmg file to be used in the capacity analysis mode.

<startload>

This tag is optional. It is relevant only in capacity analysis mode. It specifies the load level to start the capacity analysis.

<ServerInfo>

This block contains information about the server

<NodeInfo>

This tag specifies the information required to profile resource consumption by a server process. It uses the <node>, <process> and <port> tags.

  • The <Node> tag  specifies the IP address of machine on which the process to be profiled runs.
  • The <Process> tag specifies the name of server process on this machine.
  • The <Port> tag specifies port on which the profiler agent listens for commands from the Master.

The type attribute tell us whether the server process is a Java process (e.g. running within a Java VM). 

e.g.

<NodeInfo type="NonJavaNode"> Indicates a non java profiler.

<NodeInfo type="JavaNode"> Indicates a java profiler.

The Java profiler is currently not supported.

Sample excerpt from input.xml

<NodeInfo type="nonJavaNode">
    <Node>10.129.41.154</Node>
    <Process>apache2</Process>
    <Port>22221</Port>
</NodeInfo>

<Profile>

Sample excerpt from input.xml

<Profile>
    <usetransactions>
    <name>ViewEntryByWeek</name>
    <requires>Login</requires>
    </usetransactions>
    <usetransactions><name>ViewEntryByMonth</name>
    </usetransactions>
    <numusers></numusers>
    <profilethinktime></profilethinktime>
    <executioncount></executioncount>
</Profile>

<usetransactions>

<name>

This tag names the transaction to be profiled. This should correspond to the name defined in the <transaction> block. We can list multiple transactions for profiling.

<requires>

Suppose we want to profile the ViewEntryByWeek transaction.  But a ViewEntryByWeek request requires the user to be logged in. Thus, even if for profiling we need not issue other requests, we will have to issue a login request before issuing a ViewEntryByWeek request.Such a scenario would be specified by having Login in the requires tag. Here the Login transaction should be defined in the transaction tag in the Load block.

<numusers>

This tag specifies the number of users to emulate

<profilethinktime>

This tag specifies the thinktime in milliseconds relevant to the profiling mode. The default value taken is zero if this tag is empty.

<executioncount>

This tag is used to specify how many iterations of the transaction are to be performed by each of the emulated users. When no value  is specified, this count is determined automatically, in a way that, a reliable estimate of CPU execution time required on the server by this request is obtained by AutoPerf.

See Appendix 1 for a sample input.xml file.

 

4.2 config.properties

This file contains various configuration parameters and their values, to be used during execution of autoperf. The file provided along with code may be directly used by the user.

See Appendix 2 for a sample "config.properties" file.

Configuration Parameters

  • config_file_name=input.xml

Name of the .xml file which contains all the transaction definitions.

  • cbmg_file_name=cbmg.txt

Name of the file which contains the Customer Behavior Model Graph (CBMG).

  • manual_loadlevel_input_file_name

Name of the file which contains manual loadlevel input.

  • master_port

Port on which master listens for communication from the profilers.

  • infinity

Approximated value of infinity.

  • inter_profile_request_time

Duration after which AutoPerf checks if sufficient number of transactions have been executed since the last time this check was performed, so that termination criteria can be checked.

  • max_seq_var_count

Load level below which load level is varied sequentially in steps. Above this, load level is varied by applying the binary algorithm.

  • warmup_window_size

Warmup window size during profiling a single transaction

  • lower_bound_on_transaction_execution_count

When execution count is determined automatically, lower bound on it's value.

  • upper_bound_on_transaction_execution_count

When execution count is determined automatically, upper bound on it's value.

  • max_seq_var_threshold

Percentage error allowed in the predicted throughput value (when using the binary algorithm)

  • mstar_lower_bound_multiplier

The predicted value of M* is multiplied by this for slight over estimation and binary algorithm is applied in that range.

  • mstar_upper_bound_multiplier

The overestimated value of M* is multiplied by this, and capacity analysis is performed within the previously over-estimated M* and this new M*. Load level however is varied in sequence steps.

  • avg_execution_count_window_size

Window size used for execution count stabilisation detection (at each load level)

  • percentage_error_allowed_in_avg_service_time_observed

Percentage deviation allowed in each of the values in the window from the avg of the values in the window.

  • inter_service_time_measurement_execution_count

criteria for detecting if execution count is sufficient, is applied after every 'inter_service_time_measurement_execution_count * load level' increase in the value of execution count

  • maximum_execution_count_when_service_demand_zero

If observed service demand at a particular server-process is 0 even after the following number of transactions, then execution count automation criteria is not applied further for that process.

  • throughput_window_size

Window size used for throughput stabilisation detection (at each load level). The same window is also used for warmup detection in capacity analysis mode.

  • percentage_error_allowed_in_throughput_observed

Percentage deviation allowed in each of the values in the window from the avg of the values in the window.

  • inter_throughput_measurement_execution_count

Criteria for detecting if throughput has stabilised is applied after every 'inter_service_time_measurement_execution_count * load level' increase in the value of execution count

  • warmup_window_difference_threshold

If percentage error in 2 successive warmup windows falls below this, we say server has warmed up.

  • percentage_time_outs_allowed

If percentage timeouts increase beyond this, we stop.

  • check_for_percentage_time_outs_after

The last check is made after these many percentage of transactions have been executed.

 

4.3 Manual Load Level

This file (mll.txt) is read when autoperf is invoked in the "manual load generation" mode. This disables autoperf's algorithm of determining load levels, and instead has autoperf determine load levels i.e number of users to emulate, based on this file. The file contains various load levels and the corresponding duration for which the load level is to be maintained.

  • First column in file is LoadLevel.

  • Second column is corresponding time (in seconds).

e.g.,

3    100
5    150
4    50
3    120

A periodically repeating workload pattern can also be specified. For a cycle, the values of load levels and their duration are given in a repeat block. 

E.g., in the following file, the load generation starts with emulating 4 users for 10 seconds, then the cycle within the "repeat..end" block is repeated 6 times, and then 9 users are emulated for 10 seconds.

4     10
repeat  6
3     14
6     10
end 6
9     10

 

This file is referenced only when AutoPerf is run in Manual mode. Note the "input.xml" and the "cbmg.txt" files have to be configured and are used as usual in this mode.

See Appendix 3 for a sample Manual Load Level text file.

 

4.4 Configuring CBMG.txt

This file contains consumer behaviour model graph (navigation pattern of the users). Transactions given in the input file are named, to form a 2D array in which probabilities of going from one transaction to the other transaction are specified. This file is used in the capacity analysis mode but it is ignored in the profiling mode of operation. 

The following figure is the navigation pattern of the users in a particular site. The numbers against the transition arrow represent the probability with which a customer will move from current state to next state.

 

Figure 3: Customer Behaviour Graph Model

 

A Sample CBMG file for the corresponding figure should look like this:

  Entry ViewEntryByDay ViewEntryByWeek ViewEntryByMonth Exit
Entry 0 0 1 0 0
ViewEntryByDay 0 0.4 0.2 0.1 0.3
ViewEntryByWeek 0 0.3 0.3 0.3 0.1
ViewEntryByMonth 0 0.3 0.4 0.3 0
Exit 0 0 0 0 0

The Entry and the Exit state always need to be present in the CBMG.txt. As the name suggests, they mark the Entry and Exit point of entering in the state space. Thereafter the customer may navigate in the graph with respect to the probabilities given in the file.

Some rules to check the correctness of the file:

  • Row sum should always come 1; except for the Exit state.
  • A customer can't have non-zero value for first column. This implies that Entry state marks the entry into the system for consideration and will never enter the Entry= state again. It will keep navigating within other transactions and eventually Exit.
  • Once a customer enters the Exit state - it is assumed that it has exit from traversing the navigation. Therefore, the last row corresponding to Exit should always be marked 0 for all the possible transactions, including the state Exit itself.

See Appendix 4 for a sample cbmg.txt file

4.5 log4j.properties

AutoPerf-Master uses Logger class for printing the output on the console. This class provides various levels of logging information. Please refer the following link for a detailed reference.

Normally, you can output something by using function cat.warn(). The tool can be operated in debug mode by changing log4j.rootCategory=warn to log4j.rootCategory=debug in the log4j.properties file.

The file provided along with code and is placed in the <AutoPerf-Code>/AutoPerf-Master/config folder. It may be directly used by the user.

5. Running Instructions

5.1 Starting Profiler

Assuming the path to the profiler is in your $PATH, run the profiler as follows. 
prof -d [port number]

Otherwise navigate to the code directory - <AutoPerf-Code>/AutoPerf-Profiler/ and for each server process that you want to profile, initiate the profiler as:

./prof -d [port-number]
  • Specify the port number on which the profiler is to be started. This should be the same port number as was specified in input.xml file. 
  • The port number should not be in use for any other purpose.

 

5.2 Starting Master

On the client side, login and open a shell on the host machine and initiate only one instance of the autoperf from anywhere as

autoperf [OPTIONSPATTERN [FILE...]

autoperf [OPTIONS] [-fi FILE] [-fc FILE] [-fm FILE]

OPTIONS

The options corresponding to a Mode of Operation can be chosen from one of the following

-p

Profiling mode

-c

Capacity Analysis mode

-m -p

Profiling at Manual Load Level

-m -c

Capacity Analysis at Manual Load Level

File and Directory Selection

Absence of the file option(s) will make AutoPerf pick the files from the current working directory from which autoperf command was invoked.

  • -fi input.xml

Instructs autoperf to pick input.xml file

  • -fc cbmg.txt

Instructs autoperf to pick the cbmg.txt file

  • -fm mll.txt

Instructs autoperf to pick the Manual Load Level - "mll.txt" file

The input files for inputype="file" have to be present in the same path as input.xml file

EXAMPLES

  • autoperf -c

AutoPerf will run in capacity analysis mode and look for the relevant files (with standard names) from the current working directory.

  • autoperf -p -fi 1/input.xml

AutoPerf will run in profiling mode and take the "input.xml" from the ${PWD}/1 folder and the absence of the other optional switches imply that the files will be picked from the current working directory (if required).

  • autopref -m -c -fi 1/input.xml -fc 2/cbmg.txt -fm mll.txt

AutoPerf will look for "input.xml" in the ${PWD}/1 folder, "cbmg.txt" file in ${PWD}/2 folder and "mll.txt" file in the current working directory.

The corresponding options are explained in the following sections.

6 Modes of Operation

6.1 Profiling mode and at a single load level

AutoPerf can be invoked at this mode by the following command:

autoperf -p

This will generate load only at the level specified in the input XML file using the <numusers> tag. This will cause only the specific transaction to be requested and profiled.

Other load specification:

In input.xml file the <Profile> block should contain the <usetransactions> tag. The transactions corresponding to the <usetransactions> tag will be profiled in the order present in input.xml file.

6.2 Capacity Analysis mode

AutoPerf can be invoked at this mode by the following command:

autoperf -c

In this mode, a CBMG is given as an input to AutoPerf, thereby summarizing the workload characteristics. AutoPerf automatically determines the maximum throughput and saturation load level. It also reports resource profiles collected at the server. This will generate load at multiple load levels up to the point of saturation.

6.3 Profiling mode and at Manually determined Multiple load level

AutoPerf can be invoked at this mode by the following command:

autoperf -m -p

This mode of operation is similar to the Mode 1. But instead of automatically determining the load duration, load is generated at given loadlevel for given duration. Load level details and corresponding duration details are fetched from the corresponding Manual Load Level file (it should be named as "mll.txt file"). This will cause only the specific transaction to be requested and profiled for the specified amount of time.

6.4 Capacity Analysis at Manual Load level

AutoPerf can be invoked at this mode by the following command:

autoperf -m -c

This mode of operation is similar to Mode 3. However, it will generate load at multiple load levels specified by the Manual Load Level text file and for the corresponding duration specified in the same row. In this mode a mix of transactions are  requested and profiled. The load generation is a mix of transactions as is summarised in the CBMG file.

7. Output

A Sample Output contains the following values

Summary:
========

==================================================
Transaction Name : Probabilistic Session Generation
Load Level : 1
==================================================
Duration of load generation : 20571.0 [milli sec]
Total number of urls fired on the server : 63.0
Number of urls that timed out : 0
Response Time : 125.0 [milli sec]
Throughput : 3.062563803412571 [requests/sec]

Node : 10.129.41.154
Process : apache2
Service Demand : 111.11111111111111 [milli sec per transaction]
Cpu Utilization : 0.33
Virtual Memory Usage : 133944.0 [KB]
Number of disk blocks read : 0.0
Number of disk blocks written : 240.0
Number of error-free network packets read : 1068.0
Number of error-free network packets written : 1844.0

Node : 10.129.41.154
Process : mysqld
Service Demand : 0.0 [milli sec per transaction]
Cpu Utilization : 0.0
Virtual Memory Usage : 57472.0 [KB]
Number of disk blocks read : 0.0
Number of disk blocks written : 240.0
Number of error-free network packets read : 1068.0
Number of error-free network packets written : 1844.0
 
  • Duration of load generation - Time for which the URLs were fired on the profiler from the load generator for a particular load level.
  • Total number of urls fired on the server - The total number of URLs fired are determined by "Throughput convergence" and "Service Time Granularity" algorithms (in capacity analysis mode).
  • Number of URLs that timed out - Indicates the number of URLs that got timed out. The time out value is a parameter from the Config.properties file. If a particular URL's response is not received in that particular time period, it is considered as Timed Out.
  • Throughput - Throughput is measured as ((Total Number of URLs Fired - Total URLs timed out) / (Duration of the load generation))
  • Node - Represents the IP address of the host.
  • Process - Represents the name of the process whose service demand is being reported. It is a parameter in "input.xml" file.
  • CPU Utilisation - The average value of CPU utilisation observed for the period during which the test was run.

The following are the per transaction resource consumption details. In the profiling mode, these correspond to the transaction being profiled. In the capacity analysis mode, this is an average over all the transactions, with respect to the transaction mix generated (the mix is determined by the CBMG). 

Per process, per transaction detail:

  • Service Demand (in ms) - This is the average time that this process used the CPU for to service a single transaction . 
  • Virtual Memory Usage - Indicates the maximum amount of non-swapped physical memory that a task has used (in kilobytes) during its execution.

Per host detail. The following are calculated per host, and represent resource consumption as a result of execution of a single transaction. // per load level (in Capacity Analysis ) or Per Transaction Profiling run

  • Number of disk blocks read 
  • Number of disk blocks written
  • Number of error-free network packets read
  • Number of error-free network packets written 

8. Future Work

  • Self awareness about Client bottlenecking
  • Frequency scaled CPU

9. Appendix

  1. Sample Input.xml file

  2. Sample config.properties file

  3. Sample Manual_loadlevel_Input.txt file

  4. Sample CBMG.txt file

 

 

Copyright © 2013 Vikas Goel. All rights reserved