Performance tests analysis plugin for TeamCity
==============================================
This is plugin for TeamCity 8.0 that helps to organize simplest performance testing in CI. 
It has the ability to aggregate results from a log file, calculate metrics, 
compare results with reference values, monitor a remote machine with the tested application. 
Additionally, it allows viewing all the results of performance tests as charts.

Last version: https://teamcity.jetbrains.com/repository/download/TeamCityPluginsByJetBrains_JMeterPlugin_Build/.lastSuccessful/jmeter.zip

How it works
==============
The plugin has two features: 
1) Performance Metrics Calculation - a build feature to configure settings for calcuation performance metrics;
2) Performance Statistic - a tab with values of average metrics per test by builds;

3) Performance Remote Monitoring - a build feature to configure plugin settings for build configuration;
4) RemotePerfMon - a tab with monitoring results for the build.


Performance Metrics Calculation  
===========================================
required - *

Aggregation: 
-------------------------------------------
File to aggregate results: * 
	a relative path to the file with the raw test results to calculate average metrics;
Aggregate metrics: *
	average, min, max, 90% line,
	response codes: it allows to calculated count of each code in test results - http code, or some other test result.
	(for example you can log some id for exception occurred during test run: OK, InternalError, NPE)
	note: response code NOT fail build, 
	to DETECT errors or non-200 http codes and fail build in this case use assertions!  
Format settings:
	total - calculate total average values for all tests; Note: if thread groups  are used, total will be calculated by groups
	assertions - fail the build if any assertion check fails,
	thread groups - you can group tests in thread groups(20,100 threads..), in this case, test name must have the specific format at the log 
	
	used TeamCity tests format
	This option must be set if your tests are not created and run using Test framework, like TestNG, JUnit; and you aren't use service messages to log test result in build log.
	
Check reference data (optionally): 
The build will be failed if the aggregated values exceed reference values considering variation.
-------------------------------------------
Get reference values from: *
	- file - set if you have a static reference values. In this case, set relative path to reference data; 
	- builds - not supported yet
Variation: 
	the value of variation  [0..1] in decimal format; default - 0.05 (5%);
	
	
	
	
	
	
Performance Remote Monitoring: 
===========================================
allows monitoring some system and jvm statistics on the remote machine with the tested application. 
-------------------------------------------
Build step to analyze: *
	the name of the build step which starts tests. 
	Note: The plugin uses Server Agent(http://jmeter-plugins.org/wiki/PerfMonAgent/) to collect metric values on the remoted machine. So, the agent must be run on the remote machine before the build step begins.	

Remote machine: *
	host *, port * - settings for access to the running agent;
	clock delay   - if tests will be performed from another machine(non-BuildAgent), set system clock delay between build agent and test machine to sync time of monitoring.
	
Monitoring interval: * - in seconds, to collect metrics from agent;

Monitored parameters, supported by plugin:
- CPU system/user/all/iowait (all in percent)
- Memory used (bytes)
- Disk I/O reads/writes (ops)
- JMX memory heap, pools commited/usage; gc time; class loaded;	
	
	
	
Formats of the file to aggregate  results:
==========================================
The first line must contain the titles of columns. 
Delimiter - \t
Label is the same test/sample name; 
If thread groups are used; label must have format: <thread_group_name>:<label>
Don't end a label name with 'Total'!

1) Simplest format for result file:
startTime	spendTime	label	....

2) + HTTP response codes:
startTime	spendTime	label	responseCode	....

3) + assertions:
startTime	spendTime	label	isSuccsessful	....

4) + assertions and HTTP response codes:
startTime	spendTime	label	responseCode	isSuccsessful	....

Note: After required columns, you can save custom columns with data for each row. 

Examples:

start	spend	method
1383822067829	5	foo222
1383822067835	77	foo111

timeStamp	elapsed	label	responseCode	success	responseMessage	thread	dataType
1383843124555	153	50 threads: Login	200	true	OK	Thread 1-1	text
1383843126523	12	10 threads: Login	200	true	OK	Thread 1-3	text
1383843124713	3391	50 threads: Open page	200	true	OK	Thread 1-1	text
1383843128268	215	10 threads: Search query	200	true	OK	Thread 1-4	text
1383843128269	224	50 threads: Search query	200	true	OK	Thread 1-2	text
1383843128540	1457	10threads: Open page	200	true	OK	Thread 1-5	text
1383843128270	2138	50 threads: Search query	404	false	Not found!	Thread 1-3	text



Reference data format
======================
format:label	metric	value	variation(optional)

example1:
10 threads: Login	line90	120
example2: 
Home page	average	340	0.1

Delimiter - \t
All labels must  match the corresponding item labels in the raw test file. 
It is possible to set your own variation for each label. If  the value of variation isn't set, it will be extracted from feature 
the settings (by default 0.05).

Possible values for metrics: 
min 
max 
average 
line90


Statistic visualization
========================

Performance Statistics tab
-------------------------
After running configuration with Performance Metrics Calculation feature, you can see Performance Statistic tab on the build page. 
// TODO
It contains two sections if performance check is not passed
charts with Response codes (if selected) and charts for each label 
with aggregated metrics. The charts contain comparative statistics by builds. 

X-Axis settings has 2 options:
actual build numbers   - x-axis constanis the build numbers of the configuration with tests;
build numbers of the dependency - if your test configuration uses an artifact dependency from the tested application,
x-axis contains build numbers of the tested application, and the numbers will be extracted from the artifact 
dependencies.
Server Response Time - shows the distribution time for each type of test row (label); 
Request Per Seconds - show count of rows per second for each label. 

RemotePerfMon tab
-----------------
After running configuration with Performance Remote Monitoring feature, you can see RemotePerfMon tab on the build page. 
The charts with system metrics and jmx metrics are displayed here. Note, all metrics related to memory 
(system memory, jmx memory, memorypool, swap) will be on the same chart.


Useful links:
http://confluence.jetbrains.com/display/TCD8/TeamCity+Documentation
http://jmeter-plugins.org/wiki/PerfMonAgent/