Current time: 12-15-2019, 02:27 AM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Optimal Number of Test Runs
12-01-2012, 07:34 AM
Post: #1
Optimal Number of Test Runs
I've been looking at some public tests and many people seem to run only one test. Others run 3. I run 10. Is there a sweet spot?
Visit this user's website Find all posts by this user
Quote this message in a reply
12-04-2012, 02:38 AM
Post: #2
RE: Optimal Number of Test Runs
Funnily, in Pat's recent article in the Performance Calendar, he says that he runs 9 tests.

http://calendar.perfplanet.com/2012/usin...c-testing/

"Each test was configured for 9 runs so that we could collect enough data to be reasonably confident with the results..."
Visit this user's website Find all posts by this user
Quote this message in a reply
12-04-2012, 04:48 AM
Post: #3
RE: Optimal Number of Test Runs
Yeah, > 1 is absolutely critical if you are looking at the times. If you are just looking at how the page is constructed (requests, waterfall, caching, etc) then 1 works ok for a quick snapshot.

I usually do 9 instead of 10 because the times are from the median run and median makes more sense for an odd number of samples. Depending on the page, even 9 may not be enough. If you click on the "plot full results" link below the data table you can see what the distribution of the times looks like for all of the runs and where the median run times were. Eyeballing that should give you better confidence if you are looking at a representative result or a page that has wildly variable performance.
Visit this user's website Find all posts by this user
Quote this message in a reply
12-04-2012, 05:27 AM (This post was last modified: 12-04-2012 05:33 AM by rviscomi.)
Post: #4
RE: Optimal Number of Test Runs
Thanks, Pat. Nice pro tip to run an odd number of tests, now I'm curious how the median is selected for even tests. (tiebreaker?)

Higher sample size is always better, but is there an imposed/suggested upper limit to the number of tests? Would not want to hog the tool's resources.
Visit this user's website Find all posts by this user
Quote this message in a reply
12-04-2012, 05:29 AM
Post: #5
RE: Optimal Number of Test Runs
Yeah, the tool doesn't let you submit for more than 10 runs which is why you see a lot around 9 or 10 :-)

The median for an even number of tests is actually the test that's on the faster side of the midpoint I believe (not a true median since it is picking a specific run).
Visit this user's website Find all posts by this user
Quote this message in a reply
12-04-2012, 10:26 AM
Post: #6
RE: Optimal Number of Test Runs
Pat,

Would it be possible to allow users to select the criteria based upon which the median is selected? For example, Speed Index or Start Render instead of Load Time?

I believe you may be working on this already, as I stumbled across a query param somewhere (can't find it now) called something like criteria=loadTime

Best Regards,

Jason Hofmann

(12-04-2012 05:29 AM)pmeenan Wrote:  Yeah, the tool doesn't let you submit for more than 10 runs which is why you see a lot around 9 or 10 :-)

The median for an even number of tests is actually the test that's on the faster side of the midpoint I believe (not a true median since it is picking a specific run).
Find all posts by this user
Quote this message in a reply
12-11-2012, 07:10 AM
Post: #7
RE: Optimal Number of Test Runs
I don't have UI for it right now but you can pass medianMetric=X to have it use an arbitrary metric (if you look at the XMLResult for a given test you can see all of the metric names).
Visit this user's website Find all posts by this user
Quote this message in a reply
12-11-2012, 08:00 AM
Post: #8
RE: Optimal Number of Test Runs
Did I document any that aren't supported or correct? (It's hard to tell by testing, since a few of them actually would and should return the same result are the default, loadTime):

http://www.webpagetest.org/result/121210...c=loadTime
http://www.webpagetest.org/result/121210...alComplete
http://www.webpagetest.org/result/121210...SpeedIndex
http://www.webpagetest.org/result/121210...=titleTime
http://www.webpagetest.org/result/121210...ic=domTime
http://www.webpagetest.org/result/121210...ic=docTime
http://www.webpagetest.org/result/121210...ullyLoaded
http://www.webpagetest.org/result/121210...ric=render
http://www.webpagetest.org/result/121210...etric=TTFB


(12-11-2012 07:10 AM)pmeenan Wrote:  I don't have UI for it right now but you can pass medianMetric=X to have it use an arbitrary metric (if you look at the XMLResult for a given test you can see all of the metric names).
Find all posts by this user
Quote this message in a reply
12-11-2012, 10:47 AM
Post: #9
RE: Optimal Number of Test Runs
those should all work (when/if they apply - domTime doesn't always). It also works for all of the non-time measurements (bytes, requests, etc).
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)