WebPagetest Forums
Differences in API and website UI - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: Web Performance (/forumdisplay.php?fid=3)
+--- Forum: Discuss Test Results (/forumdisplay.php?fid=4)
+--- Thread: Differences in API and website UI (/showthread.php?tid=15904)

Differences in API and website UI - nmadz2 - 09-23-2019 10:53 PM


I have been collecting web page load speed data using the webpagetest.org UI by visiting the website everyday and also using the API. The issue I am having is that both methods provide different results and I am interested in the reason for this. My aim is to migrate to using the APIs to automate the collection of this data. But at the moment, the two sets of data over the past month do not correlate and I cannot explain the difference.

This is the procedure I follow when testing using the website:
1. Enter URL
2. Set test location as Ireland - EC2 (Chrome, Firefox)
3. Set browser as Chrome
4. Open advanced settings
5. Set connection to 3G Fast (1.6Mbps/768 Kbps 150ms RTT) in test settings
6. Set number of test runs to 3
7. Click first view only
8. Tick capture video
9. In the chromium tab, tick capture lighthouse report
10. Click emulate mobile browser and choose Nexus 5
11. Click capture dev tools timeline
12. Click run test and wait for results to finish
13. Note down speed index and date from table

I am also attempting to replicate the same results using the API. I am using Python and here are the parameters I am setting. Note - I am aware that some of the parameters are different, but I believe they have no real impact on the results but let me know if I’m wrong!

parameters = {
'k' : #api here,
'f' : "json",
'location' : "ec2-eu-west-1:Chrome.3GFast",
'mobileDevice' : "Nexus 5",
'url' : url,
'runs' : 6,
'lighthouse' : 1,
'timeline' : 1,
'video' : 1,
'fvonly' : 1,


I take the speed index value from this process as this is the KPI my company uses. I record one result manually and kick-off 6 automated runs using the APIs. I then average the 6 API values of speed index into 1 API result. Both processes are testing the same URL. I believe I am using the same parameters as the website so in theory I should be getting the same results, however they are different. Here are some examples in difference in speed index.

Manual result:
date: 20/09/19, 9:35:44 AM
speed index: 4017

Api results:
date: 20/9/2019 9:26:57
speed index: 3051
date: 20/9/2019 9:27:17
speed index: 2798
date: 20/9/2019 9:27:31
speed index: 2613
date: 20/9/2019 9:27:38
speed index: 2852
date:20/9/2019 9:27:40
speed index: 2608
date: 20/9/2019 9:27:53
speed index: 2775
average: 2337

Here are the graph results over a month:

Manual process
[Image: mdNf8tj.png]

[Image: AdxRO8N.png]

The results from the API has been consistently lower than manually entering in the URL and testing on the website. Could you please help me identify why the results are different? What am I doing differently in the code compared to the website?

Thanks in advance.

RE: Differences in API and website UI - pmeenan - 09-24-2019 02:37 AM

Looks like your API tests are not enabling mobile emulation (mobile=1) so it is probably running the test as a desktop test.

RE: Differences in API and website UI - nmadz2 - 09-24-2019 11:19 PM

Thank you! That looks to have increased the API results closer to the results from the manual process.