|
|
# Amazon EC2
|
|
# Amazon EC2
|
|
|
|
|
|
|
|
## Pricing
|
|
|
|
|
|
|
|
|
|
* Amazon EBS Snapshots to Amazon S3 (http://aws.amazon.com/en/pricing/ebs/).
|
|
|
|
|
* Used to store images of Linux and Windows machines. Amazon S3 storage cost.
|
|
|
|
|
|
|
|
|
|
* Amazon Virtual Private Cloud (https://aws.amazon.com/en/pricing/vpc/)
|
|
|
|
|
* Used to choose the private addresses assigned to instances. This is free.
|
|
|
|
|
|
|
|
|
|
* Amazon S3 (http://aws.amazon.com/en/s3/pricing/)
|
|
|
|
|
* mathjaxtest bucket is used to store MathJax results. Amazon S3 storage cost.
|
|
|
|
|
* There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same Region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US Standard Region.
|
|
|
|
|
|
|
|
|
|
* CloudFront (http://aws.amazon.com/en/pricing/cloudfront/)
|
|
|
|
|
* Cloudfront is used to serve public test results:
|
|
|
|
|
http://mathjaxtest.s3.amazonaws.com/index.html
|
|
|
|
|
|
|
|
|
|
* Running Instances (during testing period only)
|
|
|
|
|
* Amazon EBS Volumes (http://aws.amazon.com/en/pricing/ebs/): disks used by running instances. Storage and IO requests cost.
|
|
|
|
|
* On-Demand Instances charge (http://aws.amazon.com/en/ec2/pricing/). Per instance-hour cost.
|
|
|
|
|
* Data Transfer (http://aws.amazon.com/en/ec2/pricing/).
|
|
|
|
|
* Data transferred between instances in the same Availability Zone using private IP addresses is free.
|
|
|
|
|
* Remote access to instances (SSH, RDP)
|
|
|
|
|
* Web Interface to control the framework (HTTP)
|
|
|
|
|
* Downloading/Updating MathJax-test github repository, browsers, programs etc
|
|
|
|
|
* IP Addresses (http://aws.amazon.com/en/ec2/pricing/). Used for external access to instances (Remote access, Web Interface). We don't use fixed addresses, this is less convenient but free.
|
|
|
|
|
|
|
|
|
|
## AWS Management Console
|
|
## AWS Management Console
|
|
|
|
|
|
|
|
* Instances
|
|
* Instances
|
| ... | @@ -172,42 +146,28 @@ IPs so that we won't be charged anymore. See "Stopping the Testing Framework". |
... | @@ -172,42 +146,28 @@ IPs so that we won't be charged anymore. See "Stopping the Testing Framework". |
|
|
* Right Click on an instance and select terminate.
|
|
* Right Click on an instance and select terminate.
|
|
|
* Check "Release Elastic IPs", Yes Terminate
|
|
* Check "Release Elastic IPs", Yes Terminate
|
|
|
|
|
|
|
|
## Organization of test execution
|
|
## Pricing
|
|
|
|
|
|
|
|
In this section, we find a way to optimize the test execution on EC2. We would like to test the four main rendering engines (Gecko, Presto, Webkit, Trident) and their supported output modes (HTML-CSS+STIX, HTML-CSS+TeX, SVG). For Internet Explorer, we also have 4 document compatibily modes to test. This gives 3+2+3+9=17 tasks to execute:
|
|
* Amazon EBS Snapshots to Amazon S3 (http://aws.amazon.com/en/pricing/ebs/).
|
|
|
|
|
* Used to store images of Linux and Windows machines. Amazon S3 storage cost.
|
|
|
- {Firefox} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux
|
|
|
|
|
- {Opera} × {HTML-CSS+TeX, SVG}: Linux
|
|
* Amazon Virtual Private Cloud (https://aws.amazon.com/en/pricing/vpc/)
|
|
|
- {Chrome} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux (with Selenium 1 API) or Windows (Webdriver)
|
|
* Used to choose the private addresses assigned to instances. This is free.
|
|
|
- {Internet Explorer} × [ {HTML-CSS+STIX, HTML-CSS+TeX} × {Quirks, IE7, IE8, IE9} ∪ {(SVG, IE9)} ]: Windows
|
|
|
|
|
|
|
* Amazon S3 (http://aws.amazon.com/en/s3/pricing/)
|
|
|
Some points to consider:
|
|
* mathjaxtest bucket is used to store MathJax results. Amazon S3 storage cost.
|
|
|
|
|
* There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same Region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US Standard Region.
|
|
|
* Pricing is per instance-hour consumed for each instance, from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour.
|
|
|
|
|
* Execution of Linux is cheaper but not possible for all browsers. Moreover the task handler is installed on Linux, so when you execute a test on a Windows instance you also need to run a Linux instance at the same time.
|
|
* CloudFront (http://aws.amazon.com/en/pricing/cloudfront/)
|
|
|
* More powerful hardware is faster but more expensive.
|
|
* Cloudfront is used to serve public test results:
|
|
|
* One can execute several EC2 instances in parallel but that's less convenient to set up and manage.
|
|
http://mathjaxtest.s3.amazonaws.com/index.html
|
|
|
* Webdriver is faster, more reliable and allow parallel execution on the same machine. However, it is not supported yet for Chrome+Linux and only one instance of Internet Explorer can be executed at a time.
|
|
|
|
|
* Execution on Internet Explorer is the slowest and there are many configurations to test.
|
|
* Running Instances (during testing period only)
|
|
|
* The testing framework can not handle too many tasks at once, for otherwise it automatically stops unresponsive tasks. We have to oversee the execution and run interrupted tasks manually again.
|
|
* Amazon EBS Volumes (http://aws.amazon.com/en/pricing/ebs/): disks used by running instances. Storage and IO requests cost.
|
|
|
* In addition to test execution, you need time to set up the testing framework, save the results to AmazonS3 and stop the testing framework.
|
|
* On-Demand Instances charge (http://aws.amazon.com/en/ec2/pricing/). Per instance-hour cost.
|
|
|
|
|
* Data Transfer (http://aws.amazon.com/en/ec2/pricing/).
|
|
|
Combinations experimented:
|
|
* Data transferred between instances in the same Availability Zone using private IP addresses is free.
|
|
|
|
|
* Remote access to instances (SSH, RDP)
|
|
|
- Running Linux Standard Small On-Demand Instances, the 5 Firefox&Opera tasks.
|
|
* Web Interface to control the framework (HTTP)
|
|
|
Interruption: It seems hard for the testing framework to handle everything. Tasks are interrupted too often.
|
|
* Downloading/Updating MathJax-test github repository, browsers, programs etc
|
|
|
- Running Linux Standard Small On-Demand Instances, the 3 Firefox tasks, followed by the 2 Opera tasks.
|
|
* IP Addresses (http://aws.amazon.com/en/ec2/pricing/). Used for external access to instances (Remote access, Web Interface). We don't use fixed addresses, this is less convenient but free. |
|
|
Interruption: Opera tasks have been interrupted twice.
|
|
|
|
|
Estimated Execution time: ~100min (Firefox) + ~45min (Opera).
|
|
|
|
|
Estimated cost: $0.240
|
|
|
|
|
- Running Linux High-CPU Medium On-Demand Instances , the 5 Firefox&Opera tasks.
|
|
|
|
|
Interruption: one Opera task has been interrupted twice.
|
|
|
|
|
Estimated Execution time ~40min.
|
|
|
|
|
Estimated cost: $0.165
|
|
|
|
|
- Running Linux Standard On-Demand Instances and 2 Windows High-CPU On-Demand Instances.
|
|
|
|
|
Linux (5 tasks): Firefox, Opera
|
|
|
|
|
Windows1 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+STIX}, {Chrome}×{HTML-CSS+STIX, HTML-CSS+TeX}
|
|
|
|
|
Windows2 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+TeX}, {IE9,Chrome}×{SVG}
|
|
|
|
|
This seems to work well, except that the execution of Internet Explorer tests are still too slow.
|
|
|
|
|
- 1 Windows High-CPU Extra Large On-Demand Instance?
|
|
|
|
|
19 tasks: Firefox (3), Opera (2), IE (9), Chrome (3), Safari (2) |
|
|
|
\ No newline at end of file |
|
|
