c. yang, m. sun, j. xia, j. li, k. liu, q. huang and z. gui, 2013. chapter 12 how to test the...

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221.

How to test the readinessof cloud services?

Chaowei Yang, Min Sun, Jizhe Xia, Jing Li, KaiLiu, Qunying Huang, and Zhipeng Gui

1


Learning Objectives

Developing the fundamental technology for testing the readiness cloud services in support of geosciences1. Geoscience computing characteristics2. Cloud test environment 3. Concurrent intensity test with Clearinghouse4. Data and computing intensity test with

Climate@Home5. Comprehensive test using dust storm forecasting

2


Learning Materials•Online Videos:

o Chapter_12_Video_1.mp4o Chapter_12-Video_2-dust-storm-test.mp4o Chapter_12-Video_3-modelEtest.mp4

•Scripts, Files and others:o Chapter_12_Section12.3.zip o modelEtest_materials.zipo scripts.zip

3

http://www.youtube.com/watch?v=_USakxs1X4E

http://www.youtube.com/edit?o=U&video_id=TVlOFOdsHKI

http://stcenter.net/downloads/cloudbook/chapter12/Chapter_12_Section12.3.zip

http://stcenter.net/downloads/cloudbook/chapter12/modelEtest_materials.zip

http://stcenter.net/downloads/cloudbook/chapter12/scripts.zip


Geoscience computing characteristics

1. Concurrent intensity: for popular geospatial applications, such as data.gov or clearinghouse.

2. Data intensity: geoscience observation and research will produce significant amount of data.

3. Computing intensity: geoscience research needs significant computer power to conduct simulations.

4. Spatiotemporal intensity: geosciences involves the understanding of past, present, and future of the Earth system.

5. These characteristics are represented by the three applications of GEOSS clearinghouse, climate@home, and dust storm forecasting.

4


Learning Objectives



5


Cloud test environment

• Cloud service runs at computing centers and are accessed through the Internet

• The geographic region and computing facility will matter in the test.

• This test will look at the performance of different applications running on different cloud services

• The cloud services are located at different regions across the U.S. and connected through the national network backbone.

• Different service instance configure will produce different performance results.

6



7

The test will be conducted from each cloud and a local server to other computing service. Comparison will be made cross the cloud service and the local server.

Computing Service Geographic Location Cloud Services Host Organization

EC2 Reston, VA IaaS Amazon

Azure Chicago, IL PaaS Microsoft

Nebula Ames, CA IaaS NASA

Local Cluster Fairfax, VA Traditional Cluster GMU



8

Service names CPU Core CPU Speed (GHz) Memory (GB)

EC2 1 1 2.3 0.5

EC2 2 2 2.3 7.5

EC2 3 4 2.3 7.5

EC2 4 8 2.8 7.5

Azure 1 1 1.6 1.75

Azure 2 2 1.6 3.5

Azure 3 4 1.6 7

Azure 4 8 1.6 14

Nebula 1 1 2.9 2

Nebula 2 2 2.9 4

Nebula 3 4 2.9 8

Nebula 4 8 2.9 16

Local Server 8 2.4 23


Learning Objectives



9


Concurrent intensity test with Clearinghouse

1. Clearinghouse computing requirement2. Test Design3. Test Workflow4. Test Results Analyses

10


Concurrent intensity test with Clearinghouse: Computing Requirements

• CLH is a FGDC, GEO, and NASA collaborative project to support the sharing of Earth Observation (EO) data in the global context.

• Global user access to the clearinghouse concurrently, and the access has different spatiotemporal patterns for different regions and different metadata.

• For example, clearinghouse receives more accesses at local day time than evening hours, and the access can be massive when a significant event (e.g., earthquake) happens.

11



12

The test will be conducted from each cloud and a local server to other computing service. Comparison will be made cross the cloud service and the local server.


Concurrent intensity test with Clearinghouse: Test Design

1. To compare the capabilities of different cloud services in supporting concurrent requests, two experiments designed:

2. Matrix of concurrent search requests test: 1) With CLH running on the four computing services: three cloud services

(AWS, Azure and Nebula) and a local server (figure 12.1), 2) Concurrent requests are issued from one service to all other services.

Apache JMeter is used for testing the performance of concurrent requests between the services.

3) The test result is a 4*4 matrix which can represent the concurrent performance of each computing against other computing services.

3. Load balance and scalability tests: 1) The load balancing experiment compares concurrent performance when

different numbers of instances are used behind the Load Balancer; 2) The auto-scaling experiment tracks the change of performance when cloud

services scale new instances dynamically with the increasing number of concurrent requests.

4. http://jmeter.apache.org/ (please check relevant video and cloud images)

13

http://jmeter.apache.org/

http://jmeter.apache.org/

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. 14

2. Set up CSW GetRecords Requests

3. Set up JMeter Test Plan

4. Run JMeter

1. Start GEOSS Clearinghouse Instance

5. Analyze Jmeter Results

Matrix of concurrent search requests test: Workflow

Video: Chapter_12_Video_1.mp4




Steps of Concurrent Test:

1. Installing CLH on different cloud services by launching Instance using virtual image.

2. Setting up CSW GetRecords request. CSW GetRecords request can be issued to the CLH (refer to Chapter 8).

3. Setting up JMeter test plan. Testers can set a test plan to simulate the concurrent requests to CLH. The concurrent number can be increased, such as 1, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, and 600 by setting up thread group and controller.

4. Running Jmeter through Graphic User Interface or Command line. 5. Analyze JMeter Results (2.3.4)

15

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. 16

Load balancing and auto-scaling tests: Workflow

3. Set up CSW GetRecords Requests

4. Set up JMeter Test Plan

5. Run JMeter

1. Start GEOSS Clearinghouse Instance

2. Set up Load Balancing or Auto-Scaling

6. Analyze Jmeter Results


Steps of Load balancing and auto-scaling tests:

1. These two tests can be used to test the improvements of the performance when using load balancing or auto-scaling for CLH. Testers can step-wise to use 10 to 400 numbers of concurrent requests to do the tests.

2. Steps 1, 3, 4, 5, 6 can be referred to as Matrix test. 3. Step 2 sets up the load balancing or auto-scaling as detailed in Chapter

8. CPU utilization is recommended to be used as resource trigger.

17


Concurrent intensity test with Clearinghouse: Test Results Analyses

• The JMeter records the results information in the result file (set up in step 3). • Testers can compare the response times using the results file. • Figure 12.6 shows the average response time of GEOSS using Amazon EC2 to scale

different levels of concurrent requests. • Three groups of experiments are conducted: no scale up and scale down with just

one instance, automatically scale up with up to 2 instances, and automatically scale up to 5 instances (response/latency time longer than 4s is trigger).

• Using 2 and 5 instances to scale the concurrent access can decrease the average response time. After 50 to 60 of concurrent requests, average response time is longer than 4 seconds, group “2 instances” and “5 instances” started a new instance.

• The amazon AWS log file shows group “5 instances” start the third instance 10 minutes later after the second instance starts, the fourth instance 8 minutes later after the third instance starts and the fifth instance 9 minutes later after the fourth instance starts. Hence there are some drops when the concurrent number is 170, 210 and 270 for the group “5 instances”.

18


Learning Objectives



19


Data and computing intensity test with Climate@Home

1. Computing Requirements2. Test Design3. Test workflow4. Test Results Analyses

20


Data and computing intensity test with Climate@Home: Computing Requirements

The Climate@Home project needs multiple virtual machines to run many model runs.

• Climate model: ModelE• Minimum computing requirements: A Linux machine equipped

with one CPU core of 1.5+ Ghz speed and 1G RAM• The initial capacity of data storage: >10G • The cloud instance: sufficient hard disk space for the basic

installation, which includes the model, input data and model outputs as well.

21


Data and computing intensity test with Climate@Home: Test Design

Experiment how different cloud services support climate modeling. To be specific in three tasks:

• Identifying a cost-effective hardware configuration of the instance.

• Evaluating the stability and reliability of virtual machine instances by recording the total failure times and average time spent on executing multiple runs.

• Comparing the performance of cloud-based virtual machines and physical machines.

22


Data and computing intensity test with Climate@Home: Test workflow

23

Compile a list of configurations

Create virtual machines based on the configurations

Configure ModelE on the virtual

machine

Develop shared script

Exectute the model simulations

Collect data

Install OS and dependent packages

1: Configure computing

environment

2: Configure testing environment

3: Test multiple runs

4: Analyze resultsAnalyze data

5: Recycle machines Video: Chapter_12-Video_3-modelEtest.mp4

http://www.youtube.com/watch?v=Ttg9idRuLVg


Data and computing intensity test with Climate@Home: Test Results Analyses

• The configurations of virtual machines include 1-core, 2-core, 4-core and 8-core instances with comparable main memory .

• After executing the shared script on every instance, the time cost of each model-run is recorded in a log file tagged with the end time of each simulation period.

• When analyzing the log file, we need to clearly label the association of the log files to their machine configurations which are not captured in the file name.

24

An example of the log file (“logModel19491208.txt”, 1-core instance from Nebula).


Learning Objectives



25


Comprehensive test using dust storm forecasting

1. Computing Requirements2. Test Design3. Test Workflow4. Test Results Analyses

26


Comprehensive test using dust storm forecasting: Computing Requirements

• Dust storm forecasting involves big geoscience data in model input and output, intensive data processing and computing (Chapter 10).

• The periodic phenomena simulation of dust storm requires multiple computing resources to run the simulation in parallel for running the same set of intensive computation for many times.

• Advanced computing methodologies and techniques are normally used to address the computing challenges (Yang et al. 2011a).

• Previous study of dust storm simulation (Huang et al. 2012) found that CPU and networking speed have significant impact on the simulation performance because of massive data exchange and synchronization (as described in the Chapter 10).

• Therefore, the cloud virtual machines for dust storm forecasting should have fast CPU speed and be connected with high bandwidth network.

27


Comprehensive test using dust storm forecasting: Test Design

28

Examining the performance of local HPC cluster and the two cloud services:• Different numbers of computing nodes from different services for supporting

the same dust storm simulations. • The same amount of computing resources from different services to support

different simulation tasks. • The impact of hyperthreading.

1) a single physical processor appear to be multiple logical processors, and significantly improves performance on computing workloads (Koufaty and Marr 2003).

2) Improve processor resource utilization (Bulpin and Pratt 2004). 3) Experiment is designed to test the performance of one EC2 instance

before and after shutting down the hyperthreading capability.


Comprehensive test using dust storm forecasting: Test Workflow

29

Video: Chapter_12-Video_2-dust-storm-test.mp4

http://www.youtube.com/watch?v=TVlOFOdsHKI


Comprehensive test using dust storm forecasting: Test Results Analyses

1. Figure 12.8 shows a typical test result with different process numbers. 1) The first line “****** Test type EC2.2VMs**********” indicates that it is the

test result by EC2 cloud service with two instances. 2) The third line indicates the model running directory. 3) The fourth line means the starting time to run the model with 128 process

numbers. 4) The fifth line tells the model completing time with 128 processes. 5) The following lines indicate the starting and ending time with other process

numbers. 6) The last line with “Finish” shows the test has been finished successfully.

2. The execution time of a 3-hour and 2.3 * 3.5 degree model forecasting by EC2 cloud service using two instances and different process numbers can be calculated and analyzed (Figure 12.9 a).

3. Combining different test output files, more complex analysis can be performed (Figure 12.9 b).

30


Discussion Questions

1. What’s the different between using and without using preconfigured images for the experiments?

2. Do you know any other geoscience application that fit into the intensities of data, computing, concurrent, and spatiotemporal?

3. How to analyze the testing results? 4. How to compare the performance of different cloud services?

31


References

1. Huang, Q., Yang, C., Nebert, D., Liu, K., & Wu, H. (2010, November). Cloud computing for geosciences: deployment of GEOSS clearinghouse on Amazon's EC2. In Proceedings of the ACM SIGSPATIAL International Workshop on High Performance and Distributed Geographic Information Systems (pp. 35-38). ACM.

2. Huang, Q., Yang, C., Liu, K., Xia, J., Xu, C., Li, J., ... & Li, Z. (2013). Evaluating Open-source Cloud Computing Solutions for Geosciences. Computers & Geosciences.

3. Sun, M., Li, J., Yang, C., Schmidt, G. A., Bambacus, M., Cahalan, R., ... & Li, Z. (2012). A Web-Based Geovisual Analytical System for Climate Studies. Future Internet, 4(4), 1069-1085.

32

c. yang, m. sun, j. xia, j. li, k. liu, q. huang and z. gui, 2013. chapter 12 how to test the...

Documents

spatial cloud computing

readiness of cloud services

readiness cloud services

computing intensity

concurrent intensity

comprehensive test

kai liu

qunying huang