discovering metamorphic relations for scientific software
TRANSCRIPT
EDITORS: Jeffrey Carver, [email protected] Morris, [email protected]
SPECIAL TRACK: SOFTWARE ENGINEERING
Discovering Metamorphic Relations forScientific Software From User ForumsXuanyi Lin , University of Cincinnati, Cincinnati, OH, 45221, USA
Michelle Simon , Environmental Protection Agency, Cincinnati, OH, 45268, USA
Zedong Peng , University of Cincinnati, Cincinnati, OH, 45221, USA
Nan Niu , University of Cincinnati, Cincinnati, OH, 45221, USA
Scientific software can be used for decades and is constantly evolving. Recently,metamorphic testing, a property-based testing technique, has shown to be effectivein testing scientific software, and the necessary properties are expressed asmetamorphic relations. However, the development of metamorphic relations isdifficult: it requires considerable practical expertise for the software tester. In thisarticle, we report our experience of uncovering metamorphic relations from a userforum’s questions of the United States Environmental Protection Agency’s StormWater Management Model (SWMM). Our study not only illustrates a wealth of endusers’ expertise in interpreting software results, but also demonstrates theusefulness of classifying the user-oriented metamorphic relations into a nominal,ordinal, and functional hierarchy mainly from the software output perspective.
Scientific software development can last for deca-des. Compared to business/IT software, develop-ers of scientific software often find it difficult to
perform rigorous, systematic testing.1 A root cause is theoracle problem when the mechanism for checkingwhether the program under testing produces theexpected output. For example, the exact value of sine(12)can be hard to obtain, as determining this oracle wouldrequire knowledge about how the specific scientific soft-ware implementation handles round-off and truncation.
An emergingmethod to address the oracle problem ismetamorphic testing.2 The chief idea is to shift softwaretesting from one input at a time to multiple inputs whoseoutputs shall meet cert`ain relations. An example of test-ing sine() is asserting sine(x) ¼ sine(p-x) irrespective ofthe implementation specifics. Relations like sine(x)¼ sine(p-x) are called metamorphic relations (MRs), which arenecessary properties of the program under test over a
sequence of two or more inputs and their correspondingoutputs.
Identifying MRs is critical because without them,metamorphic testing cannot be performed. Murphy etal.3 made one of the first attempts to codify commonMRs. They enumerated six MRs applicable to numeri-cal or collection-like data types: additive, multiplica-tive, permutative, invertive, inclusive, and exclusive.These MRs focus only on change made to the input,and are used to verify the software correctness mainlyat the method and function levels.
In this article, the focus is on both software correct-ness verification and user requirements validation. Weare interested in testing scientific software as it is run bythe end users as an individual, stand-alone application. Asscientific software tends to have a large multiparameterinput space, users would focus on a subset of inputs andoutputs when executing the code. It is therefore throughthe actual usages of the scientific software that weexpect diverseMRs to be discovered.
MRS OF SCIENTIFIC SOFTWAREIn a survey conducted by Segura et al.4 covering thepapers published from 1998 (the year that metamorphic
1521-9615 � 2020 IEEEDigital Object Identifier 10.1109/MCSE.2020.3046973Date of publication 23 December 2020; date of currentversion 25 March 2021.
March/April 2021 Published by the IEEE Computer Society Computing in Science & Engineering 65
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
testingwas introduced2) to 2015, 119 primary studieswereidentified. We deem this survey’s categories of “simula-tion and modeling” and “numerical programs” as themost relevant to scientific software; these categorieswere in contrast to categories such as “web servicesand applications” and “variability and decision sup-port.” It is shown in Segura et al.4 that these two scien-tific software categories are among the most populardomains where metamorphic testing is applied. Toinclude more recent studies, we performed a citationanalysis via Google Scholar on the nine “simulation andmodeling” and “numerical programs” papers discussedin the survey of Segura et al.4 In addition, we manuallysearched the proceedings of two relevant workshopssince 2016: one on metamorphic testing (http://metwiki.net/MET19/series.html) and the other on soft-ware engineering for science (https://se4science.org/workshops/). Our citation analysis and workshop pro-ceedings search was performed inMarch 2020.
Our search resulted in 14 papers. Combined withthe nine papers discussed,4 the total number ofstudies was 23. Among this group of work applyingmetamorphic testing to simulation, modeling, andnumerical programs, we performed a final filteringwith a user-oriented focus, i.e., we paid special atten-tion to the scientific software that was developed foruse by others and not only used by the original devel-opers. To this end, we checked whether the softwarewas publicly downloadable and whether there was evi-dence about end users’ usage of the software. Apply-ing our user-oriented filtering led to 13 papers. Table 1summarizes these studies.
FromTable 1, it can be seen that each study involved ahandful MRs (from 1 to 5), which are identified manually
by testers or researchers. Although a body of work usesmachine learning to predict if a code unit (e.g., a method)exhibits certain MRs (e.g., “additive,” “inclusive,” and “per-mutative”),5 the diverse domains listed in Table 1 showthat focusing on particular input parameters and theirinfluences on specific outputs can be a useful strategyfor deriving scientific software’s MRs. Our work expandsthe discovery of MRs by turning attention to the endusers’ point of view.
SWMMDEVELOPMENT ANDEVOLUTION
The subject system of our study is the SWMM devel-oped and maintained by the United States Environ-mental Protection Agency (EPA). SWMM is a dynamicrainfall-runoff simulation model that computes runoffquantity and quality from primarily urban areas. Thedevelopment of SWMM began in 1971 and since thenthe software has undergone several major upgrades.
We studied SWMM version 5.1.015, released in July2020. The top of Figure 1 shows a screenshot of theSWMM5 graphical user interface where inputs (e.g., prop-erties of a weir) and output (e.g., link flow results) are illus-trated. The computational engine of SWMM is written inC with about 46,300 lines of code, which can be compiledeither as a DLL (dynamic link library) underWindows or asa stand-alone console application under both Windowsand Linux. The bottom of Figure 1 shows that our meta-morphic testing is file based and invokes the command-line environment to run SWMM. We share our testingcode at https://doi.org/10.7945/ravz-y408.
The users of SWMM include hydrologists, engi-neers, and water resources management specialists
TABLE 1. User-Oriented Studies Applying Metamorphic Testing to Scientific Software.
DOI link to the paper1 # of MRs # of test cases Code environment Domain
10.1109/ICECCS.2009.28 4 700 Java; JMT Queuing network10.1145/1987993.1988003 5 N/A2 Java; JSim; MATLAB; GCS3 Health care simulation10.1007/978-3-642-02138-1_19 4 N/A N/A Network simulation10.1145/1982595.1982597 5 >40 FORTRAN; MPI4 Monte Carlo modeling10.1007/s12243-014-0442-7 3 600 N/A Cloud computing10.1145/3194747.3194750 3 2336 C; Python Storm water modeling10.1145/2897676.2897678 5 12 300 Java; BBMap Bioinformatics10.1145/2896971.2896981 5 N/A C99; Cþþ; FORTRAN Light scattering10.1109/MET.2017.4 1 N/A N/A Bioinformatics10.1109/MET.2017.1 1 N/A N/A Time series analysis10.1145/3193977.3193981 3 N/A Java; LingPipe Bioinformatics10.1145/3193977.3193979 2 N/A JavaScript Data analytics10.1007/s11219-016-9337-3 5 >40 FORTRAN; MPI Monte Carlo modeling
1The top five papers are also part of the “simulation and modeling” and “numerical programs” surveyed.42N/A means not applicable.3GCS represents Glycemic Control Simulator.4MPI represents Message Passing Interface.
66 Computing in Science & Engineering March/April 2021
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
who are interested in the planning, analysis, and designrelated to storm water runoff and quality, combinedand sanitary sewers,6 and other drainage systems inurban areas. Since end users generally face the oracleproblem when validating each single execution ofSWMM, metamorphic testing can help. In fact, ourongoing work on hierarchical and exploratory MRs7,8
illustrates this by detecting defects when SWMM isintegrated with an automated parameter calibrationsystem.9 Here, our interests are in SWMM itself.
Because the development of SWMM has lasted forfive decades and continues to be active, we decide toextract potentially valuable information from userforums. A forum allows people who share similar inter-ests to discuss issues related to certain topics. ForSWMM end users, the topics typically include how torun the software and interpret its results. We relymainly on Open SWMM (https://www.openswmm.org)to extract the software usage questions valuabletoward formulating MRs.
DISCOVERING MRS FROM SWMMUSAGE QUESTIONS
The first set of questions we noticed is that the soft-ware did not run at all under certain conditions. Figure 2shows such an example where the formats of date inan external data file caused SWMM to “crash.”Although how to format time series is specified inSWMM’s user manual, which is available online(https://www.epa.gov/water-research/storm-water-management-model-swmm), it is clear that some users(e.g., User1 in Figure 2) do not follow the manual andfeel unsatisfied after the continued error appearances.
The post shown in Figure 2 led us to formulate ametamorphic relation MRdate by checking how differentdate formats could result in errors. Discovering MRdate
allowed us to run SWMM via the command line consoleautomatically on 480 test cases by considering five fac-tors (https://en.wikipedia.org/wiki/Date): basic compo-nent (year-month-day, month-day-year, or day-month-year), year (two-digit or four-digit), month (one to three
FIGURE 1. SWMM5 Graphical User Interface (top) and our file-based metamorphic testing (bottom).
March/April 2021 Computing in Science & Engineering 67
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
characters, or fully spelled out), day (one to three char-acters, or fully spelled out), and separator (“/” [slash], “.”[dot], “-“ [dash], “ “ [space], or “” [no space]).
The automated testing results of MRdate are shownin Figure 3. User1’s post corresponds to the middle barof Figure 3. We used 480 different ways to format thedate of time series in an external rainfall .dat file.SWMM encountered errors under 464 (97%) testsincluding the ones mentioned in User1’s post. It is sur-prising to realize that when the same 480 cases weretested through SWMM’s .inp input file, the error ratiodropped to 90%, as shown in the bottom bar ofFigure 3. MRdate helped detect real defects of SWMM,confirmed by an original developer. The results furthersuggest that the code written to handle the “codeinput-in” data (.inp) is more buggy than the code writ-ten to handle the external input data files (.dat). Thedevelopers should test the “internal” code more rigor-ously, leaving less room for the “inside” errors to creepin. MRdate helps to achieve that.
The second class of questions that we analyzedconcerned the SWMM runtime behavior in specific sit-uations. In contrast to the first set of questions dis-cussed above, SWMM was able to complete a run;however, certain software behaviors puzzled theusers. For example, User2 raised a question: when astorm is at its peak the total inflow rate to a node(Qin) is much greater than the total outflow rate froma node (Qout) for an open storm ditch: however, Qoutbecomes much greater than Qin when the storm’sintensity lessens. User2 expressed it as “a strangeeffect” of SWMM as shown in Figure 4 (left).
Commenter1 provided a useful hint to betterunderstand Qin and Qout: In SWMM, each nodeassembly consists of the node itself and half of thelength of each link connected to the node. As a result,SWMM exhibits a storage effect and the “strangeeffect” mentioned by User2 is actually a feature ratherthan a bug. Figure 4 (right) illustrates this storageeffect with the junction node positioned in a slopedroad. The node absorbs the storm to some extentwhen the storm becomes heavy, serving effectively asa surface water storage. In this case, Qin is muchlarger than Qout. When the storm’s intensity lessens,the junction node outflows water, resulting in agreater Qout than Qin.
The post by User2 and the related discussions aboutthe post led us to produce MRditch, which monitors howQout changes via the alternation of Qin (storm volume).Automatically testing MRditch could better delineate thebehavioral patterns of SWMM. Although no bugs wouldbe detected with MRditch, the systematic testing associ-ated with this MR could make better sense of the soft-ware behavior and further depict it in a morecomprehensive form, e.g., via a piecewise linear func-tion. Explaining what User2 referred to as the strangeeffect may require diagnostic methods like parameter
FIGURE 2. Question (https://www.openswmm.org/Topic/
11247/question-about-error-363) provoking the formulation
of MRdate.
FIGURE 3. Testing results of MRdate where each bar is based on 480 test cases.
68 Computing in Science & Engineering March/April 2021
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
tracking8 and domain knowledge like the temporarystorage effect offered by Commenter1.
The third category of usage questions thatemerged from our analysis embodies the end users’desire to understand the software output in a quanti-tative way. Figure 5 (top) presents a post where User3did not choose the correct equation to calculate“Water Head.” What User3 should be using was: WaterHead ¼ Storage Water Level – Storage Bottom Eleva-tion, but what the user actually used was: Water Head¼ Storage Water Level – Downstream Node Invert.
Although User3 perceived SWMM’s calculation tobe wrong, he made sense of the situation by diagnos-ing which incorrect equation caused the problem, dueto the visibility of “Water Head” values trackable fromSWMM’s output. However, in some situations, whatthe end users are interested in tracking may not bedirectly visible to them. The screenshot shown inFigure 1 is one such example. The user is providedwith a drop-down list of five options to select the typeof weir in the simulation. According to the specifica-tions, SWMM implements the “Weir Type” as shown inFigure 5 (bottom).
Unfortunately, “Weir Type” values are not directlyviewable to the end users. Had the user chosen anincorrect equation but was unaware of the mistake, itwould be challenging for the user to figure out thatSWMM’s visible outputs were wrong due to the “WeirType” selection. Fortunately, a class of MRs could beformed on the basis of the above “Weir Type” specifi-cations. For example, the relation between V-NOTCHand ROADWAY can be inferred by how h varies, i.e.,ðCwSh
5=2Þ=ðCwLh3=2Þ ¼ ðS=LÞh, where S and L are con-
stants. Figure 6 (left) plots MRweir between V-NOTCH
and ROADWAY. Using (S/L)h, the change fromV-NOTCH to ROADWAY, e.g., triggered by the end user atSWMM runtime, can be quantitatively characterized.
Although the values of “Weir Type” are not directlyaccessible to the end users, the quantitative MRweir
makes it easier to locate surrogate variables to trackthe switch between “Weir Type” options and henceeasier to correct the wrong choice at SWMM runtime.Figure 6 (right) shows two output variables directly vis-ible to the end users: “maximum flow” and “averagevolume” tested when “Weir Type” was switched fromV-NOTCH to ROADWAY. Although “average volume”had a nonlinear change, “maximum flow” reflected thelinear function with the slope ¼ (S/L). The outputvariable, “maximum flow,” thus effectively serves as asurrogate for tracking V-NOTCH and ROADWAY.
In summary, our analysis of SWMM user forumsidentified a broad range of questions raised by the endusers themselves that enabled us to perform auto-mated metamorphic testing. Compared to other test-ing techniques like fuzzing where invalid, unexpected,or random inputs are generated, the MRs in our workare formulated based on the actual uses of the scien-tific software. Moreover, fuzzing tends to crash or hangan application, whereas MRs attempt to check whetherthe expected output relations hold. For scientific soft-ware with a large input and a large output spaces, the“input change leading to output change” can be a moreeffective testing strategy than crashing a software sys-tem. During our collaboration meetings, we shared theMRs discovered from the user forums with the SWMMdevelopment team and received positive feedback.Although the MRs all seemed sensible, the team neverthought testing the software by using these relations.
FIGURE 4. Question (https://www.openswmm.org/Topic/19102/junction-apparently-functioning-as-a-retention-basin) provoking
the formulation of MRditch.
March/April 2021 Computing in Science & Engineering 69
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
In fact, the developers were surprised that bugs wereuncovered by MRdate and user questions were raisedregarding MRditch.
When it comes to assisting in the formulation ofMRs, we classified the information extracted from the
usage questions into three categories: those complain-ing the software is not able to run at all in certain condi-tions, those expressing the difficulty of making sense ofa peculiar behavior of the software in some situations,and those trying to figure out what might go wrong
FIGURE 5. Question (https://www.openswmm.org/Topic/11274/epa-swmm-outlet-rating-discharge-differences-between-swmm-
versions) provoking the formulation of MRweir.
FIGURE 6. Plotting V-NOTCH and ROADWAY on h (left). Testing MRweir of V-NOTCH and ROADWAY on two SWMM output varia-
bles (right).
70 Computing in Science & Engineering March/April 2021
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
with the software output from the end user’s controlperspective and how to get it right. We conclude nextby organizing these categories in a hierarchy of MRsand by sharing our lessons learned. Our proposed cate-gories are: 1) errors that prevent successful initiation orcompletion of the program, 2) potential differences inoutput from expected behavior, and 3) potential differ-ences in output related to operations, functions, or cri-teria outside the users’ control.
CONCLUDING REMARKSMetamorphic testing alleviates the oracle problemcommonly faced by scientific software.1 Selecting aproper set of MRs is useful for comprehensive soft-ware testing. While current approaches rely onresearchers’ domain knowledge or developers’ imple-mentation knowledge to identify MRs, we have shownin this article that forums where end users pose ques-tions and exchange concerns are valuable sources todiscover MRs. Compared to software documentationlike SWMM user manual, forums contain moredynamic usage scenarios, allowing some MRs to beidentified more easily. For example, the input variableof MRWeir (“Weir Type”) appears four times in theSWMM user manual (https://www.epa.gov/water-research/storm-water-management-model-swmm-version-51-users-manual); however, this input variablenever coappears with the two output variables on thesame page. The closest distance of “Weir Type” and“maximum flow” is 3 pages, and that of “Weir Type”and “average volume” is 68 pages, suggesting the diffi-culty of extracting MRWeir only from software
documentation. Our manual analysis of these SWMMforums helped extract half of a dozen MRs specific toSWMM, which we show in Table 2 (left).
Our results suggest a hierarchy of nominal, ordinal,and functional MRs as shown in Table 2 (right). Thishierarchy is rooted in the theory of measurementscales,10 where nominal groups data only by names(e.g., “run” or “not run”) and ordinal allows data to beordered (e.g., “less than” or “greater than”). The mostprecise MRs in our hierarchy are those of functionalforms. When a change is made to the input (Dx), a func-tional MR allows the corresponding output change (Dy)to be predicted in an exact manner, e.g., the “WeirType” switch from V-NOTCH to ROADWAYwould followa linear relationship. Compared with the MRs codifiedin Murphy et al.,3 the hierarchy of MRs of Table 2 focusmore on the behavior and the output of software.
Our ongoing work investigates automated ways ofmining MRs from the usage questions, exploits server-less architectures and frameworks to efficiently exe-cute metamorphic tests11 and leverages numericalregression tests12 to verify the correctness of the MRs.
ACKNOWLEDGMENTSThis work was supported in part by the U.S. NationalScience Foundation (Award CCF 1350487) and U.S.Environmental Protection Agency.
DISCLAIMERThe research described in this article has been fundedin part by the U.S. Environmental Protection Agency
TABLE 2.MRs Discovered Manually From SWMMUser Forums (Left) and a Hierarchy Focusing on Output Relations (Right).
March/April 2021 Computing in Science & Engineering 71
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.
(EPA). It has been subjected to review by the Office ofResearch and Development and approved for publica-tion. Approval does not signify that the contentsreflect the views of the Agency, nor does mention oftrade names or commercial products constituteendorsement or recommendation for use.
REFERENCES1. U. Kanewala and J. M. Bieman, “Testing scientific
software: A systematic literature review,” Inf. Softw.
Technol., vol. 56, no. 10, pp. 1219–1232, Oct. 2014.
2. T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic
testing: A new approach for generating next test
cases,” The Hong Kong Univ. Sci. Technol., Hong Kong,
China, Tech. Rep. HKUST-CS98-01, 1998.
3. C. Murphy, G. E. Kaiser, L. Hu, and L. Wu, “Properties
of machine learning applications for use in
metamorphic testing,” in Proc. 20th Int. Conf. Softw.
Eng. Knowl. Eng., San Francisco, CA, USA, Jul. 2008,
pp. 867–872.
4. S. Segura, G. Fraser, A. B. Sanchez, and A. R. Cortes, “A
survey on metamorphic testing,” IEEE Trans. Softw.
Eng., vol. 42, no. 9, pp. 805–824, Sep. 2016.
5. A. Nair, K. Meinke, and S. Eldh, “Leveraging mutants for
automatic prediction of metamorphic relations using
machine learning,” in Proc. 3rd Int. Workshop Mach.
Learn. Techn. Softw. Qual. Eval., Tallinn, Estonia,
Aug. 2019, pp. 1–6.
6. H. Gudaparthi, R. Johnson, H. Challa, and N. Niu, “Deep
learning for smart sewer systems: Assessing
nonfunctional requirements,” in Proc. 42nd Int. Conf.
Softw. Eng. Softw. Eng. Soc. Track, Seoul, South Korea,
Jun./Jul. 2020, pp. 35–38.
7. X. Lin, M. Simon, and N. Niu, “Hierarchical metamorphic
relations for testing scientific software,” in Proc. Int.
Workshop Softw. Eng. Sci., Gothenburg, Sweden,
Jun. 2018, pp. 1–8.
8. X. Lin, M. Simon, and N. Niu, “Exploratory metamorphic
testing for scientific software,” Comput. Sci. Eng.,
vol. 22, no. 2, pp. 78–87, Mar./Apr. 2020.
9. S. Kamble, X. Jin, N. Niu, and M. Simon, “A novel
coupling pattern in computational science and
engineering software,” in Proc. Int. Workshop Softw.
Eng. Sci., Buenos Aires, Argentina, May 2017, pp. 9–12.
10. S. S. Stevens, “On the theory of scales of
measurements,” Science, vol. 103, pp. 677–680,
Jun. 1946. Art. no. 2684.
11. X. Lin, M. Simon, and N. Niu, “Scientific testing goes
serverless: Creating and invoking metamorphic
functions,” IEEE Softw., vol. 38, no. 1, pp. 61–67, Jan./
Feb. 2021.
12. Z. Peng, X. Lin, and N. Niu, “Unit tests of scientific
software: A study on SWMM,” in Proc. 20th Int. Conf.
Comput. Sci., Amsterdam, The Netherlands, Jun. 2020,
pp. 413–427.
XUANYI LIN is currently a Ph.D. candidate with the Depart-
ment of Electrical Engineering and Computer Science, Univer-
sity of Cincinnati. His research interests include automated
software testing, scientific software development, and require-
ments engineering. He received the M.Sc. degree in computer
science from the University at Albany, State University of New
York, NY, USA, in 2016. Contact him at [email protected].
MICHELLE SIMON is currently a Senior Chemical Engineer
with the U.S. Environmental Protection Agency Office of
Research and Development. She is also with the Center for
Environmental Solutions and Emergency Response, Water
Infrastructure Division, Cincinnati, OH, USA. He is a profes-
sional Engineer. Her research interests include stormwater
modeling, green infrastructure, and wastewater infrastruc-
ture. She received the B.S. degree in chemical engineering
from the University of Notre Dame, the M.S. degree in chemi-
cal engineering from the Colorado School of Mines, and the
Ph.D. degree in environmental science from the University of
Arizona . Contact her at [email protected].
ZEDONG PENG is currently a Ph.D. student with the Depart-
ment of Electrical Engineering and Computer Science,
University of Cincinnati. His research interests include
requirements engineering, metamorphic testing, and natural
language processing. He received the M.Sc. degree in com-
puter science from the Ball State University, Muncie, IN, USA,
in 2019. Contact him at [email protected].
NAN NIU is currently an Associate Professor with the Depart-
ment of Electrical Engineering and Computer Science,
University of Cincinnati. His research interests include
requirements engineering, scientific software development,
and human-centric computing. He received the Ph.D. degree
in computer science from the University of Toronto . Contact
him at [email protected].
72 Computing in Science & Engineering March/April 2021
SOFTWARE ENGINEERING
Authorized licensed use limited to: University of Cincinnati. Downloaded on March 29,2021 at 23:26:37 UTC from IEEE Xplore. Restrictions apply.