![Page 1: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/1.jpg)
Performance Comparison of
a Custom Emulation-based
Test Environment Against a
Real-world LTE Testbed
1
Sérgio Massami Sakai – Software Engineer
CPqD Foundation
![Page 2: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/2.jpg)
Motivation
• LTE´s high complexity poses computational and time
burdens on testing tasks involving real-world platforms.
• Simulation is the tool most widely used for this purpose.
• Emulation in research has been ramping up recently as
alternatives to pure simulation.
• Most simulators found in the literature tend to rely on
built-in traffic generation processes.
2
![Page 3: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/3.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
3
![Page 4: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/4.jpg)
LENA
Overview
• LENA: ns-3 module whose use has been leveraged
within the community in view of the LTE functionalities.
• Simulates evolved packet core (EPC) network, evolved
node B (eNodeB), user equipment (UE), and respective
protocol stacks.
• Does not support the use of external traffic entities in
conjunction with the LTE simulation.
4
![Page 5: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/5.jpg)
LENA
Network Topology
Source: http://networks.cttc.es/mobile-networks/software-tools/lena/
5
![Page 6: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/6.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
6
![Page 7: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/7.jpg)
Experiments
Proposal
• Customize LENA for use in an emulation-based test
Environment.
• Connect the ns-3 server to other computers running
industry-standard testing tools.
• Assess the performance of the emulator in terms of
voice quality, latency, and throughput. Focus on VoIP
applications.
• Validate the results obtained through emulation against
an (real) LTE testbed.
7
![Page 8: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/8.jpg)
Experiments
Overview
• Two different experiments:
• Emulation-based: Simulated LTE Network
• Testbed-based: Real-world LTE Network
• Same functionality is implemented in both experiments.
• Same traffic generation and test tools are used in both
experiments:
• IxCharriot
• Iperf
8
![Page 9: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/9.jpg)
Experiments
LTE Configuration
Same configuration in both experiments:
• Band 31 (450-470 MHz)
• 5MHz bandwidth
• Frequency Division Duplexing (FDD) mode
• Adaptative modulation
• Round robin scheduler
• SISO operation mode
9
![Page 10: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/10.jpg)
Experiments
General Architecture
The measurement architecture depicted
here is common to both experiments.
10
![Page 11: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/11.jpg)
Experiments
Experiment I: Simulated LTE Network
• “LTE Network” is replaced with ns-3 server
• SNR is set by changing the simulator parameters related
to line of sight distance between eNodeB and UE.
11
![Page 12: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/12.jpg)
Experiments
LENA Customization and Configuration
• Use class “FdNetDevice” to create in LTE simulation two
ethernet interfaces mapped onto two real physical
ethernet interfaces on the server:
• One interface created in an UE modem (LTE user side) – mapped
onto “eth1” physical ethernet interface.
• One interface created in a host connected to the EPC (LTE core
network side) – mapped onto “eth0” physical ethernet interface.
• Customize and configure simulation data plane & routing
to support LTE data flow between the two ethernet
interfaces.
• Configure LENA operation mode to realtime mode.
• Use mapped physical ethernet interfaces to connect
simulated elements (UE and host) to the external traffic
generation/test tools. 12
![Page 13: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/13.jpg)
Experiments
Experiment II: Real-world LTE Network
• “LTE Network” is replaced with real LTE testbed
• SNR is set by changing attenuation between eNodeB
and UE. 13
![Page 14: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/14.jpg)
Experiments
Laboratory Environment
14
![Page 15: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/15.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
15
![Page 16: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/16.jpg)
Measurements
VoIP
• VoIP traffic test elements:
• IxCharriot server
• IxCharriot endpoint 1 – UE side PC
• IxCharriot endpoint 2 – EPC side PC
• IxCharriot endpoints generate 1-10 VoIP calls:
• RTP
• G.711 codec
• SNR range: 5-25 dB
• Performance metrics: mean opinion score (MOS) and
average latency estimated by IxCharriot.
Obs. 1: Each SNR x Data Traffic point: 3 measurements, 1 minute duration.
Obs. 2: IxCharriot is a commercial test tool used to predict device and system performance.
Obs. 3: IxCharriot MOS estimation takes into consideration end-to-end delay, packeterization delay,
jitter buffer delay, additional fixed delay, data loss, and jitter.
16
![Page 17: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/17.jpg)
Measurements
Throughput
UDP data traffic elements:
• Iperf client at UE side PC
• Iperf server at EPC side PC
Configured data rate: 20 Mbps
SNR range: 5-25 dB
Direction: Downlink
Performance metrics: data rate received at iperf server side
17
![Page 18: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/18.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
18
![Page 19: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/19.jpg)
Results
VoIP MOS measurements
• The perceived voice quality
is above "good"
(MOS=4:00) and below
"excellent“ (MOS=5:00) for
both experiments.
• No significant MOS
performance difference
between Emulation and
Real-world testbed (0.01).
19
![Page 20: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/20.jpg)
Results
VoIP latency measurements
• Average Latencies are no
longer than 20 ms.
• Emulation latency almost
constant.
• Performance of the Real-
world Testbed is shown to
be slightly better than that
of the Emulated
Environment due to the
characteristic latency of the
latter, which does not suffer
significant variations as the
number of simultaneous
calls is varied.
20
![Page 21: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/21.jpg)
Results
Throughput measurements
• Achieved throughput
increases as a function of
the SNR.
• Emulated Environment
slightly outperforms the
Real-world Testbed along
all the SNRs considered.
• Slight advantage of the
Emulation over Real-world
testbed caused by
simplications of the
simulation implementation.
21
![Page 22: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/22.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
22
![Page 23: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/23.jpg)
Concluding Remarks
• Initial results confirms the suitability of emulation
approach as a viable means to predict performance in
real LTE networks.
• ns-3 LTE models can deliver MOS voice quality and
latency as good as an experimental testbed using actual
LTE equipment over a range of SNRs.
• The good agreement of our experimental results is
possible not only because the same functionality is
implemented in both experiments but due to the use of
the same traffic generation tools in the simulated and
real-world LTE networks (not possible in standard LENA
simulation).
23
![Page 24: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/24.jpg)
Agenda
• LENA
• Experiments
• Measurements
• Results
• Concluding Remarks
• Future Works
• Questions
24
![Page 25: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/25.jpg)
Future works
• Future work suggestions could involve extensions of the
experiments proposed in this paper to different types of
data traffic, e:g: web browsing, le transfers, and video.
• Another research path that may be worth investigating is
to consider multiple UEs and verify the validity of the
comparisons made here for the case of multiuser
environments.
25
![Page 26: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/26.jpg)
Questions
26
![Page 27: Performance Comparison of a Custom Emulation-based Test ... · VoIP latency measurements • Average Latencies are no longer than 20 ms. • Emulation latency almost constant. •](https://reader033.vdocuments.site/reader033/viewer/2022060421/5f18615ffe49dc5da44109ea/html5/thumbnails/27.jpg)
Thank you!
www.cpqd.com.br
27