c++ network programming mastering complexity with ace & patterns dr. douglas c. schmidt...

229
C++ Network C++ Network Programming Programming Mastering Complexity with Mastering Complexity with ACE & Patterns ACE & Patterns Dr. Douglas C. Schmidt [email protected] www.cs.wustl.edu/~schmidt/tutorials- ace.html Professor of EECS Vanderbilt University Nashville, Tennessee

Upload: lionel-norton

Post on 13-Dec-2015

230 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

C++ Network C++ Network ProgrammingProgramming

Mastering Complexity with Mastering Complexity with ACE & PatternsACE & Patterns

Dr. Douglas C. [email protected]

www.cs.wustl.edu/~schmidt/tutorials-ace.html

Professor of EECS Vanderbilt University Nashville, Tennessee

Page 2: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

2

Motivation: Challenges of Networked ApplicationsObservation• Building robust, efficient, & extensible concurrent & networked applications is hard• e.g., we must address many complex topics that are less problematic for non-concurrent, stand-alone applications

Accidental Complexities• Low-level APIs•Poor debugging tools•Algorithmic decomposition•Continuous re-invention/discovery of core concepts & components

Inherent Complexities• Latency•Reliability•Load balancing•Causal ordering•Scheduling & synchronization•Deadlock

Complexities in networked applications

Page 3: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

3

Presentation Outline

•Patterns, which embody reusable software architectures & designs

•ACE wrapper facades, which encapsulate OS concurrency & network programming APIs

Cover OO techniques & language features that enhance software quality

•OO language features, e.g., classes, dynamic binding & inheritance, parameterized types

Presentation Organization

1. Background

2. Concurrent & network challenges & solution approaches

3. Patterns & wrapper facades in ACE + applications

Page 4: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

4

CPUs & networks have increased by 3-7 orders of magnitude in the past decade2,400 bits/sec to

1 Gigabits/sec

10 Megahertz to 1 Gigahertz

These advances stem largely from standardizing hardware & software APIs & protocols, e.g.:

The Evolution of Information Technologies

• TCP/IP, ATM• POSIX & JVMs• Middleware & components

• Intel x86 & Power PC chipsets

• Quality of service aspects

Extrapolating this trend to 2010 yields

• ~100 Gigahertz desktops• ~100 Gigabits/sec LANs• ~100 Megabits/sec wireless• ~10 Terabits/sec Internet backbone

In general, software has not improved as rapidly or as

effectively as hardware

In general, software has not improved as rapidly or as

effectively as hardware

Increasing software productivity & QoS depends heavily on COTSIncreasing software productivity & QoS depends heavily on COTS

Page 5: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

5

There are multiple COTS layers & research/

business opportunities

Historically, mission-critical apps were built directly atop hardware & OS•Tedious, error-prone, & costly over lifecycles

Standards-based COTS middleware helps:•Control end-to-end resources & QoS•Leverage hardware & software technology advances

•Evolve to new environments & requirements

•Provide a wide array of reuseable, off-the-shelf developer-oriented services

There are layers of middleware, just like there are layers of networking protocols

Component Middleware Layers

Page 6: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

6

Operating System & Protocols•Operating systems & protocols provide mechanisms to manage endsystem resources, e.g.,•CPU scheduling & dispatching•Virtual memory management•Secondary storage, persistence, & file systems•Local & remove interprocess communication (IPC)

•OS examples•UNIX/Linux, Windows, VxWorks, QNX, etc.

•Protocol examples•TCP, UDP, IP, SCTP, RTP, etc.

RTP

DNS

HTTP

UDP TCP

IP

TELNET

Ethernet ATM FDDI

Fibre Channel

FTP

INTERNETWORKING ARCH

TFTP

20th Century

Win2K Linux LynxOS

Solaris VxWorks

Middleware

MiddlewareServices

MiddlewareApplications

MIDDLEWARE ARCH

21st Century

Page 7: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

7 www.cs.wustl.edu/~schmidt/ACE.html

Host Infrastructure Middleware•Host infrastructure middleware encapsulates & enhances native OS mechanisms to create reusable network programming components•These components abstract away many tedious & error-prone aspects of low-level OS APIs

Domain-SpecificServices

CommonMiddleware Services

DistributionMiddleware

Host InfrastructureMiddleware

Synchronization

MemoryManagement

PhysicalMemoryAccess

AsynchronousEvent Handling

Scheduling

AsynchronousTransfer of

Control

www.rtj.org

•Examples•Java Virtual Machine (JVM), Common Language Runtime (CLR), ADAPTIVE Communication Environment (ACE)

Page 8: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

8

Distribution Middleware•Distribution middleware defines higher-level distributed programming models whose reusable APIs & components automate & extend native OS capabilities

Domain-SpecificServices

CommonMiddleware Services

DistributionMiddleware

Host InfrastructureMiddleware

Distribution middleware avoids hard-coding client & server application dependencies on object location, language, OS, protocols, & hardware

•Examples•OMG Real-time CORBA & DDS, Sun RMI, Microsoft DCOM, W3C SOAP

ClientOBJREF

Object(Servant)

in argsoperation()

out args + return

IDLSTUBS

IDLSKEL

Object Adapter

ORB CORE GIOP

Protocol Properties

End-to-End PriorityPropagation

ThreadPools

StandardSynchronizersExplicit

Binding

Portable Priorities

SchedulingService

en.wikipedia.org/wiki/Data_Distribution_Servicerealtime.omg.org/

Page 9: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

9

Common Middleware Services•Common middleware services augment distribution middleware by defining higher-level domain-independent services that focus on programming “business logic”

Domain-SpecificServices

CommonMiddleware Services

DistributionMiddleware

Host InfrastructureMiddleware

•Common middleware services support many recurring distributed system capabilities, e.g.,•Transactional behavior

•Authentication & authorization,

•Database connection pooling & concurrency control

•Active replication

•Dynamic resource management

•Examples•CORBA Component Model & Object Services, Sun’s J2EE, Microsoft’s .NET, W3C Web Services

Page 10: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

10

Domain-Specific Middleware

Modalitiese.g., MRI, CT, CR,

Ultrasound, etc.

Siemens MED Syngo•Common software platform for distributed electronic medical systems

•Used by all ~13 Siemens MED business units worldwide

•Domain-specific middleware services are tailored to the requirements of particular domains, such as telecom, e-commerce, health care, process automation, or aerospace

Domain-SpecificServices

CommonMiddleware Services

DistributionMiddleware

Host InfrastructureMiddleware

Boeing Bold Stroke • Common software platform for Boeing avionics mission computing systems

•Examples

Page 11: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

11

•Present solutions to common software problems arising within a certain context

Overview of Patterns

•Capture recurring structures & dynamics among software participants to facilitate reuse of successful designs

The Proxy Pattern

1 1Proxy

service

Service

service

AbstractService

service

Client

•Help resolve key software design forces

•Flexibility•Extensibility•Dependability•Predictability•Scalability•Efficiency

•Generally codify expert knowledge of design strategies, constraints & “best practices”

Page 12: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

12

Overview of Pattern Languages

Motivation•Individual patterns & pattern catalogs are insufficient

•Software modeling methods & tools that just illustrate how, not why, systems are designed

Benefits of Pattern Languages• Define a vocabulary for talking about software development problems• Provide a process for the orderly resolution of these problems, e.g.:

• What are key problems to be resolved & in what order• What alternatives exist for resolving a given problem• How should mutual dependencies between the problems be handled• How to resolve each individual problem most effectively in its context

• Help to generate & reuse software architectures

Page 13: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

13

Taxonomy of Patterns & Idioms

Type Description Examples

Idioms Restricted to a particular language, system, or tool

Scoped locking

Design patterns

Capture the static & dynamic roles & relationships in solutions that occur repeatedly

Active Object, Bridge, Proxy, Wrapper Façade, & Visitor

Architectural patterns

Express a fundamental structural organization for software systems that provide a set of predefined subsystems, specify their relationships, & include the rules & guidelines for organizing the relationships between them

Half-Sync/Half-Async, Layers, Proactor, Publisher-Subscriber, & Reactor

Optimization principle patterns

Document rules for avoiding common design & implementation mistakes that degrade performance

Optimize for common case, pass information between layers

Page 14: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

14

The Layered Architecture of ACEFeatures•Open-source•200,000+ lines of C++

•40+ person-years of effort

•Ported to many OS platforms

•Large open-source user community• www.cs.wustl.edu/~schmidt/ACE-users.html

•Commercial support by Riverace• www.riverace.com/

www.cs.wustl.edu/~schmidt/ACE.html

Page 15: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

15

Sidebar: Platforms Supported by ACE

•ACE runs on a wide range of operating systems, including:

•PCs, e.g., Windows (all 32/64-bit versions), WinCE; Redhat, Debian, & SuSE Linux; & Macintosh OS X

•Most versions of UNIX, e.g., Solaris, SGI IRIX, HP-UX, Digital UNIX (Compaq Tru64), AIX, DG/UX, SCO OpenServer, UnixWare, NetBSD, & FreeBSD

•Real-time operating systems, e.g., VxWorks, OS/9, LynxOS, Pharlap TNT, QNX Neutrino & RTP, & RTEMS

•Large enterprise systems, e.g., OpenVMS, MVS OpenEdition, Tandem NonStop-UX, & Cray UNICOS

•ACE can be used with all of the major C++ compilers on these platforms

•The ACE Web site at http://www.cs.wustl.edu/~schmidt/ACE.html contains a complete, up-to-date list of platforms, along with instructions for downloading & building ACE

Page 16: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

16

Event Handling & IPCService Access & Control

Concurrency Synchronization

Key Capabilities Provided by ACE

Page 17: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

17

The Pattern Language for ACEPattern Benefits• Preserve crucial design information used by applications & middleware frameworks & components

• Facilitate reuse of proven software designs & architectures

• Guide design choices for application developers

Page 18: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

18

POSA2 Pattern AbstractsService Access & Configuration Patterns

The Wrapper Facade design pattern encapsulates the functions & data provided by existing non-object-oriented APIs within more concise, robust, portable, maintainable, & cohesive object-oriented class interfaces.

The Component Configurator design pattern allows an application to link & unlink its component implementations at run-time without having to modify, recompile, or statically relink the application. Component Configurator further supports the reconfiguration of components into different application processes without having to shut down & re-start running processes.

The Interceptor architectural pattern allows services to be added transparently to a framework & triggered automatically when certain events occur.

The Extension Interface design pattern allows multiple interfaces to be exported by a component, to prevent bloating of interfaces & breaking of client code when developers extend or modify the functionality of the component.

Event Handling Patterns

The Reactor architectural pattern allows event-driven applications to demultiplex & dispatch service requests that are delivered to an application from one or more clients.

The Proactor architectural pattern allows event-driven applications to efficiently demultiplex & dispatch service requests triggered by the completion of asynchronous operations, to achieve the performance benefits of concurrency without incurring certain of its liabilities.

The Asynchronous Completion Token design pattern allows an application to demultiplex & process efficiently the responses of asynchronous operations it invokes on services.

The Acceptor-Connector design pattern decouples the connection & initialization of cooperating peer services in a networked system from the processing performed by the peer services after they are connected & initialized.

Page 19: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

19

POSA2 Pattern Abstracts (cont’d)Synchronization Patterns

The Scoped Locking C++ idiom ensures that a lock is acquired when control enters a scope & released automatically when control leaves the scope, regardless of the return path from the scope.

The Strategized Locking design pattern parameterizes synchronization mechanisms that protect a component’s critical sections from concurrent access.

The Thread-Safe Interface design pattern minimizes locking overhead & ensures that intra-component method calls do not incur ‘self-deadlock’ by trying to reacquire a lock that is held by the component already.

The Double-Checked Locking Optimization design pattern reduces contention & synchronization overhead whenever critical sections of code must acquire locks in a thread-safe manner just once during program execution.

Concurrency Patterns

The Active Object design pattern decouples method execution from method invocation to enhance concurrency & simplify synchronized access to objects that reside in their own threads of control.

The Monitor Object design pattern synchronizes concurrent method execution to ensure that only one method at a time runs within an object. It also allows an object’s methods to cooperatively schedule their execution sequences.

The Half-Sync/Half-Async architectural pattern decouples asynchronous & synchronous service processing in concurrent systems, to simplify programming without unduly reducing performance. The pattern introduces two intercommunicating layers, one for asynchronous & one for synchronous service processing.

The Leader/Followers architectural pattern provides an efficient concurrency model where multiple threads take turns sharing a set of event sources in order to detect, demultiplex, dispatch, & process service requests that occur on the event sources.

The Thread-Specific Storage design pattern allows multiple threads to use one ‘logically global’ access point to retrieve an object that is local to a thread, without incurring locking overhead on each object access.

Page 20: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

20

The Frameworks in ACE

ACE Framework Inversion of Control

Reactor & Proactor Calls back to application-supplied event handlers to perform processing when events occur synchronously & asynchronously

Service Configurator Calls back to application-supplied service objects to initialize, suspend, resume, & finalize them

Task Calls back to an application-supplied hook method to perform processing in one or more threads of control

Acceptor-Connector Calls back to service handlers to initialize them after they are connected

Streams Calls back to initialize & finalize tasks when they are pushed & popped from a stream

Page 21: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

21

Example: Applying ACE in Real-time Avionics

Key System Characteristics• Deterministic & statistical deadlines

• ~20 Hz• Low latency & jitter

• ~250 usecs• Periodic & aperiodic processing• Complex dependencies• Continuous platform upgrades

• Test flown at China Lake NAWS by Boeing OSAT II ‘98, funded by OS-JTF• www.cs.wustl.edu/~schmidt/TAO-boeing.html

• Also used on SOFIA project by Raytheon• sofia.arc.nasa.gov

• First use of RT CORBA in mission computing• Drove Real-time CORBA standardization

• Test flown at China Lake NAWS by Boeing OSAT II ‘98, funded by OS-JTF• www.cs.wustl.edu/~schmidt/TAO-boeing.html

• Also used on SOFIA project by Raytheon• sofia.arc.nasa.gov

• First use of RT CORBA in mission computing• Drove Real-time CORBA standardization

Key Results

Goals• Apply COTS & open systems to mission-critical real-time avionics

Page 22: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

22

Time-critical targets require immediate response because:•They pose a clear & present danger to friendly forces &•Are highly lucrative, fleeting targets of opportunity

Example: Applying ACE to Time-Critical Targets

A d a p t e d f r o m “ T h e F u t u r e o f A W A C S ” ,b y L t C o l J o e C h a p a

J o i n t F o r c e sG l o b a l I n f o G r i d

J o i n t F o r c e sG l o b a l I n f o G r i d C h a l le n g e

is t o m a k e t h is p o s s ib le !

Challenges are also relevant to TBMD & NMD

Goals• Detect, identify, track, & destroy time-critical targets

• Real-time mission-critical sensor-to-shooter needs

• Highly dynamic QoS requirements & environmental conditions

• Multi-service & asset coordination

Key System Characteristics

Key Solution Characteristics•Efficient & scalable•Affordable & flexible•COTS-based

•Efficient & scalable•Affordable & flexible•COTS-based

•Adaptive & reflective•High confidence •Safety critical

•Adaptive & reflective•High confidence •Safety critical

Page 23: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

Example: Applying ACE to Large-scale Routers

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

IOM

BSE

BSE

BSE

BSE

BSE

BSE

BSEBSE

BSEGoal•Switch ATM cells + IP packets at terabit rates

Key Software Solution Characteristics

•High confidence & scalable computing architecture•Networked embedded processors•Distribution middleware•FT & load sharing•Distributed & layered resource management

•Affordable, flexible, & COTS

•High confidence & scalable computing architecture•Networked embedded processors•Distribution middleware•FT & load sharing•Distributed & layered resource management

•Affordable, flexible, & COTS

Key System Characteristics•Very high-speed WDM links

•102/103 line cards•Stringent requirements for availability

•Multi-layer load balancing, e.g.:•Layer 3+4•Layer 5

www.arl.wustl.edu

Page 24: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

24

Example: Applying ACE to Hot Rolling Mills

Goals• Control the processing of molten steel moving through a hot rolling mill in real-time

System Characteristics• Hard real-time process automation requirements• i.e., 250 ms real-time cycles

• System acquires values representing plant’s current state, tracks material flow, calculates new settings for the rolls & devices, & submits new settings back to plant

Key Software Solution Characteristics

•Affordable, flexible, & COTS• Product-line architecture• Design guided by patterns & frameworks

•Affordable, flexible, & COTS• Product-line architecture• Design guided by patterns & frameworks

• Windows NT/2000• Real-time CORBA (ACE+TAO)

www.siroll.de

Page 25: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

25

Example: Applying ACE to Real-time Image Processing

Goals• Examine glass bottles for defects in real-time

System Characteristics• Process 20 bottles per sec• i.e., ~50 msec per bottle

• Networked configuration

• ~10 cameras

Key Software Solution Characteristics

•Affordable, flexible, & COTS• Embedded Linux (Lem)• Compact PCI bus + Celeron processors

•Affordable, flexible, & COTS• Embedded Linux (Lem)• Compact PCI bus + Celeron processors

• Remote booted by DHCP/TFTP• Real-time CORBA (ACE+TAO)

www.krones.com

Page 26: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

26

Networked Logging Service ExampleKey Participants

• Client application processes

•Generate log records

• Server logging daemon

•Receive, process, & store log records

• The logging server example in C++NPv2 is more sophisticated than the one in C++NPv1

C++ code for all logging service examples are in

• ACE_ROOT/examples/ C++NPv1/

• ACE_ROOT/examples/ C++NPv2/

• There’s an extra daemon involved

Page 27: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

27

Patterns in the Networked Logging Service

Reactor

Acceptor-Connector

ComponentConfigurator

MonitorObject

ActiveObject

Proactor

Pipes &Filters

WrapperFacade

StrategizedLocking Scoped

Locking

Thread-safeInterface

Half-Sync/Half-Async

Leader/Followers

Page 28: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

28

ACE Basics: Logging

• ACE’s logging facility usually best for diagnostics• Can customize logging sinks• Filterable logging severities• Portable printf()-like format directives (thread/process ID,

date/time, types)• Serializes output across multiple threads• Propagates settings to threads created via ACE• Can log across a network

•ACE_Log_Msg class; use thread-specific singleton most of the time, via ACE_LOG_MSG macro

• Macros encapsulate most usage. Most common:•ACE_DEBUG ((severity, format [, args…]));•ACE_ERROR[_RETURN] ((severity, format [,args…])[, return-value]);

• See ACE Programmer’s Guide (APG) tables 3.1 (severities), 3.2 (directives), 3.3 (macros)

Page 29: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

29

ACE Logging Usage

•The ACE logging API is similar to printf(), e.g.:

ACE_ERROR ((LM_ERROR, "(%t) fork failed"));

generates:

Oct 31 14:50:13 [email protected]@2766@LM_ERROR@client::(4) fork failed and

ACE_DEBUG ((LM_DEBUG, "(%t) sending to server %s", host));

generates:

Oct 31 14:50:28 [email protected]@1832@LM_DEBUG@drwho::(6) sending to server tangoFormat Action

%l Displays the line number where the error occurred

%N Displays the file name where the error occurred

%n Displays the name of the program

%P Displays the current process ID

%p Takes a const char * argument & displays it & the error string corresponding to errno (similar to perror())

%T Displays the current time

%t Displays the calling thread’s ID

Page 30: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

30

Logging Severities• You can control which severities are

seen at run time using two masks :

• Process-wide mask (defaults to all severities enabled)

• Per-thread mask (defaults to all severities disabled)

• Message is displayed is logged severity is enabled in either mask

• Set process/instance mask with:•ACE_Log_Msg::priority_mask (u_long mask, MASK_TYPE which);

•MASK_TYPE is ACE_Log_Msg::PROCESS or ACE_Log_Msg::THREAD

• Per-thread mask initializer can be adjusted (default is all severities disabled):•ACE_Log_Msg::disable_ debug_messages();

•ACE_Log_Msg::enable_ debug_messages();

• These static methods set & clear a (set of) bits instead of replacing the mask, as priority_mask() does

• Any set of severities can be specified (OR’d together)

• Since default is to enable all severities process-wide, all severities are logged in all threads unless you change it

• See ACE_Logging_Strategy for interesting ways of changing it…

Page 31: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

31

Logging Severities Example

• To allow threads to decide their own logging, the desired severities must be:

• Disabled at process level & enabled in the thread(s) to display them.

• e.g.,

ACE_LOG_MSG->priority_mask (0, ACE_Log_Msg::PROCESS);

ACE_Log_Msg::enable_debug_messages ();

ACE_Thread_Manager::instance ()->spawn (service);

ACE_Log_Msg::disable_debug_messages ();

ACE_Thread_Manager::instance ()->spawn_n (3, worker);

• LM_DEBUG severity (only) logged in service thread

• LM_DEBUG severity (and all others) not logged in worker threads

• Note how severities are “inherited” when threads spawned

Page 32: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

32

Redirect Logging to a File

• Default logging sink is stderr

• Redirect to a file by setting the OSTREAM flag & assigning a stream

• Flag can be set in two ways:

•ACE_Log_Msg::open (const ACE_TCHAR *prog_name, u_long options_flags = ACE_Log_Msg::STDERR, const ACE_TCHAR *logger_key = 0);

•ACE_Log_Msg::set_flags (u_long flags);• Assign a stream:

•ACE_Log_Msg::msg_ostream (ACE_OSTREAM_TYPE *);(Optional 2nd arg to tell ACE_Log_Msg to delete the ostream)

•ACE_OSTREAM_TYPE is ofstream where supported, else FILE*

• To also stop output to stderr, use open() without STDERR flag, or ACE_Log_Msg::clr_flags (STDERR)

Page 33: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

33

Redirect Logging to Syslog

• Redirected log output to ACE_Log_Msg::SYSLOG goes to:

• Windows NT4 & up: system’s Event Log

• UNIX/Linux: syslog facility (uses LOG_USER syslog facility)

• Can’t set this with set_flags/clr_flags; must open. For example:

•ACE_LOG_MSG->open(argv[0], ACE_Log_Msg::SYSLOG, ACE_TEXT (“syslogTest”));

• Windows: 3rd arg, if supplied, replaces 1st as program name in event log

• To turn it off, call open() again with different flag(s)

• This seems odd, but you’re effectively resetting the logging… think of it as reopen().

Page 34: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

34

Logging Callbacks

• Logging callbacks are useful for adding special processing or filtering to log output

• Derive a class from ACE_Log_Msg_Callback & reimplement:

•virtual void log (ACE_Log_Record &log_record);

• Use ACE_Log_Msg::msg_callback() to register callback

• Also call ACE_Log_Msg::set_flags() to add ACE_Log_Msg::MSG_CALLBACK flag

• Beware…

• Callback registration is specific to each ACE_Log_Msg instance

• Callbacks are not inherited when new threads are created

• See ACE_ROOT/examples/Log_Msg for an example

Page 35: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

35

Useful Logging Flags

• There are some other ACE_Log_Msg flags that add useful functionality to ACE’s logging:

•VERBOSE: Prepends program name, timestamp, host name, process ID, & message priority to each message

•VERBOSE_LITE: Prepends timestamp & message priority to each message (this is what ACE test suite uses)

•SILENT: Don’t display any messages of any severity

•LOGGER: Write messages to the local client logger daemon

• Example of using VERBOSE_LITE:

ACE_DEBUG ((LM_DEBUG, ACE_TEXT ("This is ACE Version %u.%u.%u\n\n"),

ACE_MAJOR_VERSION, ACE_MINOR_VERSION, ACE_BETA_VERSION));

Outputs:

Mar 29 09:50:34.891 2005@LM_DEBUG@This is ACE Version 5.4.4

Page 36: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

36

Tracing & Logging• ACE’s tracing facility logs function/method entry & exit

• Uses logging with severity LM_TRACE, so output can be selectively disabled

• Just put ACE_TRACE macro in the method:

#include “ace/Log_Msg.h”

void foo (void) {

ACE_TRACE (“foo”);

// … do stuff

}

Says:

(1024) Calling foo in file ‘test.cpp’ on line 3

(1024) Leaving foo

• Clever indenting by call depth makes output easier to read

• Huge amount of output, so tracing no-op’d out by default

• Enable by rebuilding with config.h having: #define ACE_NTRACE 0

• See ACE_ROOT/examples/Misc for an example

Page 37: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

37

Networked Logging Service ExampleKey Participants

• Client application processes

•Generate log records

• Server logging daemon

•Receive, process, & store log records

• We’ll develop architecture similar to ACE’s, but not same implementation.

C++ code for all logging service examples are in

• ACE_ROOT/examples/ C++NPv1/

• ACE_ROOT/examples/ C++NPv2/

Page 38: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

38

Network Daemon Design Dimensions

•Communication dimensions address the rules, form, & level of abstraction that networked applications use to interact

•Concurrency dimensions address the policies & mechanisms governing the proper use of processes & threads to represent multiple service instances, as well as how each service instance may use multiple threads internally

•Service dimensions address key properties of a networked application service, such as the duration & structure of each service instance

•Configuration dimensions address how networked services are identified & the time at which they are bound together to form complete applications

Page 39: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

39

Communication Design Dimensions

•Communication is fundamental to networked application design

•The next three slides present a domain analysis of communication design dimensions, which address the rules, form, & levels of abstraction that networked applications use to interact with each other

•We cover the following communication design dimensions:

•Connectionless versus connection-oriented protocols

•Synchronous versus asynchronous message exchange

•Message-passing versus shared memory

Page 40: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

40

Connectionless vs. Connection-oriented Protocols

•Connection-oriented applications must address two additional design issues:•Data framing strategies, e.g., bytestream vs. message-oriented•Connection multiplexing (muxing) strategies, e.g., multiplexed vs. nonmultiplexed

•A protocol is a set of rules that specify how control & data information is exchanged between communicating entities

3-way handshake in TCP/IP

Connector Acceptor

SYN

SYN/ACK

ACK

• Connection-oriented protocols provide a reliable, sequences, non-duplicated delivery service, which is useful for applications that can’t tolerate data loss

• Examples include TCP & ATM

• Connectionless protocols provide a message-oriented service in which each message can be routed & delivered independently

• Examples include UDP & IP

Page 41: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

41

Alternative Connection Muxing Strategies

•In multiplexed connections all client requests emanating from threads in a single process pass through one TCP connection to a server process

•Pros: Conserves OS communication resources, such as socket handles & connection control blocks

•Cons: harder to program, less efficient, & less deterministic

•In nonmultiplexed connections each client uses a different connection to communicate with a peer service

•Pros: Finer control of communication priorities & low synchronization overhead since additional locks aren't needed

•Cons: use more OS resources, & therefore may not scale well in certain environments

Page 42: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

42

Sync vs. Async Message Exchange

•Synchronous request/response protocols are the simplest form to implement

•Requests & responses are exchanged in a lock-step sequence.

•Each request must receive a response synchronously before the next is sent

•Asynchronous request/response protocols stream requests from client to server without waiting for responses synchronously

•Multiple client requests can be transmitted before any responses arrive from a server

•These protocols therefore often require a strategy for detecting lost or failed requests & resending them later

Page 43: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

43

Message Passing vs. Shared Memory

•Message passing exchanges data explicitly via the IPC mechanisms

•Application developers generally define the protocol for exchanging the data, e.g.:

•Format & content of the data

•Number of possible participants in each exchange (e.g., point-to-point unicast), multicast, or broadcast)

•How participants begin, conduct, & end a message-passing session

•Shared memory allows multiple processes on the same or different hosts to access & exchange data as though it were local to the address space of each process

•Applications using native OS shared memory mechanisms must define how to locate & map the shared memory region(s) & the data structures that are placed in shared memory

Page 44: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

44

Sidebar: C++ Objects & Shared Memory

•General responsibilities using placement new operator

•Pointer passed to placement new operator must point to a memory region that is big enough & is aligned properly for the object type being created

•The placed object must be destroyed by explicitly calling the destructor

•Pitfalls initializing C++ objects with virtual functions in shared memory

•The shared memory region may reside at a different virtual memory location in each process that maps the shared memory

•The C++ compiler/linker need not locate the vtable at the same address in different processes that use the shared memory

•ACE wrapper façade classes that can be initialized in shared memory must therefore be concrete data types

• i.e., classes with only non-virtual methods

Allocating a C++ Object in shared Memory

void *obj_buf = … // Get a pointer to location in shared memoryABC *abc = new (obj_buf) ABC; // Use C++ placement new operator

Page 45: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

45

Overview of the Socket API (1/2)

Sockets are the most common network programming API available on operating system platforms

•Originally developed in BSD Unix as a C language API to TCP/IP protocol suite

•The Socket API has approximately two dozen functions classified in five categories

•Socket is a handle created by the OS that associates it with an end point of a communication channel

•A socket can be bound to a local or remote address

•In Unix, socket handles & I/O handles can be used interchangeably in most cases, but this is not the case for Windows

Page 46: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

46

Network addressing

Options management

Data transfer mechanisms

Overview of the Socket API (2/2)

Connection establishment & termination

Local context management

Page 47: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

47

Taxonomy of Socket DimensionsThe Socket API can be decomposed into the following dimensions:

•Type of communication service• e.g., streams versus datagrams versus connected datagrams

•Communication & connection role• e.g., clients often initiate connections actively, whereas servers often accept them passively

•Communication domain • e.g., local host only versus local or remote host

Page 48: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

48

Limitations with the Socket APIs (1/2)

Poorly structured, non-uniform, & non-portable

• API is linear rather than hierarchical

• i.e., the API is not structured according to the different phases of connection lifecycle management & the roles played by the participants

• No consistency among the names

• Non-portable & error-prone

• Function names: read() & write() used for any I/O handle on Unix but Windows needs ReadFile() & WriteFile()

• Function semantics: different behavior of same function on different OS e.g., accept () can take NULL client address parameter on Unix/Windows, but will crash on some operating systems, such as VxWorks

• Socket handle representations: different platforms represent sockets differently e.g., Unix uses unsigned integers whereas Windows uses pointers

• Header files: Different platforms use different names for header files for the socket API

Page 49: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

49

Limitations with the Socket APIs (2/2)Lack of type safety• I/O handles are not amenable to strong type checking at compile time

• e.g., no type distinction between a socket used for passive listening & a socket used for data transfer

Steep learning curve due to complex semantics• Multiple protocol families & address families

• Options for infrequently used features such as broadcasting, async I/O, non blocking I/O, urgent data delivery

• Communication optimizations such as scatter-read & gather-write

• Different communication & connection roles, such as active & passive connection establishment, & data transfer

Too many low-level details• Forgetting to use the network byte order before data transfer

• Possibility of missing a function, such as listen()

• Possibility of mismatch between protocol & address families

• Forgetting to initialize underlying C structures e.g., sockaddr

• Using a wrong socket for a given role

Page 50: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

50

1 #include <sys/types.h>

2 #include <sys/socket.h>

3

4 const int PORT_NUM = 10000;

5

6 int echo_server ()

7 {

8 struct sockaddr_in addr;

9 int addr_len;

10 char buf[BUFSIZ];

11 int n_handle;

12 // Create the local endpoint.

Example of Socket API Limitations (1/3)

Forgot to initialize to sizeof (sockaddr_in)

Use of non-portable handle type

Possible differences in header file names

Page 51: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

51

13 int s_handle = socket (PF_UNIX, SOCK_DGRAM, 0);

14 if (s_handle == -1) return -1;

15

16 // Set up address information where server listens.

17 addr.sin_family = AF_INET;

18 addr.sin_port = PORT_NUM;

19 addr.sin_addr.addr = INADDR_ANY;

20

21 if (bind (s_handle, (struct sockaddr *) &addr,

22 sizeof addr) == -1)

23 return -1;

24

Use of non-portable return value

Unused structure members not zeroed out

Protocol & address family mismatch

Wrong byte order

Missed call to listen()

Example of Socket API Limitations (2/3)

Page 52: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

52

25 // Create a new communication endpoint.

26 if (n_handle = accept (s_handle, (struct sockaddr *) &addr,

27 &addr_len) != -1) {

28 int n;

29 while ((n = read (s_handle, buf, sizeof buf)) > 0)

30 write (n_handle, buf, n);

31

32 close (n_handle);

33 }

34 return 0;

35 }

SOCK_DGRAM handle illegal here

Reading from wrong handle

No guarantee that “n” bytes will be written

Example of Socket API Limitations (3/3)

Page 53: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

53

ACE Socket Wrapper Façade Classes

These classes are designed in accordance

with the Wrapper Facade design pattern

ACE defines a set of C++ classes that address the limitations with the Socket API•Enhance type-safety•Ensure portability•Simplify common use cases

•Building blocks for higher-level abstractions

Page 54: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

54

The Wrapper Façade Pattern (1/2)

Problem•The diversity of hardware & operating systems makes it hard to build portable & robust networked application software

•Programming directly to low-level OS APIs is tedious, error-prone, & non-portable

Context •Networked applications must manage a variety of OS services, including processes, threads, socket connections, virtual memory, & files

•OS platforms provide low-level APIs written in C to access these services

Win2K Linux LynxOS

Solaris VxWorks

Applications

Page 55: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

55

The Wrapper Façade Pattern (2/2)

This pattern encapsulates data & functions provided by existing non-OO APIs within more concise, robust, portable, maintainable, & cohesive OO class interfaces

: Application

method()

: WrapperFacade

: APIFunctionA

functionA()

: APIFunctionB

functionB()

Applicationcalls methods

callsAPI FunctionA()

callsAPI FunctionB()

calls API FunctionC()

void methodN(){functionA();

}

void method1(){functionA();

}functionB();

Wrapper Facade

data

method1()…methodN()

Solution•Apply the Wrapper Facade design pattern (P2) to avoid accessing low-level operating system APIs directly

Solution•Apply the Wrapper Facade design pattern (P2) to avoid accessing low-level operating system APIs directly

Page 56: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

56

ACE Socket Wrapper Façades Taxonomy•The structure of the ACE Socket wrapper facades reflects the domain of networked IPC properties

•The ACE Socket wrapper façade classes provide the following capabilities:

•ACE_SOCK_* classes encapsulate Internet-domain Socket API functionality

•ACE_LSOCK_* classes encapsulate UNIX-domain Socket API functionality

•ACE also has wrapper facades for datagrams•e.g., unicast, multicast, broadcast

Page 57: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

57

Roles in the ACE Socket Wrapper Facade

•The active connection role (ACE_SOCK_Connector) is played by a peer application that initiates a connection to a remote peer

•The passive connection role (ACE_SOCK_Acceptor) is played by a peer application that accepts a connection from a remote peer &

•The communication role (ACE_SOCK_Stream) is played by both peer applications to exchange data after they are connected

Page 58: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

58

ACE Socket Addressing Classes (1/2)

Motivation

•Network addressing is a trouble spot in the Socket API

•To minimize the complexity of these low-level details, ACE defines a hierarchy of classes that provide a uniform interface for all ACE network addressing objects

Page 59: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

59

ACE Socket Addressing Classes (2/2)

Class Capabilities

•The ACE_Addr class is the root of the ACE network addressing hierarchy

•The ACE_INET_Addr class represents TCP/IP & UDP/IP addressing information

•This class eliminates many subtle sources of accidental complexity

Page 60: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

60

ACE I/O Handle Classes (1/2)Motivation

•Low-level C I/O handle types are tedious, error-prone, & non-portable

•Even the ACE_HANDLE typedef is still not sufficiently object-oriented & typesafe

int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0);

s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; write (handle, buf, n); }}

Reading from wrong handle

int is not portable to Windows

Page 61: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

61

ACE I/O Handle Classes (2/2)Class Capabilities

•ACE_IPC_SAP is the root of the ACE hierarchy of IPC wrapper facades

•It provides basic I/O handle manipulation capabilities to other ACE IPC wrapper facades

•ACE_SOCK is the root of the ACE Socket wrapper facades & it provides methods to

•Create & destroy socket handles

•Obtain the network addresses of local & remote peers

•Set/get socket options, such as socket queue sizes,

•Enable broadcast/multicast communication

•Disable Nagle‘s algorithm

Page 62: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

62

The ACE_SOCK_Connector ClassMotivation

•There is a confusing asymmetry in the Socket API between (1) connection roles & (2) socket modes

•e.g., an application may accidentally call recv() or send() on a data-mode socket handle before it's connected

•This problem can't be detected until run time since C socket handles are weakly-typed

int buggy_echo_client (u_short port_num, const char *s) { int handle = socket (PF_UNIX, SOCK_DGRAM, 0); write (handle, s, strlen (s) + 1);

sockaddr_in s_addr; memset (&s_addr, 0, sizeof s_addr); s_addr.sin_family = AF_INET; s_addr.sin_port = htons (port_num); connect (handle, (sockaddr *) &s_addr, sizeof s_addr);}

Operations called in wrong order

Page 63: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

63

The ACE_SOCK_Connector ClassClass Capabilities

•ACE_SOCK_Connector is factory that establishes a new endpoint of communication actively & provides capabilities to

•Initiate a connection with a peer acceptor & then to initialize an ACE_SOCK_Stream object after the connection is established

•Initiate connections in either a blocking, nonblocking, or timed manner

•Use C++ traits to support generic programming techniques that enable wholesale replacement of IPC functionality

Page 64: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

64

Sidebar: Traits for ACE Wrapper Facades (1/2)

•ACE uses the C++ generic programming idiom to define & combine a set of characteristics to alter the behavior of a template class

•In C++, the typedef & typename language feature is used to define a trait

•A trait provides a convenient way to associate related types, values, & functions with template parameter type without requiring that they be defined as members of the type

•Traits are used extensively in the C++ Standard Template Library (STL)

Page 65: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

65

Sidebar: Traits for ACE Wrapper Facades (2/2)

•ACE Socket wrapper facades use traits to define the following associations

•PEER_ADDR – this trait defines the ACE_INET_Addr class associated with the ACE Socket Wrapper Façade

•PEER_STREAM – this trait defines the ACE_SOCK_Stream data transfer class associated with the ACE_SOCK_Acceptor & ACE_SOCK_Connector factories

class ACE_SOCK_Connector {public: typedef ACE_INET_Addr PEER_ADDR; typedef ACE_SOCK_Stream PEER_STREAM; // ...

class ACE_TLI_Connector {public: typedef ACE_INET_Addr PEER_ADDR; typedef ACE_TLI_Stream PEER_STREAM; // ...

Page 66: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

66

Using the ACE_SOCK_Connector (1/3)

int main (int argc, char *argv[]) { const char *pathname = argc > 1 ? argv[1] : “/index.html"; const char *server_hostname = argc > 2 ? argv[2] : “www.dre.vanderbilt.edu"; typedef ACE_SOCK_Connector CONNECTOR; CONNECTOR connector; CONNECTOR::PEER_STREAM peer; CONNECTOR::PEER_ADDR peer_addr; if (peer_addr.set (80, server_hostname) == -1) return 1; else if (connector.connect (peer, peer_addr) == -1) return 1;

• Block until connection established or connection request failure

• Instantiate the connector, data transfer, & address objects

•This example shows how the ACE_SOCK_Connector can be used to connect a client application to a Web server

Page 67: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

67

Using the ACE_SOCK_Connector (2/3) // Designate a nonblocking connect. if (connector.connect (peer, peer_addr, &ACE_Time_Value::zero) == -1) { if (errno == EWOULDBLOCK) { // Do some other work ...

// Now, try to complete connection establishment, // but don't block if it isn't complete yet. if (connector.complete (peer, 0, &ACE_Time_Value::zero) == -1)

// Designate a timed connect. ACE_Time_Value timeout (10); // 10 second timeout. if (connector.connect (peer, peer_addr, &timeout) == -1) { if (errno == ETIME) { // Timeout, do something else

• Perform a non-blocking connect

• If connection not established, do other work & try again without blocking

• Perform a timed connect e.g., 10 seconds in this case

Page 68: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

68

Using the ACE_SOCK_Connector (3/3)

•The ACE_SOCK_Connector can be passed the following values to control its timeout behavior

Page 69: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

69

The ACE_SOCK_Stream Class (1/2)

Motivation •Developers can misuse sockets in ways that can't be detected during compilation

•An ACE_SOCK_Stream object can't be used in any role other than data transfer without violating its (statically type-checked) interface

int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0); s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; write (handle, buf, n); }}

Reading from wrong handle

Page 70: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

70

The ACE_SOCK_Stream Class (2/2)Class Capabilities

•Encapsulates data transfer mechanisms supported by data-mode sockets to provide the following capabilities:

•Support for sending & receiving up to n bytes or exactly n bytes

•Support for “scatter-read,” which populate multiple caller-supplied buffers instead of a single contiguous buffer

•Support for ``gather-write'' operations, which transmit the contents of multiple noncontiguous data buffers in a single operation

•Support for blocking, nonblocking, & timed I/O operations

•Support for generic programming techniques that enable the wholesale replacement of functionality via C++ parameterized types

Page 71: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

71

Using the ACE_SOCK_Stream (1/2)

// ...Connection code from example in Section 3.5 omitted... char buf[BUFSIZ]; iovec iov[3]; iov[0].iov_base = (char *) "GET "; iov[0].iov_len = 4; // Length of "GET ". iov[1].iov_base = (char *) pathname; iov[1].iov_len = strlen (pathname); iov[2].iov_base = (char *) " HTTP/1.0\r\n\r\n"; iov[2].iov_len = 13; // Length of " HTTP/1.0\r\n\r\n";

if (peer.sendv_n (iov, 3) == -1) return 1;

for (ssize_t n; (n = peer.recv (buf, sizeof buf)) > 0; ) ACE::write_n (ACE_STDOUT, buf, n);

return peer.close () == -1 ? 1 : 0;}

• Initialize the iovec vector for scatter-read & gather-write I/O

• Perform blocking gather-write on ACE_SOCK_Stream

• Perform blocking read on ACE_SOCK_Stream

•This example shows how an ACE_SOCK_Stream can be used to send & receive data to & from a Web server

Page 72: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

72

Using the ACE_SOCK_Stream (2/2)

• Blocking & non-blocking I/O semantics can be controlled via the ACE_SOCK_STREAM enable() & disable() methods, e.g.,

•peer.enable (ACE_NONBLOCK); // enables non blocking

peer.disable (ACE_NONBLOCK); // disable non blocking

•If the I/O operation blocks, it returns a -1 & errno is set to EWOULDBLOCK

• I/O operations can involve timeouts, e.g.,

ACE_Time_Value timeout (10); // 10 second timeout

If (peer.sendv_n (iov, 3, &timeout) == -1) {

// check if errno is set to ETIME,

// which indicates a timeout

}

// similarly use timeout for receiving data

Page 73: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

73

Sidebar: Working with (& Around) Nagle’s Algorithm

Nagle’s Algorithm

•Problem: Need to tackle the send-side silly window syndrome, where small data payloads, such as a keystroke, result in transmissions of large packets & causing unnecessary waste of network resources & congestion

•Solution: The OS kernel buffers a # of small-sized application messages & concatenates them into a larger size packet that can then be transmitted

•Consequences: Although network congestion is minimized, it can lead to higher & unpredictable latencies, as well as lower throughput

Controlling Nagle’s Algorithm via ACE

•Use the set_option() method of the ACE_SOCK class e.g., int nodelay = 1; // Disable Nagle’s algorithm ACE_SOCK_Stream option_setter (handle); if (-1 == option_setter.set_option (ACE_IPPROTO_TCP, TCP_NODELAY, &nodelay, sizeof (nodelay))) ...

Page 74: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

74

The ACE_SOCK_Acceptor Class (1/2)

Motivation

•The C functions in the Socket API are weakly typed, which makes it easy to apply them incorrectly in ways that can’t be detected until run-time

•The ACE_SOCK_Acceptor class ensures type errors are detected at compile-time

int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0); s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; write (handle, buf, n); }}

Reading from wrong handle

Page 75: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

75

Class Capabilities

•This class is a factory that establishes a new endpoint of communication passively & provides the following capabilities:

The ACE_SOCK_Acceptor Class (2/2)

•It accepts a connection from a peer connector & then initializes an ACE_SOCK_Stream object after the connection is established

•Connections can be accepted in either a blocking, nonblocking, or timed manner

•C++ traits are used to support generic programming techniques that enable the wholesale replacement of functionality via C++ parameterized types

Page 76: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

76

Using the ACE_SOCK_Acceptor

extern char *get_url_pathname (ACE_SOCK_Stream *);int main (){ typedef ACE_SOCK_Acceptor ACCEPTOR; ACCEPTOR::PEER_ADDR server_addr; ACCEPTOR acceptor; ACCEPTOR::PEER_STREAM peer;

if (server_addr.set (80) == -1) return 1; if (acceptor.open (server_addr) == -1) return 1;

for (;;) { if (acceptor.accept (peer) == -1) return 1; peer.disable (ACE_NONBLOCK); // Ensure blocking <send_n>.

ACE_Auto_Array_Ptr<char> pathname (get_url_pathname (peer)); ACE_Mem_Map mapped_file (pathname.get ());

if (peer.send_n (mapped_file.addr (), mapped_file.size ()) == -1) return 1; peer.close (); }

return acceptor.close () == -1 ? 1 : 0;}

• Instantiate the acceptor, data transfer, & address objects

• Initialize a passive mode endpoint to listen for connections on port 80

• Accept a new connection

• Send the requested data

• Close the connection to the sender

• Stop receiving any connections

•This example shows how an ACE_SOCK_Acceptor & ACE_SOCK_Stream can be used to accept connections & send/receive data to/from a web client

Page 77: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

77

Sidebar: The ACE_Mem_Map Class

Memory Mapped Files

•Many modern operating systems provide a mechanism for mapping a file’s contents directly into a process’s virtual address space

•This memory-mapped file mechanism can be read from or written to directly by referencing the virtual memory

•e.g., via pointers instead of using less efficient I/O functions

•The file manager defers all read/write operations to the virtual memory manager

•Contents of memory mapped files can be shared by multiple processes on the same machine

•It can also be used to provide a persistent backing store

ACE_Mem_Map Class

•A wrapper façade that encapsulates the memory mapped file system mechanisms on different operating systems

•Relieves application developers from having to manually perform bookkeeping tasks

•e.g., explicitly opening files or determining their lengths

•The ACE_Mem_Map class offers multiple constructors with several signature variants

Page 78: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

78

The ACE_Message_Block Class (1/2)

•Storing messages in buffers as they are received from the network or from other processes

•Adding/removing headers/trailers from messages as they pass through a user-level protocol stack

•Fragmenting/reassembling messages to fit into network MTUs•Storing messages in buffers for transmission or retransmission•Reordering messages that were received out-of-sequence

Motivation•Many networked applications require a means to manipulate messages efficiently, e.g.:

MESSAGES IN TRANSIT MESSAGES IN TRANSIT

MESSAGES MESSAGES BUFFERED FOR BUFFERED FOR TRANSMISSION TRANSMISSION

MESSAGES MESSAGES BUFFERED BUFFERED AWAITING AWAITING

PROCESSING PROCESSING

Page 79: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

79

The ACE_Message_Block Class (2/2)Class Capabilities•This class is a composite that enables efficient manipulation of messages via the following operations:•Each ACE_Message_Block contains a pointer to a reference-counted ACE_Data_Block which in turn points to the actual data associated with a message

•It allows multiple messages to be chained together into a composite message

•It allows multiple messages to be joined together to form an ACE_Message_Queue

•It treats synchronization & memory management properties as aspects

Page 80: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

80

Allocators & ACE_Message_Block

• You can set new allocation strategies for:

• Allocating ACE_Message_Block objects

• Allocating ACE_Data_Block objects referred to by an ACE_Message_Block

• Allocating the memory chunk controlled by ACE_Data_Block

• They all default to ACE_Allocator::instance(), which defaults to ACE_New_Allocator, i.e., C++ operator new

• Any class conforming to the ACE_Allocator interface may be used, including:

•ACE_New_Allocator: uses C++ operator new

•ACE_Static_Allocator: uses a fixed pool of fixed-size blocks

•ACE_Cached_Allocator: uses operator new, but caches blocks

•ACE_Allocator_Adapter: adapts other pools, such as shared memory, to ACE_Allocator interface

Page 81: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

81

Two Kinds of Message Blocks

•Composite messages contain multiple ACE_Message_Blocks •These blocks are linked together in accordance with the Composite pattern

•Composite messages often consist of a control message that contains bookkeeping information •e.g., destination addresses, followed by one or more data messages that contain the actual contents of the message

•ACE_Data_Blocks can be referenced counted

•Simple messages contain a one ACE_Message_Block

•An ACE_Message_Block points to an ACE_Data_Block

•An ACE_Data_Block points to the actual data payload

Page 82: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

82

ACE_Message_Block Areas• There are a number of size-related methods to obtain these values:

•length(): Amount of data between rd_ptr() & wr_ptr()

•size(): Total amount of useable allocated area

•capacity(): Total amount of allocated area (usually same as size(), only different if size is decreased after creating the block)

•space(): Free space after the wr_ptr(); how many bytes can be added

length() space()

size()

base() rd_ptr() wr_ptr() end()

capacity()

Page 83: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

83

Using the ACE_Message_Block (1/2)

int main (int argc, char *argv[]){ ACE_Message_Block *head = new ACE_Message_Block (BUFSIZ); ACE_Message_Block *mblk = head;

for (size_t recvd = 0; ; recvd = 0) { ssize_t nbytes = ACE::read_n (ACE_STDIN, mblk->wr_ptr (), mblk->size (), &recvd); if (nbytes <= 0) break; // Break out at EOF or error.

mblk->wr_ptr (recvd);

• Allocate an ACE_Message_Block whose payload is of size BUFSIZ

• Read data from standard input into the message block starting at write pointer (wr_ptr ())

• Advance write pointer by the number of bytes read to end of buffer

•The following program reads all data from standard input into a singly linked list of dynamically allocated ACE_Message_Blocks

•These ACE_Message_Blocks are chained together by their continuation pointers

Page 84: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

84

Using the ACE_Message_Block (2/2)

mblk->cont (new ACE_Message_Block (BUFSIZ));

mblk = mblk->cont ();

}

// Print the contents of the list to the standard output.

for (mblk = head; mblk != 0; mblk = mblk->cont ())

ACE::write_n (ACE_STDOUT, mblk->rd_ptr (), mblk->length ());

head->release (); // Release all the memory in the chain.

return 0;

}

• Allocate a new ACE_Message_Block of size BUFSIZ & chain it to the previous one at the end of the list

• For every message block, print mblk->length() amount of contents starting at the read pointer (rd_ptr ())

• Can also use ACE::write_n (head) to write entire chain…

• Advance mblk to point to the newly allocated ACE_Message_Block

Page 85: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

85

ACE CDR StreamsMotivation

•Networked applications that send/receive messages often require support for

•Linearization

•To handle the conversion of richly typed data to/from raw memory buffers

•(De)marshaling

•To interoperate with heterogeneous compiler alignments & hardware instructions with different byte-orders

class ACE_Log_Record{ ...private: ACE_UINT type_; ACE_UINT pid_; ACE_Time_Value timestamp_; char msg_data_[ACE_MAXLOGMSGLEN];};

Page 86: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

86

ACE CDR Streams (cont’d)

•ACE_OutputCDR creates a CDR buffer from a data structure (marshaling)

•ACE_InputCDR extracts data from a CDR buffer (demarshaling)

• The ACE_OutputCDR & ACE_InputCDR classes provide a highly optimized, portable, & convenient means to marshal & demarshal data

• These classes use the standard OMG CORBA Common Data Representation (CDR)

• Unlike blindly always sending in network byte order, CDR uses a “receiver makes right” approach

Page 87: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

87

The ACE_OutputCDR & ACE_InputCDR Classes Class Capabilities•ACE_OutputCDR & ACE_InputCDR support the following features:

•Operations to (de)marshal types:

•Primitive types, e.g., booleans; 16-, 32-, & 64-bit integers; 8-bit octets; single & double precision floating point numbers; characters; & strings

•Arrays of primitive types

•The insertion (<<) & extraction (>>) operators can marshal & demarshal primitive types, using the same syntax as the C++ iostream components•ACE_Message_Block chains are used internally to minimize mem copies

•Use CORBA CDR alignment & byte-ordering rules to avoid memory copying & byte-swapping operations, respectively

•Provide optimized byte swapping code that uses inline assembly language instructions for common hardware platforms (such as Intel x86) & standard hton*()& ntoh*() macros/functions on other platforms

•Support zero copy marshaling & demarshaling of octet buffers

•Users can define custom character set translators for platforms that do not use ASCII or Unicode as their native character sets

Page 88: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

88

Sidebar: Log Record Message Structure

class ACE_Log_Record{private: ACE_UINT type_; ACE_UINT pid_; ACE_Time_Value timestamp_; char *msg_data_; // Defaults to ACE_MAXLOGMSGLEN

public: ACE_UINT type () const; ACE_UINT pid () const; const ACE_Time_Value timestamp () const; const char *msg_data () const;};

ACE_Log_Record is a type that ACE uses internally to keep track of the fields in a log record

•This example uses a 8-byte, CDR encoded header followed by the payload

•Header includes byte order, payload length, & other fields

Page 89: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

89

Using ACE_OutputCDR

int operator<< (ACE_OutputCDR &cdr, const ACE_Log_Record &log_record){ size_t msglen = log_record.msg_data_len ();

// Insert each <log_record> field into the output CDR stream. cdr << ACE_CDR::Long (log_record.type ()); cdr << ACE_CDR::Long (log_record.pid ()); cdr << ACE_CDR::Long (log_record.time_stamp ().sec ()); cdr << ACE_CDR::Long (log_record.time_stamp ().usec ()); cdr << ACE_CDR::ULong (msglen);

cdr.write_char_array (log_record.msg_data (), msglen);

return cdr.good_bit ();}

After marshaling all the fields of the log record into the CDR stream, return the success/failure status

•We show the ACE CDR insertion (operator<<) & extraction (operator>>) operators for ACE_Log_Record that's used by client application & logging server

Page 90: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

90

Using ACE_InputCDRint operator>> (ACE_InputCDR &cdr, ACE_Log_Record &log_record){ ACE_CDR::Long type; ACE_CDR::Long pid; ACE_CDR::Long sec, usec; ACE_CDR::ULong buffer_len;

// Extract each field from input CDR stream into <log_record>. if ((cdr >> type) && (cdr >> pid) && (cdr >> sec) && (cdr >> usec) && (cdr >> buffer_len)) { ACE_TCHAR *log_msg = new [buffer_len + 1]; log_record.type (type); log_record.pid (pid); log_record.time_stamp (ACE_Time_Value (sec, usec)); cdr.read_char_array (log_msg, buffer_len); log_msg[buffer_len] = '\0'; log_record.msg_data (log_msg); } return cdr.good_bit ();}

After demarshaling all the fields of the log record from the CDR stream, return the success/failure status

Temporaries used during demarshaling (not always necessary)

Page 91: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

91

Implementing the Client Application (1/6)

class Logging_Client {

public:

// Send <log_record> to the server.

int send (const ACE_Log_Record &log_record);

// Accessor method.

ACE_SOCK_Stream &peer () { return logging_peer_; }

// Close the connection to the server.

~Logging_Client () { logging_peer_.close (); }

private:

ACE_SOCK_Stream logging_peer_; // Connected to server.

};

Header file: “Logging_Client.h”

•The following client application illustrates how to use the ACE C++ Socket wrapper facades & CDR streams to establish connections, marshal log records, & send the data to our logging server

•This example behaves as follows:1.Reads lines from standard input2.Sends each line to the logging

server in a separate log record & 3.Stops when it reads EOF from

standard input

Page 92: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

92

Implementing the Client Application (2/6)

1 int Logging_Client::send (const ACE_Log_Record &log_record) { 2 const size_t max_payload_size = 3 4 // type() 4 + 8 // timestamp 5 + 4 // process id 6 + 4 // data length 7 + ACE_Log_Record::ACE_MAXLOGMSGLEN // data 8 + ACE_CDR::MAX_ALIGNMENT; // padding; 9 10 ACE_OutputCDR payload (max_payload_size);11 payload << log_record;12 ACE_CDR::ULong length = payload.total_length ();13

The Logging_Client::send() method behaves as follows:

1.Computes the size of the payload (lines 2 – 8)

2.Marshals the header & data into an output CDR (lines 10 – 16) &

3.Sends it to the logging server (lines 18 – 24)

First marshal the payload to contain the linearized ACE_Log_Record

Page 93: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

93

Implementing the Client Application (3/6)

14 ACE_OutputCDR header (ACE_CDR::MAX_ALIGNMENT + 8);

15 header << ACE_OutputCDR::from_boolean (ACE_CDR_BYTE_ORDER);

16 header << ACE_CDR::ULong (length);

17

18 iovec iov[2];

19 iov[0].iov_base = header.begin ()->rd_ptr ();

20 iov[0].iov_len = 8;

21 iov[1].iov_base = payload.begin ()->rd_ptr ();

22 iov[1].iov_len = length;

23

24 return logging_peer_.sendv_n (iov, 2);

25 }

4. Then marshal the header info that includes byte order & payload length

5. Construct an iovec of size 2 with header & payload info

6. Send entire message to logging server

Since TCP/IP is a bytestream protocol (i.e., without any message boundaries) the logging service uses CDR as a message framing protocol to delimit log records

Page 94: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

94

Implementing the Client Application (4/6)

1 int main (int argc, char *argv[]) 2 { 3 u_short logger_port = 4 argc > 1 ? atoi (argv[1]) : 0; 5 const char *logger_host = 6 argc > 2 ? argv[2] : ACE_DEFAULT_SERVER_HOST; 7 int result; 8 9 ACE_INET_Addr server_addr;10 11 if (logger_port != 0)12 result = server_addr.set (logger_port, logger_host);13 else14 result = server_addr.set ("ace_logger", logger_host);15 if (result == -1) 16 ACE_ERROR_RETURN((LM_ERROR,17 "lookup %s, %p\n",18 logger_port == 0 ? "ace_logger" : argv[1],19 logger_host), 1);20

The Logging_Client main() program

Page 95: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

95

Sidebar: ACE Debugging & Error Macros

•Consolidates printing of debug & error messages via a printf ()-like format e.g., ACE_DEBUG, ACE_ERROR (& their *_RETURN counterparts) that encapsulate the ACE_Log_Msg::log() method

•Arguments are enclosed in a double set of parentheses to make it appear as one argument to the C++ preprocessor

•First argument is the severity code; second one is a format string supporting a superset of printf() conversion specifiers

Format Action

%l Displays the line number where the error occurred

%N Displays the file name where the error occurred

%n Displays the name of the program

%P Displays the current process ID

%p Takes a const char * argument & displays it & the error string corresponding to errno (similar to perror())

%T Displays the current time

%t Displays the calling thread’s ID

Page 96: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

96

Implementing the Client Application (5/6)

21 ACE_SOCK_Connector connector;22 Logging_Client logging_client;23 24 if (connector.connect (logging_client.peer (),25 server_addr) < 0)26 ACE_ERROR_RETURN ((LM_ERROR,27 "%p\n",28 "connect()"),29 1);3031 // Limit the number of characters read on each record.32 cin.width (ACE_Log_Record::MAXLOGMSGLEN);

Use the ACE_SOCK_Connector wrapper façade to connect to the logging server

Contents of the message to be sent to logging server are obtained from standard input

Page 97: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

97

Implementing the Client Application (6/6)33 for (;;) {34 std::string user_input;35 getline (cin, user_input, '\n');36 37 if (!cin || cin.eof ()) break;38 39 ACE_Time_Value now (ACE_OS::gettimeofday ());40 ACE_Log_Record log_record (LM_INFO, now,41 ACE_OS::getpid ());42 log_record.msg_data (user_input.c_str ());43 44 if (logging_client.send (log_record) == -1)45 ACE_ERROR_RETURN ((LM_ERROR,46 "%p\n", "logging_client.send()"), 1);47 }48 49 return 0; // Logging_Client destructor 50 // closes TCP connection.51 }

Send log_record to logging server

Create log_record

Page 98: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

98

The figure below illustrates our Logging_Server abstract base class, the Logging_Handler class we'll describe shortly, & the concrete logging server

classes that we'll develop in subsequent sections of the tutorial

The Logging_Server Classes

Page 99: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

99

Implementing the Logging_Server (1/5)

// Forward declaration.class ACE_SOCK_Stream;

class Logging_Server{public: // Template Method that runs logging server's event loop. virtual int run (int argc, char *argv[]);

protected: // The following four methods are ``hooks'' that can be // overridden by subclasses. virtual int open (u_short logger_port = 0); virtual int wait_for_multiple_events () { return 0; } virtual int handle_connections () = 0; virtual int handle_data (ACE_SOCK_Stream * = 0) = 0;

Header file “Logging_Server.h”

•This example uses the ACE_Message_Block & ACE CDR classes in a common base class that simplifies logging server implementations in the examples

Page 100: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

100

Sidebar: Template Method Pattern• Intent

• Define the skeleton of an algorithm in an operation, deferring some steps to subclasses

• Context

• You have a fixed algorithm structure with variations possible for individual steps

• Problem

• You want to plug in & out steps of the algorithm without changing the algorithm itself

• Solution

• Define a fixed base class function that calls virtual “hook” methods that derived classes can override

Abstract Classtemplate_method ();hook_method_1();hook_method_2();

Concrete Class 1

hook_method_1();hook_method_2();

Concrete Class 2

hook_method_2();

...hook_method_1();...hook_method_2();...

Page 101: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

101

Implementing the Logging_Server (2/5)

// This helper method can be used by the hook methods. int make_log_file (ACE_FILE_IO &, ACE_SOCK_Stream * = 0);

// Close the socket endpoint & shutdown ACE. virtual ~Logging_Server () { acceptor_.close (); }

// Accessor. ACE_SOCK_Acceptor &acceptor () { return acceptor_; }

private: // Socket acceptor endpoint. ACE_SOCK_Acceptor acceptor_; };

Header file “Logging_Server.h” (cont’d)

Page 102: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

102

Implementing the Logging_Server (3/5)

#include "ace/FILE_Addr.h"#include "ace/FILE_Connector.h"#include "ace/FILE_IO.h"#include "ace/INET_Addr.h"#include "ace/SOCK_Stream.h"#include "Logging_Server.h"

int Logging_Server::run (int argc, char *argv[]){ if (open (argc > 1 ? atoi (argv[1]) : 0) == -1) return -1;

for (;;) { if (wait_for_multiple_events () == -1) return -1; if (handle_connections () == -1) return -1; if (handle_data () == -1) return -1; } return 0;}

Implementation file “Logging_Server.cpp”

• Template method providing the skeleton of the algorithm to use

• Hook methods will be overridden by subclasses unless default is ok to use

Three hook methods that can be overridden in subclasses

Page 103: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

103

Implementing the Logging_Server (4/5)

int Logging_Server::open (u_short logger_port){ // Raises the number of available socket handles to // the maximum supported by the OS platform. ACE::set_handle_limit (); ACE_INET_Addr server_addr; int result;

if (logger_port != 0) result = server_addr.set (logger_port, INADDR_ANY); else result = server_addr.set ("ace_logger", INADDR_ANY); if (result == -1) return -1;

// Start listening & enable reuse of listen address // for quick restarts. return acceptor_.open (server_addr, 1);}

Initialize the acceptor so it can accept connections from any server network interface

Page 104: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

104

Implementing the Logging_Server (5/5)int Logging_Server::make_log_file (ACE_FILE_IO &logging_file, ACE_SOCK_Stream *logging_peer){ std::string filename (MAXHOSTNAMELEN, ’\0’);

if (logging_peer != 0) { // Use client host name as file name. ACE_INET_Addr logging_peer_addr; logging_peer->get_remote_addr (logging_peer_addr); logging_peer_addr.get_host_name (filename.c_str (), filename.size ()); filename += ".log"; } else filename = "logging_server.log";

ACE_FILE_Connector connector; return connector.connect (logging_file, ACE_FILE_Addr (filename.c_str ()), 0, // No time-out. ACE_Addr::sap_any, // Ignored. 0, // Don't try to reuse the addr. O_RDWR|O_CREAT|O_APPEND, ACE_DEFAULT_FILE_PERMS);}

Create the log file using the ACE_FILE_Connector factory

Page 105: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

105

Sidebar: The ACE File Wrapper Facades

•ACE file wrapper facades encapsulate platform mechanisms for unbuffered file operations

•The design of these wrapper facades is very similar to ACE IPC wrapper facades

•The ACE File classes decouple:

•Initialization factories: e.g., ACE_FILE_Connector, which opens and/or creates files

•Data transfer classes: e.g., ACE_FILE_IO, which applications use to read/write data from/to files opened using ACE_FILE_Connector

•This generality in ACE’s design of wrapper facades helps strategize higher-level ACE framework components

•e.g., ACE_Acceptor, ACE_Connector, & ACE_Svc_Handler

Page 106: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

106

Implementing the Logging_Handler (1/6)

#include "ace/FILE_IO.h"#include "ace/SOCK_Stream.h"

class ACE_Message_Block; // Forward declaration.

class Logging_Handler{protected: // Reference to a log file. ACE_FILE_IO &log_file_;

// Connected to the client. ACE_SOCK_Stream logging_peer_;

Header file “Logging_Handler.h”

This class is used by the logging server to encapsulate the I/O & processing of log records

Page 107: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

107

Implementing the Logging_Server (2/6)

// Receive one log record from a connected client. <mblk> // contains the hostname, <mblk->cont()> contains the log // record header (the byte order & the length) & the data. int recv_log_record (ACE_Message_Block *&mblk);

// Write one record to the log file. The <mblk> contains the // hostname & the <mblk->cont> contains the log record. int write_log_record (ACE_Message_Block *mblk);

// Log one record by calling <recv_log_record> and // <write_log_record>. int log_record (); };

Header file “Logging_Handler.h” cont’d

When a log record is received it is stored as an ACE_Message_Block chain

Page 108: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

108

Implementing the Logging_Server (3/6)

1 int Logging_Handler::recv_log_record (ACE_Message_Block *&mblk) 2 { 3 ACE_INET_Addr peer_addr; 4 logging_peer_.get_remote_addr (peer_addr); 5 mblk = new ACE_Message_Block (MAXHOSTNAMELEN + 1); 6 peer_addr.get_host_name (mblk->wr_ptr (), MAXHOSTNAMELEN); 7 mblk->wr_ptr (strlen (mblk->wr_ptr ()) + 1); // Go past name. 8 9 ACE_Message_Block *payload =10 new ACE_Message_Block (ACE_DEFAULT_CDR_BUFSIZE);11 // Align Message Block for a CDR stream.12 ACE_CDR::mb_align (payload);1314 if (logging_peer_.recv_n (payload->wr_ptr (), 8) == 8) {15 payload->wr_ptr (8); // Reflect addition of 8 bytes.1617 ACE_InputCDR cdr (payload);18

1. Receive incoming data & use the input CDR class to parse header 2. Then payload based on the framing protocol & 3. Finally save it in an ACE_Message_Block chain

Force proper alignment

First save the peer hostname

Receive the header info (byte order & length)

Page 109: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

109

Implementing the Logging_Server (4/6)19 ACE_CDR::Boolean byte_order;20 // Use helper method to disambiguate booleans from chars.21 cdr >> ACE_InputCDR::to_boolean (byte_order);22 cdr.reset_byte_order (byte_order);23 24 ACE_CDR::ULong length;25 cdr >> length;2627 payload->size (8 + ACE_CDR::MAX_ALIGNMENT + length);2829 if (logging_peer_.recv_n (payload->wr_ptr(), length) > 0) {30 payload->wr_ptr (length); // Reflect additional bytes.31 mblk->cont (payload); // Chain the header & payload.32 return length; // Return length of the log record.33 }34 }35 payload->release ();36 mblk->release ();37 payload = mblk = 0;38 return -1;39 }

On error, free up allocated message blocks

Resize message block to be the right size for payload & that’s aligned properly

Demarshal header info

Page 110: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

110

Implementing the Logging_Server (5/6)

1 int Logging_Handler::write_log_record (ACE_Message_Block *mblk) 2 { 3 if (log_file_->send_n (mblk) == -1) return -1; 4 5 if (ACE::debug ()) { 6 ACE_InputCDR cdr (mblk->cont ()); 7 ACE_CDR::Boolean byte_order; 8 ACE_CDR::ULong length; 9 cdr >> ACE_InputCDR::to_boolean (byte_order);10 cdr.reset_byte_order (byte_order);11 cdr >> length;12 ACE_Log_Record log_record;13 cdr >> log_record; // Extract the <ACE_log_record>.14 log_record.print (mblk->rd_ptr (), 1, cerr);15 }16 17 return mblk->total_length ();18 }

1. Send the message block chain to the log file, which is stored in binary format2. If debug flag is set, print contents of the message

Page 111: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

111

Implementing the Logging_Server (6/6)

int Logging_Handler::log_record (){ ACE_Message_Block *mblk = 0; if (recv_log_record (mblk) == -1) return -1; else { int result = write_log_record (mblk); mblk->release (); // Free up the entire contents. return result == -1 ? -1 : 0; }}

1. Receives the message

2. Demarshals it into a ACE_Message_Block &

3. Writes it to the log file

Later on we’ll see the virtue of splitting the recv_log_record() & write_log_record() logic into two methods …

Page 112: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

112

Iterative Logging Server

•This is the simplest possible logging server implementation

•The iterative server implementation simply accepts & processes one client connection at a time

•Clearly, this approach does not scale up for non-trivial applications of the logging service!!!

•Subsequent implementations will enhance this version, yet still use the logging server framework

ACE_SOCK_Stream

ACE_SOCK_Acceptor

Only one client is accepted/processed at a time

Page 113: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

113

Implementing the Iterative_Logging_Server (1/3)#include "ace/FILE_IO.h"#include "ace/INET_Addr.h"#include "ace/Log_Msg.h"#include "Logging_Server.h"#include "Logging_Handler.h"

class Iterative_Logging_Server : public Logging_Server {public: Iterative_Logging_Server (): logging_handler_ (log_file_) {}

Logging_Handler &logging_handler () { return logging_handler_; }

protected: ACE_FILE_IO log_file_; Logging_Handler logging_handler_; // Other methods shown below...};

Header file: Iterative_Logging_Server.h

Page 114: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

114

Implementing the Iterative_Logging_Server (2/3) virtual int open (u_short logger_port) { if (make_log_file (log_file_) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "make_log_file()"), -1); return Logging_Server::open (logger_port); }

virtual int handle_connections () { ACE_INET_Addr logging_peer_addr;

if (acceptor ().accept (logging_handler_.peer (), &logging_peer_addr) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "accept()"), -1); ACE_DEBUG ((LM_DEBUG, "Accepted connection from %s\n", logging_peer_addr.get_host_name ())); return 0; }

Override the handle_connections() hook method to handle one connection at a time

Override & “decorate” the Logging_Server::open() method

Page 115: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

115

Implementing the Iterative_Logging_Server (3/3) virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) continue;

logging_handler_.close (); // Close the socket handle. return 0; }

#include "ace/Log_Msg.h"#include "Iterative_Logging_Server.h"

int main (int argc, char *argv[]){ Iterative_Logging_Server server;

if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); return 0;}

Main program of iterative logging server

Delegate I/O to Logging_Handler

Page 116: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

116

Concurrency Design Dimensions

•Concurrency is essential to develop scalable & robust networked applications, particularly servers

•The next group of slides present a domain analysis of concurrency design dimensions that address the policies & mechanisms governing the proper use of processes, threads, & synchronizers

•We cover the following design dimensions in this chapter:• Iterative versus concurrent versus reactive servers•Processes versus threads•Process/thread spawning strategies•User versus kernel versus hybrid threading models •Time-shared versus real-time scheduling classes•Task- versus message-based architectures

Page 117: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

117

Iterative vs. Concurrent Servers

•Iterative/reactive servers handle each client request in its entirety before servicing subsequent requests

•Best suited for short-duration or infrequent services

•Concurrent servers handle multiple requests from clients simultaneously

•Best suited for I/O-bound services or long-duration services

•Also good for busy servers

Page 118: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

118

Multiprocessing vs. Multithreading

•A process provides the context for executing program instructions

•Each process manages certain resources (such as virtual memory, I/O handles, & signal handlers) & is protected from other OS processes via an MMU

•IPC between processes can be complicated & inefficient

•A thread is a sequence of instructions in the context of a process

•Each thread manages certain resources (such as runtime stack, registers, signal masks, priorities, & thread-specific data)

•Threads are not protected from other threads

•IPC between threads can be more efficient than IPC between processes

Page 119: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

119

Thread Pool Eager Spawning Strategies•This strategy prespawns one or more OS processes or threads at server creation time

•These``warm-started'' execution resources form a pool that improves response time by incurring service startup overhead before requests are serviced

•Two general types of eager spawning strategies are shown below:

•These strategies based on Half-Sync/Half-Async & Leader/Followers patterns

Page 120: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

120

The Half-Sync/Half-Async Pattern

•The Half-Sync/Half-Async pattern decouples async & sync service processing in concurrent systems to simplify programming without unduly reducing performance

This pattern yields two primary benefits:

1.Threads can be mapped to separate CPUs to scale up server performance via multi-processing

2.Each thread blocks independently, which prevents a flow-controlled connection from degrading the QoS that other clients receive

SyncServiceLayer

AsyncService Layer

QueueingLayer

<<read/write>><<read/write>>

<<read/write>>

<<dequeue/enqueue>> <<interrupt>>

Sync Service 1 Sync Service 2 Sync Service 3

ExternalEvent Source

Queue

Async Service

Page 121: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

121

Half-Sync/Half-Async Pattern Dynamics

•This pattern defines two service processing layers—one async & one sync—along with a queueing layer that allows services to exchange messages between the two layers

: External EventSource

: Async Service : Queue

notification

read()

enqueue()

message

: Sync Service

work()

message

read()

message

work()

notification

•The pattern allows sync services (such as processing log records from different clients) to run concurrently, relative both to each other & to async/reactive services (such as event demultiplexing)

Page 122: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

122

Drawbacks with Half-Sync/Half-Async

Solution•Apply the Leader/Followers architectural pattern to minimize server threading overhead

Problem•Although Half-Sync/Half-Async threading model is more scalable than the purely reactive model, it is not necessarily the most efficient design

•CPU cache updates

<<get>><<get>>

<<get>>

<<put>>

Worker Thread 1

Worker Thread 3

Event source

Request Queue

acceptorhandlers

Worker Thread 2

•e.g., passing a request between the async thread & a worker thread incurs:

•This overhead can make server latency unnecessarily high

•Dynamic memory (de)allocation,

•A context switch, &•Synchronization operations,

Page 123: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

123

The Leader/Followers Pattern•The Leader/Followers architectural pattern is more efficient than Half-Sync/Half-Async

•Multiple threads take turns sharing event sources to detect, demux, dispatch, & process service requests that occur on the event sources

•This pattern eliminates the need for—& the overhead of—a separate Reactor thread & synchronized request queue used in the Half-Sync/Half-Async pattern

Handles

Handle SetsConcurrent Handles Iterative Handles

Concurrent Handle Sets

UDP Sockets + WaitForMultipleObjects(

)

TCP Sockets + WaitForMultpleObjects()

Iterative Handle Sets

UDP Sockets + select()/poll()

TCP Sockets + select()/poll()

Handleuses

demultiplexes

*

*

Handle Sethandle_events()deactivate_handle()reactivate_handle()select()

Event Handler

handle_event ()get_handle()

Concrete Event Handler B

handle_event ()get_handle()

Concrete Event Handler A

handle_event ()get_handle()

Thread Pool

join()promote_new_leader()

synchronizer

Page 124: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

124

Leader/Followers Pattern Dynamics: Concrete

Event Handler

join()

handle_event()

: ThreadPool

: HandleSet

join()

thread 2 sleepsuntil it becomesthe leader

event

thread 1 sleepsuntil it becomesthe leader

deactivate_handle()

join()

Thread 1 Thread 2

handle_events() reactivate_

handle()

handle_event()

event

thread 2waits for anew event,thread 1processescurrentevent

deactivate_handle()

handle_events()

new_leader()

1.Leader thread demuxing

2.Follower thread promotion

3.Event handler demuxing & event processing

4.Rejoining the thread pool

promote_

Page 125: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

125

Thread-per-Request On-demand Spawning Strategy

•On-demand spawning creates a new process or thread in response to the arrival of client connection and/or data requests

•Typically used to implement the thread-per-request & thread-per-connection models

•The primary benefit of on-demand spawning strategies is their reduced consumption of resources

•The drawbacks, however, are that these strategies can degrade performance in heavily loaded servers & determinism in real-time systems due to costs of spawning processes/threads & starting services

Page 126: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

126

The N:1 & 1:1 Threading Models•OS scheduling ensures applications use host CPU resources suitably•Modern OS platforms provide various models for scheduling threads •A key difference between the models is the contention scope in which threads compete for system resources, particularly CPU time

•The two different contention scopes are shown below:

• Process contention scope (aka “user threading”) where threads in the same process compete with each other (but not directly with threads in other processes)

• System contention scope (aka “kernel threading”) where threads compete directly with other system-scope threads, regardless of what process they’re in

Page 127: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

127

The N:M Threading Model

•Some operating systems (such as Solaris) offer a combination of the N:1 & 1:1 models, referred to as the ``N:M'‘ hybrid-threading model

•When an application spawns a thread, it can indicate in which contention scope the thread should operate

•The OS threading library creates a user-space thread, but only creates a kernel thread if needed or if the application explicitly requests the system contention scope

•When the OS kernel blocks an LWP, all user threads scheduled onto it by the threads library also block

•However, threads scheduled onto other LWPs in the process can continue to make progress

Page 128: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

128

Task- vs. Message-based Concurrency Architectures

•A concurrency architecture is a binding between:•CPUs, which provide the execution context for application code •Data & control messages, which are sent & received from one or more applications & network devices

•Service processing tasks, which perform services upon messages as they arrive & depart

•Task-based concurrency architectures structure multiple CPUs according to units of service functionality in an application

•Message-based concurrency architectures structure the CPUs according to the messages received from applications & network devices

Page 129: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

129

Overview of OS Concurrency Mechanisms

•Networked applications, particularly servers, must often process requests concurrently to meet their quality of service requirements

•This section presents an overview of the

•Synchronous event demultiplexing

•Multiprocessing

•Multithreading &

•Synchronization

mechanisms available to implement those designs

•We also discuss common portability & programming problems that arise when networked applications are developed using native C-level concurrency APIs

Page 130: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

130

Synchronous Event Demultiplexing• Synchronous event demuxers wait for certain events to occur on a set of event sources, where the caller is returned the thread of control whenever one or more event sources become active

•e.g., poll() on System V UNIX, WaitForMultipleObjects() on Windows, & select()

•select() is the most common event demultiplexing API for I/O handles

int select (int width, // Maximum handle plus 1 fd_set *read_fds, // Set of "read" handles fd_set *write_fds, // Set of "write" handles fd_set *except_fds, // Set of "exception" handles struct timeval *timeout);// Time to wait for events

•fd_set is a structure representing the set of handles to check for I/O events, such as ready for reading, writing, or exception events

•select() modifies the fd_set depending on the active/inactive handles as follows:

• If a handle is not active in the fd_set, it is ignored & select will keep it inactive in the fd_set

• If a handle is active, select() will determine if there are pending events. If there is one, the appropriate fd_set has the handle activated else its value is made inactive in the returned fd_set

Page 131: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

131

Multiprocessing Mechanisms

• Multiprocessing mechanisms include the features provided by the OS for creating & managing the execution of multiple processes, e.g.,

•Process lifetime operations – such as fork() & exec*() on POSIX & CreateProcess() on Windows create a new process address space for programs to run

• The initiating process can set command line arguments, environment variables & working directories for the new process

• The new process can terminate voluntarily by reaching the end of its execution or be involuntarily killed via signals (in POSIX) or TerminateProcess() (in Windows)

•Process synchronization options – provided by the OS to retain the identity & exit status of a process & report it to the parent, e.g.,

• POSIX wait() & waitpid()

• Windows WaitForSingleObject() & WaitForMultipleObjects()

•Process property operations – used to get/set process properties, such as default file access permissions, user identification, resource limits, scheduling priority & current working directory

Page 132: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

132

Multithreading Mechanisms (1/2)• Multithreading mechanisms are provided by the OS to handle thread lifetime management, synchronization, priorities, & thread specific storage

•Thread lifetime operations – include operations to create threads, e.g., pthread_create() (PThreads) & CreateThread() (Windows)

• Thread termination is achieved in the following manner:

• Voluntarily – by reaching the end point of the thread entry function or calling pthread_exit() (Pthreads) or ExitThread() (Windows)

• Involuntarily – by being killed via a signal or an aynchronous thread cancelation operations, such as pthread_cancel() (Pthreads) & TerminateThread() (Windows)

•Thread synchronization operations – that allow created threads to be

• Detached – where the OS reclaims storage used for the thread’s state & exit status after it has exit

• Joinable – where the OS retains identity & exit status of a terminating thread so other threads can synchronize with it

• Other operations that allow threads to suspend & resume each other, or send signals to other threads

Page 133: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

133

Multithreading Mechanisms (2/2)

•Thread property operations – includes operations to set & get thread properties, such as priority & scheduling class.

•Thread-specific storage – is similar to global data except that the data is global in the scope of the executing thread.

• Each thread has its own copy of a TSS data e.g., errno

• Each TSS item has a key that is global to all threads within a process

• A thread uses this key to access its copy of the TSS data

• Keys are created by factory functions, such as pthread_key_create() (Pthreads) or TlsAlloc() (Windows).

• Key/pointer relationships are managed by TSS set/get functions, such as pthread_getspecific() & pthread_setspecific() (Pthreads) or TlsGetValue() & TlsSetValue() (Windows)

Page 134: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

134

Synchronization Mechanisms (1/2)

• Synchronization mechanisms allow processes & threads to coordinate their execution order & the order in which they access shared resources, such as files, network devices, database records, & shared memory

•Mutexes – serialize execution of multiple threads by defining a critical section of code that can be executed by only one thread at a time. A thread owning a mutex must release it

•There are two kinds of mutexes:

• Nonrecursive mutex – that will deadlock or fail if the thread currently owning the mutex tries to reacquire it without first releasing it

• Recursive mutex – that will allow the thread owning the mutex to reacquire it without deadlocking

• The owner thread is responsible to release it the same number of times it has acquired it

Page 135: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

135

Synchronization Mechanisms (2/2)•Readers/writer locks – allows access to a shared resource by either multiple threads simultaneously having read-only access or only one thread at a time having a read-write access

• They help improve performance for applications where resources are read more often than modified

• They can be implemented to give more preference to either the readers or the writer

•Semaphores – is a non negative integer that can be incremented & decremented atomically

• A thread blocks when it tried to decrement a semaphore whose value is 0

• A block thread makes progress when another thread increments the value of the semaphore

• Usually implemented using sleep locks, that trigger a context switch

•Condition variables – allows a thread to coordinate & schedule its own processing

• A thread can wait on complex expressions to attain a desired state

• Used to build higher level patterns such as active object & monitor objects

Page 136: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

136

Sidebar: Evaluating Synchronization Mechanisms

• Performance of synchronization mechanisms depends on the OS implementation & hardware

• Some general issues to keep in mind:

•Condition variables & semaphores – generally have a higher overhead than mutexes

•Native OS implementations usually perform better than emulated behavior

•Mutexes versus Reader/Writer Locks – mutexes generally have the lower overhead than reader/writer locks.

•On a multiprocessor platform, reader/writer locks scale well since multiple readers can execute in parallel

•Nonrecursive mutexes – are more efficient than recursive mutexes

•Moreover, subtle errors can be caused using recursive mutexes due to mismatch in the number of lock & unlock operations

Page 137: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

137

Sidebar: ACE API Error Propagation Strategies

•Error reporting strategies usually differ across different concurrency APIs & OS

•e.g., UI & Pthreads return 0 on success & a non-zero number on failure whereas Windows returns 0 on failure & conveys the error value via thread specific storage

•This makes code non-portable & filled with accidental complexities

•ACE concurrency wrapper facades solve this problem by returning -1 on error & setting the errno variable in thread specific storage

Page 138: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

138

The ACE Event Demuxing Wrapper Facades

•The reactive server model can be thought of as “lightweight multitasking,” where a single-threaded server communicates with multiple clients in a round-robin manner without introducing the threading & synchronization overhead & complexity

•This server concurrency strategy uses an event loop that examines & reacts to events from its clients continuously

•An event loop demultiplexes input from various event sources so they can be processed in an orderly way

•Event sources in networked applications are primarily socket handles

•The most popular event demultipelxing function is select(), which provides the basis for the ACE classes described below

Page 139: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

139

The ACE_Handle_Set Class (1/2)

Motivation

•The fd_set represents a source of accidental complexity in the following areas:

•The code to scan for active handles is often a “hot spot” since it executes continually in a tight loop

•The macros supplied to manipulate & scan an fd_set must be used carefully to avoid processing handles that aren't active & to avoid corrupting an fd_set

•The fd_set is defined in system-supplied header files whose representation is exposed to programmers

•There are subtle nonportable aspects of fd_set when used in conjunction with select()

Page 140: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

140

The ACE_Handle_Set Class (2/2)Class Capabilities

•The ACE_Handle_Set class uses the Wrapper Façade pattern to encapsulate fd_sets & provide the following capabilities:

•It enhances portability, ease of use, & type safety of event-driven applications that use of fd_set & select() across OS platforms

•It tracks & adjusts the fd_set size-related values automatically as handles are added & removed

•ACE_Handle_Set_Iterator is an optimized iterator for ACE_Handle_Set

Page 141: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

141

Reactive Logging Server Version 1

•This example enhances the earlier iterative logging server implementation by using select() together with the ACE_Handle_Set & ACE_Handle_Set_Iterator classes

This server demultiplexes the following two types of events:

• Arrival of new connections from clients

• Arrival of log records on client connections

Page 142: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

142

Using the ACE_Handle_Set (1/4)

class Reactive_Logging_Server : public Iterative_Logging_Server {protected: // Keeps track of the acceptor socket handle & all the // connected stream socket handles. ACE_Handle_Set master_handle_set_;

// Keep track of handles marked as active by <select>. ACE_Handle_Set active_handles_;

typedef Iterative_Logging_Server PARENT;

// Other methods shown below...};

•We use a pair of ACE_Handle_Set data members since select() modifies the fd_set parameters passed to it, so we need to keep a master copy.

Page 143: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

143

Using the ACE_Handle_Set (2/4)

virtual int open (u_short logger_port) { PARENT::open (logger_port); master_handle_set_.set_bit (acceptor ().get_handle ()); acceptor ().enable (ACE_NONBLOCK); return 0; }

virtual int wait_for_multiple_events () { active_handles_ = master_handle_set_; if (select (active_handles_.max_set () + 1, active_handles_.fdset (), 0, // no write_fds 0, // no except_fds 0) == -1) // no timeout return -1; active_handles_.sync ((ACE_HANDLE) active_handles_.max_set () + 1); return 0; }

•open() method uses ACE_Handle_Set to track the acceptor’s handle in fd_set

•Note the use of non-blocking acceptor...

• Use select() to determine all active handles

sync() resets handle count in ACE_Handle_Set after the select() call

• Override hook method

Page 144: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

144

Sidebar: Motivation for Non-blocking Acceptors

Context

• An acceptor socket is passed to select() & gets marked as active when a connection is received

•Many servers use this event to call accept() without blocking

Problem

•Client disconnects exactly within the time interval between the server making the select() & accept() calls

•Possible race condition due to asynchronous behavior of TCP/IP leading to accept() call being blocked forever & hanging the application

Solution

•Acceptor sockets should always be set in non-blocking mode

•Achieved portably in ACE via the enable()method of ACE_IPC_SAP class passing it the ACE_NONBLOCK flag

Page 145: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

145

Using the ACE_Handle_Set (3/4)

virtual int handle_connections () {

if (active_handles_.is_set (acceptor ().get_handle ())) {

while (acceptor ().accept (logging_handler ().peer ()) == 0)

master_handle_set_.set_bit

(logging_handler ().peer ().get_handle ());

// Remove acceptor handle from further consideration.

active_handles_.clr_bit (acceptor ().get_handle ());

}

return 0;

}

• If the acceptor() handle is active, iteratively accept all the connections & save them in master_handle_set_

•Override the hook method

Page 146: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

146

Using the ACE_Handle_Set (4/4)

#include "ace/Log_Msg.h"

#include "Reactive_Logging_Server.h"

int main (int argc, char *argv[])

{

Reactive_Logging_Server server;

if (server.run (argc, argv) == -1)

ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1);

return 0;

}

Reactive logging server main program

handle_data() method shown later using ACE_Handle_Set_Iterator class

Page 147: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

147

Using the ACE_Handle_Set_Iterator (1/2)

virtual int handle_data (ACE_SOCK_Stream *) { for (ACE_HANDLE handle = acceptor ().get_handle () + 1; handle < active_handles_.max_set () + 1; handle++) {

if (active_handles_.is_set (handle)) { logging_handler ().peer ().set_handle (handle);

if (logging_handler ().log_record () == -1) { // Handle connection shutdown or comm failure. master_handle_set_.clr_bit (handle); logging_handler ().close (); } } } return 0; }

•Non-portable assumption that socket handles are a contiguous range integers•Cannot assume acceptor socket has lowest number

It’s inefficient to sequentially search for active handles in the large, but sparsely populated handle set

This method has problems. Do not copy this example!

Page 148: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

148

Using the ACE_Handle_Set_Iterator (2/2)

virtual int handle_data (ACE_SOCK_Stream *) { ACE_Handle_Set_Iterator peer_iterator (active_handles_);

for (ACE_HANDLE handle; (handle = peer_iterator ()) != ACE_INVALID_HANDLE; ) { logging_handler ().peer ().set_handle (handle);

if (logging_handler ().log_record () == -1) { // Handle connection shutdown or comm failure. master_handle_set_.clr_bit (handle); logging_handler ().close (); } } }

Overcoming drawbacks outlined before in handle_data() method by using the ACE_Handle_Set_Iterator class

•ACE_Handle_Set_Iterator optimizes searching for active handles based on the underlying platform’s representation of fd_set

•These optimizations are encapsulated within a common interface, making the class portable

Page 149: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

149

Reactive Logging Server Version 2

•We now extend our reactive server example to write log records from different clients to different log files, one for each connected client

•This reactive server implementation maintains a map container that allows a logging server to keep separate log files for each of its clients

•The figure also shows how we use the ACE::select() wrapper method & the ACE_Handle_Set class to service multiple clients via a reactive server model

Page 150: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

150

Implementing the Reactive Logging Server (1/6)

#include "ace/ACE.h"#include "ace/Handle_Set.h"#include "ace/Hash_Map_Manager.h"#include "ace/Synch.h"#include "Logging_Server.h"#include "Logging_Handler.h"

typedef ACE_Hash_Map_Manager<ACE_HANDLE, ACE_FILE_IO *, ACE_Null_Mutex> LOG_MAP;

class ACE {public: static int select (int width, ACE_Handle_Set &rfds, const ACE_Time_Value *tv = 0); static int select (int width, ACE_Handle_Set *rfds, ACE_Handle_Set *wfds = 0, ACE_Handle_Set *efds = 0, const ACE_Time_Value *tv = 0); // ... Other methods omitted ...};

Reactive logging server using ACE::select() & hash map container classes

Page 151: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

151

Implementing the Reactive Logging Server (2/6)

class Reactive_Logging_Server_Ex : public Logging_Server{protected: // Associate an active handle to an <ACE_FILE_IO> pointer. LOG_MAP log_map_;

// Keep track of acceptor socket & all the connected // stream socket handles. ACE_Handle_Set master_handle_set_;

// Keep track of read handles marked as active by <select>. ACE_Handle_Set active_read_handles_;

typedef Logging_Server PARENT;

// Other methods shown below...};

Association between each connected peer & its log file is maintained in a hash map

Page 152: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

152

Implementing the Reactive Logging Server (3/6)

virtual int open (u_short logger_port) { PARENT::open (logger_port); master_handle_set_.set_bit (acceptor ().get_handle ()); acceptor ().enable (ACE_NONBLOCK); return 0; }

virtual int wait_for_multiple_events () { active_read_handles_ = master_handle_set_; int width = (int) active_read_handles_.max_set () + 1;

return ACE::select (width, active_read_handles_); }

Note use of ACE::select(), which calls active_read_handles_.sync() automatically

open() method similar to previous one for first version of reactive logging server

Page 153: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

153

Implementing the Reactive Logging Server (4/6)

virtual int handle_connections () { ACE_SOCK_Stream logging_peer;

while (acceptor ().accept (logging_peer) != -1) { ACE_FILE_IO *log_file = new ACE_FILE_IO;

// Use the client's hostname as the logfile name. make_log_file (*log_file, &logging_peer);

// Add the new <logging_peer>'s handle to the map & // to the set of handles we <select> for input. log_map_.bind (logging_peer.get_handle (), log_file); master_handle_set_.set_bit (logging_peer.get_handle ()); } // Remove acceptor handle from further consideration... active_read_handles_.clr_bit (acceptor ().get_handle ()); return 0; }

•This version of reactive logging server creates a log file when a peer establishes connection

•The association between the log file & the handle is maintained in the hash map

Page 154: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

154

Implementing the Reactive Logging Server (5/6)

virtual int handle_data (ACE_SOCK_Stream *) { ACE_Handle_Set_Iterator peer_iterator (active_read_handles_);

for (ACE_HANDLE handle; (handle = peer_iterator ()) != ACE_INVALID_HANDLE; ) { ACE_FILE_IO *log_file;

log_map_.find (handle, log_file); Logging_Handler logging_handler (*log_file, handle);

if (logging_handler.log_record () == -1) { logging_handler.close (); master_handle_set_.clr_bit (handle); log_map_.unbind (handle); log_file->close (); delete log_file; } } return 0; }

Identify the log file corresponding to the peer who sent the data

Free up resources on error

Page 155: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

155

Implementing the Reactive Logging Server (6/6)

int main (int argc, char *argv[]){ Reactive_Logging_Server_Ex server;

if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); return 0;}

Reactive logging server main program (this should start looking rather familiar by now… ;-))

Page 156: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

156

The ACE Process Wrapper FacadesOS multiprocessing support helps to• Enable concurrency by scheduling & running separate processes on different CPUs• Increase robustness by using memory-management unit (MMU) hardware to protect separate process address spaces from accidental or malicious corruption by other active processes in the system

• Enhance security by allowing each process to verify or control per-user or per-session security & authentication information

We now cover the following ACE classes that can use to spawn & manage one or more processes:

These classes are related as follows:

Page 157: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

157

The ACE_Process Class (1/2)

Motivation

•OS process management mechanisms differ syntactically & semantically

•e.g., the UNIX fork() system function is very different than Windows CreateProcess()

•Addressing these platform variations in each application is difficult, tedious, error prone, & unnecessary since ACE provides the ACE_Process class

UNIX

Windows

Page 158: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

158

The ACE_Process Class (2/2)

Class Capabilities

•This class encapsulates the variation among different OS multiprocessing APIs in accordance with the Wrapper Facade pattern by defining methods that provide the following capabilities:

•Spawn & terminate processes

•Synchronize on process exit

•Access process properties, such as process ID

•Most methods are portable, though some are platform-specific

Page 159: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

159

Sidebar: POSIX Portability ChallengesContext

•Many Unix applications use POSIX fork() system call to build concurrent process-based servers

Problem

•Portability is a concern if we rely on fork() to both:

•Have a parent process fork() a child process with a duplicate of the parent’s address space & all I/O handles

•Have both parent & child return from fork() with the same location but with different return values, which can then run different parts of the program

Forces

•fork() is the only system call that allows duplicating an address space

• I/O handles of the parent are often required in the child process

Solution

•Use the ACE_Process_Options class that provides methods such as pass_handle(), working_directory(), setenv() & others

Page 160: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

160

Using ACE_Process

int main (int argc, char *argv[]){ ACE_Process_Options options; char *n_env = 0; int n; if (argc == 1) { // Top-level process. n_env = ACE_OS::getenv ("FACTORIAL"); n = n_env == 0 ? 0 : atoi (n_env); options.command_line ("%s %d", argv[0], n == 0 ? 10 : n); } else if (atoi (argv[1]) == 1) return 1; // Base case. else { n = atoi (argv[1]); options.command_line ("%s %d", argv[0], n - 1); } ACE_Process child; child.spawn (options); // Make the ``recursive'' call. child.wait (); return n * child.exit_code (); // Compute n factorial.}

Pass command line for spawned child to use

• This program uses ACE_Process to computes factorials “recursively”

Page 161: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

161

Class Capabilities •This class unifies how process properties are passed to ACE_Process & ACE_Process_Manager to provide the following capabilities:•Enable an application to specify desired process control information•Allow process control items to be expanded as platforms change•Provide a decoupling mechanism that enables ACE to offer these capabilities without varying the process creation interface

The ACE_Process_Options Class (2/2)

Page 162: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

162

The ACE_Process_Options Class (1/2)

Motivation

•Operating systems provide various methods for setting the properties of newly-created processes, including

•Program image, which program should the new process execute?

•Open I/O handles, e.g., should the child process inherit open I/O handles or other OS objects; should it close some or all of its inherited open handles?

•Working directory, e.g., should the child process run in the same or different directory as its parent?

•Process relationship, e.g., should the child process run in the background as an independent daemon or as part of a related group?

Page 163: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

163

Using ACE_Process_Options (1/2)

int main (int argc, char *argv[]){ ACE_Process_Options options; FILE *fp = 0; char *n_env = 0; int n;

if (argc == 1) { // Top-level process. n_env = ACE_OS::getenv ("FACTORIAL"); n = n_env == 0 ? 0 : atoi (n_env); options.command_line ("%s %d", argv[0], n == 0 ? 10 : n); const char *working_dir = ACE_OS::getenv ("WORKING_DIR"); if (working_dir) options.working_directory (working_dir); fp = fopen ("factorial.log", "a"); options.setenv ("PROGRAM=%s", ACE::basename (argv[0])); }

•This example demonstrates how to use ACE_Process_Options to pass environment, working directory, & command line to spawned child process

Change the working directory

Page 164: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

164

Using ACE_Process_Options (2/2)else { fp = fopen ("factorial.log", "a"); if (atoi (argv[1]) == 1) { fprintf (fp, "[%s|%d]: base case\n", ACE_OS::getenv ("PROGRAM"), ACE_OS::getpid ()); fclose (fp); return 1; // Base case. } else { n = atoi (argv[1]); options.command_line ("%s %d", argv[0], n - 1); } } ACE_Process child; child.spawn (options); // Make the ``recursive'' call. child.wait (); int factorial = n * child.exit_code (); // Compute n factorial. fprintf (fp, "[%s|%d]: %d! == %d\n", ACE_OS::getenv ("PROGRAM"), ACE_OS::getpid (), n, factorial); fclose (fp); return factorial; }

Update the command-line

Wait for child to exit so we can get its exit value (which will be a factorial value it has computed for “n-1”)

Pass options to spawn()

Page 165: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

165

The ACE_Process_Manager Class (1/2)Motivation

•Complex networked applications often require groups of processes to coordinate to provide a particular service, e.g.,

•A multistage workflow automation application may spawn multiple processes to work on different parts of a large problem

•One master process may wait for the entire group of worker processes to exit before proceeding with the next stage in the workflow

Page 166: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

166

Class Capabilities

•This class uses the Wrapper Facade pattern to combine the portability & power of ACE_Process with the ability to manage groups of processes as a unit & provide the following capabilities:

•It provides internal record keeping to manage & monitor groups of processes that are spawned by the ACE_Process class

•It allows one process to spawn a group of process & wait for them to exit before proceeding with its own processing

The ACE_Process_Manager Class (2/2)

Page 167: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

167

Multiprocessing Logging Server•This revision of the logging server uses a process-per connection concurrency model

•The master process spawns a new worker process for each accepted connection to the logging service port

•The master process then continues to accept new connections•Each worker process handles all logging requests sent by a client across one connection; the process exits when this connection closes

Page 168: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

168

Using ACE_Process_Manager (1/6)

class Process_Per_Connection_Logging_Server : public Logging_Server{public: // … Methods shown below …protected: std::string prog_name_;};

virtual int run (int argc, char *argv[]) { prog_name_ = argv[0]; // Ensure NUL-termination. prog_name_[prog_name_.size () - 1] = '\0'; if (argc == 3) return run_worker (argc, argv); // Only on Windows. else return run_master (argc, argv); }

•Process-per-connection logging server that uses the ACE_Process_Manager

Page 169: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

169

Process Creation on POSIX & Windows

•Master/Worker Process Creation Sequence for POSIX

•Note use of child copy of parent’s process image

UNIX

•Master/Worker Process Creation Sequence for Windows

•Child doesn’t automatically inherit the parent process’s image, so we must take other steps… •e.g., pass the socket handle Windows

Page 170: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

170

Using ACE_Process_Manager (2/6) int run_master (int argc, char *argv[]) { u_short logger_port = 0; if (argc == 2) logger_port = atoi (argv[1]); if (open (logger_port) == -1) return -1;

for (;;) if (handle_connections () == -1) return -1;

return 0; }

int run_worker (int argc, char *argv[]) { ACE_HANDLE socket_handle = ACE_static_cast (ACE_HANDLE, atoi (argv[2])); ACE_SOCK_Stream logging_peer (socket_handle);

handle_data (&logging_peer); logging_peer.close (); return 0; }

Method only gets invoked for Windows

Page 171: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

171

Context

•C language cast operator used frequently for type casting

Problem

•Can lead to subtle errors due to type violations caused by misuse

•Although C++ introduces several keywords to allow robust type casting, they are not supported across the spectrum of C++ compilers

Solution

•Use ACE cast macros shown below

•These methods are largely deprecated in ACE since new C++ compilers are better

Sidebar: Portable Casting on C++ Compilers

ACE Cast Macro C++ Cast Used if Available

ACE_const_cast (TYPE, EXPR) const_cast<TYPE> (EXPR)

ACE_static_cast (TYPE, EXPR) static_cast<TYPE> (EXPR)

ACE_dynamic_cast (TYPE, EXPR) dynamic_cast<TYPE> (EXPR)

ACE_reinterpret_cast (TYPE, EXPR) reinterpret_cast<TYPE> (EXPR)

Page 172: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

172

Using ACE_Process_Manager (3/6) 1 virtual int handle_connections () {

2 ACE_SOCK_Stream logging_peer;

3 if (acceptor ().accept (logging_peer) == -1)

4 return -1;

5

6 Logging_Process *logger =

7 new Logging_Process (prog_name_, logging_peer);

8 ACE_Process_Options options;

9 pid_t pid = ACE_Process_Manager::instance ()->spawn

10 (logger, options);

11 if (pid == 0) {

12 acceptor().close ();

13 handle_data (&logging_peer);

14 delete logger;

15 ACE_OS::exit (0);

16 }

Create a Logger_Process that the process manager will manage

On POSIX, this will be a child, so close the acceptor, handle the data, & release resources

On Windows, the child process created after the spawn call above will end up running the run_worker() method shown earlier instead of lines 11 thru 16

spawn() invokes hook methods on logger to initialize options properly

Note use of Singleton pattern

Page 173: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

173

Singleton Pattern

• Intent

• Ensure a class has only one instance & provide a global point of access to it

• Context

• Need to address initialization versus usage ordering

• Problem

• Want to ensure a single instance of a class, shared by all uses throughout a program

class Singleton {public: static Singleton *instance (){ if (instance_ == 0) {

instance_ = new Singleton; } return instance_; } void method_1 (); // … Other methods omitted …private: // Private constructor Singleton (); // Initialized to 0 by linker. static Singleton *instance_;};• Solution

• Provide a global access method (static in C++)

• First use of the access method instantiates the class

• Constructors for instance are made private

Page 174: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

174

Using ACE_Process_Manager (4/6)17 logging_peer.close ();18 if (pid == -1)19 ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "spawn()"), -1);2021 return ACE_Process_Manager::instance ()->wait 22 (0, ACE_Time_Value::zero);23 }

virtual int handle_data (ACE_SOCK_Stream *logging_peer) { // Ensure blocking <recv>s. logging_peer->disable (ACE_NONBLOCK); ACE_FILE_IO log_file; make_log_file (log_file, logging_peer); Logging_Handler logging_handler (log_file, *logging_peer);

while (logging_handler.log_record () != -1) continue;

log_file.close (); return 0; }

Parent does not need the peer

handle_data() method similar to before (uses blocking I/O)

Page 175: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

175

Using ACE_Process_Manager (5/6)class Logging_Process : public ACE_Process{private: Logging_Process (); // Force desired constructor to be used. std::string prog_name_; ACE_SOCK_Stream logging_peer_;public: Logging_Process (const std::string &prog_name, const ACE_SOCK_Stream &logging_peer) : logging_peer_ (logging_peer.get_handle ()) { prog_name_ (prog_name); }

virtual int prepare (ACE_Process_Options &options) { if (options.pass_handle (logging_peer_.get_handle ()) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "pass_handle()"), -1); options.command_line ("%s", prog_name_.c_str ()); options.avoid_zombies (1); options.creation_flags (ACE_Process_Options::NO_EXEC); return 0; }

prepare() hook method is invoked by spawn() prior to the actual spawning

pass_handle() adds “+H <handle>” to command line on Windows

Logging_Process class extends ACE_Process

Don’t exec() after fork() on POSIX

Page 176: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

176

Using ACE_Process_Manager (6/6)virtual void unmanage () { delete this; }

static void sigterm_handler (int /* signum */) { /* No-op. */ }

int main (int argc, char *argv[]){ // Register to receive the <SIGTERM> signal. ACE_Sig_Action sa (sigterm_handler, SIGTERM);

Process_Per_Connection_Logging_Server server;

if (server.run (argc, argv) == -1 && errno != EINTR) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1);

// Barrier synchronization. return ACE_Process_Manager::instance ()->wait ();}

Invoked by process manager as it removes the managed process

Page 177: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

177

The ACE Threading Wrapper Facades•Multithreading is useful for servers that manage connection-oriented or connectionless network associations for many clients simultaneously

•Today's powerful OS support for multithreading helps networked applications to:

• Leverage hardware advances, such as symmetric multiprocessing that enables true execution parallelism

• Increase performance by overlapping computation & communication

• Improve response time for GUIs & network servers to ensure that time-sensitive tasks are scheduled as needed

• Simplify program structure by enabling applications to use intuitive synchronous programming mechanisms, rather than more complex asynchronous programming mechanisms

•Below we describe the following ACE classes that networked applications can use to spawn & manage one or more threads of control within a process:

Page 178: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

178

The ACE_Thread_Manager Class (1/2)

Motivation

•Different operating systems use different APIs to create & manage threads, which causes two types of non-portable variability:

•Syntactic, e.g., the Windows CreateThread() & the Pthreads pthread_create() functions provide similar thread creation capabilities, even though their APIs differ syntactically

•Semantic, e.g., both Pthreads & UI threads support detached threads, whereas Windows does not, & VxWorks supports only detached threads

Page 179: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

179

Class Capabilities•This class uses the Wrapper Facade pattern to encapsulate variation among different OS multithreading APIs & provide the following portable capabilities:•Spawns one or more threads, each running an application-designated function concurrently

•Alters the most common thread attributes, for example, scheduling priority & stack size, for each of the spawned threads

The ACE_Thread_Manager Class (2/2)

•Spawns & manages a set of threads as a cohesive collection, called a thread group

•Manages the threads in an ACE_Task (discussed in C++NPv2 tutorial)•Facilitates cooperative cancelation of threads, •Waits for one or more threads to exit

Page 180: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

180

ACE_Thread_Manager::spawn() Options

The ACE_Thread_Manager::spawn() method can be passed a set of flags to specify the properties of the created thread, whose value is a bit-wise inclusive ``or'' of the flags shown in the following table:

Page 181: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

181

Thread-per-Connection Logging Server•This example illustrates the thread-per-connection concurrency model

•The master thread runs continuously & plays the role of a factory that

1.Accepts a connection & creates an ACE_SOCK_Stream object dynamically

2.Spawns a worker thread that uses this object to handle this client's logging

•The worker thread performs all subsequent log record processing on the ACE_SOCK_Stream & destroys it when the connection is closed

Thread_Per_Connection_

Logging_Server

Page 182: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

182

Sidebar: Traps & Pitfalls of Mixing Objects & Threads

Context

•Lifecycle of objects passed as parameters to thread entry point functions must be managed carefully

Problem

•Programmers forget to make the following distinction

•A thread is a unit of execution

•An object is a chunk of memory with associated methods

•There is no implicit connection between a thread & any object that the thread accesses during execution

Forces

•A thread should not be able access an object after the latter has been deleted

Solution

•Dynamically allocate the object passing it to the thread function as parameter & let the latter delete it

Page 183: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

183

Sidebar: How Threads are Spawned in ACE

The figure below show the calls that occur when ACE_Thread_Manager:: spawn() is invoked on a platform configuration that uses the UI Threads thr_create() system function:

Regardless of OS, the following steps occur to spawn a thread:•The OS creates a thread execution context•The OS allocates memory for the thread's stack•The new thread's register set is prepared so that when it's scheduled into execution, it will call the thread entry point function supplied as a parameter to spawn()

•The thread is marked runnable so the OS can start executing it

Page 184: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

184

Using ACE_Thread_Manager (1/5)

class Thread_Per_Connection_Logging_Server : public Logging_Server{private: class Thread_Args { public: Thread_Args (Thread_Per_Connection_Logging_Server *lsp) : this_ (lsp) {} Thread_Per_Connection_Logging_Server *this_; ACE_SOCK_Stream logging_peer_; }; // Passed as a parameter to <ACE_Thread_Manager::spawn>. static void *run_svc (void *arg);

protected: // Other methods shown below...};

•This example uses the ACE_Thread_Manager to implement our first multithreaded logging server based on a thread-per-connection concurrency model

Page 185: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

185

Using ACE_Thread_Manager (2/5)

virtual int handle_connections () { auto_ptr<Thread_Args> thread_args (new Thread_Args (this));

if (acceptor ().accept (thread_args->logging_peer_) == -1) return -1; if (ACE_Thread_Manager::instance ()->spawn ( // Pointer-to-function entry point. Thread_Per_Connection_Logging_Server::run_svc, // <run_svc> parameter. ACE_static_cast (void *, thread_args.get ()),

THR_DETACHED | THR_SCOPE_SYSTEM) == -1) return -1; thread_args.release (); // Spawned thread now owns memory return 0; }

Arguments to pass to thread entry point function

Create a detached kernel thread

Page 186: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

186

Using ACE_Thread_Manager (3/5)

void *Thread_Per_Connection_Logging_Server::run_svc (void *arg){ auto_ptr<Thread_Args> thread_args (ACE_static_cast (Thread_Args *, arg));

thread_args->this_->handle_data (&thread_args->logging_peer_); thread_args->logging_peer_.close (); return 0; // Return value is ignored.}

•Thread entry point “adapter” method that handles incoming data from peer

•This method will also clean up resources, such as closing the peer connection

We’re responsible for deleting this dynamically allocated memory when we’re done, so we use an auto_ptr

Page 187: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

187

Using ACE_Thread_Manager (4/5)virtual int handle_data (ACE_SOCK_Stream *client) { ACE_FILE_IO log_file; // Client's hostname used as logfile name. make_log_file (log_file, client); // Place the connection into blocking mode. client->disable (ACE_NONBLOCK);

Logging_Handler logging_handler (log_file, *client); ACE_Thread_Manager *tm = ACE_Thread_Manager::instance (); ACE_thread_t me = ACE_OS::thr_self ();

// Keep handling log records until client closes connection // or this thread is asked to cancel itself. while (!tm->testcancel (me) && logging_handler.log_record () != -1) continue;

log_file.close (); return 0; }

Note the use of the Singleton pattern

Note the use of cooperative thread cancellation

Page 188: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

188

Sidebar: Serializing Singletons in ACE

•As discussed earlier, the Singleton pattern ensures that a class has only one instance & provides a global point of access to it

•ACE singletons use the Double-Checked Locking Optimization pattern to reduce contention & synchronization overhead when critical sections of code must acquire locks in a thread-safe manner just once during program execution

ACE_Thread_Manager *ACE_Thread_Manager::instance () {

if (ACE_Thread_Manager::thr_mgr_ == 0) {

ACE_GUARD_RETURN (ACE_Recursive_Thread_Mutex, ace_mon,

*ACE_Static_Object_Lock::instance(),

0));

if (ACE_Thread_Manager::thr_mgr_ == 0)

ACE_Thread_Manager::thr_mgr_ = new ACE_Thread_Manager;

}

return ACE_Thread_Manager::thr_mgr_;

} ACE_Static_Object_Lock instance is created prior to execution of main() program

Page 189: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

189

Using ACE_Thread_Manager (5/5)

int main (int argc, char *argv[]){ // Register to receive the <SIGTERM> signal. ACE_Sig_Action sa (sigterm_handler, SIGTERM);

Thread_Per_Connection_Logging_Server server; if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1);

// Cooperative thread cancelation. ACE_Thread_Manager::instance ()->cancel_all ();

// Barrier synchronization, wait no more than a minute. ACE_Time_Value timeout (ACE_OS::gettimeofday ()); timeout += 60; return ACE_Thread_Manager::instance ()->wait (&timeout);}

Thread-per-connection logging server main program

Page 190: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

190

The ACE_Sched_Params Class (1/2)

Motivation

•Certain types of networked applications, particularly those with real-time requirements, need strict control over their thread priorities

•It’s important that this control be as portable as possible

Page 191: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

191

Class Capabilities

•This class uses the Wrapper Facade pattern to encapsulate OS scheduling class APIs & can be used with the ACE_OS::sched_params() method to provide the following capabilities:

The ACE_Sched_Params Class (2/2)

•A portable means to specify scheduling policies, such as FIFO

•A way to specify a time-slice quantum for the round-robin scheduling policy

•A way to specify the scope in which the policy applies, for example, to the current process or the current thread

•A consistent representation of scheduling priorities, which is necessary since higher scheduling priorities are indicated by lower priority values on some OS platforms

Page 192: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

192

Using ACE_Sched_Params (1/2)

class RT_Thread_Per_Connection_Logging_Server : public Thread_Per_Connection_Logging_Server{public: virtual int open (u_short port) { ACE_Sched_Params fifo_sched_params (ACE_SCHED_FIFO, ACE_Sched_Params::priority_min (ACE_SCHED_FIFO), ACE_SCOPE_PROCESS);

if (ACE_OS::sched_params (fifo_sched_params) == -1) { if (errno == EPERM || errno == ENOTSUP) ACE_DEBUG ((LM_DEBUG, "Warning: user's not superuser, so " "we'll run in the time-shared class\n")); else

•This example demonstrates a real-time thread-per-connection logging server whose scheduling parameters can be controlled via ACE_Sched_Params

Page 193: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

193

Using ACE_Sched_Params (2/2) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "ACE_OS::sched_params()"), -1); } // Initialize the parent classes. return Thread_Per_Connection_Logging_Server::open (port); }

virtual int handle_data (ACE_SOCK_Stream *logging_client) { int prio = ACE_Sched_Params::next_priority (ACE_SCHED_FIFO, ACE_Sched_Params::priority_min (ACE_SCHED_FIFO), ACE_SCOPE_THREAD);

ACE_OS::thr_setprio (prio); return Thread_Per_Connection_Logging_Server::handle_data (logging_client); }};

Data handling is done at a higher priority than connections

Page 194: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

194

The ACE_TSS Class (1/2)

Motivation

•While C++ global variables can be useful, their potential for harmful side-effects & undefined initialization semantics are problematic

•These problems are exacerbated in multithreaded applications, e.g., when multiple threads access unsynchronized global variables simultaneously

•The use of synchronizers in these situations can often cause more problems than they solve…

•The classic example is errno// ...if (recv (……) == -1 && errno != EWOULDBLOCK) ACE_ERROR ((LM_ERROR, “recv failed, %p”, errno));};

Page 195: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

195

The ACE_TSS Class (2/2)

Class Capabilities•This class implements the Thread-Specific Storage pattern, which encapsulates & enhances the native OS Thread-Specific Storage (TSS) APIs to provide the following capabilities:

•It supports data that are ``physically'' thread specific, that is, private to a thread, but allows them to be accessed as though they were ``logically'' global to a program

•It uses the C++ delegation operator, operator->(), to provide thread-specific smart pointers

•It encapsulates the management of the keys associated with TSS objects

•For platforms that lack adequate TSS support natively (such as VxWorks) ACE_TSS emulates TSS efficiently

Page 196: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

196

The Thread-Specific Storage Pattern• The Thread-Specific Storage pattern allows multiple threads to use one ‘logically global’ access point to retrieve an object that is local to a thread, without incurring locking overhead on each object access

The application thread identifier, thread-specific object set, &

proxy cooperate to obtain the correct thread-specific object

key 1

key n

thread m

Thread-SpecificObject

Thread-SpecificObject Proxy

Thread-SpecificObject Set

accesses

manages

[k,t]

thread 1

: Thread-SpecificObject Proxy

method()

: ApplicationThread

: Thread-SpecificObject Set

: Thread-SpecificObject

key

set()

create_key()

: KeyFactory

keyTSObject

Page 197: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

197

Using ACE_TSS (1/3)

template <class TYPE> TYPE *ACE_TSS<TYPE>::operator-> () { if (once_ == 0) { // Ensure that we're serialized. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, keylock_, 0);

if (once_ == 0) { ACE_OS::thr_keycreate (&key_, &ACE_TSS<TYPE>::cleanup); once_ = 1; } }

• In this implementation, each thread gets its own request count that resides in thread-specific storage to alleviate race conditions on the request count without requiring a mutex

• This example illustrates how to implement & apply ACE_TSS to our thread-per-connection logging server

We used the double-checked locking optimization pattern here

Page 198: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

198

Using ACE_TSS (2/3)

TYPE *ts_obj = 0;

// Initialize <ts_obj> from thread-specific storage. ACE_OS::thr_getspecific (key_, (void **) &ts_obj);

// Check if this method's been called in this thread. if (ts_obj == 0) { // Allocate memory off the heap & store it in a pointer. ts_obj = new TYPE;

// Store the dynamically allocated pointer in TSS. ACE_OS::thr_setspecific (key_, ts_obj); } return ts_obj;}

Page 199: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

199

Using ACE_TSS (3/3)class Request_Count {public: Request_Count (): count_ (0) {} void increment () { ++count_; } int value () const { return count_; }

private: int count_;};

static ACE_TSS<Request_Count> request_count;

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) // Keep track of number of requests. request_count->increment ();

ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", request_count->value ())); }

This call increments variable in thread-specific storage

Page 200: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

200

The ACE Synchronization Wrapper Facades

•Different operating systems provide different synchronization mechanisms with different semantics using different APIs•Some of these APIs conform to international standards, such as Pthreads•Other APIs conform to de facto standards, such as Windows

•Below we describe the following ACE classes that networked applications can use to synchronize threads and/or processes portably

Page 201: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

201

The ACE_Lock* Pseudo-Class

•The ACE mutex, readers/writer, semaphore, & file lock mechanisms all support the ACE_LOCK* interface shown below

•ACE_LOCK* is a “pseudo-class,” i.e., it's not a real C++ class in ACE

•We use it to illustrate the uniformity of the signatures supported by many of the ACE synchronization classes

•e.g., ACE_Thread_Mutex, ACE_RW_Thread_Mutex, ACE_Process_Mutex, & ACE_Thread_Semaphore

Page 202: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

202

The ACE_Guard ClassesMotivation

•When acquiring & releasing locks explicitly, it can be hard to ensure that all paths through the code release the lock, especially when C++ exceptions are thrown

•ACE provides the ACE_Guard class & its associated subclasses to help assure that locks are acquired & released properly

Class Capabilities•These classes implement the Scoped Locking idiom, which leverages the semantics of C++ class constructors & destructors to ensure a lock is acquired & released automatically upon entry to & exit from a block of C++ code, respectively

Page 203: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

203

The Scoped Locking IdiomMotivation•Code that shouldn’t execute concurrently must be protected by some type of lock that is acquired/released when control enters/leaves a critical section

•If programmers must acquire & release locks explicitly, it is hard to ensure that the locks are released in all paths through the code

•e.g., in C++ control can leave a scope due to a return, break, continue, or goto statement, as well as from an unhandled exception being propagated out of the scopevoid method () { lock_.acquire (); // The implementation may return prematurely… lock_.release (); } •The Scoped Locking idiom defines a guard class whose constructor

automatically acquires a lock when control enters a scope & whose destructor automatically releases the lock when control leaves the scope void method () { ACE_Guard <ACE_Thread_Mutex> guard (lock_); // The lock is released when the method returns}

Page 204: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

204

Implementing Scoped Locking in ACEtemplate <class LOCK> class ACE_Guard {public: // Store a pointer to the lock & acquire the lock. ACE_Guard (LOCK &lock) : lock_ (&lock) { lock_->acquire (); }

// Release the lock when the guard goes out of scope, ~ACE_Guard () { lock_->release (); }

// Other methods omitted…

private: // Pointer to the lock we’re managing. LOCK *lock_;};

Generic ACE_Guard Wrapper Facade

•ACE_Write_Guard & ACE_Read_Guard acquire write locks & read locks, respectively

•Instances of the ACE_Guard* classes can be allocated on the run-time stack to acquire & release locks in method or block scopes that define critical sections

Page 205: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

205

Sidebar: Overview of ACE_GUARD Macros

•ACE provides macros to simply the use of the different ACE_Guard classes

•These macros check for deadlock & failures in lock operations# define ACE_GUARD(MUTEX,OBJ,LOCK) \ ACE_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return;# define ACE_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN;# define ACE_WRITE_GUARD(MUTEX,OBJ,LOCK) \ ACE_Write_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return;# define ACE_WRITE_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Write_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN;# define ACE_READ_GUARD(MUTEX,OBJ,LOCK) \ ACE_Read_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return;# define ACE_READ_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Read_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN;

Page 206: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

206

The ACE Mutex ClassesMotivation

•Most operating systems provide some form of mutex mechanism that concurrent applications can use to serialize access to shared resources

•As with most of the other platform-specific capabilities, there are subtle variations in syntax & semantics between different OS platforms

•Mutexes also have different initialization requirements

Class Capabilities

•ACE uses the Wrapper Facade pattern to guide the encapsulation of native OS mutex synchronization mechanisms with the ACE_Process_Mutex & ACE_Thread_Mutex classes

•These classes implement nonrecursive mutex semantics portably at system scope & process scope, respectively

•They can be used to serialize thread access to critical sections across processes or in one process

•The interface for the these classes is identical to the ACE_LOCK* pseudo-class

•The ACE_Null_Mutex class implements all of its methods as ``no-op'' inline functions

Page 207: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

207

Using ACE_Thread_Mutex (1/3)#include "ace/Synch.h"

typedef u_long COUNTER;static COUNTER request_count; // File scope global variable.// Mutex protecting request_count (constructor initializes).static ACE_Thread_Mutex m;

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) { // Try to acquire the lock. if (m.acquire () == -1) return 0; ++request_count; // Count # of requests m.release (); // Release lock. } m.acquire (); int count = request_count; m.release (); ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", count)); logging_handler_.close (); return 0; }

This code is tedious & error-prone to write & maintain!

Page 208: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

208

Using ACE_Thread_Mutex (2/3)

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) { // Acquire lock in constructor. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, m, -1); ++request_count; // Count # of requests // Release lock in destructor. }

int count; { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, m, -1); count = request_count; } ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", count)); }

handle_data() method shown using ACE_GUARD macros

m must be initialized properly (C++ does not guarantee any order)

Source of accidental complexity

Obtrusive change

Page 209: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

209

Using ACE_Thread_Mutex (3/3)

// ... while (logging_handler_.log_record () != -1) { // Acquire lock in constructor. ACE_GUARD_RETURN (ACE_Recursive_Thread_Mutex, guard, ACE_Static_Object_Lock::instance (), -1); ++request_count; // Count # of requests // Release lock in destructor. } // ... ACE_Static_Object_Lock gets preinitialized before

main() runs

Ensuring consistent initialization order of global & static members using ACE_Object_Manager

Page 210: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

210

The ACE Readers/Writer Classes

Motivation

•Readers/writer locks allow efficient concurrent access to resources whose contents are searched much more often than they are changed

•Operating systems support readers/writer semantics in their file-locking APIs

•Involving the file system in synchronization activities is unnecessarily inefficient, however, & can block under unpredictable situations

•Moreover, file-locking mechanisms work only at the system-scope level, rather than at process scope

Class Capabilities

•ACE encapsulates the native readers/writer lock mechanisms with the ACE_RW_Thread_Mutex & ACE_RW_Process_Mutex classes.

•These classes apply the Wrapper Facade pattern to implement the semantics of process- & system-scoped readers/writer locks portably

•The interface for these classes is identical to the signatures of ACE_LOCK* pseudo-class

•The ACE readers/writer implementation gives preference to writers, i.e., if there are multiple readers & a single writer waiting on the lock, the writer will acquire it first

Page 211: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

211

Using ACE_RW_Thread_Mutex (1/3)

class Atomic_Op{public: // Initialize <count_> to <count>. Atomic_Op (long count = 0) : count_ (count) {}

// Atomically pre-increment <count_>. long operator++ () { // Use the <acquire_write> method to acquire a write lock. ACE_WRITE_GUARD_RETURN (ACE_RW_Thread_Mutex, guard, lock_, -1); return ++count_; }

Implementing atomic operations using reader/writer locks

ACE provides the ACE_Atomic_Op<> template that’s based on this approach.

Note the use of the Scoped Locking idiom

Page 212: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

212

Using ACE_RW_Thread_Mutex (2/3)

// Atomically return <count_>. operator long () { // Use the <acquire_read> method to acquire a read lock. ACE_READ_GUARD_RETURN (ACE_RW_Thread_Mutex, guard, lock_, 0); return count_; }

// ... Other arithmetic operators omitted.

private: // Readers/writer lock. ACE_RW_Thread_Mutex lock_;

// Value of the <Atomic_Op> count. long count_;};

Multiple threads can be reading this value concurrently

Page 213: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

213

Using ACE_RW_Thread_Mutex (3/3)

typedef Atomic_Op COUNTER;

static COUNTER request_count; // File scope global variable.

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) // Keep track of number of requests. ++request_count; // Actually calls <Atomic_Op::operator++>.

ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", // Actually calls <Atomic_Op::operator long>. (long) request_count)); }

Overcoming the need for obtrusive changes using class Atomic_Op

Page 214: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

214

The ACE Semaphore Classes

Motivation

•Semaphores are a powerful mechanism used to lock and/or synchronize access to shared resources in concurrent applications

•A semaphore contains a count that indicates the status of a shared resource

•Application designers assign the meaning of the semaphore's count, as well as its initial value

•Semaphores can therefore be used to mediate access to a pool of resources

Class Capabilities

•The ACE_Thread_Semaphore & ACE_Process_Semaphore classes portably encapsulate process-scoped & system-scoped semaphores, respectively, in accordance with the Wrapper Façade pattern

•These class interfaces are largely the same as the ACE_LOCK* pseudo-class

•The ACE_Null_Semaphore class implements all of its methods as ``no-op'' inline functions

Page 215: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

215

Using ACE_Thread_Semaphore (1/8)class Message_Queue{public: // Default high & low water marks. enum { DEFAULT_LWM = 0, // 0 is the low water mark. DEFAULT_HWM = 16 * 1024 // 16 K is the high water mark. };

// Initialize. Message_Queue (size_t = DEFAULT_HWM, size_t = DEFAULT_LWM);

// Destroy. ~Message_Queue ();

// Checks if queue is full/empty. int is_full () const; int is_empty () const;

Message queue implementation using ACE_Thread_Semaphore

Page 216: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

216

Using ACE_Thread_Semaphore (2/8)

// Interface for enqueueing & dequeueing ACE_Message_Blocks. int enqueue_tail (ACE_Message_Block *, ACE_Time_Value * = 0); int dequeue_head (ACE_Message_Block *&, ACE_Time_Value * = 0);

private: // Implementations that enqueue/dequeue ACE_Message_Blocks. int enqueue_tail_i (ACE_Message_Block *, ACE_Time_Value * = 0); int dequeue_head_i (ACE_Message_Block *&, ACE_Time_Value * = 0);

// Implement the checks for boundary conditions. int is_empty_i () const; int is_full_i () const;

// Lowest number before unblocking occurs. int low_water_mark_; // Greatest number of bytes before blocking. int high_water_mark_;

Note use of Thread-Safe Interface pattern

Page 217: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

217

Thread-Safe Interface Pattern

• Interface methods check•All interface methods, such as C++ public methods, should only acquire/release component lock(s), thereby performing synchronization checks at the ‘border’ of the component.

This pattern structures all components that process intra-component method invocations according two design conventions:

• Implementation methods trust • Implementation methods, such as C++ private & protected methods, should only perform work when called by interface methods.

Context•Components in multi-threaded applications that contain intra-component method calls

Problem•Thread-safe components should be designed to avoid unnecessary locking

•Thread-safe components should be designed to avoid “self-deadlock”

Solution•Apply the Thread-safe Interface design pattern (P2) to minimize locking overhead & ensure that intra-component method calls do not incur ‘self-deadlock’ by trying to reacquire a lock that is held by the component already

Page 218: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

218

Using ACE_Thread_Semaphore (3/8)

// Current number of bytes in the queue. int cur_bytes_; // Current number of messages in the queue. int cur_count_; // Number of threads waiting to dequeue a message. size_t dequeue_waiters_; // Number of threads waiting to enqueue a message. size_t enqueue_waiters_;

// C++ wrapper facades to coordinate concurrent access. mutable ACE_Thread_Mutex lock_; ACE_Thread_Semaphore notempty_; ACE_Thread_Semaphore notfull_;

// Remaining details of queue implementation omitted....};

Using ACE_Thread_Semaphore

Page 219: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

219

Using ACE_Thread_Semaphore (4/8)Message_Queue::Message_Queue (size_t hwm, size_t lwm) : low_water_mark_ (lwm), high_water_mark (hwm), cur_bytes_ (0), cur_count_ (0), dequeue_waiters_ (0), enqueue_waiters_ (0), notempty_ (0), notfull_ (1){ /* Remaining constructor implementation omitted ... */ }

int Message_Queue::is_empty () const { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); return is_empty_i ();}

int Message_Queue::is_full () const { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); return is_full_i ();}

Page 220: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

220

Using ACE_Thread_Semaphore (5/8)int Message_Queue::is_empty_i () const { return cur_bytes_ <= 0 && cur_count_ <= 0; }

int Message_Queue::is_full_i () const { return cur_bytes_ >= high_water_mark_;}

int Message_Queue::enqueue_tail (ACE_Message_Block *new_mblk, ACE_Time_Value *timeout){ ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); int result = 0;

// Wait until the queue is no longer full. while (is_full_i () && result != -1) { ++enqueue_waiters_; guard.release (); result = notfull_.acquire (timeout); guard.acquire (); }

Page 221: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

221

Using ACE_Thread_Semaphore (6/8) if (result == -1) { if (enqueue_waiters_ > 0) --enqueue_waiters_; if (errno == ETIME) errno = EWOULDBLOCK; return -1; }

// Enqueue the message at the tail of the queue. int queued_messages = enqueue_tail_i (new_mblk);

// Tell any blocked threads that the queue has a new item! if (dequeue_waiters_ > 0) { --dequeue_waiters_; notempty_.release (); } return queued_messages; // guard's destructor releases lock_.}

Page 222: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

222

Using ACE_Thread_Semaphore (7/8)int Message_Queue::dequeue_head (ACE_Message_Block *&first_item, ACE_Time_Value *timeout){ ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); int result = 0;

// Wait until the queue is no longer empty. while (is_empty_i () && result != -1) { ++dequeue_waiters_; guard.release (); result = notempty_.acquire (timeout); guard.acquire (); }

if (result == -1) { if (dequeue_waiters_ > 0) --dequeue_waiters_; if (errno == ETIME) errno = EWOULDBLOCK; return -1; }

Page 223: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

223

Using ACE_Thread_Semaphore (8/8)

// Remove the first message from the queue. int queued_messages = dequeue_head_i (first_item);

// Only signal if we've fallen below the low water mark. if (cur_bytes_ <= low_water_mark_ && enqueue_waiters_ > 0) { enqueue_waiters_--; notfull_.release (); } return queued_messages; // <guard> destructor releases <lock_>}

Page 224: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

224

ACE Condition Variable Classes (1/2)Motivation

•Condition variables allow threads to coordinate & schedule their processing efficiently

•Condition variables are more appropriate than mutexes or semaphores when complex condition expressions or scheduling behaviors are needed

•e.g., condition variables are often used to implement synchronized message queues that provide “producer/consumer” communication to pass messages between threads

usesuses 2

Request Queue

put()get()

ACE_Thread_Mutex

acquire()release()

Producer Thread

ACE_Thread_Condition

wait()signal()broadcast()

ConsumerThread

<<put>> <<get>>

Page 225: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

225

Class Capabilities

•The ACE_Condition_Thread_Mutex uses the Wrapper Façade pattern to guide its encapsulation of process-scoped condition variable semantics

•The ACE_Null_Condition is a zero-cost class whose interface conforms to the ACE_Condition_Thread_Mutex

ACE Condition Variable Classes (2/2)

Page 226: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

226

Using ACE_Condition_Thread_Mutex (1/3)

Class ACE_Recursive_Thread_Mutex {public: ACE_Recursive_Thread_Mutex (const char *name, void *arg); int acquire (); int release (); // ...private: int nesting_level_; ACE_thread_t owner_id_; ACE_Thread_Mutex lock_; ACE_Condition_Thread_Mutex lock_available_;};

Demonstrating use of ACE condition variable wrapper facade

ACE_Recursive_Thread_Mutex class definition

ACE also has an ACE_Condition_Recursive_Thread_Mutex class!

Page 227: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

227

Using ACE_Condition_Thread_Mutex (2/3)

ACE_Recursive_Thread_Mutex::ACE_Recursive_Thread_Mutex (const char *name, void *arg) : nesting_level_ (0), owner_id_ (0), lock_ (name, arg), // Initialize the condition variable. lock_available_ (lock_, name, arg) { }

int ACE_Recursive_Thread_Mutex::release (){ // Automatically acquire mutex. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1);

nesting_level_--; if (nesting_level_ == 0) { lock_available_.signal (); // Inform waiters the lock is free. owner_id_ = 0; } return 0; // Destructor of <guard> releases the <lock_>.}

Demonstrating use of condition variables

Page 228: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

228

Using ACE_Condition_Thread_Mutex (3/3)

1 int ACE_Recursive_Thread_Mutex::acquire () 2 { 3 ACE_thread_t t_id = ACE_OS::thr_self (); 4 5 ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); 6 7 if (nesting_level_ == 0) { 8 owner_id_ = t_id; 9 nesting_level_ = 1;10 }11 else if (t_id == owner_id_)12 nesting_level_++;13 else {14 while (nesting_level_ > 0)15 lock_available_.wait ();16 17 owner_id_ = t_id; 18 nesting_level_ = 1;19 }20 return 0; 21 }

Demonstrating use of condition variables

Page 229: C++ Network Programming Mastering Complexity with ACE & Patterns Dr. Douglas C. Schmidt d.schmidt@vanderbilt.edu schmidt/tutorials-ace.html

229

•Patterns & frameworks for concurrent & networked objects•www.cs.wustl.edu/~schmidt/POSA/

•ACE & TAO open-source middleware•www.cs.wustl.edu/~schmidt/ACE.html•www.cs.wustl.edu/~schmidt/TAO.html

•ACE research papers•www.cs.wustl.edu/~schmidt/ACE-papers.html

•Extended ACE & TAO tutorials•UCLA extension, Feb, 2007•www.cs.wustl.edu/~schmidt/UCLA.html

•ACE books•www.cs.wustl.edu/~schmidt/ACE/

Additional Information