oneone 22 oneoneERLANG USER CONFERENCEERLANG USER CONFERENCE

3rd October 20003rd October 2000

Use of Erlang Use of Erlang for IN Service for IN Service CreationCreation

Sean Hinde Sean Hinde IN Services Manager IN Services Manager

One 2 OneOne 2 One

A Bit about Intelligent NetworksA Bit about Intelligent Networks

IN provides a high level abstraction of the functionality of a Telephony Network by defining a set of building blocks which make up a call, and allowing them to be put together in new and interesting ways.

It should not be possible to break the network by doing something wrong in an IN service!

What is it?

A Bit About Intelligent NetworksA Bit About Intelligent Networks

Architecture and Abbreviations

SCP (Service Control Point)

MSC/SSF (Mobile Switching Centre/Service Switching Function)

C7 INAP Signalling

SDP (Service Data Point)

A Bit about Intelligent NetworksA Bit about Intelligent Networks

Put a new box in the network (SCP) and interface this into the existing switching network using a standard (ish) protocol.

Provide a GUI based Service Creation Environment for the SCP.

The SCP contains the real time data base and logic program which controls the SSF.

Optional Service Data Point (SDP) - external database for holding large amounts of customer data

How does it work?

Properties of IN and GSM systemsProperties of IN and GSM systems

Real Time Response - a few millisecs

Massively High Availability - 99.999%

Enormous complexity - GSM!

Great Flexibility - Service differentiation

Is it impossible to achieve all these at once??

One 2 One SDP ImplementationOne 2 One SDP Implementation

One 2 One started out with a home grown SDP implementation which existed before the introduction of IN into the network.

Based on a commercial relational database/ big multiprocessor UNIX machine.

The SCP interface and some service logic is home grown and coded in straight C.

Uses a separate c7 to TCP/IP protocol converter

IN Service fails safe if SDP times out

Experiences of own SDPExperiences of own SDP

One read failure led to others - multithread!

Random explosions in the number of threads - file descriptors..

Commercial Failover never worked

CPU load 100% - buying more processors didn’t help - spinlock problem.

Still slows down - Full time DBAs to keep it running/optimised.

Experiences of own SDP - 2Experiences of own SDP - 2

Now have team of 20 IT contractors writing C code

Conclusion from the telecoms designer perspective was that the experiment was not a great success. Hard and expensive to do well.

What was the alternative though?

Commercial SDPs not flexible enough

SCP internal database not suitable/flexible enough

Needed a Telecoms tool running on IT platforms with IT flexibility...

ERLANGERLANG

Browsing Ericsson Web site - stumbled across a new platform - Open Telecom Platform..

It seemed to have many of the properties we were looking for, but what actually was it??

Downloaded and read more and more

Some time approved to investigate further

ERLANG - The learning ERLANG - The learning experienceexperience

Started out as a non programmer (except for Ericsson SMAS IN SCE) and learned by reading the Erlang book part 1 PDF and the source.

Three weeks to get multithreaded TCP/IP server running.

Real confidence that code would carry on working.

Bought commercial licence and got The Book.

ERLANG - Initial ConclusionsERLANG - Initial Conclusions

The conclusion from the initial investigations was that there was minimal work required to turn Erlang/OTP into a high availability SDP with sophisticated logic.

SCP TCP/IP Interface and service logic wrapper Alarm Interface - TCP/IP (now SNMP) Stats Interface - Files Provisioning - Same as SCP Event logging via Socket. Web based monitoring Service logic written in very few lines of easily testable

and maintainable code

Current ApplicationsCurrent Applications

Corporate VPN service Short Code dialling Black/White lists Short code presentation number Intranet based provisioning system (uses enhanced esp) 3 Nodes

WAP RADIUS Whole customer base - 24 nodes Authentication with non standard logic mnesia table split into 64 fragments across 8 pairs

Soon to be in serviceSoon to be in service

Mini Prepayment system Checks usage at start of call and writes new total at the end Intranet Page to view customers with highest usage Uses several ordered set tables to achieve ordered queries 3 Nodes

Number Translation Another with whole customer base - 22 nodes All voicemail calls will absolutely rely on it. mnesia table split into 64 fragments across 8 pairs

RADIUS Management RADIUS Management Customer DetailsCustomer Details

RADIUS Management RADIUS Management TCP/IP Client statusTCP/IP Client status

RADIUS Management RADIUS Management Current StatsCurrent Stats

RADIUS Management RADIUS Management System AlarmsSystem Alarms

Operational ExperienceOperational Experience

First 5 months of WAP system - several node failures, no missed calls.

The day after sending the paper for publication - disaster.

Major LAN problems affected Solaris/OTP so badly that heart or h/w watchdog kicked in on several nodes.

Erlang went into cyclic restart on several nodes due to corrupt SNMP ets disk file preventing clean startup.

LAN up and down - so many partitioned network entries we lost count.

Failed to recover database consistency despite repeated attempts to set_master_nodes, mnesia deadlock

Disk Failure on one machine meant we couldn’t load fallback until this was also fixed.

Lessons from OutageLessons from Outage

Plan to put a check to stop cyclic restarts.

Need to apply an SNMP fix which was already available - and any others which may affect startup.

More thought needed about how to fix partitioned networks.

Proved that restore from fallback works faultlessly :-) 40 mins for 370M backup file.

Having Said That...Having Said That...

Very few problems have been found with the whole system over two years of use.

Wishes:

Multipath distribution mechanism or something similar.

Better memory handling - Occasional excessive memory usage with potentially serious consequences.

Mnesia online table transform

Future Plans Future Plans

Plan to continue the use of Erlang/OTP in the Telecoms domain in One 2 One:

Mass SMS Sending Tool More functionality for existing systems Tuxedo interface.. Possible migration of existing work into 3G

architecture Other odd projects!

ConclusionsConclusions

Erlang/OTP has proved itself to be well up to the task of being an IN Service Data Point.

The Erlang language has proved to be an ideal and safe environment for writing and maintaining IN service logic.

It has also proved very quick and easy to write simple Internet protocol front ends to mnesia with no software faults found to date.

Quote from Support Engineer:

“We didn’t know much about how to fix it because it never goes wrong”...

oneone 22 oneone