E-Genting Programming Competition 2005

Public Lecture by Jonathan Searcy

21 January 2006

Competition Questions

No. Title and Description Marks

1. Mytel MobileA commercial reporting program with data accessing complications.

250

2. Project Scheduling ProgramA problem in parsing, logic and data formatting.

350

3. Failsafe StorageA class for accessing a small amount of non-volatile storage.

100

4. Vending Machine MultiplexerA program to allow a vending machine to accept money from both a coin acceptor and a smart card reader

500

Failsafe Storage

• An application programmer required a small amount of failsafe storage;

• The storage had to be completely updated or completely unchanged;

• If the power failed during a write, the update had to be rolled back.

Failsafe Storage, the Design

Where did the Contestants have Difficulty?

• getting the data into and out of the disk files;

• calculating and validating the CRC;

• evaluating the modulo 232 comparison.

The Nature of Computer Storage

The Nature of Magnetic Storage

FM Encoding Scheme for Magnetic Media

What is Formatting?

Which Functions should We Use?

Language Unformatted I/O Formatted I/O

Javaread, write (various classes)

print, println, readInt, etc.

Cread, write, fread, fwrite

fprintf, fscanf

C++ifstream::read, ofstream::write

<<, >>

Cyclic Redundancy Check

class Checksums {// ...static short crc16 (

byte [] data, int length);// ...

}

If the CRC calculated by the above function is appended to the data in little-endian byte order, the CRC of the concatenated data and CRC is zero.

Modular Arithmetic

Properties of Modulo-2 Arithmetic

(a + b) mod 2 = (a – b) mod 2

For example:

(5 + 3) mod 2 = 8 mod 2 = 0 = 2 mod 2 = (5 – 3) mod 2

And therefore (by letting a equal zero):

–b mod 2 = b mod 2

What is a CRC?A CRC is the modulo-2 remainder of the product of a polynomial representation of the data and the order of magnitude of a CRC polynomial divided by the CRC polynomial.

For example, if the data to be hashed is the bit string 1010, the polynomial representation of the data would be:

1x3 + 0x2 + 1x1 + 0x0

= x3 + x

Applying the CRC polynomial for the USB CRC, which is:

x5 + x2 + 1

The product of the polynomial representation of the data and the order of magnitude of the CRC polynomial would be:

(x3 + x) x5

And the USB CRC of 1010 would be:

((x3 + x) x5) % (x5 + x2 + 1) (mod 2)

Where % is the remainder operator.

Hand-Calculating a CRC

Divisor Dividend

x5 x4 x3 x2 x1 x0 | x8 x7 x6 x5 x4 x3 x2 x1 x0

1 0 0 1 0 1 | 1 0 1 0 0 0 0 0 0

1 0 0 1 0 1

1 1 0 1 0 0 0

1 0 0 1 0 1

1 0 0 0 1 0

1 0 0 1 0 1

0 0 1 1 1

Therefore the USB CRC of 1010 is 00111

CRC(x + CRC(x)) = 0

Divisor Dividend

x5 x4 x3 x2 x1 x0 | x13 x12 x11 x10 x9 x8 x7 x6 x5 x4 x3 x2 x1 x0

1 0 0 1 0 1 | 1 0 1 0 0 0 1 1 1 0 0 0 0 0

1 0 0 1 0 1

1 1 0 1 1 1 1 0 0 0 0 0

1 0 0 1 0 1

1 0 0 1 0 1 0 0 0 0 0

1 0 0 1 0 1

0

Structure of the Disk File

The following items need to be stored:1. serial number;2. data;3. CRC.

We can store them in six alternative orders:

serial, data, CRC data, serial, CRC CRC, serial, data

serial, CRC, data data, CRC, serial CRC, data, serial

serial, data, CRC CRC, serial, data

Writing to the Disk FilebyteCnt = 4 + length + 2;byteArr = new byte [ byteCnt ];byteArr[0] = (byte)(serialNo & 0xff);byteArr[1] = (byte)(serialNo >> 8 & 0xff);byteArr[2] = (byte)(serialNo >> 16 & 0xff);byteArr[3] = (byte)(serialNo >> 24 & 0xff);for (i = 0; i < length; i++)

byteArr[4+i] = data[i];crc = Checksums.crc16 (byteArr, 4+length);byteArr[4+length] = (byte)(crc & 0xff);byteArr[4+length+1] = (byte)((crc >> 8) & 0xff);

Serial Number Comparison

The serial number comparison was complicated by two factors:

1. Signed integers are stored in two’s complement format;

2. The serial numbers had to be compared in a modular domain.

Two’s ComplimentThree-Bit Integer

Bit Pattern Unsigned value

Signed value

0 0 0 0 0

0 0 1 1 1

0 1 0 2 2

0 1 1 3 3

1 0 0 4 – 4

1 0 1 5 – 3

1 1 0 6 – 2

1 1 1 7 – 1

Comparing Serial Numbers

Selecting the more Recent Serial Number

The question paper specified that:

If ((A – B) mod 232) < 231, then A is the more recent serial number, otherwise B is the more recent serial number.

Converting to the 3-bit domain:

If ((A – B) mod 23) < 22, then A is the more recent …

Applying to the example:

If ((0 – 7) mod 8) < 4, then 0 is the more recent …

If (1 < 4), then 0 is the more recent …

Therefore serial no 0 is more recent than serial no 7.

Coding the Serial Number Comparison

The formula in the question paper:

((A – B) mod 232) < 231

Many contestants converted it to:

(A – B) % pow(2,32) < pow(2,31) WRONG

(A – B) % 2^32 < 2^31 WRONG

The correct way to code the formula is:

((A – B) & 0xffffffff) < 0x80000000 RIGHT

I.e.:

x mod 2n ≡ x & (2n – 1)

InitialisationWrite-function pseudo-code:

1. If the class is not initialised:a. Execute a read cycle to

discoverthe latest serial number andwhich of the two files was lastupdated.

2. Increment the serial number.3. Create a byte array of the data to be

written.4. Write the byte array to the file that was

least recently updated.

Summary

• The nature of computer storage and the system calls for transferring binary data.

• The distinction between formatting and input/output.

• The nature of a CRC and some of its properties.• The effects of storing data in different sequences.• The nature and consequences of the two’s

complement integer format.• Various characteristics of modular arithmetic.• The need to consider initialisation.

Mytel Mobile• Mytel Mobile operates a mobile telephone service.• Subscribers can earn points by making calls on

their mobile phones.• Mytel wants to set up a point redemption

arrangement with an airline so that its subscribers can exchange points for flights.

• To convince the airline to enter into the arrangement, Mytel needs a report that displays the destination and frequency of its subscribers’ flights.

• Mytel has a database that contains information about each call, including the identity of the subscriber, the time of the call and the city from which the call was made.

Data Dictionary1. date and time the report was generated;

2. reporting period (from-date and to-date);

3. for each point balance range:

a. the point balance range;

b. number of subscribers;

c. for each base city:

i. for each other city:

1. number of trips;

d. total number of trips;

4. totals for all point balance ranges.

1 2 3 4 5 1...5....0....5....0....5....0....5....0....5....0....5.DD-MM-YY HH:MM TRIP ANALYSIS REPORT PAGE X for XX-XX-XX to XX-XX-XX

No of Base Number of trips to Total noPoint range subs city KUL SIN BKK of trips

0 to 99,999 X,XXX KUL 0 X,XXX X,XXX X,XXX SIN X,XXX 0 X,XXX BKK X,XXX X,XXX 0

100k to 299k X,XXX KUL 0 X,XXX X,XXX X,XXX SIN X,XXX 0 X,XXX BKK X,XXX X,XXX 0

300k or more X,XXX KUL 0 X,XXX X,XXX X,XXX SIN X,XXX 0 X,XXX BKK X,XXX X,XXX 0

Total X,XXX KUL 0 X,XXX X,XXX X,XXX SIN X,XXX 0 X,XXX BKK X,XXX X,XXX 0

05-01-06 15:24 TRIP ANALYSIS REPORT PAGE 1 for 01-01-04 to 31-12-04  No of Base Number of trips to Total noPoint range subs city KUL SIN BKK of trips 0 to 100k 2,438 KUL 0 2,510 2,504 14,367 SIN 2,351 0 2,392 BKK 2,336 2,274 0 100k to 299k 5,025 KUL 0 5,165 5,139 29,449 SIN 4,837 0 4,823 BKK 4,830 4,655 0 300k or more 2,537 KUL 0 2,574 2,530 14,983 SIN 2,428 0 2,433 BKK 2,491 2,527 0 Total 10,000 KUL 0 10,249 10,173 58,799 SIN 9,616 0 9,648 BKK 9,657 9,456 0

Database Schemacreate table subscribers (

subId integer not null,subName char(40) not null,subBal integer not null

);create table callRegister (

crSubId integer not null,crFromCity char(3) not null,crCalledNo char(20) not null,crConnectTime integer not null,crDisconTime integer not null

);

Singleton Selectexec sql begin declare section; long idVar; long balVar;exec sql end delcare section;//...idVar = 100321;exec sql select subBal into :balVar from subscribers where subId = :idVar;if (SQLCODE != 0) // subscriber not found

JDBCStatement stmt; // JDBC statementResultSet rs; // JDBC result setint idVar; // Sub id variableint balVar; // Point bal variable//...idVar = 100321;rs = stmt.executeQuery (

"select subBal from subscribers" +"where subId = " + idVar

);if ( ! rs.next())

{/* row not found */}balVar = rs.getInt ("subBal");

Errors that an embedded SQL pre-processor can detect at compile time that raw JDBC calls cannot detect until run time:

•incorrectly spelled reserved words such as ‘select’ and ‘where’;

•syntax errors in SQL statements;

•misspelled column names;

•ambiguous column names (i.e. a reference to an unqualified column name that is a valid column name in two tables in the query);

•incompatible database column and host variable data types.

Cursorsexec sql declare callCur cursor for

select crSubId, crFromCity, crConnectTimefrom callRegisterorder by crConnectTime;

exec sql open callCur;for (;;) {

exec sql fetch callCurinto :idVar, :cityVar, :timeVar;

if (SQLCODE != 0) break; 

// Process the row}exec sql close callCur;

Selecting the Base City

The reporting program must determine each subscriber’s base city by assuming that the subscriber’s base city is the city from which he made the greatest number of calls.

Counting Trips

The reporting program must determine the number of trips made by each subscriber by scanning the call register and counting a new trip whenever a call is made from a new city other than the base city.

Red Herring

From the Concise Oxford Dictionary:red herring ● n. 1 a dried smoked herring. 2 a misleading clue or distraction. [so named from the practice of using the scent of red herring in training hounds.]

Process for Each SubscriberSet the current city to null;

For each call the subscriber made in connection time order:

Increment the call count for the from-city;

If the from-city is different to the current city:

Increment the trip count for the from-city;

Set the current city to the from-city;

End if;

End for.

But the Call Register is Huge

“It would take several years to execute the following SQL statement for each customer on the database”

select crConnectTime, crFromCityfrom callRegisterwhere crSubId = :subId and

crConnectTime between:fromDate and :toDate

Order by crConnectTime;

Parallel Processes

Process PseudocodeCentral Database Reader:

For each call register row in the reporting period:Fetch the call register row from the database;Send the call register row to the Subscriber’sCall and Trip Counter;

End for. Subscriber’s Call and Trip Counter:

Set the current city to null;Loop:

Receive a call register row from the CentralDatabase Reader;Increment the call count for the from-city;If the from-city is different to the current city:

Increment the trip count for the from-city;Set the current city to the from-city;

End if;End loop.

Client and Server-Mode Processes

• Client-mode processes execute top-to-bottom. When they need to communicate with another process they call a function to transfer the data.

• Server-mode processes wait in an idle state to be called by a client-mode process. When called, they perform whatever function is needed and then return control to the client-mode process.

Client-Client Standoff

Conversion to a Finite State MachineClient-Mode Pseudo-Code State Processing[INITIAL state]Set the current city to null;Loop: [RECEVING state] Receive a call register row from the Central Database Reader; Increment the call count for the from-city; If the from-city is different to the current city: Increment the trip count for the from-city; Set the current city to the from-city; End if;End loop.

INITIAL Set the current city to null;

State = RECEVING.

RECEVING Increment the call count for the from-city;If the from-city is different to the current city: Increment the trip count for the from- city; Set the current city to the from-cityEnd if.

Summary

• Data dictionary interpretation;• Report layout interpretation;• Database accessing using SQL;• Dataflow analysis;• Using parallel processing to overcome

serial throughput constraints;• Converting top-to-bottom pseudo-code

into an event-driven finite state machine.

Project Scheduling Program

• accept an input file name as an argument;

• read and analyse the input file;

• generate a project schedule listing.

Input File Formatstart 30-08-05; # Project start dateholidays { # Public holidays

31-08-05, # Hari merdeka01-11-05 # Deepavali

}tasks { # Task list

func "Functional specification“ MDJ 10;design "Internal design“ AB 12

needs func;screens "Program data entry screens“ JOJ 8

needs design;reports "Program reports“ AB 10

needs design;tests "Acceptance tests“ MDJ 12

needs screens, reports;}

BNF of the Input Syntaxinput_file:

start_stmt holidays_stmt tasks_stmtstart_stmt:

start start_date ;holidays_stmt:

holidays { holiday_listopt }

holiday_list:holiday_dateholiday list , holiday_date

tasks_stmt:tasks { task_listopt }

…

Report Layout

1 2 3 4 5 1...5....0....5....0....5....0....5....0....5....0....5.DD-MM-YY HH:MM PROJECT SCHEDULE LISTING PAGE X for X------------------X Task name Description Person Days Start Finish X--------X X---------------X X-X X-X XX-XX-XX XX-XX-XXX--------X X---------------X X-X X-X XX-XX-XX XX-XX-XXX--------X X---------------X X-X X-X XX-XX-XX XX-XX-XX ----Project duration X—-X

Dataflow Analysis

Token Types• words (keywords, task names, person

identifiers, numbers and dates);

• strings (task descriptions);

• semicolon;

• comma;

• open brace;

• close brace;

• end-of-file.

Lexical AnalyserSkip white space and comments;If the look ahead character is a letter, digit, underscore or dash:

Do:Append the look-ahead character to the word;Read the next look-ahead character;

Until the look-ahead character is not a valid wordcharacter;Return a WORD token;

Else if the look-ahead character is a double quote:Read the string from the input file;Return a STRING token;

Else if the look-ahead character is a semicolonRead the next look-ahead character;Return a SEMICOLON token;

Else …

Syntax Diagram

Parser for the Holidays Statementholidays = new TreeSet ();if (tokenType != TT_WORD || !tokenValue.equals("holidays")) throw new Error ("no holidays keyword");readToken ();if (tokenType != TT_OPEN) throw new Error ("no open brace");readToken ();if (tokenType != TT_CLOSE) { for (;;) { if (tokenType != TT_WORD) throw new Error ("date expected"); holidays.add (getDateValue()); readToken (); if (tokenType != TT_COMMA) break; readToken (); } if (tokenType != TT_CLOSE) throw new Error ("no close brace");}readToken ();

The Schedulerprivate voidscheduleTask ( Task task) // Task to be scheduled{ int startDay; // Start day number Task needTask; // Needed task reference int i; // General purpose index  startDay = 0; for (i = 0; i < task.taskNeeds.size(); i++) { needTask = (Task) task.taskNeeds.get(i); if ( ! needTask.taskScheduled) scheduleTask (needTask); if (needTask.taskFinishDay >= startDay) startDay = needTask.taskFinishDay + 1; } task.taskStartDay = startDay; task.taskFinishDay = startDay + task.taskDuration - 1; task.taskScheduled = true;}

Creating Filler TasksFor each person:

Create a list of the tasks assigned to the person sortedby start date;Set a previous task variable to the first task assigned to the person;For each subsequent task assigned to the person:

If there is a gap between the finish date of theprevious task and the start date of the currenttask:

Create a filler task to fill in the gap;Set the previous task to the current task;

End for;End for.

Mapping Day Numbers to Physical Dates

Brute force and ignorance:

Start on the start day and count days that are not public holidays or weekends until you get to the day number you need.

Summary• Reading and understanding BNF.• Dataflow analysis of a straightforward

application program;• Creating a server-mode lexical analyser;• Creating a client-mode parser;• Using a recursive or iterative process to

order tasks based on an ordering rule;• Dealing with irritations such as filler tasks

and public holidays;• Converting an internal data structure into

a legible report.

Vending Machine Multiplexer

• A public health authority has several hundred vending machines that sell health-related paraphernalia;

• The existing vending machines can be connected to a coin acceptor or an e-purse interface, but not both at the same time, because the vending machine only has one port;

• The task is to design a multiplexer that will allow the vending machine to be connected to both a coin acceptor and an electronic purse interface at the same time.

The Protocol

Positive Acknowledgement with RetransmissionLoss of Request

Positive Acknowledgement with RetransmissionLoss of Response

Dataflow Analysis

The Robustness Principle

Be conservative in what you do, be liberal in what you accept from others.

Jon Postel, RFC 793, September 1981.

Client Interface

Loop:Receive a request from the client device;Send the request to the Server Interface;Wait to receive a response from the Server Interface;Send the response to the client device;

End loop.

Unpacking a Request

serial = (long)msgBuf[0] << 24 | (long)msgBuf[1] << 16 | (long)msgBuf[2] << 8 | (long)msgBuf[3];value = (long)msgBuf[4] << 24 | (long)msgBuf[5] << 16 | (long)msgBuf[6] << 8 | (long)msgBuf[7];

Listener

Loop:Accept a new connection;Send an accepted connection message to theServer Interface;

End loop.

Request Queue

Server InterfaceLoop:

Receive a message from the Request Queue;If the message is an accepted connection:

Initiate a new Client Interface to service the connection;Else if the message is a client request:

If the client request has a new serial number:Allocate a new serial number to the request;Send the request to the vending machine;Receive a response from the vending machine;Restore the original serial number;Return the response to the Client Interface;

Else:Return the old response to the Client Interface;

End if;End if.

End loop.

Surviving Power Failures• Data to be saved:

– last serial number sent to the vending machine;– for each client:

• client state (no request, processing, done),• last request,• last response.

• Save the data before submitting a new request to the vending machine;

• Save the data after receiving a response from the vending machine but before sending the response to the client;

• If a request was being sent to the vending machine at the time of the failure, resend the request when the multiplexer is restarted.

Summary• How to use the operating system infrastructure to

communicate over a TCP connection;• A basic understanding of how to resynchronise a

connection after a communications failure;• How to divide a large problem into workable components

by the use of dataflow analysis;• How to initiate and use concurrent processes or threads;• How to pack and unpack binary messages;• How to set up a message queue and/or how to use

semaphores to control access to shared resources;• How to make a system survive transient power outages

and communications failures.