solving neural odes using fixed-point arithmetic with

Solving neural ODEs using fixed-point arithmetic with stochastic roundingMantas [email protected] with: Michael Hopkins, Dave Lester, Steve Furber @ CS Dept.

NLA group meetingManchester, 24 October 2019

mailto:[email protected]

2/23

Mantas Mikaitis Stochastic rounding in fixed-point arithmetic

https://en.wikipedia.org/wiki/White_stork

3/23


• Hardware and software for simulating large-scale biological neural networks• 1Million ARM968 (2004 ARM processors)

The SpiNNaker project

Images: Furtak, S. (2018). Neurons. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology

4/23


Generalized fixed-point number representation

2"# 2"$ …… 2# 2% .… …𝑠

𝑖 𝑝• 𝑖 - number of integer bits• 𝑝 - number of fractional bits (precision)• Range:

[−2+, 2+ − 2"-] (signed),[0, 2+ − 2"-] (unsigned)

• 𝜖 = 2"- (machine epsilon)• 𝜖 is a gap between any two consecutive

values• Accuracy is measured in 𝜖.• The maximum error is: 2

$

Converting to decimals:Given a binary number: < 𝑠|𝑢, 𝑖, 𝑝 > − 𝑠𝐼+"#𝐼+"$ ⋯ 𝐼%. 𝐹#𝐹$ ⋯𝐹-,if signed format and 𝑠 is set

𝑣𝑎𝑙𝑢𝑒 = ?@A%

+"#

𝐼@2@ +?CA#

-

𝐹C2"C − 𝑠2+, 𝑒𝑙𝑠𝑒 𝑣𝑎𝑙𝑢𝑒 = ?@A%

+

𝐼@2@ +?CA#

-

𝐹C2"C

𝑤

5/23


GCC types accum and long fract

• accum <s, 16, 15>:𝜖 = 2"#E ≈ 0.0000305176…

Range: [−2#M = −65536, 2#M − 2"#E ≈ 65535.99996948… ]

0 0.0000305176 0.00006103516-0.0000305176

• long fract <s, 0, 31>:𝜖 = 2"Q# ≈ 0.000000000465661

Range: [−1, 2% − 2"Q# ≈ 0.999999999534339… ]

00.000000000465661

0.000000000931322−0.000000000465661

𝜖

6/23


Izhikevich neuron model: Cortical spiking patterns (Izhikevich 2003)

(RK2 Midpoint ODE solver,Hopkins & Furber, 2015)

(on spike: 𝑉 = 𝑐,𝑈 = 𝑈 + 𝑑)

7/23


Error measure: spike time lag/lead (when I=4.775 at 60th ms and kept constant)

Arithmetic errors in spike lag

8/23


Errors when I=4.775 at 60th ms and kept constant (relative to FP64)

9/23


Some improvements to the constants• Instead of 0.04 -> 0.040008544921875 (error of 0.28𝜖).• GCC rounds down by default, returning an error of 0.72𝝐• Vague specification in the ISO/IEC 18037 standard:

“If the source value cannot be represented exactly by the fixed-point type, the source value is rounded to either the

closest fixed-point value greater than the source value (rounded up) or to the closest fixed-point value less than

the source value (rounded down).”

• Use fixed-point type < 𝑢, 0, 32 >, i.e. 0.040000000037252902984619140625.

• Requires multiplications to be done in mixed precision.

10/23

Mantas Mikaitis Stochastic rounding in fixed-point arithmeticM

SB

LSB

×

=

LSB

MSB

≫ (Round and saturate) Use these bits for rounding.

32-bit accum

64-bit answer

binary point

Fixed-point multiplier

11/23


Given the output from multiplier:

answer somewhere in this gapaccum range

ignore

Round-down (RD)

12/23



?


0 1

Round-to-nearest (RN)

13/23


Reduced error: running longer (0.1ms timestep).

14/23


Stochastic rounding (SR)

Given a real number 𝑥, a random number 𝑃 ∈ 0,1 from a uniform distribution and a fixed-point destination format < 𝑠, 𝑖, 𝑝 > with 𝜖 = 2"-,

Over many roundings:𝔼 𝑆𝑅 𝑥,< 𝑠, 𝑖, 𝑝 > = 𝑥.

𝑆𝑅 𝑥, < 𝑠, 𝑖, 𝑝 > =𝑥 𝑖𝑓 𝑃 ≥

𝑥 − 𝑥𝜖

,

𝑥 + 𝜖 𝑖𝑓 𝑃 <𝑥 − 𝑥𝜖

.

15/23


Given the output from the multiplier:


Round-off bitsIf < round-off valueround up, else round down.

Use these bits as probability of

rounding up, [0,1).

SR at bit level

16/23


SR algorithm: by addition

17/23


Testing method of Izhikevich ODE solutions

• Four ODE solvers: RK2 Midpoint, RK2 Trapezoid, RK3 Heun, Chan-Tsai

• Two different neuron types (regular and fast spiking -RS/FS)

• Five arithmetics: IEEE 754 double (reference), float, fixed-point {round-down, round-to-nearest, stochastic}

• Run for ~1min to produce around 600 spikes.• For SR, repeat experiment 100 times with different

PRNG seeds• Report mean and std.dev. spike lag of each spike.

18/23


Results: 2nd order solvers (h=0.1ms)

Spike number

Spik

e la

g/le

ad c

ompa

red

to d

oubl

e re

f (m

s).

19/23


Results: 3rd order solvers (h=0.1ms)

Spike number

Spik

e la

g/le

ad c

ompa

red

to d

oubl

e re

f (m

s).

20/23


Comparison of SR resolutions


Use SOME of these bits as probability

of rounding up, [0,1).

21/23


Psuedo-random number generation

• Marsaglia’s KISS64 (expensive but passes randomness tests)• Linear congruential generator (modular arithmetic by

overflow of 32-bit registers):

• For our tests, no significant difference (mean & std.dev.)!

22/23


Efficiency

• Fixed-point + SR 2.6× faster than soft-float and 4.2× faster than soft-double running RK2 Midpoint

• Previously: use fixed-point for speed, soft-float for accuracy• Now: use fixed-point for speed, fixed-point+SR for accuracy

23/23


Summary

• Stochastic rounding is very beneficial in solving this ODE• Fixed-point arithmetic similar in accuracy to double-precision• Only 6 random bits per multiplication required• Implemented in hardware on SpiNNaker2 (64/32bit -> 32/16bit

and float -> bfloat16).• Next questions:

1. Where else does SR apply?2. Is the pseudo-randomness in arithmetic a difficulty?3. Is SR less expensive than other approaches for similar

problems?

24/23


References

• Izhikevich. Simple Model of Spiking Neurons. 2003. IEEE Tran. Neur. Net.

• Hopkins & Furber. Accuracy and Efficiency in Fixed-Point Neural ODE Solvers. 2015. https://www.ncbi.nlm.nih.gov/pubmed/26313605

• Hopkins, Mikaitis, Lester & Furber. Stochastic rounding and reduced-precision arithmetic for solving neural ODEs. 2019. https://arxiv.org/pdf/1904.11263.pdf

• PD ISO/IEC TR 18037:2008, Programming languages. C. Extensions to support embedded processors. 2008. https://bsol.bsigroup.com/Bibliographic/BibliographicInfoData/000000000030185055

https://www.ncbi.nlm.nih.gov/pubmed/26313605

https://arxiv.org/pdf/1904.11263.pdf

https://bsol.bsigroup.com/Bibliographic/BibliographicInfoData/000000000030185055

25/23


References

• Higham & Srikara. Simulating low precision floating point arithmetic. 2019. http://eprints.maths.manchester.ac.uk/2692/

http://eprints.maths.manchester.ac.uk/2692/

solving neural odes using fixed-point arithmetic with

Documents