mu uuuu ui iiui iiui mu lull iuu mu uui iuu iuui uu uii mi(75) inventors: dwayne westinskow, salt...

mu uuuu ui iiui iiui mu lull iuu mu uui iuu iuui uu uii mi

(12) United States Patent (1o) Patent No.: US 7,654,966 B2Westinskow et al. (45) Date of Patent: Feb. 2, 2010

(54) METHOD AND APPARATUS FOR (56) References CitedMONITORING DYNAMICCARDIOVASCULAR FUNCTION USING U.S. PATENT DOCUMENTS

N-DIMENSIONAL REPRESENTATIVES OF 4,598,706 A * 7/1986 Darowsld et al. ...... 128/205.24CRITICAL FUNCTIONS 4,752,893 A 6/1988 Guttag et al.

4,772,882 A 9/1988 Mica](75) Inventors: Dwayne Westinskow, Salt Lake City, 4,813,013 A 3/1989 Dunn

UT (US); James Agutter, Salt Lake 4,823,283 A 4/1989 Diehm et al.City, UT (US); Noah Syroid, Salt Lake 4,915,757 A 4/1990 RandoCity, UT (US); David Strayer, Salt Lake 4,939,757 A * 7/1990 Nambu .......................... 378/8City, UT (US); Robert Albert, Salt Lake 4,984,158 A * 1/1991 Hillsman ............... 128/200.14City, UT (US); S. Blake Wachter, 5,021,976 A 6/1991 Wexelblat et al.Huntsville, UT (US); Frank Drews, Salt 5,121,469 A 6/1992 Richards et al.Lake City, UT (US) 5,129,401 A * 7/1992 Corenman et al. .......... 600/529

(73) Assignee: University of Utah Research5,262,944 A 11/1993 Weisner et al.

Foundation, Salt Lake City UT (US) 5,317,321 A 5/1994 Sass5,333,106 A * 7/1994 Lanpher et al . ............. 600/538

(*) Notice: Subject to any disclaimer, the term of this 5,491,779 A 2/1996 Bezjian

patent is extended or adjusted under 35 5,515,301 A 5/1996 Corby, Jr. et al.

U.S.C. 154(b) by 1393 days.

(21) Appl. No.: 10/269,423 (Continued)

(22) Filed: Oct. 11, 2002Primary Examiner Charles A Marmor, IIAssistant Examiner Navin Natnithithadha

(65) Prior Publication Data (74)Attorney,Agent, orFirm—Thorpe North& Western LLP

US 2003/0227472 Al Dec. 11, 2003 (57) ABSTRACT

Related U.S. Application DataA method, system, apparatus and device for the monitoring,

(63) Continuation-in-part of application No. 09/457,068, diagnosis and evaluation of the state of a dynamic pulmonaryfiled on Dec. 7, 1999, now abandoned. system is disclosed. This method and system provides the

(60) Provisional application No. 60/328,880, filed on Oct. processing means for receiving sensed and/or simulated data,12, 2001. converting such data into a displayable object format and

displaying such objects in a manner such that the interrela-(51) Int. Cl. tionships between the respective variables can be correlated

A 61 5108 (2006.01) and identified by a user. This invention provides for the rapid(52) U.S. Cl . ........................ 600/532; 600/538; 600/529 cognitive grasp of the overall state of a pulmonary critical

(58) Field of Classification Search ................. 600/485, function with respect to a dynamic system.

600/529-543; 434/262-275

See application file for complete search history. 16 Claims, 33 Drawing Sheets

YDATA

STOMIZEfO1b

RECORDED SPEED 101SIM 106 N106 REPRESENTATION

101 SENSORS +

REAL TIME MODELING ® 109

DIGITAL 103DATA

0^fOfd FORMA104

108

NATURALSYSTEM

US 7,654,966 B2Page 2

U.S. PATENT DOCUMENTS 6,158,432 A * 12/2000 Biondi etal. .......... 128/204.216,201,542 B1 3/2001 Arai et al.

5,588,104 A 12/1996 Lainier et al. 6,222,547 B1 4/2001 Schwuttke et al.5,592,195 A 1/1997 Misono et al. 6,417,854 B1 7/2002 Isowaki et al.5,596,694 A 1/1997 Capps 6,463,930 B2 * 10/2002 Biondi et al. .......... 128/204.215,731,819 A 3/1998 Gagne et al. 6,575,918 B2 * 6/2003 Kline ......................... 600/5325,751,931 A 5/1998 Cox et al. 6,584,973 B1 * 7/2003 Biondi et al. .......... 128/204.215,768,552 A 6/1998 Jacoby 6,612,995 B2 * 9/2003 Leonhardt et al............ 600/5325,774,878 A 6/1998 Marshall 6,745,764 B2 * 6/2004 Hickle ................... 128/203.125,796,398 A 8/1998 Zimmer 7,066,892 B2 * 6/2006 Kline ......................... 600/5325,812,134 A 9/1998 Poo ser et al. 7,128,578 B2 * 10/2006 Lampotang et al. ......... 434/3655,830,150 A 11/1998 Palmer et al. 2003/0060725 Al* 3/2003 Kline ......................... 600/5295,923,330 A 7/1999 Tarthon et al.5,931,160 A * 8/1999 Gilmore et al. ........ 128/204.21 * cited by examiner

U.S. Patent Feb. 2, 2010 Sheet 1 of 33 US 7,654,966 B2

CUSTOMIZE101 -101

REPRESENTATION

f02, SENSORS

f09

1018 FORMAT

NATURALSYSTEM

FIGURE 1a

SIM

1108SENSORS

RECORDEDSPEED

1 f4 N115

REAL TIME

DATA 117FORMAT 113

ARTIFICIALfff

SYSTEM

FIGURE 1b

CUSTOMIZE

116REPRESENTATION

DATAMODELING ® ff8

111


A !f•

^ L JALLJ\^ Q^ m

^YJ Q = AW_^

C.-no ^ N O

aohAQ

_CC-cc AJ

U dJ

U dO cf^o O yJ Q J O

NACGQO QO

Cl CU DU4A

Nw

Q O1---- O=Q CL-CM W 'pd hA

CDZ _

WQ W QG U C hO hd' Aa

-Cc ar-- QQ CD

OO ^'hw

UW

n

hA QVQ WD Ck--

AA Q

Q a^ 0

Z

Q _OQ Qp W

CCU

C",110ZWN

hA

wA

J Jd N a ZD D

A A

ODQZZ^

^ C', ccaQODUZ


tf•!f•A

0 COW a O. Li

F- OCCn Q

N QJd ^V1D^•

A ^(Q

QA

a

4tQ a-A A

4. N 1 -

W

Aa CDAn

0 W CD^, N ZQJ C F-Q ^ Z QO ^Z ^

A ^

Q U^ WQ JC JOU

HN • • • • • ^ZZ Z ZW W W

A A \+

QOA

201

9na

208

203

14 Ac

X4^^2f1

216


202y

z

X

204 IVV

ENGINEx i = ENGINE TEMPERATUREy, = ENGINE RPMz, : ENGINE EXHAUST

GAS VOLUME

FIGURE 2a

f0

CARDIAC SYSTEM FUNCTIONX2 = z 2 = HEART RATE I SECONDY2 = STROKE VOLUME

SPHERICAL VOLUME = CARDIAC OUTPUT

FIGURE 2b

-144

212

Y3

Z' 214

213 -----X3

SALES DEPARTMENT OPERATION

x 3 = AVERAGE TIME /CONTRACT

y3 : # OF CONTRACTS

Z3 = AVG. REVENUEICONTRACT

RESPIRATORY FUNCTION

X4 = AVERAGE TIME I CONTRACT

Y4 = FCN OF xy AND RESP, VOLUME

Z4 = +I• INHALATION I EXHALATION

SLAB VOLUME=RESPIRATORY VOLUME

FIGURE 2c

FIGURE 2d

@ OTN*

Ox = 112 REP. SPACE(H•PIC. SPACE)

Oy = BLOOD PRESSURE(UP-DOWN)

Oz-- OXYGEN CONCENTRATIONIN THE BLOOD(BACK TO FRONT)

@ ATN*

Px = 112 REP. SPACE(H•PIC, SPACE)

Py = SALES PROFIT(UP-DOWN)

Pz = PRODUCTS IN STOCK(BACK TO FRONT)


909

311-

310•

305

301312

FIGURE 3


V Oc

OcLZI

2

OcLZI 3

Ocl Y4 —

IZY

n

400; z7 2 3 z4Ocn

n

t0 Atl At2 At3 A4 Otn

401

FIGURE 4a

411

408 409 410

Atl Ott At3

X i = zi [X2=22] >[Xi ^zll [X3=23] >[X2=22] stn

y Y2>Yi Y3>Y2 [Xn^2n]

sop 51076508C1414 ed

7

W I I

f-i I

51;

512

W

ffe

U.S. Patent Feb. 2,2010 Sheet 7 of 33 US 7,654,966 B2

502

Z§503

FIGURE 5a

504 905

,4§ Z§FIGURE 5b

506

FIGURE 5c

FIGURE 5d

MR

519 510

528

52;

532

53,

530

53;

U.S. Patent

Feb. 2, 2010

Sheet 8 of 33

US 7,654,966 B2

FIGURE 5e

522 513

6^14

FIGURE 5f

A6

515 516

A6FIGURE 5g

521

FIGURE 5h

i

T

s^oyJ

Z--613)50

FIGURE 6


706

TA Id

tuo

FIGURE 7



T-N

t

X (TIME)

802

T `t-X

PRESENT

f 80>

* •Y

FIGURE 8a,y

800

808

FIGURE 8b

904

903

FIGURE 9a

901906

908909

U.S. Patent Feb. 2, 2010 Sheet 12 of 33

US 7,654,966 B2

FIGURE 9b


0

W

LL-

1 f08'

1f05-,,^ //AP


1101

)a --- 1101M rc5-

1104 f

C1103

FIGURE 11a

FIGURE 11b

72

1203


UIV/

FIGURE 12b


INTERFACE MODE

INTERFACE MODE II

(e.g. MEDICINE)

(e.g. CORPORATE DASHBOARD)

GIVEN:

GIVEN:

- CRITICAL FUNCTIONS

- DEFAULTIGENERIC l-SPACEIH-SPACE

(UNCHANGEABLE)

- PHYSIOLOGIC DATA COLLECTED

. SYMBOLIC SYSTEM STANDARD

USER DETERMINES:

- CRITICAL FUNCTIONS

- VITAL SIGNS TO BE COLLECTED

• SYMBOLIC SYSTEM TO BE USED

- IDEAL VALUESIALARMS

- OBJECTS/ATTRIBUTES SPACE

- REFERENTIAL FRAMEWORK

IDEAL VALUESIALARMS

(CHANGEABLE)

• PARTICULAR VALUES

OBJECT ATTRIBUTES

130f')1301

COMMON INTERFACE FEATURES

• L-SPACE

• H•SPACE

- ZOOMISPEED

• VIEWPOINTS

FIGURE 13

U.S. Patent Feb. 2, 2010 Sheet 17 of 33

POSS. 1401OPTIONS

#11402 PATIENT

1416 1404METI PATIENT

PATIENT SIMULATORDIGITAL SENSORS

SENSORS N CTR BOX

1405 AID MANNEQUINCONVERTER

TRADITIONAL 1406PATIENT MONITOR

1401POSS.

OPTIONS#2

US 7,654,966 B2

1403

NETWORK t-, f0gSERVER

140POSS.

OPTIONS#3

1410 BUFFER I I DIALE

'1411

1412 3-DIM.DISPLAY PGM

1413POSS.

OP#l4

NS

FIGURE 14

1414 DISPLAY I I R INTOERFACE R

1415


READ START-UP FILE 1501ENTER PATIENT DATA

INITIALIZE INTERNAL [_-, 1502DATA TABLES

1506ENTER EVENT 1503

WAIT UNTIL TIME LOOP

FOR NEW DATA REFRESH

INITIALIZE 1504WINDOWS

1506 i DATAFILE

GET NEW 1509DATA DATA

1501DATA

L ------- COMPUTE(PROPERTIES

1510

COMPUTE SOUND 1511PROPERTIES

RENDER 1512PROPERTIES

PRODUCE 1513SOUNDS

1514SMOOT

N ANIMATION Y151SELECTED

1516SHIFT OBJECT NEW N

LEFT DATA 1516REFRESH

CALCULATE LEFT

1511Y SHIFT AND APPLY

CALCULATE LEFTSHIFT & APPLY

FIGURE 15


1601-1- GET WINDOW10

160? FIND DATA SET CURRENT F- 1611STRUCTURE POSITION

1603 SET SET CARDIAC 1613VIEW OBJECT

1604 SET ROTATION RENDER CARDIAC 1614MATRIX OBJECT

1605 ET PROJECTION RENDER 1615MATRIX QUADS

1606 SET SET MATERIAL 1616LIGHTS PROPERTIES

1607 CLEAR BACK SEND VERTICES 1617BUFFER COMPOSE REF, PLANES

1608 PROCESS SWAP 1618OBJECT BUFFERS

1609 SETVIEW

1610 RENDEROBJECT

1611 RENDERGRIDS

REFFIGURE 16

U.S. Patent Feb. 2,2010 Sheet 20 of 33 US 7,654,966 B2

C>cc C,

ll^lCIO r1i L rvl

c"IOcOC"

CD

cc

(DOco LC13

ccCIO

O

7

C13

C-DO ca

CL-CO

03WZE 2-- r-j

A

tl cccu

LLA

r-j

B e

^ac

. . . . . .. . C36.C31

cc

co

cC

A

co

4:1.

w ^U

N CQC/1

z. .•a. .^.- G

A

CIO

LLB

rt

UCdr-dOHcOd Q-

N 7 ^

^ x QdR O O td C _, c IA Ogo

X Y


y CCa

V O^^H C N CO CV

7 CU. dd«? a 3

CkN d dOt y

dO

ONto

\^ C.>C E-=CV O

CO Cn CVN O

coH

cnC, Q

na O_ dE Q4

^ ^ O^ O ^w


OA

0Ro^ca

y c ta) X3^ XR O 'CCH_O R

C^i R CC:, Ot N ._&3- 'O CQx ca^ of^ a c

OC! = O)- O GR R O

an U 'Q

FFF T••• ^^^ III

Q1r

Wr'Y

CDLL_

AO^

RC"HRa

-o00m

d C

_^ N1"^ Z11 11

X Y

ydR

dO'CCRtzdeaO30tH

HWC

J

N>

dN

GO

cy

CV

XW

O

CQ ccCa R

C NCCGddNdO

O X

CO.Q NNOO_

L n

O=

e

c0U

O CVcO

F— CIO

11 1

X N

ca^

C%)JW

WJ

dW

OJ

DZQtYWdd

OSU

Z

O

I—ccOa

CWc!3.Q


A

h

^— 0hG O

cesa^'a

Eo

hOh

OON

aOh


90Oh

ON

Oh CCUCl*-0

LuO ^

h ^a

x

Oc cas ^ coas

cn^• dd ^C

CQ^.0 ^ N R^CC

O R y

OO

^CO

C R 7^ d— U

ON


2301b 31/13 29070 > 2301d 108/692301a 2300

9 23048 '§'@ 23058 230623018 CVP PAP LAP MAP

0 0 23068

0 0

0 02301 2303 HR 97 2304 2305

CO23028 2302 2303b SV 52 ^90

2301e

FIGURE 23a

1817 231 fa 73/406 231'08

PAP 2314 23102312J23128 2313CVP A LAP MA

23088 -- -

771=5̂-- 231382308

230982309 23f 1

HR 91 2312 CO

SV 502315

FIGURE 23b

1217 23198 70/41

2322 @ 2323 11 2319 g j 23208

23f6a CVP PAP LAP MAP 2324 2321

1316 02321a

2311231'8 HR 65 2320 CO

23118 23188SV 30 PQ

FIGURE 23c


30/170 2326 ^^

0i;

2327a 95/45

23261 329

CVP PAP 329123301

?32512330

2325

2331 2332

HR 31 CO

SV 50

FIGURE 23d

35/18 0

2J40

2336 2331 90/42

S6 61 @@ 2338

CVP PAP MAP2339

23341 23998

2334?341

2331,7 2338a

2335 80 CO2335a SV 30

FIGURE 23e

22/8 95/3523451 2345 23461110 2343,7

LAP ?3441 2341a

CVP PAP 2343 2344 MAP

?3462341

2342C

23421 002348 2349

HR 90 CO

SV 25

FIGURE 23f

2400

2605

r 26042601 - - - - - - - II 1


2400

I I

r------------I I-------11 II I ------- 1II II II II II II I^ II II II II II II II 1- ----- I ^ II IL ---- ------^ L -----^I I

I I

2401 2402 2403 L Z 1404

FIGURE 24

1

Ir-----,1 - 2500I II II

L -

I

2501 2502 -- - - - - J 2400

2503

FIGURE 25

i I PAP LAP I-----^ MAPI I

HR

CVP SV

FIGURE 26


FIGURE 27

2100

2

?800

-------r------r-----r-----^--,

I I I I I 1

I I I I I I

I I I I I 1

I I I I I 1

I I I I I 1

I I I I I I

I I I I I 1---J-- ---^-- --J---r,-1-

280fa 2801b 2801c 2801d 280M

FIGURE 28

?901

^ 2900

I II I I I

I II I I I

I II I I I

I II I I I

I II I I II

1-T --

2902a 2902b 2902c 2902d

FIGURE 29

/ I

1 I I I

I I I I

I / I

3001 3005 3002 3004 3003

FIGURE 30

3000

3104

Sheet 30 of 33

3101

"Airw a'3102

3108

ExpiredGas

q3113

V033101

FIGURE 31

U.S. Patent

Feb. 2, 2010

3109

3106

31103100 -^

3111-47-

US 7,654,966 B2

—3112

3105

RR U

VT F-

3201-----Ravr

3201

^ 3206

PIP _3202

3203

F102

3205

3204

FIGURE 32

3208

3403 3402


VTRR 13

3304

PIP

Ravr 3302

3303

J

F102 ETCO2

330f

FIGURE 33

'1110

FIGURE 34

3602

P03


RR 9

VT3301

PIP 3502

Ravr

U. 3503

F102Lt:

3204

FIGURE 36

RR 13 3601

FA02

FIGURE 36

3102

3802

3103

FIGURE 37

RR 13 3801

JOUJ

FIGURE 38


US 7,654,966 B21

METHOD AND APPARATUS FORMONITORING DYNAMIC

CARDIOVASCULAR FUNCTION USINGN-DIMENSIONAL REPRESENTATIVES OF

CRITICAL FUNCTIONS

CROSS REFERENCE TO RELATEDAPPLICATIONS

This patent application is a continuation-in-part patentapplication of copending U.S. patent application Ser. No.09/457,068, which was filed on Dec. 7, 1999 and of copend-ing Provisional Patent Application Ser. No. 60/328,880, filedon Oct. 12, 2001. Priority is hereby claimed to all commonmaterial disclosed in these pending patent cases.

FEDERAL RESEARCH STATEMENT

Some of the technology described in this patent applicationwas funded in part by National Institute of Health Grant No.1 R24 HL 64590 and NASA Grant No. NGTA40101.

BACKGROUND OF INVENTION

1. Field of the InventionThis invention relates to the visualization, perception, rep-

resentation and computation of data relating to the attributesor conditions constituting the health state of a dynamic sys-tem. More specifically, this invention relates to the displayand computation of pulmonary and cardiovascular data, inwhich variables constituting attributes and conditions of adynamic physiological system can be interrelated and visu-ally correlated in time as three-dimensional objects.

2. Description of the Related ArtA variety of methods and systems for the visualization of

data have been proposed. Traditionally, these methods andsystems fail to present in a real-time multi-dimensional for-mat that is directed to facilitating a user's analysis of multiplevariables and the relationships between such multiple vari-ables. Moreover, such prior methods and systems tend not tobe specifically directed to display of a patient's cardiovascu-lar and pulmonary system by showing such variables as bloodpressure, blood flow, vascular tone and the like. Prior methodstypically do not process and display data in real-time, ratherthey use databases or spatial organizations of historical data.Generally, they also simply plot existing information in twoor three dimensions, but without using three-dimensionalgeometric objects to show the interrelations between data.Often previous systems and methods are limited to pie charts,lines or bars to represent the data. Also, many previous sys-tems are limited to particular applications or types of data.The flexibility and adaptability of the user interface and con-trol is typically very limited, and may not provide flexiblecoordinate systems and historical-trend monitors. Other sys-tems, which have a flexible user interface, generally requiresubstantial user expertise in order to collect and evaluate thedata, including the pre-identification of data ranges and reso-lution. Another common limitation of previous systems andmethods is that they provide only a single or predeterminedviewpoint from which to observe the data. Typically, priorsystems and methods do not provide data normalcy frame-works to aid in the interpretation of the data. Furthermore,most prior methods use "icons," shapes, lines, bars, or graphs.

The state-of-the-art for pulmonary monitored data in bothtraditional intensive care units and anesthesiology units is thewave form and numeric display monitors. These monitorsrepresent information in discrete variables associated with

2each sensor using the display paradigm of a "single-sensor-single-indicator-display." As a result, current monitors lackintegrated data. The anesthesiologist is left to collect andprocess all of the different data herself in a timely manner.

5 The anesthesiologist is faced with a fire-hose of informa-tion during every case. Numerical data, waveforms, controldials, ventilator breaths, heart rate beeps, and alarm soundsbombard the anesthesiologist from all directions. During anuneventful case, the anesthesiologist successfully monitors

io all the available data at a low cognitive level. However, shouldan unexpected event happen, the anesthesiologist mustquickly ascertain if a problem exists, the causality, and theneeded corrective action. Should the anesthesiologist fail atany of these tasks, an error will occur. Human error is asso-

15 ciated with more than 80% of critical anesthesia incidents andmore than 50% of anesthetic deaths.

Studies have shown that integrated information repre-sented in a graphical metaphor does reduce detection timeand time to accurately diagnose a problem in anesthesia.

20 However, the graphical metaphors studied to date tend to becomplex and lack usability testing to drive designs. As aresult, the graphical interface may inadvertently increase thecognitive workload of the anesthesiologists.

For general background material, the reader is directed to25 U.S. Pat. Nos. 3,908,640, 4,193,393, 4,464,122, 4,519,395,

4,619,269, 4,752,893, 4,772,882, 4,813,013, 4,814,755,4,823,283, 4,832,038, 4,875,165, 4,880,013, 4,915,757,4,930,518, 4,989,611, 5,012,411, 5,021,976, 5,103,828,5,121,469, 5,162,991, 5,222,020, 5,224,481, 5,262,944,

3o 5,317,321, 5,425,372, 5,491,779, 5,568,811, 5,588,104,5,592,195, 5,596,694, 5,626,141, 5,634,461, 5,751,931,5,768,552, 5,774,878, 5,796,398, 5,812,134, 5,830,150,5,836,884, 5,913,826, 5,923,330, 5,961,467, 6,042,548, and6,090,047 each of which is hereby incorporated by reference

35 in its entirety for the material disclosed therein.As this disclosure employs a number of terms, which may

be new to the reader, the reader is directed to the applicants'definitions section, which is provided at the beginning of thedetailed description section.

40SUMMARY OF INVENTION

It is desirable to provide a method, system, and apparatus,which facilitates the rapid and accurate analysis of complex

45 and quickly changing data. Moreover, it is desirable that sucha system and method include a graphic element that depictsthe status of a patient's cardiovascular system by graphicallyshowing blood pressure, blood flow, vascular tone and othercardiovascular variables. It is important that such a graphic

50 element provide an anesthesiologist with the means toquickly assess the patient's status. It is also desirable that theelement by comprised of subcomponents, which are linkedtogether to show thereby the relationships of the variouscardiovascular and pulmonary variables. Also, it is desirable

55 that system and method be capable of analyzing time based,real-time, and historical data and that it be able to graphicallyshow the relationships between various data.

Research studies have indicated that the human mind isbetter able to analyze and use complex data when it is pre-

60 sented in a graphic, real world type representation, rather thanwhen it is presented in textual or numeric formats. Researchin thinking, imagination and learning has shown that visual-ization plays an intuitive and essential role in assisting a userassociate, correlate, manipulate and use information. The

65 more complex the relationship between information, themore critically important is the communication, includingaudio and visualization of the data. Modern human factors

US 7,654,966 B23

theory suggests that effective data representation requires thepresentation of information in a manner that is consistent withthe perceptual, cognitive, and response-based mental repre-sentations of the user. For example, the application ofpercep-tual grouping (using color, similarity, connectedness, motion,sound etc.) can facilitate the presentation of information thatshould be grouped together. Conversely, a failure to use per-ceptual principles in the appropriate ways can lead to errone-ous analysis of information.

The manner in which information is presented also affectsthe speed and accuracy of higher-level cognitive operations.For example, research on the "symbolic distance effect" sug-gests that there is a relationship between the nature of thecognitive decisions (for example, is the data increasing ordecreasing in magnitude?) and the way the information ispresented (for example, do the critical indices become largeror smaller, or does the sound volume or pitch rise or fall?).Additionally, "population stereotypes" suggest that there areways to present information that are compatible with well-learned interactions with other systems (for example, anupwards movement indicates an increasing value, while adownwards movement indicates a decreasing value).

Where there is compatibility between the information pre-sented to the user and the cognitive representations presentedto the user, performance is often more rapid, accurate, andconsistent. Therefore, it is desirable that information be pre-sented to the user in a manner that improves the user's abilityto process the information and minimizes any mental trans-formations that must be applied to the data.

Therefore, it is the general object of this invention to pro-vide a method and systems for presenting a three-dimensionalvisual and/or possibly an audio display technique that assistsa doctor in the monitoring of a patient's cardiovascular andpulmonary function.

It is a further object of this invention to provide a methodand system that assists in the monitoring of a patient's car-diovascular and/or pulmonary system through the use of athree-dimensional graphic element.

It is another object of this invention to provide a methodand system that assists in the management of anesthesia careof patients, by presenting a display, which quickly shows therelationships of various cardiovascular or pulmonary vari-ables.

It is a still further object of this invention to provide amethod and system that assists in the determination of the"health" of a dynamic cardiovascular or pulmonary system,by providing visual information related to the nature or qual-ity of the soundness, wholeness, or well-being of the systemas related to historical or normative values.

Another object of this invention is to provide a method andsystem that assists in the determination of the functioning ofa cardiovascular or pulmonary system by measuring the inter-action among a set of "vital-signs" normally associated withthe health of the cardiovascular system.

A still further obj ect of this invention is to provide a methodand system, which provides the gathering and use of sensormeasured data, as well as the formatting and normalization ofthe data in a format suitable to the processing methodology.

A further object of this invention is to provide a method andsystem, which organizes a cardiovascular or pulmonary sys-tem's data into relevant data sets or critical functions asappropriate.

Another object of this invention is to provide a method andsystem, which provides a three-dimensional health-space formapping the cardiovascular or pulmonary system data.

It is another object of this invention to provide a methodand system, which provides three-dimensional objects that

4are symbols of the critical functioning of the cardiovascularor pulmonary system being monitored.

It is an object of this invention to provide a method andsystem that shows the relationships between several critical

5 functions that a user wishes to monitor.It is a further object of this invention to provide a method

and system that permits an integrated and overall holisticunderstanding of the cardiovascular or pulmonary processbeing monitored.

10 A further object of this invention is to provide a method andsystem where three-dimensional objects are built from three-dimensional object primitives, including: cubes, spheres,pyramids, n-polygon prisms, cylinders, slabs.

A still further object of this invention is to provide a methodis and system, wherein three-dimensional objects are placed

within health-space based on the coordinates of their geomet-ric centers, edges, vertices, or other definite geometric vari-ables.

It is a further object of this invention to provide a method20

and system, which has three-dimensional objects that havethree spatial dimensions, as well as geometric, aesthetic andaural attributes, to permit the mapping of multiple data func-tions.

25 It is another object of this invention to provide a methodand system, which shows increases and decreases in datavalues using changes in location, size, form, texture, opacity,color, sound and the relationships thereof in their context.

It is a still further object of this invention to provide a

30 method and system, wherein the particular three-dimensionalconfiguration of three-dimensional objects can be associatedwith a particular time and health state.

A still further object of this invention is to provide a methodand system that permits the simultaneous display of the his-

35 tory of data objects.Another object of this invention is to provide a method and

system that provides for the selection of various user select-able viewports.

It is a further object of this invention to provide a method4o and system that provides both a global and a local three-

dimensional coordinate space.It is another object of this invention to provide a method

and system that permits the use of time as one of the coordi-nates.

45 It is a still further object of this invention to provide amethod and system that provides a reference framework ofnormative values for direct comparison with the measureddata.

It is a further object of this invention to provide a methodSo and system where normative values are based on the average

historical behavior of a wide population of healthy systemssimilar to the system whose health is being monitored.

A further object of this invention is to provide a method and

55 system that provides viewpoints that can be selected to beperspective views, immersive Virtual Reality views, or anyorthographic views.

Another object of this invention is to provide a method andsystem that permits the display of a layout of multiple time-

60 space viewpoints.A still further object of this invention is to provide a method

and system that provides for zooming in and out of a timeand/or space coordinate.

It is another object of this invention to provide a method65 and system that permits temporal and three-dimensional

modeling of data "health" states based on either pre-recordeddata or real-time data, that is as the data is obtained.

US 7,654,966 B25

6Another object of this invention is to provide a method and

FIG. 14 is a hardware system flow diagram showing vari-

system that presents the data in familiar shapes, colors, and

ous hardware components of the preferred embodiments oflocations to enhance the usability of the data. the invention.

A still further object of the invention is to provide a method

FIG. 15 is a software flow chart showing the logic steps ofand system that uses animation, and sound to enhance the 5 a preferred embodiment of the invention.usefulness of the data to the user. FIG. 16 is a softwareblock diagram showing the logic steps

It is an object of this invention to provide a method and

of the image computation and rendering process of a pre-

system for the measurement, computation, display and user

ferred embodiment of the invention.

interaction, of complex data sets that can be communicated

FIG. 17 is a photograph of the 3-dimensional display of aand processed at various locations physically remote from io preferred embodiment of the invention.

each other, over a communication network, as necessary for

FIG. 18 is a close-up front view of the cardiac object and

the efficient utilization of the data and which can be dynami- the associated reference grid of a preferred embodiment of thecally changed or relocated as necessary. invention.

It is a still further object of this invention to provide a

FIG. 19 is a view of the front view portion of the display ofmethod and system for the display of data that provides both 15 a preferred embodiment of the present invention showing the

a standard and a customized interface mode, thereby provid- cardiac object in the foreground and the respiratory object ining user and application flexibility. the background.

These and other objects of this invention are achieved by

FIG. 20 is a view of the top view portion of the display of

the method and system herein described and are readily a preferred embodiment of the present invention showing theapparent to those of ordinary skill in the art upon careful 20 cardiac object toward the bottom of the view and the respira-

review of the following drawings, detailed description and

tory object toward the top of the view.claims.

FIG. 21 is a view of the side view portion of the display ofa preferred embodiment of the present invention showing the

BRIEF DESCRIPTION OF DRAWINGS cardiac object to the left and the respiratory object to the right.25 FIG. 22 is a view of the 3-D perspective view portion of the

In order to show the manner that the above recited and other display of a preferred embodiment of the invention showing

advantages and objects of the invention are obtained, a more the cardiac object in the left foreground and the respiratory

particular description of the preferred embodiment of the object in the right background.

invention, which is illustrated in the appended drawings, is FIG. 23a is a view of the preferred graphic element of thisdescribed as follows. The reader should understand that the 30 invention in a normal cardiovascular system.

drawings depict only a preferred embodiment of the inven- FIG. 23b is a view of the preferred graphic element of this

tion, and are not to be considered as limiting in scope. A brief

invention in a cardiovascular system showing anaphylaxis.description of the drawings is as follows: FIG. 23c is a view of the preferred graphic element of this

FIG. la is a top-level representative diagram showing the invention in a cardiovascular system showing hypovolemia.data processing paths of the preferred embodiment of this 35 FIG. 23d is a view of the preferred graphic element of thisinvention. invention in a cardiovascular system showing bradycardia.

FIG. lb is a top-level block diagram of the data processing FIG. 23e is a view of the preferred graphic element of thisflow of the preferred embodiment of this invention. invention in a cardiovascular system showing ischemia.

FIG. lc is a top-level block diagram of one preferred pro- FIG. 23f is a view of the preferred graphic element of thiscessing path of this invention. 40 invention in a cardiovascular system showing pulmonary

FIG. ld is a top-level block diagram of a second preferred embolism.processing path of this invention. FIG. 24 is a view of the preferred reference grid of this

FIGS. 2a, 2b, 2c, and 2d are representative 3-D objects embodiment of the invention.representing critical functions. FIG. 25 is a view of the preferred reference grid showing

FIG. 3 is a representation of data objects in H-space. 45 object placement in this preferred embodiment of the inven-

FIGS. 4a and 4b are representative views of changes in data tion.objects in time. FIG. 26 is a view of the preferred reference grid showing

FIGS. 5a, 5b, 5c, 5d, 5e, 5f, 5g and 5h are representative the functional object relationships in this preferred embodi-

views of properties of data objects provided in the preferred ment of the invention.embodiment of this invention. 50 FIG. 27 is a representative three-dimensional object used

FIG. 6 shows a 3-D configuration of the objects in H-space in the present preferred embodiment of the invention.in the preferred embodiment of the invention. FIG. 28 is a representative view of the normalization of the

FIG. 7 shows H-space with a time coordinate along with present preferred embodiment of the invention.local-space coordinates. FIG. 29 is an integrated view showing numeric information

FIGS. 8a and 8b show the global level coordinate system of 55 in the present preferred embodiment of the invention.the preferred embodiment of this invention. FIG. 30 is a view showing the addition of slopes to show the

FIGS. 9a and 9b show various viewpoints of the data restriction of blood vessels.

within H-space in the preferred embodiment of this invention. FIG. 31 shows a an embodiment of a pulmonary metaphor

FIG. 10 shows the transformation of an object in space in 3100 with anatomical representations.context, with a reference framework, in the preferred embodi- 60 FIG. 32 shows an embodiment of a pulmonary metaphorment of this invention. where various features are abnormal.

FIG. 1 l shows the zooming out function in the invention. FIG. 33 shows an embodiment of a pulmonary metaphor

FIG. llb shows the zooming in function in the invention. where the intrinsic PEEP has increased.

FIGS. 12a and 12b show a 3-D referential framework of

FIG. 34 shows a preferred embodiment of a pulmonarynormative values. 65 metaphor.

FIG. 13 shows the interface modes of the preferred

FIG. 35 shows an embodiment of a pulmonary metaphorembodiment of this invention. wherein the patient has pulmonary fibrosis.

US 7,654,966 B27

FIG. 36 shows an alternative embodiment of a pulmonarymetaphor wherein the patient has pulmonary fibrosis.

FIG. 37 shows an embodiment of a pulmonary metaphorwherein the patient has emphysema.

FIG. 38 shows an alternative embodiment of a pulmonary 5metaphor wherein the patient has emphysema.

DETAILED DESCRIPTION

This invention is a method, system and apparatus for the 10

visual display of complex sets of dynamic data. In particular,this invention provides the means for efficiently analyzing,comparing and contrasting data, originating from either natu-ral or artificial systems. In its most common use the preferredembodiment of this invention is used to produce an improved 15cardiovascular or pulmonary display of a human or animalpatient. This invention provides n-dimensional visual repre-sentations of data through innovative use of orthogonalviews, form, space, frameworks, color, shading, texture,transparency, sound and visual positioning of the data. The 20

preferred system of this invention includes one or a pluralityof networked computer processing and display systems,which provide real-time as well as historical data, and whichprocesses and formats the data into an audio-visual formatwith a visual combination of objects and models with which 25the user can interact to enhance the usefulness of the pro-cessed data. While this invention is applicable to a widevariety of data analysis applications, one important applica-tion is the analysis of health data. For this reason, the exampleof a medical application for this invention is used throughout 30

this description. The use of this example is not intended tolimit the scope of this invention to medical data analysisapplications only, rather it is provided to give a context to thewide range of potential application for this invention.

This invention requires its own lexicon. Forthe purposes of 35

this patent description and claims, the inventors intend thatthe following terms be understood to have the followingdefinitions.

An "artificial system" is an entity, process, combination of 40

human designed parts, and/or environment that is created,designed or constructed by human intention. Examples ofartificial systems include manmade real or virtual processes,computer systems, electrical power systems, utility and con-struction systems, chemical processes and designed combi-

45nations, economic processes (including, financial transac-tions), agricultural processes, machines, and human designedorganic entities.

A "natural system" is a functioning entity whose origin,processes and structures were not manmade or artificially 50created. Examples of natural systems are living organisms,ecological systems and various Earth environments.

The "health" of a system is the state of being of the systemas defined by its freedom from disease, ailment, failure orinefficiency. A diseased or ill state is a detrimental departure 55from normal functional conditions, as defined by the nature orspecifications of the particular system (using historical andnormative statistical values). The health of a functioning sys-tem refers to the soundness, wholeness, efficiency or wellbeing of the entity. Moreover, the health of a system is deter- 60mined by its functioning.

"Functions" are behaviors or operations that an entity per-forms. Functional fitness is measures by the interactionamong a set of "vital-signs" normally taken or measuredusing methods well known in the art, from a system to estab- 65lish the system's health state, typically at regular or definedtime intervals.

8"Health-space" or "H-space" is the data representation

environment that is used to map the data in three or moredimensions.

"H-state" is a particular 3-D configuration or compositionthat the various 3-D objects take in H-space at a particulartime. Inotherwords, H-stateis a3-D snapshot ofthe system'shealth at one point of time.

"Life-space" or "L-space" provides the present and pasthealth states of a system in a historical and comparative viewof the evolution of the system in time. This 3-D representationenvironment constitutes the historical or Life-space of adynamic system. L-space allows for both continuous andcategorical displays of temporal dependent complex data. Inother words, L-space represents the health history or trajec-tory of the system in time.

"Real-Time Representation" is the display of a represen-tation of the data within a fraction of a second from the timewhen the event of the measured data occurred in the dynamicsystem.

"Real-Time User Interface" is the seemingly instantaneousresponse in the representation due to user interactivity (suchas rotation and zooming).

A "variable" is a time dependent information unit (one unitper time increment) related to sensing a given and constantfeature of the dynamic system.

"Vital signs" are key indicators that measure the system'scritical functions or physiology. In the preferred embodi-ments of this invention, data is gathered using methods orprocesses well known in the art or as appropriate and neces-sary. For example, in general, physiologic data, such as heartrate, respiration rate and volume, blood pressure, and the like,is collected using the various sensors that measure the func-tions of the natural system. Sensor-measured data is elec-tronically transferred and translated into a digital data formatto permit use by the invention. This invention uses thereceived measured data to deliver real-time and/or historicalrepresentations of the data and/or recorded data for laterreplay. Moreover, this invention permits the monitoring of thehealth of a dynamic system in a distributed environment. Bydistributed environment, it is meant that a user or users inter-acting with the monitoring system may be in separate loca-tions from the location of the dynamic system being moni-tored. In its most basic elements, the monitoring system ofthis invention has three major logical components: (1) thesensors that measure the data of the system; (2) the networkedcomputational information systems that computes the repre-sentation and that exchanges data with the sensors and theuser interface; and (3) the interactive user interface that dis-plays the desired representation and that interactively acceptsthe users' inputs. The components and devices that performthe three major functions of this invention may be multiple,may be in the same or different physical locations, and/or maybe assigned to a specific process or shared by multiple pro-cesses.

FIG. la is a top-level representative diagram showing thedata processing paths of the preferred embodiment of thisinvention operating on a natural system. The natural system101a is shown as a dynamic entity whose origin, processesand structures (although not necessarily its maintenance)were not manmade or artificially created. Examples of naturalsystems are living organisms, ecological systems, and variousEarth environments. In one preferred embodiment of theinvention, a human being is the natural system whose physi-ology is being monitored. Attached to the natural system 101aare a number of sensors 102. These sensors 102 collect thephysiologic data, thereby measuring the selected criticalfunctions of the natural system. Typically, the data gathering

US 7,654,966 B29

of the sensors 102 is accomplished with methods or tech-niques well known in the art. The sensors 102 are typicallyand preferably electrically connected to a digital data format-ter 103. However, in other embodiments of this invention, thesensors may be connected using alternative means includingbut not limited to optical, RE and the like. In many instances,this digital data formatter 103 is a high-speed analog to digitalconverter. Also, connected to the digital data formatter 103 isthe simulator 101b. The simulator 101b is an apparatus orprocess designed to simulate the physiologic process under-lying the life of the natural system 101a. A simulator 101b isprovided to generate vital sign data in place of a naturalsystem 101a, for such purposes as education, research, sys-tem test, and calibration. The output of the digital data for-matter 103 is Real-Time data 104. Real-Time data 104 mayvary based on the natural system 101a being monitored or thesimulator 101b being used and can be selected to follow anydesired time frame, for example time frames ranging fromone-second periodic intervals, for the refreshment rates ofpatients in surgery, to monthly statistics reporting in an eco-logical system. The Real-Time data 104 is provided to a datarecorder 105, which provides the means for recording data forlater review and analysis, and to a data modeling processorand process 108. In the preferred embodiments of this inven-tion the data recorder 105 uses processor controlled digitalmemory, and the data modeling processor and process 108 isone or more digital computer devices, each having a proces-sor, memory, display, input and output devices and a networkconnection. The data recorder 105 provides the recorded datato a speed controller 106, which permits the user to speed-upor slow-down the replay of recorded information. Scalarmanipulations of the time (speed) in the context of the 3-Dmodeling of the dynamic recorded digital data allows for newand improved methods or reviewing the health of the systems101a,b. A customize/standardize function 107 is provided topermit the data modeling to be constructed and viewed in awide variety of ways according to the user's needs or inten-tions. Customization 107 includes the ability to modify spa-tial scale, such modifying includes but is not limited to zoom-ing, translating, and rotating, attributes and viewports inaddition to speed. In one preferred embodiment of the inven-tion, the range of customization 107 permitted for monitoringnatural systems 101a physiologic states is reduced and isheavily standardized in order to ensure that data is presentedin a common format that leads to common interpretationsamong a diverse set of users. The data modeling processorand process 108 uses the prescribed design parameters, thestandardized/customize function and the received data tobuild a three-dimensional (3-D) model in real-time and todeliver it to an attached display. The attached display of thedata modeling processor and process 108 presents a repre-sentation 109 of 3-D objects in 3-D space in time to providethe visual representation of the health of the natural system101a in time, or as in the described instances of the simulated101b system.

FIG. lb is a top-level block diagram of the data processingflow of the preferred embodiment of this invention operatingon an artificial system. An artificial system is a dynamic entitywhose origin, processes and structure have been designed andconstructed by human intention. Examples of artificial sys-tems are manmade real or virtual, mechanical, electrical,chemical and/or organic entities. The artificial system 110a isshown attached to a number of sensors 111. These sensors111 collect the various desired data, thereby measuring theselected critical functions of the artificial system. Typically,the data gathering of the sensors 111 is accomplished withmethods or techniques well known in the art. The sensors 111

10are connected to a data formatter 112, although alternativeconnection means including optical, RE and the like may besubstituted without departing from the concept of this inven-tion. In many instances, this digital data formatter 112 is a

5 high-speed analog to digital converter. Although, in certainapplications of the invention, namely stock market transac-tions, the data is communicated initially by people makingtrades. Also connected to the digital data formatter 112 is thesimulator 110b. The simulator 110b is an apparatus orprocess

10 designed to simulate the process underlying the state of theartificial system 110a. The simulator 110b is provided togenerate vital data in place of the artificial system 110a, forsuch purposes as education, research, system test, and cali-

15 bration. The output of the digital data formatter 112 is Real-Time data 113. Real-Time data 113 may vary based on theartificial system 110a being monitored or the simulator 110bbeing used and can be selected to follow any desired timeframe, for example time frames ranging from microsecond

20 periodic intervals, for the analysis of electronic systems, todaily statistics reported in an financial trading system. TheReal-Time data 113 is provided to a data recorder 114, whichprovides the means for recording data for later review andanalysis, and to a data modeling processor and process 117. In

25 the preferred embodiments of this invention the data recorder114 uses processor controlled digital memory, and the datamodeling processor and process 117 is one or more digitalcomputer devices, each having a processor, memory, display,input and output devices and a network connection. The data

30 recorder 114 provides the recorded data to a speed controller115, which permits the user to speed-up or slow-down thereplay of recorded information. Scalar manipulations of thetime (speed) in the context of the 3-D modeling of the

35 dynamic recorded digital data allows for new and improvedmethods or reviewing the health of the system 110a,b. Acustomize/standardize function 116 is provided to permit thedata modeling to be constructed and viewed in a wide varietyof ways according to the user's needs or intentions. Customi-

40 zation 116 includes the ability to modify spatial scale (suchmodification including, but not limited to translating, rotat-ing, and zooming), attributes, other structural and symbolicparameters, and viewports in addition to speed. The range ofcustomization form monitoring artificial systems' 110a,b

45 states is wide and not as standardized as that used in thepreferred embodiment of the natural system 101a,b monitor-ing. In this Free Customization, the symbolic system anddisplay method is fully adaptable to the user's needs andinterests. Although this invention has a default visualization

50 space, its rules, parameters, structure, time intervals, andoverall design are completely customizable. This interfacemode customize/standardize function 116 also allows theuser to select what information to view and how to display thedata. This interface mode customization 116 may, in some

55 preferred embodiments, produce personalized displays thatalthough they may be incomprehensible to other users, facili-tate highly individual or competitive pursuits not limited tostandardized interpretations, and therefore permit a user tolook at data in a new manner. Such applications as analysis of

60 stock market data or corporation health monitoring may bewell suited to the flexibility of this interface mode. The datamodeling processor and process 117 uses the prescribeddesign parameters, the customize/standardized function 116and the received real-time data 113 to build a three-dimen-

65 sional (3-D) model in time and to deliver it to a display. Thedisplay of the data modeling processor and process 117 pre-sents a representation 118 of 3-D objects in 3-D space in time

US 7,654,966 B211

to provide the visual representation of the health of the arti-ficial system 110a in time, or as in the described instances ofthe simulated 110b system.

FIG. lc is a top-level block diagram of one preferred pro-cessing path of this invention. Sensors 119 collect the desired 5signals and transfer them as electrical impulses to the appro-priate data creation apparatus 120. The data creation appara-tus 120 converts the received electrical impulses into digitaldata. A data formatter 121 receives the digital data from thedata creation apparatus 120 to provide appropriate formatted iodata for the data recorder 122. The data recorder 122 providesdigital storage of data for processing and display. A dataprocessor 123 receives the output from the data recorder 122.The data processor 123 includes a data organizer 124 forformatting the received data for further processing. The data 15modeler 125 receives the data from the data organizer andprepares the models for representing to the user. The com-puted models are received by the data representer 126, whichformats the models for presentation on a computer displaydevice. Receiving the formatted data from the data processor 20123 are a number of data communication devices 127, 130.These devices 127, 130 include a central processing unit,which controls the image provided to one or more local dis-plays 128, 131. The local displays may be interfaced with acustom interface module 129 which provides user control of 25such attributes as speed 131, object attributes 132, viewports133, zoom 134 and other like user controls 135.

FIG. ld is a top-level block diagram of a second preferredprocessing path of this invention. In this embodiment of theinvention a plurality of entities 136a,b,c are attached to sen- 30sors 137a,b,c which communicate sensor data to a data col-lection mechanism 138, which receives and organizes thesensed data. The data collection mechanism 138 is connected139 to the data normalize and formatting process 140. Thedata normalize and formatting process 140 passes the nor- 35malized and formatted data 141 to the distributed processors142. Typically and preferably the processing 142 is distrib-uted over the Internet, although alternative communicationnetworks may be substituted without departing from the con-cept of this invention. Each processing unit 142 is connected 40to any of the display devices 143a,b,c and receives commandcontrol from a user from a number of interface units 144a, b, c,each of which may also be connected directly to a displaydevices 143a,b,c. The interface units 144a,b,c receive com-mands 145 from the user that provide speed, zoom and other 45visual attributes controls to the displays 143a,b,c.

FIGS. 2a, 2b, 2c, and 2d are representative 3-D objectsrepresenting critical functions. Each 3-D object is provided asa symbol for a critical function of the entity whose health isbeing monitored. The symbol is created by selecting the inter- 50dependent variables that measure a particular physiologicfunction and expressing the variable in spatial (x,y,z) andother dimensions. Each 3-D object is built from 3-D objectprimitives (i.e., a cube, a sphere, a pyramid, a n-polygonprism, a cylinder, a slab, etc.). More specifically, the spatial 55dimensions (extensions X, Y and Z) are modeled after themost important physiologic variables based on (1) data inter-dependency relationships, (2) rate, type and magnitude ofchange in data flow, (3) geometric nature and perceptualpotential of the 3-D object, for example a pyramid versus a 60cylinder, (4) potential of the object's volume to be a data-variable itself by modeling appropriate data into x, y and zdimensions (e.g., in one preferred application of the inven-tion, cardiac output is the result of heart rate (x and y dimen-sions) and stroke volume (z)), (5) orthographic viewing 65potential (see viewport) and (6) the relationship with thenormal values framework.

12The first representative object 201, shown in FIG. 2a, is an

engine process. The object 201 representing this process isprovided on a standard x-y-z coordinate axis 202. The corre-lation between temperature, shown in the xl -dimension 204,engine RPM, shown in the yl -dimension 205 and exhaust gasvolume, shown in the A -dimension 203 is shown by changesin the overall sizes and proportion of the object 201. In theshown example object 201 the engine gas volume 203 is large,when RPM 205 is low and the engine temperature 204 is in themiddle range. This combination of values, even without spe-cific identified values suggests an engine's starting point.

The second representative object 206, shown in FIG. 2b, isan object representing cardiac function using stroke volume,in the y2-dimension 209, and the heart rate per second, shownas the x2, z2 dimensions. The total cardiac volume is shown asthe total spherical volume 208.

The third representative object 211, shown in FIG. 2c,represents the interaction between the number of contracts,shown in the y3-dimension 212, the average revenue percontract, shown in the z3-dimension 214, and the averagetime per contract, shown in the x3-dimension 213. Assessingthe interaction among these variables is important in moni-toring of a sales department's operations.

The fourth representative object 215 is shown in FIG. 2d,shows the respiratory function generated by the respiratoryrate, shown in x4-dimension 216, the respiratory volume,shown in the y4-dimension 216, and inhalation/exhalations,shown in the z4-dimension 218.

FIG. 3 is a representation of data objects in H-space 301.Data sets are represented as 3-D objects of various character-istics and relationships within a 3-D representation space.The data representation environment in this figure is used tomap the physiologic data in 3-D and is what is referred to as"Health-space" or "H-space "301. The 3-D objects are placedwithin H-space on the 3 coordinates of their geometric cen-ters. The coordinates for an object's geometric centerdepends on the relevant data associated to the particular criti-cal function the object represents. For example, in the pre-ferred embodiment, the cardiac function object, shown as aspherical object 302, is placed in H-space 301 based on MeanBlood Pressure, designated as Oy 306 and Oxygen Saturationin the Blood, shown as Oz 307. In the other example object,the prism 309 is placed in H-space 301 depending on salesprofit, shown as Py 312, and products in stock, shown as Pz,311. The location of 3-D objects in H-space 301 allows for theoverall extension envelope of H-space, the relationshipbetween 3-D objects and spaces within H-space 301, theviewport display areas and the departure from normativevalues. Typically and preferably the centers of the objects302, 309 are located in the middle of the x-dimension ofH-space 301.

FIGS. 4a and 4b are representative views of changes in dataobjects in time. In FIG. 4a, the x-coordinate 400 is used tomeasure the temporal dimension of an objects 402 trajectory.The y-z plane 401a determines the location of an object'sgeometric center within H-space. Increases or decreases indata values associated with the coordinates of the object'sgeometric center that make that object's location change intime as shown in path line 401b. In this view, the object 402 ispresented in four different time intervals 403, 404, 405, 406,thereby creating a historical trajectory. The time intervals atwhich the object 402 is shown are provided 407. In FIG. 4b,increases in size and proportion are presented, 408, 409, 410,411 providing an example of changes in values. The moni-toring of these changes in time assists the user establish andevaluate comparative relationships within and acrossH-states.

US 7,654,966 B213

14FIGS. 5a, 5b, 5c, 5d, 5e, 5f, 5g and 5h are representative textual to a particular object 608 and need not be the same for

views of properties of data objects provided in the preferred

all objects, except in the object's 608 extension measuring

embodiment of this invention. In addition to the three x-y-z properties. Fixed x- and z-dimension values 609a and 609bspatial dimensions used for value correlation and analysis, are shown as constant. The y-value 610 of this object 608

3-D objects may present data value states by using other 5 changes to fluctuating values or data type that results in the

geometric, aesthetic, and aural attributes that provide for the

height of the object 608 increasing or decreasing. This view

mapping of more physiologic data. These figures show some shows another object 611 showing the relationship between

of the representative other geometric, aesthetic, and aural

the three dimensions. Constant x- and y-values 612a and

attributes supported for data presentation in this invention. 612b are shown. The z-value 613 of this object 611 changes toFIG. 5a shows changes in apparent volumetric density. A io fluctuating values or data types that result in the width of the

solid object 501 is shown in relation to a void object 502 and

object 611 increasing or decreasing. An overlapping view 614an intermediate state 503 object. FIG. 5b shows changes in of an obj ect 615 that has extended past the H-space limitation.

apparent 3-D enclosure. An open object 504, a closed object

A limit of H-space 616 with a spherical object 617 located

505, and an intermediate state 506 is shown. FIG. 5c shows

inside H-space 616 shown with the degree of extension shownthe apparent degree of formal deformation. A normal object 15 in shaded circles.

507, a distorted object 508, a transformed object 509, and a FIG. 7 shows a series of H-spaces 701, 702, 703, 704, 705,destroyed object 510 are shown in comparison. FIG. 5d

706 along a global time coordinate 708, and the local-space

shows secondary forms of the objects. "Needles" 513 pro- coordinates 707 that governs each H-space. Each of these

truding through a standard object 512 in combination 511 is

H-spaces represents progressive states of the dynamic systemshown in comparison with a boundary 515 surrounding a 20 at pre-established temporal intervals (T o, T_,, T_z, ... T_„)

standard object 514 and a bar 517 protruding into the original

and the six 701, 702, 703, 704, 705, 706 together show the

form object 518 forming a new combination object 516 are evolution of that system overtime, demonstrating the histori-

shown providing additional combination supported in this cal representation of individual H-states within an overall

invention. FIG. 5e shows the various degrees of opacity of the

"Life-space" or "L-space." At the global level (or L-space),object's surface, showing an opaque object 519, a transparent 25 one of the coordinates, typically x, is always time. The tem-

object 520 and an intermediate state object 521. FIG. 5f shows poral coordinate is scaled based on the intervals at which a

the various degrees of texture supported by the object display particular functions system's physiologic data are collected

of this invention, including a textured object 522, a smooth

by the art or as appropriate. This interval or module is fixed

object 523 and an intermediate textured object 524. FIG. 5g is and constant across L-space and provides the necessary tem-intended to represent various color hue possibilities sup- 30 poral frame of reference for comparing different H-spaces.

ported for objects in this invention. An object with color hue

The fixed temporal interval also determines the maximum

is represented 525 next to a value hue object 526 and a x-extension of the representation envelope of H-space. The

saturation hue object 527 for relative comparison. Naturally, other two coordinates, y and z, provide L-space with exten-

in the actual display of this invention colors are used rather sion and are not fixed. The three coordinates thus describedthan simply the representation of color shown in FIG. 5g. 35 provide a regulating 3-D environment within which the

FIG. 5h shows the atmospheric density of the representation

H-states can be visualized and related to each other.

space possible in the display of objects in this invention. An

FIGS. 8a and 8b show the global level coordinate system of

empty-clear space 528, a full-dark space 530 and an interme- the preferred embodiment of this invention. FIG. 8a shows

diate foggy space 523 are shown with 3-D objects shown the L-space coordinate system 801 in its preferred embodi-within the representative space 529, 531, 533. 40 ment. The x-dimension 802 of L-space is mapped to a con-

Aural properties supported in this invention include, but stant time interval, set by means standard in the art or other-are not limited to pitch, timbre, tone and the like. wise as appropriate. The present position of H-state is also

FIG. 6 shows the 3-D configuration of the objects in

indicated on the x-dimension 802. The y-dimension 803 in

H-space in the preferred embodiment of the invention. In this

both positive and negative extensions is measured, up andview the local level, H-space 601 is shown within which the 45 down from the x-axis. This dimension 803 can be mapped to

3-D objects 602, 603, and 604 are located. Object 602 repre- a data variable within particular 3D object in space. The

sents the respiratory function of an individual. Its 602 x-y-z z-dimension 804 is shown inboth positive and negative exten-

dimensions change based on the parameter-based dimensions measured forwards and backwards from the intersect-

sional correlation. The object 603 represents the efficiency of

ing x-axis. This dimension 804 can be mapped to a datathe cardiac system by varying the x,y,z coordinates of the 50 variable within a particular 3D object in space. Now for FIG.

object. The object 604 represents a humanbrainfunction, also 8b aprismatic object 800 represents a critical function, whose

with the x,y,z dimensions changing based on the parameter- evolution is being monitored in L-space, of a given dynamic

based dimensional correlation. In this way the user can easily system. The front view 805 shows the different H-states of the

view the relative relationships between the three physiologi- prism/function 800 using a time T to T-n historical trend. Thecal objects 602, 603, 604. Within H-space 601, the temporal 55 level of intersection and separation between the front views of

coordinate (i.e., periodic time interval for data capturing that

the prism indicate abnormal health states of the critical func-

defines how H-space is plotted in Live-space see FIG. 7) is a tion the object 800 represents. No separation or intersection

spatial dimension on which data is mapped. The x-dimension shows normal function conditions. The trajectory in the y-di-

of 605 of the H-space 601 can be mapped to another indepen- mension of the prism (i.e., H-states of the critical function)dent variable such as heart rate period, blood pressure or the 6o are mapped to a variable that cause their relative position to

like. The location of an object in the y-dimension 606 of

change in the + and y dimension. The current state 806 of the

H-space 601 can be mapped to additional variables that are prism is shown in this front view 805. A top view of 809 of the

desired to be monitored such as SaO2 content, CaO2 content, three-dimensional L-space is shown, showing the evolution

or temperature in the blood. The location of an object in the of the prism 800 backward in time and showing a T to T-Nz-dimension 607 of the H-space 601 can also be mapped to 65 historical trend. The level of intersection and separation indi-

additional variables that the user desires to monitor. A hypo- cate abnormal health states of the particular critical function

thetical object 608 shows that the three coordinates are con- the prism represents. No separation or intersection shows

US 7,654,966 B215

16normal conditions. The trajectory in the z-dimension of theobject is mapped to a variable that causes their position tochange in the + and z dimension. This top view shows both thez and y trajectories in one comprehensive view. The perspec-tive view 808 of L-space gives a comprehensive view of theinteraction of the prisms (the H-states of the function) andtheir movement in all dimensions. The side view 807 ofL-space shows the prisms and their positions in L-space giv-ing a simultaneous view of z and y trajectories.

FIGS. 9a and 9b shows various viewpoints in which thedata may be visualized in the preferred embodiment of thisinvention. This figure shows representations of a data object(a prism) and is provided to show that there are two basictypes of viewports: orthographic and perspectival. The ortho-graphic viewports 906, 907, 908, of FIG. 9b use a parallelsystem of projection to generate representations of H-spacethat maintains dimensional constancy without deformation.Some examples of orthographic views include traditionalarchitectural or engineering views of objects, such as a topview, a front view, and a side view. The orthographic viewportallows for accurate and focused 2-D expressions of the actual3-D object. The perspectival viewport 909, shown in FIG. 9buses a focal system ofprojection to generate depictions analo-gous to our perception of reality but at the cost of deformationand lack of dimensional constancy. For example, the top view902 along with the side view 903 and the front view of 904 ofthe 3-D data object 901 are shown in FIG. 9a. FIG. 9b showsthree orthogonal views 906, 907, 908 along with aperspectiveview 909 of the current data object. The number and types ofviewports used in a particular embodiment of the inventionmay range from one type, for example a perspective viewportallowing immerse virtual reality, to combinations of bothtypes. In the preferred current embodiment, there are the fourviewports shown in FIG. 9b. Given the 3-D nature of dataobjects and H-space, viewports provide the user with differ-ent depictions of the same data.

FIG. 10 shows the transform of an object in space in con-text, with a reference framework, in the preferred embodi-ment of this invention. The referential framework 1010 istypically set based on population normals or patient normals.This framework assists the user to see deviations from normalvery quickly. An individual spherical object 1011 that repre-sents cardiac function is shown located in L-space and inrelation to the referential framework. A side view 1012 isshown along with several cardiac objects. In this view thereferential framework provides a center target point so that auser can make the necessary corrections to bring the objectback to the ideal center of the framework. A perspectival view1013 of the framework is also shown along with severalcardiac objects. The top view 1014 of the framework is shownwith several spherical objects (representing cardiac states).This figure demonstrates the variety of viewports provided tothe user by this invention, which provides enhanced flexibil-ity of analysis of the displayed data.

FIG. 1 l shows the zooming out function in the invention.This invention provides a variety of data display functions.This figure shows the way views may be zoomed in and outproviding the relative expansion or compression of the timecoordinate. Zooming out 1101 permits the user to look at theevolution of the system's health as it implies the relativediminution of H-states and the expansion of L-space. Thisview 1101 shows a zoomed out view of the front view show-ing a historical view of many health states. A side view 1102zoomed out view is provided to show the historical trendstacking up behind the current view. A 3-D perspectival,zoomed out view 1103 showing the interaction of H-states

over a significant amount of time is provided. A zoomed outtop view 1104 shows the interaction of H-states over a largeamount of time.

FIG. llb shows the zooming in function of the invention.5 The zooming in front view 1105 is shown providing an

example of how zooming in permits a user to focus in on oneor a few H-states to closely study specific data to determinewith precision to the forces acting on a particular H-state. Azoomed in side view 1106 is provided showing the details of

io specific variables and their interactions. A zoomed in 3-Dperspective view 1107 of a few objects is also shown. Alsoshown is a zoomed in top view 1108 showing the details ofspecific variables and their interaction.

FIG. 12a shows a 3-D referential framework of normative15 values that is provided to permit the user a direct comparison

between existing and normative health states, thereby allow-ing rapid detection of abnormal states. The reference frame-work 1201 works at both the global L-space level and thelocal H-space level. "Normal" values are established based on

20 average historical behavior of a wide population of systemssimilar to the one whose health is being monitored. Thisnormal value constitutes the initial or by-default ideal value,which, if necessary may be adjusted to acknowledge theparticular characteristics of a specific system or to follow

25 user-determined specifications. The highest normal value ofvital sign A"1202 (+y) is shown, along with the lowestnormal value of `B"1203 (—z), the lowest normal value ofvital sign A"1204 (—y) and the highest normal value of vitalsign `B" 1205 (+z). In FIG. 12b, abnormal values of A" and

30 `B" are shown in an orthogonal view. An abnormally highvalue of A"1206, an abnormally low value of `B"1207, anabnormally low value of A" 1208 and an abnormally highvalue of `B"1209 are shown.

FIG. 13 shows a comparison of the interface modes of the35 preferred embodiment of this invention. This invention pro-

vides two basic types of interface modes: (a) standardized orconstrained customization; and (b) free or total customiza-tion. Each is directed toward different types of applications.The standardized or constrained customization 1301 uses a

40 method and apparatus for user interface that is set a-priori bythe designer and allows little customization. This interfacemode establishes a stable, common, and standard symbolicsystem and displaying method that is "user-resistant". Thefundamental rules, parameters, structure, time intervals, and

45 overall design of L-space and H-space are not customizable.Such a normalized symbolic organization creates a commoninterpretative ground upon which different users may arrive atsimilar conclusions when provided common or similar healthconditions. This is provided because similar data flows will

50 generate similar visualization patterns within a standardizedsymbolic system. This interface method is intended for socialdisciplines, such as medicine in which common and agree-able interpretations of the data are highly sought after toensure appropriate and verifiable monitoring, diagnosis and

55 treatment of health states. The customization permitted in thismode is minimal and is never threatening to render the moni-toring device incomprehensible to other users.

The free or total customization interface mode 1302 pro-vides a symbolic system and displaying method that is

60 changeable according to the user's individual needs and inter-ests. Although the invention comes with a default symbolicL-space and H-space, its rules, parameters, structure, timeintervals, and overall design are customizable. This interfacemode also permits the user to select what information the user

65 wishes to view as well as how the user wishes to display it.This interface mode may produce personalized displays thatare incomprehensible to other users, but provides flexibility

US 7,654,966 B217

that is highly desired in individual or competitive pursuits thatdo not require agreeable or verifiable interpretations.Examples of appropriate applications may include the stockmarket and corporate health data monitoring.

FIG. 14 is a hardware system flow diagram showing vari- 5ous hardware components of the preferred embodiments ofthe invention in a "natural system" medical application. Ini-tially a decision 1401 is made as to the option of using datamonitored on a "real' system, that is a real patient, or datafrom the simulator, for anesthesiology training purposes. If iothe data is from a real patient, then the patient 1402 is pro-vided with patient sensors 1404, which are used to collectphysiological data. Various types of sensors, including but notlimited to non-invasive BP sensors, ECG leads, SaO2 sensorsand the like may be used. Digital sensors 1416 may also 15provide physiological data. An A/D converter 1405, is pro-vided in the interface box, which receives the analog sensorsignals and outputs digital data to a traditional patient monitor1406. If the data is produced 1401 by the simulator 1403, acontrol box and mannequins are used. The control box con- 20trols the scenarios simulated and the setup values of eachphysiological variable. The mannequins generate the physi-ological data that simulates real patient data and doctorscollect the data through different, but comparable sensors.The traditional patient monitor 1406 displays the physiologi- 25cal data from the interface box on the screen. Typically andpreferably, this monitor 1406 is the monitor used generally inan ICU. A test 1407 is made to determine the option of wherethe computations and user interface are made, that is whetherthey are made on the network server 1408 or otherwise. If a 30network server 1408 is used, all or part of the data collectionand computation may be performed on this computer server1408. An option 1409 is proved for running a real time rep-resentation versus a representation delayed or replayed fromevents that previously occurred. For real time operation, a 35data buffer 1410 is provided to cache the data so that therepresentation is played in real time. For the replay of previ-ous events, a data file 1411 provides the means for penna-nently storing the data so that visualization is replayed. Thevisualization software 1412 runs on a personal computer and 40can display on its monitor or on remote displays via theinternet or other networking mechanism. Typically the physi-ological data measured on either a real patient or the simulatorare fed to the personal computer from the traditional datamonitor. A standard interface such as RS232, the Internet, or 45via a server, which receives data from the monitor, may serveas the communication channel to the personal computer run-ning the visualization software 1412. This program 1412 isthe heart of the invention. The program 1412 computes therepresentation and processes the user interface. An option 501413 is provided for computing and user interface on the localdesktop personal computer or for distribution across the Inter-net or other network mechanism. If a local desktop personalcomputer is selected, the personal computer 1414 with anadequate display for computation of the visualization and 55user interface is provided. If a remote user interface 1415 isselected the display and user interface is communicatedacross the Internet.

FIG. 15 is a software flow chart showing the logic steps ofa preferred embodiment of the invention. The preferred 60embodiment of this invention begins by reading the startupfile 1501, which contains the name of the window and theproperties associated with the invention. The properties asso-ciated with the a window include formulas to set object prop-erties, text that is to be rendered in the scene, the initial size of 65the window, the initial rotation in each window, zoom, light-ing and patient data that describes the normal state of each

18variable. Internal data tables are next initialized 1502. Foreach new window encountered in the startup file a new win-dow object is made and this window object is appended to thelist of windows. The window object contains an uninitializedlist of properties describing the state of the window, which isfilled with data from the startup file. The event loop is entered1503. This is a window system dependent infinite loop fromwhich the program does not exit. After some initialization, theprogram waits for user input and then acts on this input. Theprogram then takes control of the event loop for continuousrendering that is if there is no interactivity in the program.Initialization 1504 of windows is next performed. Thisinvolves calls to the window system dependent functions(these are functions that are usually different on differentcomputational platforms) that creates the windows and dis-plays them on the computer screen. In the current preferredembodiment of the invention, OpenGL is required, althoughalternative embodiments using other 3D application pro-gramming interfaces, such as PEX or DirectX, could be sub-stituted without departing from the concept of this invention.Also, in the preferred embodiment of this invention, a per-sonal computer graphics card is preferred in the personalcomputer so as to permit smooth animation with multiplewindows. Although the lack of such a card is not absolutelyrequired for operation of this invention. New data is received1509, typically from the data file 1506 or the data buffer 1507.This new data 1509 can come from any source that generatesfloating-point numbers. The preferred line of data is com-posed of columns of floating point numbers separated byspace. At this point the current time is also stored so that thenext line of data can be obtained at the next user defined timeinterval, which is typically set at about I second. Objectproperties are next computed 1510. This is performed byusing formulas that are specified in the startup file to computeproperties of objects. Data fields in the formulas are specifiedby writing the column number preceded by a dollar sign. Forexample, $1/20.0 would divide the first field by 20.0. Thespecific properties in this application are: cardiac objectdimensions, material properties, and position. Material prop-erties can include the red, green, and blue components as theyappear under ambient, diffuse, and specular light, as well astransparency. The cardiac object position includes the y and zpositions as well as an x shift. If four or more lines of datahave been acquired, the respiratory object properties are com-puted. A delay is necessary because a cubic spline is fitted,using four data points to do the fit, to the data points togenerate a smooth respiratory object. Therefore, until fourtime steps have passed, the curtain is not rendered. Thereafter,it is rendered every time new data is acquired. Cardiac objectproperties include material properties and the height of thecolor bands. Blood pressure object length and materials arethe thin cylinders that go through the top and bottom of eachellipsoid. Next, reference grid properties are computed. Allobjects, except the cardiac object reference are stationary, inthe current implementation. The cardiac object reference canmove according to the movement of the cardiac object if theuser specifies this movement in the startup file. Next, soundsare computed 1511 and made audible 1513. Objects andreference grids are rendered 1512. Before rotation the newestobject appears at the right side of the screen and oldest objectis at the left side of the screen. Sound is produced 1513 next.A test 1514 is next made to determine if smooth animation isselected. If smooth animation is selected the scene will scrollduring the time the program is waiting to get new data. Theprogram, using available computing resources, selects theminimum time increment so that the shift of the objects can berendered within the increment, but limiting the increment to

US 7,654,966 B219

20the smallest increment that human eyes can detect. If smoothanimation is not selected, objects are shifted to the left 1515such that the distance from the center of the newest cardiacobject to that of the former cardiac object is equal to theinter-cardiac spacing. The process waits 1508 until the cur-rent time minus the time since data was last obtained equalsthe data acquisition period specified by the user. If the currenttime minus the time when the data was last acquired equalsthe user specified data acquisition period then a new line ofdata is acquired. If smooth animation is selected, then thecardiac objects are shifted to the left by computing 1516 tothat when it is time to get the next line of data, the cardiacobjects have moved 1517, 1518 such that the distance fromthe rightmost cardiac object to the position where the newcardiac object will appear is equal to the inter-cardiac-objectdistance. For example, if it takes 0.20 seconds to render theprevious scene, the period of data acquisition is 1.0 seconds,and the x shift of the rightmost cardiac object is 0.1 units thenthe program will shift the scene left (0.20/(1.0+0.20)*(1.00.1)-0.15. The formula in the denominator is (1.0+0.20instead of 0.8 because, if the scene has been shifted left suchthat, when new data is acquired, the shifting has stopped(because the position of the cardiac objects satisfies the cri-teria that the distance from the center of the rightmost cardiacobject to the center point where the new cardiac object will berendered— I unit) then the animation will no longerbe smooth,that is, when new data is acquired the animation will appear tostop. Note, that the respiratory object is never entirelysmoothly shifted because no data is available to render theobject at the intermediate time steps.

FIG. 16 is a software block diagram showing the logic stepsof the image computation and rendering process of a pre-ferred embodiment of the invention. This process begins withacquiring the window identification 1601 of the current ren-dering context. Next, the data structure is found 1602 corre-sponding to the current window identification. After which,the view is set 1603. A rotation matrix is set 1604. A projec-tion matrix is set 1605. Lights are set 1606. The back buffer iscleared 1607. Object processing 1608 begins, and includesfor each cardiac object, calling OpenGL to see material prop-erties; shift left one inter-cardiac-object distance; push themodelview matrix, shift x,y, and z directions; call OpenGLutility toolkit to render the cardiac object; set the top cardiacobject material properties, call OpenGL quadries function torender top cardiac object; set top cardiac object materialproperties, call OpenGL quadrics function to render bottomcardiac object and pop modelview matrix. Next, the view isset 1609, as above. The respiratory object is rendered 1610,by setting OpenGL to render quad strips, for each polygonstrip set material properties, and send vertex to OpenGL.Reference grids are rendered 1611 by setting material prop-erty of the cardiac reference grid. The current position is set1612 to be the ideal position of the newest cardiac object, thatis the position corresponding to a patient in ideal health. Thecardiac object material properties are set 1613. The OpenGLutility toolkit is called to render 1614 the cardiac object. Next,OpenGL is set to render quads 1615. After which the materialpropert

mu uuuu ui iiui iiui mu lull iuu mu uui iuu iuui uu uii mi(75) inventors: dwayne westinskow, salt...

Documents