osmius 9 - ucmantares.sip.ucm.es/tarot09/index_files/osmius_tarot09.pdf · integrated data...
TRANSCRIPT
Osmius 9.07The OpenSource Monitoring Tool
José Luis Marina :: 2009 Osmius 9.07
You can't manage what you can't see.
Introducción::ÍndiceReinforcement Learning Bots
2Osmius 9.07: Professional Monitoring July 2009
What is Osmius?
Osmius allows you to monitor and keep track
of everything connected to a network
Introducción::ÍndiceReinforcement Learning Bots
3July 2009
What is Osmius?
We've designed Osmius to monitor:Computers and Network devicesApplications and DatabasesUser transactions...
But also:Industrial sensors.Energy consumptions.Pollution levels in the neighbour.Stock shares.Any idea?
Osmius 9.07: Professional Monitoring
Introducción::ÍndiceReinforcement Learning Bots
4Osmius 9.07: Professional Monitoring July 2009
Why choose Osmius?
Briefly, with Osmius you can:
Monitor thousands of devices in Real Time. Align IT with the Business (SLA Management). Data Mining and Business Intelligence. Use the power of GIS to see/analyze data. Monitor new “things” in minutes. Send automatic notifications to the staff. Predict Capacity shortages (ITIL). Access ALL the code and documentation.
Introducción::ÍndiceReinforcement Learning Bots
5Osmius 9.07: Professional Monitoring July 2009
Who is behind Osmius?
Peopleware: Created in 2000
~ 30 people~ 2,000,000 € Last year billing
Services: Administration, Tuning and IT OperationsSoftware Development. Performance Demanding.Free Open Source Software.Information Security.R&D in collaboration with several Universities.
More:UNEENISO/9001:2000 in IT Consultancy & Operations
PeoplewareFunds to Help R&D
Investments
Technological Partners
Geographic Systems
Business Intelligence
Osmius Partners
InstallationAdministration
SupportTraining
DeploymentDevelopment
MultiplatformInstallers
Introducción::ÍndiceReinforcement Learning Bots
6Osmius 9.07: Professional Monitoring July 2009
Who is behind Osmius?
Community
In Osmius everything is Open... not just the “core” !!
First Stable version released in July 2008More than 4.000 downloadsSourceForgeRanking : ~ Top 50100
Introducción::ÍndiceReinforcement Learning Bots
7
MA
AG1
AGn
MA
Osmius 9.07: Professional Monitoring July 2009
Architecture & Tecnology
With Osmius you can:
Monitor thousands of devices in Real Time. Align IT with the Business (SLA Management). Data Mining and Business Intelligence. Use the power of GIS to see/analyze data. Monitor new “things” in minutes. Send automatic notifications to the staff. Predict Capacity shortages (ITIL). Access ALL the code and documentation.
AG1
AGn
Master AgentPoint from which you want to monitor and deploy Agents.You can use 1 to N
MA
AG1
AGn
CS
Central ServerProcess to receive andcorrelate events andto manage all the agents
MySql
SSL
DataBaseOpen and documentedDataModel.Events, SLA, etc
TomCat
J2EE
WebConsoleJ2EE connecting tothe DBase.
InstancesSomething you want tomonitor (Web, DB, Temp)
AgentSpecialized on monitoringspecific instances likeMySQL, Web sites...
Introducción::ÍndiceReinforcement Learning Bots
8
MA
AG1
AGn
MA
Osmius 9.07: Professional Monitoring July 2009
AG1
AGn
Master AgentC++ & ACEMultiplatformPersistent QueuesFault Tolerant
MA
AG1
AGn
CS
Central ServerC++ & ACE & MySql APIFast insert and CorrelationSecure Comms
MySql
SSL
DataBaseNormalizedExplain everythingStored Procs
TomCat
J2EE
WebConsoleJava J2EESprings – HibernateOpenLayers
InstancesUse APIReuse Connections
AgentC++ & ACEAPI (MySql, Curl,etc)Only new code
Architecture & Tecnology
Introducción::ÍndiceReinforcement Learning Bots
9Osmius 9.07: Professional Monitoring July 2009
Concepts: Easy to understand
Instance: Everything you want to monitor.
Instance Type or Class: Defines the kind of questions you can make to the instanceMySQL Database | Windows Server | Exchange
Instance Events: Depending on its type an Instance can receive certain eventsMySQL Database: # Users, Free Space, Kbytes Out,...Parameters: Every 10 seconds – Customized Warning & Critical Threshold
Event 1 :: Database Connections is: 250Event 2 :: Uptime is : 8h 30'
Event 1 :: Time to load Web is : 15 s
Event 1 :: CPU Load is : 80%
Introducción::ÍndiceReinforcement Learning Bots
10Osmius 9.07: Professional Monitoring July 2009
Concepts: Easy to understand
Correlation Engine I:The Active Events view should be as clean as possible:
Show only 1 row for repeated events.Automatic clean “bad” events when arrives a “good” one.If Temperature is OK clean previous CRITICAL events.
Active andHistorical
ViewsAccumulateOnly 1 row
Correlation Engine II (*)Identify dependencies between Instances:
If this servers fails DataBase and WebServer also fail.→Manual and automatic identification.
Introducción::ÍndiceReinforcement Learning Bots
11July 2009
Concepts: Easy to understand
Service: A group of instances joined to offer a service to some usersNormally represents some functional group into your business
The Intranet...Will be down this afternoon from 15:00 to 17:00due to the upgrade of the Oracle Database from 9i to 10gDon't take this unavailability into account in the SLAs
SLA – Service Level Agreement: Defines objectives to your services.
The Intranet cannot be down more than 99.999 of its timeService Working Time
Are we using the Intranet 24x7? Services must accomplish SLAs within their Working Time Osmius also implements: Planned DownTimes
Osmius 9.07: Professional Monitoring
Introducción::ÍndiceReinforcement Learning Bots
12
Firewall
July 2009
Concepts: Easy to understand
IntranetDB
SLA – Service Level Agreement: Defines objetives to your services.
The Intranet cannot be down more than 99.999 of its time
Service Working Time Are we using the Intranet 24x7?
Services must acomplish SLAs within their Working Time Osmius also implements: Planned DownTimes
IntranetIntranet
GoldGold SilverSilver
eComeCom eMaileMailCRMCRM
99.999 Up99.900 Ok
99.999 Up99.900 Ok
Osmius 9.07: Professional Monitoring
24x7 24x7 24x7
HostMoon
ExchangeServer
Exch.FrontE
WinHost
...
From M to FFrom 9 to 17
SLAManagement
ServiceLevel
Instance &TechnicalLevel
Events sent by the agents
RulesPropagation
Introducción::ÍndiceReinforcement Learning Bots
13July 2009
Concepts: Easy to Manage
Osmius 9.07: Professional Monitoring
Manage the Configuration: Distribute changes from the console to the distributed agentsDistribute or upgrade agents with one click.
Create your own...Events using your customized scripts and distribute them.Agents: Calling Peopleware or building them yourself using the
Osmius Development Framework.
Introducción::ÍndiceReinforcement Learning Bots
14July 2009
Integration
How to deal with millions of events? Round Robin Database for the individual events.
You don't need to know wich was the exact CPU load on 01012008 at 8:00.The older the data the lesser the detail you need.
Osmius 9.07: Professional Monitoring
Integrated Data WarehouseSpecific Osmius processes to aggregate daily info (seconds in each state, etc)
Silent ModeEvents in “silent mode” are only sent when there are changes in the Severity.“Send me the Temperature only when it raises 30 C”Prevents network and storage resources starvation.
User Notificacions & SubscriptionsLet Osmius inform you about changes.
email
SMS....
Service Availability Changes
Instance State Change
Subscribe to Reports
Working time
Not Working time
Out of officeJabber
Global Mark
Sending Osmius Event to othersStandalone Agents: Integrate the agents directly with email, Nagios, …Open Tickets automatically using notifications.Send Exchange Critical Events to another Monitoring system.
IntranetIntranet
eCommeComm
Receiving Events from othersWeb ServicesCommand Line Interface
whenusing
Introducción::ÍndiceReinforcement Learning Bots
15July 2009
Osmius Advantages
Osmius 9.07: Professional Monitoring
Deal with thousands of Instances and millions of events.Integrates SLA management and ITIL best practices.Manage everything from a Central Point.Business Intelligence mining tool.Leverage the power of GIS.Real Open: There is no community downgraded version.You can influence the RoadMap.Well documented. We care about documentation.
16
Introducción
Osmius is supported by:Ministry of Science and Education
Ministry of Industry and TradeCDTI
European Funds
Next Step: Osmius and Passive Testing
What is PasTe?
PasTe: Passive Testing tool.
A library to perform passive testing approach for timedsystems.
Integrated into Osmius as an agent.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 17
What is PasTe?
PasTe: Passive Testing tool.
A library to perform passive testing approach for timedsystems.
Integrated into Osmius as an agent.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 17
What is PasTe?
PasTe: Passive Testing tool.
A library to perform passive testing approach for timedsystems.
Integrated into Osmius as an agent.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 17
Passive testing
In testing, there is usually a distinction between two approaches:
Active testing.
We can interact with the system (apply tests).We have an assesment of correctness before the system isrunning.
Passive testing.
The system is already running.We cannot interact directly with the system.We can observe its traces.
Example
Large systems working 24x7 since this interaction might produce awrong behavior of the system.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 18
Passive testing
In testing, there is usually a distinction between two approaches:
Active testing.
We can interact with the system (apply tests).We have an assesment of correctness before the system isrunning.
Passive testing.
The system is already running.We cannot interact directly with the system.We can observe its traces.
Example
Large systems working 24x7 since this interaction might produce awrong behavior of the system.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 18
Timed Framework I
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
Invariantsφ1 = login/option screen/[20, 40], ?/[0,∞],
disconnection 7→ {welcome screen}/[10, 20] B [35,∞]
φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 19
Timed Framework II
Implementation
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 20
Time Finite State Machines I
Extension of the classical finite state machine model.
The main difference with FSMs is the addition of time.
M = (S , I,O,Tr , sin)
S is a finite set of states
I is the set of input actions
O is the set of output actions
Tr is the set of transitions
sin is the initial state
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 21
Time Finite State Machines II
Transition: tr = (s, s ′, i , o, t)
s, s ′ ∈ S are the initial and final states
i ∈ I is the input action
o ∈ O is the output action
t ∈ IR+ is the time that the transition needs to be completed
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 22
Specification of SSadmin
s0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
i1= connecti2= logini3= disconnectioni4= profilei5= datai6= canceli7= savei8= marksi9= register
i10= data subjecti11= save registrationi12= return optiono1= welcome screeno2= option screeno3= error usero4= profile screeno5= marks screeno6= register screeno7= confirm screeno8= no confirmation screeno9= not spected
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 23
Usual interaction between a student andSSadmin I
Welcome page
The student wants to connect to the system.
She connects to the web page and the system shows thewelcome screen.
s0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
i1= connecto1= welcome screen
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 24
Usual interaction between a student andSSadmin II
Logging into the system
When the student sees the welcome screen she can login into thesystem. If an erroneous login is introduced then the system showsthe error user.
s0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
i2= logino2= option screeno3= error user
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 25
Usual interaction between a student andSSadmin III
Interaction
If the student introduces a correct login, then the system willshow the option screen (in 30 seconds).
If the student wants to disconnect, then she only has to pressdisconnection and SSadmin returns to thewelcome screen (in 15 seconds).
s0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
i2= logini3= disconnectiono1= welcome screeno2= option screen
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 26
Specifying properties: Invariants
Each time that a user applies a and observes z and the time required by thetransition verifies the restriction p.
Some actions are performed and the time required verifies the time restriction p.
If after performing some operations the user inputs i then he must observe andoutput in O, the time required by this transiction satifies the restriction p andthe time for all the sequence satisfies the restriction t.
Formal Definition of Invariants
φ is defined according to the following EBNF:
φ ::= a/z/p, φ |
? /p, φ′ | i 7→ O/p B tφ′ ::= i/z/p, φ | i 7→ O/p B t
p, t ∈ IR, i ∈ I, a ∈ I ∪ {?}, z ∈ O ∪ {?}, and O ⊆ O.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 27
Specifying properties: Invariants
Each time that a user applies a and observes z and the time required by thetransition verifies the restriction p.
Some actions are performed and the time required verifies the time restriction p.
If after performing some operations the user inputs i then he must observe andoutput in O, the time required by this transiction satifies the restriction p andthe time for all the sequence satisfies the restriction t.
Formal Definition of Invariants
φ is defined according to the following EBNF:
φ ::= a/z/p, φ | ? /p, φ′ |
i 7→ O/p B t
φ′ ::= i/z/p, φ | i 7→ O/p B t
p, t ∈ IR, i ∈ I, a ∈ I ∪ {?}, z ∈ O ∪ {?}, and O ⊆ O.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 27
Specifying properties: Invariants
Each time that a user applies a and observes z and the time required by thetransition verifies the restriction p.
Some actions are performed and the time required verifies the time restriction p.
If after performing some operations the user inputs i then he must observe andoutput in O, the time required by this transiction satifies the restriction p andthe time for all the sequence satisfies the restriction t.
Formal Definition of Invariants
φ is defined according to the following EBNF:
φ ::= a/z/p, φ | ? /p, φ′ | i 7→ O/p B tφ′ ::= i/z/p, φ | i 7→ O/p B t
p, t ∈ IR, i ∈ I, a ∈ I ∪ {?}, z ∈ O ∪ {?}, and O ⊆ O.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 27
Invariant φ1
If we see in the log the input login followed byoption screen and an amount of time included in [20,40],...
and we observe any sequence of input/output/time and thenthe input disconnection ...
... then if we observe disconnection we have to see theoutput welcome screen in an amount of time belonging to[10, 20].
... In addition, the sum of all time values observed from thelogin until the welcome screen output belongs to [35,∞].
Invariant
φ1 = login/option screen/[20, 40],
? /[0,∞], disconnection7→ {welcome screen}/[10, 20]B [35,∞]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 28
Invariant φ1
If we see in the log the input login followed byoption screen and an amount of time included in [20,40],...
and we observe any sequence of input/output/time and thenthe input disconnection ...
... then if we observe disconnection we have to see theoutput welcome screen in an amount of time belonging to[10, 20].
... In addition, the sum of all time values observed from thelogin until the welcome screen output belongs to [35,∞].
Invariant
φ1 = login/option screen/[20, 40],? /[0,∞], disconnection
7→ {welcome screen}/[10, 20]B [35,∞]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 28
Invariant φ1
If we see in the log the input login followed byoption screen and an amount of time included in [20,40],...
and we observe any sequence of input/output/time and thenthe input disconnection ...
... then if we observe disconnection we have to see theoutput welcome screen in an amount of time belonging to[10, 20].
... In addition, the sum of all time values observed from thelogin until the welcome screen output belongs to [35,∞].
Invariant
φ1 = login/option screen/[20, 40],? /[0,∞], disconnection7→ {welcome screen}/[10, 20]
B [35,∞]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 28
Invariant φ1
If we see in the log the input login followed byoption screen and an amount of time included in [20,40],...
and we observe any sequence of input/output/time and thenthe input disconnection ...
... then if we observe disconnection we have to see theoutput welcome screen in an amount of time belonging to[10, 20].
... In addition, the sum of all time values observed from thelogin until the welcome screen output belongs to [35,∞].
Invariant
φ1 = login/option screen/[20, 40],? /[0,∞], disconnection7→ {welcome screen}/[10, 20]B [35,∞]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 28
Invariant φ2
After observing any occurrence of login in the log,
then we have to observe option screen or error user.
The amount of time associated with this input/output mustbelong to the time interval [10, 40].
Invariant
φ2 = login
7→ {option screen, error user}/[10, 40] B [10, 40]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 29
Invariant φ2
After observing any occurrence of login in the log,
then we have to observe option screen or error user.
The amount of time associated with this input/output mustbelong to the time interval [10, 40].
Invariant
φ2 = login 7→ {option screen, error user}
/[10, 40] B [10, 40]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 29
Invariant φ2
After observing any occurrence of login in the log,
then we have to observe option screen or error user.
The amount of time associated with this input/output mustbelong to the time interval [10, 40].
Invariant
φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 29
Invariant φ3
After inserting the last change into the profile screen,
and after saving the current state of the system, theoption screen
and the time associated with these operations are included inthe intervals [10, 20] and [20, 40] respectively.
In addition, the total amount of time to perform this activitymust belong to the interval [35, 50].
Invariant
φ3 = data/profile screen
/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 30
Invariant φ3
After inserting the last change into the profile screen,
and after saving the current state of the system, theoption screen
and the time associated with these operations are included inthe intervals [10, 20] and [20, 40] respectively.
In addition, the total amount of time to perform this activitymust belong to the interval [35, 50].
Invariant
φ3 = data/profile screen
/[10, 20]
, save 7→ {option screen}
/[20, 40] B [35, 50]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 30
Invariant φ3
After inserting the last change into the profile screen,
and after saving the current state of the system, theoption screen
and the time associated with these operations are included inthe intervals [10, 20] and [20, 40] respectively.
In addition, the total amount of time to perform this activitymust belong to the interval [35, 50].
Invariant
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40]
B [35, 50]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 30
Invariant φ3
After inserting the last change into the profile screen,
and after saving the current state of the system, theoption screen
and the time associated with these operations are included inthe intervals [10, 20] and [20, 40] respectively.
In addition, the total amount of time to perform this activitymust belong to the interval [35, 50].
Invariant
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 30
Invariant φ4
When the student is in the option screen (after any input),
if she inserts the input interaction marks, then themarks screen will appear within 20 and 40 time units.
All the interaction must take between 30 and 70 time units.
Invariant
φ4 = ?/option screen/[5, 35],
marks 7→ {marks screen}/[20, 40] B [30, 70]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 31
Invariant φ4
When the student is in the option screen (after any input),
if she inserts the input interaction marks, then themarks screen will appear within 20 and 40 time units.
All the interaction must take between 30 and 70 time units.
Invariant
φ4 = ?/option screen/[5, 35],marks 7→ {marks screen}/[20, 40]
B [30, 70]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 31
Invariant φ4
When the student is in the option screen (after any input),
if she inserts the input interaction marks, then themarks screen will appear within 20 and 40 time units.
All the interaction must take between 30 and 70 time units.
Invariant
φ4 = ?/option screen/[5, 35],marks 7→ {marks screen}/[20, 40] B [30, 70]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 31
Algorithm: Correctness of invariants
Input: M = (S, I,O,Tr, sin), φ = {a1, . . . , an−1, in 7→ M} // either ak = ik/ok/Fk or ak = ?Output: Bool.I ′ ← I ; S′ ← S ; j ← 1; S′′ ← ∅; error ← false;
While (j < n)If (Head(I ′) = ?)
While (S′ 6= ∅)Choose s ∈ S′ S′ ← S′ \ {s}S′′ ← S′′ ∪ afterInp{s, ij+1}
elseWhile (S′ 6= ∅)
Choose sa ∈ S′ S′ ← S′ \ {sa}Tr′ ← afterCond{sa, ij , oj}
While(Tr′ 6= ∅)Choose (sa, sb, ij , oj , F ′) ∈ Tr′
Tr′ ← Tr′ \ {(sa, sb, ij , oj , F ′)}If(F ′ = Fj ) then S′′ ← S′′ ∪ {sb}
I ′ = Tail(I ′) j ← j + 1; S′ ← S′′; S′′ ← ∅;
If (S′ = ∅) error ← true
While (S′ 6= ∅)Choose sa ∈ S′ S′ ← S′ \ {sa} Tr′ ← afterCond{sa, in, ?}While(Tr′ 6= ∅)
Choose (sa, sb, in, o, F ′) ∈ Tr′ Tr′ ← Tr′ \ {(sa, sb, in, o, F ′)}If (〈o, F ′〉 ∈ M)
S′′ ← S′′ ∪ {sb}else error ← true
If (S′′ = ∅) then error ← trueReturn (¬error)
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 32
Global view, specification, invariant,implementation, and logs
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
Invariantsφ1 = login/option screen/[20, 40], ?/[0,∞],
disconnection 7→ {welcome screen}/[10, 20] B [35,∞]
φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
m
Implementation
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 33
Global view, specification, invariant,implementation, and logs
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
Invariantsφ1 = login/option screen/[20, 40], ?/[0,∞],
disconnection 7→ {welcome screen}/[10, 20] B [35,∞]
φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
m
Implementation
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 33
Global view, specification, invariant,implementation, and logs
Do not have a fullspecification
Partialspecification.
Only interested insome properties.
Invariantsφ1 = login/option screen/[20, 40], ?/[0,∞],
disconnection 7→ {welcome screen}/[10, 20] B [35,∞]
φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
m
Implementation
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 33
Trace files
Since we assume that we have TFSM, logs will follow this pattern:
Input1/ Output1/ Time1,Input2/ Output2/ Time2,. . .
Inputn/ Outputn/ Timen
Trace
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 34
Algorithm: Correctness of traces with res-pect to invariants
Theorem
Let S (a specification) and I (animplementation) be two Timed FiniteState Machines,
Let φ be a correct timed invariant withrespect to S .
Let e be a log recorded from I .
If the invariant φ does not match e then I doesnot conform to S .
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 35
Core of PasTe
Invariants
Correctspec?
Spec
File
Correcttrace?
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 36
Core of PasTe
InvariantsCorrectspec?
Spec
File
Correcttrace?
Traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 36
Classification of different Invariant Suites
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
⇔
Invariant Suite 2φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
Invariant Suite 1φ3 = data/profile screen/[10, 20], save 7→ {option screen}
/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
Question
Which one do we prefer? Invariant Suite 1 or Invariant Suite 2?
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 37
Classification of different Invariant Suites
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
⇔
Invariant Suite 2φ2 = login 7→ {option screen, error user}/[10, 40] B [10, 40]
φ3 = data/profile screen/[10, 20], save 7→ {option screen}/[20, 40] B [35, 50]
Invariant Suite 1φ3 = data/profile screen/[10, 20], save 7→ {option screen}
/[20, 40] B [35, 50]
φ4 = ?/option screen/[5, 35], marks 7→ {marks screen}/[20, 40] B [30, 70]
Question
Which one do we prefer? Invariant Suite 1 or Invariant Suite 2?
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 37
Mutants: Evaluating invariants I
Mutants are produced from the specification.
We generate traces from mutants.
An invariant suite is better if it kills more mutants.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 38
Mutants: Evaluating invariants II
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
ApplyingMutation ⇒Operators
Mutant 1s0s1
s2s3
s4
s5
s6s7
i1, o3, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
...
Mutant ns0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 25
i10, o6, 15
i11, o7, 80i11, o8, 80
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 39
Mutants: Evaluating invariants III
Changing an output
qa
qb
qc
a,x,1b,y,2
a,x,3
b,x,2
b,z,1a,y,3
qa
qb
qc
a,x,1b,y,2
a,y,3
b,x,2
b,z,1a,y,3
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 40
Mutants: Evaluating invariants IV
Changing the target state in a transition
qa
qb
qc
a,x,1b,y,2
a,x,3
b,x,2
b,z,1a,y,3
qa
qb
qc
a,x,1
b,y,2
a,y,3
b,x,2
b,z,1a,y,3
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 41
Mutants: Evaluating invariants V
Altering the time value
qa
qb
qc
a,x,1b,y,2
a,x,3
b,x,2
b,z,1a,y,3
qa
qb
qc
a,x,1b,y,2
a,x,3
b,x,17
b,z,1a,y,3
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 42
Core of PasTe
InvariantsCorrectspec?
Spec
Mutants
Traces
FileCorrecttrace?
generate
extract
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 43
Core of PasTe
InvariantsCorrectspec?
Spec
Mutants
Traces
FileCorrecttrace?
generate
extract
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 43
Integration into Osmius
Masteragent
OsmiusLog Agent
PasTedaemon
Log fileImpl. traces
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 44
Running Example I
Mutant of SSadmin specification executing in real time.
Paste daemon detecting periodically errors in the trace of themutant.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 45
Running Example II
Specification
TFSM
q0
q0 CONNECT WELCOMESCREEN 10 q1
q1 LOGIN ERRORUSER 30 q1
q1 LOGIN OPTIONSCREEN 30 q2
q2 DISCONNECTION WELCOMESCREEN 10 q1
q2 PROFILE PROFILESCREEN 20 q3
q2 MARKS MARKSCREEN 30 q4
q2 REGISTER REGISTERSCREEN 200 q5
q3 DATA PROFILESCREEN 10 q3
q3 SAVE OPTIONSCREEN 30 q2
q3 RETURNOPTION OPTIONSCREEN 10 q2
q3 RETURNOPTION NOTSPECTED 10 q2
q4 RETURNOPTION OPTIONSCREEN 10 q2
q5 DATASUBJECT REGISTERSCREEN 10 q6
q5 RETURNOPTION OPTIONSCREEN 10 q2
q6 DATASUBJECT REGISTERSCREEN 10 q6
q6 SAVEREGISTRATION CONFIRMSCREEN 80 q7
q6 SAVEREGISTRATION NOCONFIRMSCREEN 80 q7
q6 RETURNOPTION OPTIONSCREEN 10 q2
q7 RETURNOPTION OPTIONSCREEN 10 q2
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 46
Running Example III
Invariants
INVARIANTSI
LOGIN OPTIONSCREEN [ 20 40 ] , * [ 0 -1 ] ,
DISCONNECTION - < WELCOMESCREEN > [ 10 20 ] - [ 35 -1 ]
LOGIN - < OPTIONSCREEN , ERRORUSER > [ 10 40 ] - [ 10 40 ]
DATA PROFILESCREEN [ 10 20 ] , SAVE - < OPTIONSCREEN > [ 20 40 ] - [ 35 50 ]
? OPTIONSCREEN [ 05 35 ] , MARKS - < MARKSCREEN > [ 20 40 ] - [ 30 70 ]
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 47
Running Example IV
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 48
Current work
Planned applications
Analyze implementations of the Optimized Link State RoutingProtocol (OLSR). No formal specification(http://www.ietf.org/rfc/rfc3626.txt) currentresearch with Professor Ana Cavalli research group(Department of Network Software, TELECOM &Management SudParis).
Intrusion detection system (IDS). Invariants to expressbehavioral patterns of crackers, malware and/or disgruntledemployees.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 49
Current work I
Planned applications
Analyze implementations of the Optimized Link State RoutingProtocol (OLSR). No formal specification(http://www.ietf.org/rfc/rfc3626.txt) currentresearch with Professor Ana Cavalli research group(Department of Network Software, TELECOM &Management SudParis).
Intrusion detection system (IDS). Invariants to expressbehavioral patterns of crackers, malware and/or disgruntledemployees.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 49
Current Work II
Methods to extract a set of invariants
Extract⇒
Invariant Suite
φ1
. . .
φn
If we have a complete formal specification we have to checkthe correctness of the proposed invariant suite.
If we do not have a complete formal specification we assumethat the invariant suite is correct.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 50
Current Work III
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
Extract⇒
Invariant Suite
φ1
. . .
φn
We automatically derive invariants from the specification...but there are a lot!!
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 51
Current Work IV
Specifications0s1
s2s3
s4
s5
s6s7
i1, o1, 15
i2, o2, 30
i2, o3, 30
i3, o1, 15
i4, o4, 25i8, o5, 30
i9, o6, 200
i7, o2, 30
i5, o4, 15
i10, o6, 15
i10, o6, 15
i11, o7, 80i11, o8, 80
+
User Model
⇒
InvariantSuite
φ1
. . .
φn
We automatically derive invariants from the specification.
We combine this technique with other formal approaches likeusers models in order to obtain a relevant set of behaviours.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 52
Current work V
Develop a native Osmius agent
Masteragent
OsmiusLog Agent
PasTedaemon
Log fileImpl. traces
OsmiusPasteAgent
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 53
Current work VI
Develop a native Osmius agent
Masteragent
OsmiusLog Agent
PasTedaemon
Log file
Impl. traces
OsmiusPasteAgent
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 53
References
Osmius http://www.osmius.net
Paste http://kimba.mat.ucm.es/paste/
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 54
Publications
amn08b C. Andres, M.G. Merayo, and M. Nunez. Passive testing of timed systems. In 6th Int.Symposium on Automated Technology for Verification and Analysis, ATVA’08, LNCS 5311,pages 418–427. Springer, 2008.
amn09b C. Andres, M.G. Merayo, and M. Nunez. Formal correctness of a passive testing approachfor timed systems. In 5th Workshop on Advances in Model Based Testing, A-MOST’09 .IEEE Computer Society Press, 2009.
amn09c C. Andres, M.G. Merayo, and M. Nunez. Using a mining frequency patterns model toautomate passive testing of real-time systems. In 21st Int. Conf. on Software Engineeringand Knowledge Engineering, SEKE’09. Knowledge Systems Institute, 2009.
amn09d C. Andres, M.G. Merayo, and M. Nunez. Supporting the extraction of timed properties forpassive testing by using probabilistic user models. In Submitted to QSIC’09, 2009.
alr09 C. Andres, L. Llana, and I. Rodrıguez. Formally transforming user-model testing problemsinto implementer-model testing problems and viceversa. Journal of Logic and AlgebraicProgramming, pages 425-453, 2009.
amm09 C. Andres, M.G. Merayo, and C. Molinero. Advantages of mutation in passive testing: Anempirical study. In 4th Workshop on Mutation Analysis, Mutation’09. IEEE ComputerSociety Press, 2009.
amn09 C. Andres, M.G. Merayo, and M. Nunez. Passive testing of stochastic timed systems. In 2ndInt. Conf. on Software Testing, Verification, and Validation, ICST’09, pages 71–80. IEEEComputer Society Press, 2009.
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 55
Thank you for your attention!
The PASsive TEsting tool and Osmius Las Navas del Marques, July 2009 56