gregor stewart - osira 2014
DESCRIPTION
Basis Technology's OSINT is a web-based application that enables analysts to extract and resolve named entities, including people, places, and organizations, from large quantities of multilingual text. OSINT is deployable in the cloud or on premises, and can be customized to address different data sources, knowledge bases, and workflow models.TRANSCRIPT
A Collabora)ve, En)ty-‐Centric Approach to Mining OSINT Data
Gregor Stewart
Director of Product Management
Basis Technology Corpora=on
Open Source Intelligence !and Research Association Inaugural Meeting
Key Issue: Things not Strings!
2
§ Brief Introduc=on to Basis § Review two technologies that take us from Strings to Things: – Extrac=on: finding names and types in text – Resolu=on: connec=ng names to things
§ A Search Example, illustra=ng: – How En=ty-‐based querying can help find things quickly – How human feedback can be used to improve accuracy over =me
Overview
3
Introduc)on: Basis Technology
4
English
Pronoun Verb Name
Pronoun+VerbVerb NounConjunction Title Name
Pronoun Verb NameDeterminer Noun
Verb Name Title Name NamePrep.
Noun VerbRel. Pronoun Verb Verb Prep. Verb Noun
Pronoun+Verb Adjective Noun Prep. Adjective Adjective
What is Big Text ?
It's huge volumes of multilingual , unstructured
text that must be processed to deliver insights
and build connections . It’s President Clinton
helping Malawi . Secretary Clinton in .
The福島 meltdown . This is Big Text .
دابآ ملسا
Concept
Person
ConceptPlace
Place Person Place
Urdu: “Islamabad”
Japanese:"Fukushima”
Introduc)on: Gregor Stewart
5
En)ty Extrac)on: What is it?
6
Unhasu Orchestra
Kim Jong-ilKim
HyunRi Sol-ju
Wangjeasan Light Music Band
South KoreaChosun Ilbo
Chosun
North KoreaChina
Hyon Song-wolKim Jong-un
Kim Jong-un
Unhasu Orchestra
Chosun Ilbo
China
Hyon
Kim Jong-ilKim
Ri Sol-ju
Wangjeasan Light Music Band Twitter
South Korea
Chosun
North Korea
Kim Jong-unKim Jong-un
Unhasu OrchestraUnhasu Orchestra Chosun Ilbo
Chosun Ilbo
ChinaChina
HyunHyon Song-wol
Hyon
En)ty Extrac)on: How is it Done?
7
Probabilis=c Extractor
Supervised Model
Unsupervised Model
Determinis=c Extractor
Exact Match (GazeOeer)
PaOern Match (Regex)
En=ty Re
dactor
En=ty Joining Input Text
Filtering
Overlap Adjudica=on
Tagged Text
Domain Text
Output Text
User Defined Lists
User Defined PaOerns
Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
FISHING NEWS
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
FISHING NEWS
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
George H. W. Bush41st U.S. PresidentID: USPRES41DOB: June 12, 1924
George Herbert Walker Bush is an American politician who served as the 41st President of the United States.
George W. Bush43rd U.S. PresidentID: USPRES43DOB: July 6, 1946
George Walker Bush is an American politician and businessman who was the 43rd President of the United States of America from 2001 to 2009 and the 46th Governor of Texas from 1995 to 2000.
FISHING NEWS
Noelle BushID: BD239852
Unknown Person
Unknown OrganizationPenobscot Boat BuildersID: TF354723
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
1 Identifying Known Entities
2 Inferring New Entities
En)ty Resolu)on: What is it? (1)
En)ty Resolu)on: What is it? (2)
Alberto
Alberto
Alberto Alberto
Alberto Amos Fernandez…
Alberto M. Fernandez…
Alberto Fernandez…
Alberto Fernandiz…
Albert Fernandez…
Alberto
Alberto
Alberto Alberto
Alberto Fernandez… … Chief of Cabinet … Argen=na… …Prof of Criminal Law…
Alberto Fernandez… … born Sept 7, 1984 … cycling … Madrid
Alberto Fernandez… … born in Cuba … US Ambassador
Alburto Fernandez…
Alberto
Alberto Fernandez de la Puebla…
Alberto
Ratio of Politicians to Sportsmen?
2:1
Alberto Fernandez…
Sportsmen? YES
Nickname “El Galleta?”
?
En)ty Resolu)on: How is it Done? (1)
En)ty Resolu)on: How is it Done? (2)
11
En)ty Resolu)on: How is it Done? (3)
12
En)ty Resolu)on: How is it Done? (4)
13
Resolu=on Engine Candidate Selec=on
En=ty Index
En=ty Men=on
+ Context
Link or Ghost
Ranking
Knowledge Base
Learned
Seeded
!
§ In a far away place, not so long ago… the CIA was asked by the US President to assess the likelihood that the Syrian opposi=on would use chemical weapons by mid-‐2014.
§ As part of building that analysis, and because there are Al-‐Qaeda elements in the Syrian opposi=on, Alice the analyst was asked to: characterize Al-‐Qaeda’s a2tude to using chemical weapons against Middle Eastern governments.
A (Convenient) Fic)on…
14
15
A document that Alice needs to read…
16
From: Ayman Al-‐Zawahiri (?) To: “Hafiz Sultan”
17
From: Ayman Al-‐Zawahiri (?) To: “Hafiz Sultan”
Dear Brother, We need guidance from you on the issue of using chlorine gas technology. It was reported that the brothers in Iraq have used it, but this was implicitly denied in a statement issued by the Islamic State of Iraq. The brothers where Mahmud is have the poten=al to use chlorine gas on the forces of the apostates, Jalal Talabani and Mas'ud Barzani, and have already considered using it. However, I informed them that maOers as serious as this require centralized [coordina=on] and permission from the senior [al-‐Qa'ida] leadership, because the gas could be difficult to control and might harm some people, which could tarnish our image, alienate people from us, and so on.”
18
From: Ayman Al-‐Zawahiri (?) To: “Hafiz Sultan”
Dear Brother, We need guidance from you on the issue of using chlorine gas technology. It was reported that the brothers in Iraq have used it, but this was implicitly denied in a statement issued by the Islamic State of Iraq. The brothers where Mahmud is have the poten=al to use chlorine gas on the forces of the apostates, Jalal Talabani and Mas'ud Barzani, and have already considered using it. However, I informed them that maOers as serious as this require centralized [coordina=on] and permission from the senior [al-‐Qa'ida] leadership, because the gas could be difficult to control and might harm some people, which could tarnish our image, alienate people from us, and so on.”
19
From: Ayman Al-‐Zawahiri (?) To: “Hafiz Sultan”
3/28/07
20
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
…characterize Al-‐Qaeda’s a2tude to using chemical weapons against Middle Eastern governments.
21
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
22
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
23
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
24
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
25
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
26
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
27
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
28
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
29
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
30
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
31
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
32
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
33
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
34
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
35
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
36
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
37
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
38
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
39
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
40
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
41
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
42
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
43
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
44
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
45
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
46
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
47
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
48
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
49
3/28/07 Today 5/2/11 5/3/11 5/12/11 5/14/11
Summary
§ By Connec=ng Strings to Things in text, exis=ng knowledge can be applied: – And we can know where exis=ng knowledge is lacking (ghosts)
§ Indexing text by Things allows each query to be more precise and produc=ve – Fewer queries, fewer adjustments, fewer things to read
§ By using abundant human feedback, resolu=on accuracy can get beOer over =me without effort. – More subtle dis=nc=ons between en==es can be learned by example
§ But…
…it’s not the shoes
51
Thank You! Gregor Stewart ([email protected]) Director of Product Management
Basis Technology Corpora=on
Open Source Intelligence !and Research Association Inaugural Meeting