![Page 1: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/1.jpg)
Open Source Intelligence:
Presented by Abe Lederman, President and CTO
Deep Web Technologies, LLC
IOP ’06 Sheraton Premier, Tysons Corner, Virginia January 16-20
Access All Intelligence, in All Languages, All the Time
![Page 2: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/2.jpg)
About Deep Web Technologies (DWT)
• Deployed first “federated search” portal in the Federal Government, 1999
• Major clients include:– DOE Office of Scientific & Technical Information– Defense Technical Information Center– Science.gov Alliance– DOE Office of Science– National Agricultural Library
DWT is a New Mexico based company focused on providing state-of-the-art software solutions which search, retrieve, aggregate, and analyze content.
![Page 3: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/3.jpg)
Open Source Intelligence
The Problem:
• Collecting and analyzing enormous quantities of information in any language, in myriad formats, located anywhere, accessible through a large variety of means, with a majority not accessible through the Internet
![Page 4: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/4.jpg)
Shared Challenge: OSINT and Knowledge Discovery/Diffusion
OSINT Challenges
Knowledge Discovery/ Diffusion
Challenges
DWT for the past six years has been the lead technical organization addressing these challenges in collaboration with DOE Office of Scientific & Technical Information
![Page 5: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/5.jpg)
The DWT Proposition
To apply DWT’s technology, expertise and ongoing innovations* to address the challenges of OSINT
*Developed in partnership with DOE/OSTI
![Page 6: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/6.jpg)
Challenges in Working with Thousands of Data Sources
Locate Reliable Sources
Categorize Sources by Content
Configure Sources for Searching
Maintain Sources
![Page 7: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/7.jpg)
Challenges in Searching Thousands of Sources
Automatically Select Sources to Search
Perform Many Searches in Parallel
Translate, Analyze and Organize Results
Relevance Rank
Cluster/ Visualize
Extract Key Information
![Page 8: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/8.jpg)
DWT’s State-of-the-art Federated Search Engine
• Scalable, grid-computing based federated search engine
• Sophisticated Search Conductor• Supports custom connectors• Multi-tier relevance ranking• Framework accepts integration of advanced
linguistic, analyses, and visualization modules
ResearchAssistantTM
![Page 9: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/9.jpg)
Grid Computing: Distributing the Workload
![Page 10: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/10.jpg)
Search ConductorSelect sources
to search
Perform search
Deliver results to user
Can I get more results from “good”
sources?
Enough good
results?
YES
YES
NO
NO
![Page 11: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/11.jpg)
Multi-tier Relevance Ranking
• QuickRankTM – Ranks results based on occurrence of search terms in title and snippet
• MetaRankTM – Ranks results utilizing custom algorithms applied to metadata
• DeepRankTM – Downloads and indexes full-text documents
![Page 12: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/12.jpg)
Science.gov Alliance Consortium of 12 Federal Government Agencies
Dept of Agriculture
Dept of Commerce
Dept of Defense
Dept of Education
Dept of Energy
Dept of Health/Human Services
Dept of Interior
Environmental Protection Agency
NASA
National Science Foundation
US Government Printing Office
National Archives & Records Administration
Sponsoring
Science.gov Portal
(Access to most of Federal Government R&D
![Page 13: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/13.jpg)
Science.gov Advanced Search Page
![Page 14: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/14.jpg)
Science.gov Results Page
![Page 15: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/15.jpg)
A Science.gov Document
![Page 16: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/16.jpg)
Next Steps
Identify Sponsors and development partners that can collaborate on the development of a pilot that integrates best-of-breed technologies of value to OSINT.
This pilot will result in a portal that aggregates content of different types, generating actionable intelligence.
![Page 17: Open Source Intelligence: Presented by Abe Lederman, President and CTO Deep Web Technologies, LLC IOP 06 Sheraton Premier, Tysons Corner, Virginia January](https://reader035.vdocuments.site/reader035/viewer/2022062404/55149f4d550346ea6e8b58f5/html5/thumbnails/17.jpg)
Contact Us
Abe Lederman
122 Longview Drive
Los Alamos, NM 87544
www.deepwebtech.com
http://www.deepwebtech.com/talks/IOP.ppt