a review paper on open source intelligence: an ......3.5. open source intelligence (osint): open...

12
DOI:10.23883/IJRTER.2018.4261.SBMQL 463 A Review Paper on Open Source Intelligence: An intelligence sustenance Annie Ahuja Department of Computer Science,Guru Nanak Dev University AbstractOpen Source Intelligence is the process of identifying, collecting data and then exploit that data in order to use in an intelligent manner. The value of Open Source Intelligence with the advancement of technology has become quite significant. In the recent years, it has seen enormous growth in comparison to the Closed Source Intelligence that emphasis on secretive information which is even complex to handle. Open Source Intelligence has become a key component in national security and also find its place in organizations and businesses to empower the decision making process by gathering the relevant strategic information. By making the use of Open Source Intelligence tools, we can analyze the unstructured information elicited from various publicly and freely available sources and make that structured one which can then help businesses to become more efficient in taking strategic decisions. This paper will review the process, methods, tools used, pros, cons, future challenges in this growing area of research and also highlights the traditional and emerging information gathering intelligence methodologies. KeywordsHUMINT, SIGINT, OSINT, SOCMINT, MASINT, TECHINT Crowd Source intelligence, Deep web, Dark web, GEOINT, Market Intelligence, Maltego, Shodan, Metagoofil, FOCA, EXIF data viewers I. INTRODUCTION OF OSINT Open source is the information source which is freely available or open for accessing to everyone. Open Source Information (OSINF) is the key component of Open Source Intelligence (OSINT). The information elicited from open sources is unclassified that is not structured like newspapers, reports, public data, maps, academic sites, social networking sites, blogs, apps and web-based communities. The transition from Open Source Information to Open Source Intelligence occurs when the gathered information is exploited and disseminated within time constraints to a selective audience and the purpose is to address a specific intelligence requirement [17]. Unlike Closed Source Intelligence or Classified Information which is secretive in nature, Open Source Intelligence is based on unclassified information which is freely available and within everyone’s reach. Earlier about 20 years back, the issue was lack of available information that hinders the decision making process of an organization or individual or big businesses. With the passage of time, this issue is not more an issue as with the growth of IT sector, there are many information sources available to you in the form of newspapers, emails, blogs, social networking sites like Face book, Twitter, etc. Now the concern is having too much flow of information and which is freely available to you on just one click and you are unable to decide whether this information is accurate or not. OSINT is now everywhere in a seamless manner. Even if you are looking for some specific information on the web and you are analyzing by yourself that this particular information is quite reliable, this is also one form of Open Source Intelligence which you are doing at your own end and you are unaware of that. The traditional methods of data collection such as Using Search Engines like Google, Yahoo, etc. news sites like CNN, BBC, etc. and Corporate and Government Websites, Blogs, etc.

Upload: others

Post on 24-May-2020

36 views

Category:

Documents


1 download

TRANSCRIPT

DOI:10.23883/IJRTER.2018.4261.SBMQL 463

A Review Paper on Open Source Intelligence:

An intelligence sustenance

Annie Ahuja

Department of Computer Science,Guru Nanak Dev University

Abstract— Open Source Intelligence is the process of identifying, collecting data and then exploit

that data in order to use in an intelligent manner. The value of Open Source Intelligence with the

advancement of technology has become quite significant. In the recent years, it has seen enormous

growth in comparison to the Closed Source Intelligence that emphasis on secretive information

which is even complex to handle. Open Source Intelligence has become a key component in national

security and also find its place in organizations and businesses to empower the decision making

process by gathering the relevant strategic information. By making the use of Open Source

Intelligence tools, we can analyze the unstructured information elicited from various publicly and

freely available sources and make that structured one which can then help businesses to become

more efficient in taking strategic decisions. This paper will review the process, methods, tools used,

pros, cons, future challenges in this growing area of research and also highlights the traditional and

emerging information gathering intelligence methodologies.

Keywords— HUMINT, SIGINT, OSINT, SOCMINT, MASINT, TECHINT Crowd Source

intelligence, Deep web, Dark web, GEOINT, Market Intelligence, Maltego, Shodan, Metagoofil,

FOCA, EXIF data viewers

I. INTRODUCTION OF OSINT

Open source is the information source which is freely available or open for accessing to everyone.

Open Source Information (OSINF) is the key component of Open Source Intelligence (OSINT). The

information elicited from open sources is unclassified that is not structured like newspapers, reports,

public data, maps, academic sites, social networking sites, blogs, apps and web-based communities.

The transition from Open Source Information to Open Source Intelligence occurs when the gathered

information is exploited and disseminated within time constraints to a selective audience and the

purpose is to address a specific intelligence requirement [17].

Unlike Closed Source Intelligence or Classified Information which is secretive in nature, Open

Source Intelligence is based on unclassified information which is freely available and within

everyone’s reach. Earlier about 20 years back, the issue was lack of available information that

hinders the decision making process of an organization or individual or big businesses. With the

passage of time, this issue is not more an issue as with the growth of IT sector, there are many

information sources available to you in the form of newspapers, emails, blogs, social networking

sites like Face book, Twitter, etc. Now the concern is having too much flow of information and

which is freely available to you on just one click and you are unable to decide whether this

information is accurate or not. OSINT is now everywhere in a seamless manner. Even if you are

looking for some specific information on the web and you are analyzing by yourself that this

particular information is quite reliable, this is also one form of Open Source Intelligence which you

are doing at your own end and you are unaware of that.

The traditional methods of data collection such as Using Search Engines like Google, Yahoo, etc.

news sites like CNN, BBC, etc. and Corporate and Government Websites, Blogs, etc.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 464

The modern methods of data collection such as Advanced search engines like Bing, Social Media

Sites like Facebook, Twitter, etc and Deep Web /Dark Web searching [1].

II. EVOLUTION OF OSINT

There were many unusual challenges faced by Intelligence Community since its establishment in

1947. It faced a lot more criticism that got intensified at the end of Cold War. The Community faced

unusual challenges that includes discontinuation of the former Soviet Union, the 9/11 terrorist

attacks, and the Iraq war of 2003 [15] and its after consequences. The occurrence of all these events

forced to reform the working scenario of Intelligence Community. To affect the Intelligence

Community, it took around 25 years after the existence of Internet and Personal computer impact

continues.

Intelligence Community has been using open source information, still the progress has been slow in

recognizing its value due to the reason that it always considers closed sources of collecting

information are more valuable as it is difficult to gather information from such sources. With the

enormous growth in Internet and thus in availability of information increased manifold.

The dependence of Intelligence Community on open source information also increased. The required

information was mostly available openly. Shifting to open source scenario was accepted publicly and elated by open source advocates such as Robert Steele of Open Source Solutions, who brings

together experts from all over the world.

The National Intelligence Council former head, Joseph Nye, in the 1990s captures the prevailing

scenario of Intelligence Community when he stated, “Open source intelligence is the outer pieces of

the jigsaw puzzle, without which one can neither begin nor complete the puzzle…open source

intelligence is the critical foundation for the all-source intelligence product, but it cannot ever replace

the totality of the all-source effort” [15].

Open Source Intelligence is that which involves unclassified information and freely available but it is

not always easily found. The search engines like Google do the surface web searching but not the

deep web searching. Surface web searching refers to the searching when we type into the search box

of the Google and then it retrieves the information for which we are looking into. Sometimes, the

information like UGC approved list of journals that we are looking for, it just gives back the website

reference as outcome but not the desired information. The next step we perform is the within website

search box we type our search query and then we get back the desired result. This is called deep web

searching. Without following the methods, process, tools and techniques the navigation of OSINT

would not have been possible.

III. LITERATURE SURVEY

OSINT has gained importance few years back with the upcoming challenges in gathering

information from various sources. Before the upbringing of Open Source Intelligence, the traditional

intelligence (INTs) gathering sources include: [4] [13]

3.1. Signal Intelligence (SIGINT):

It is derived from the information gathered by signal interception. SIGINT capabilities are

maintained by various countries like U.S., Germany, Canada, China, Israel, India, and Germany and

so on. In recent times, with the enormous growth at technological front, involves a higher proportion

of national intelligence budgets. In the UK, it is the key source of intelligence as it accounts for

around 80 percent of information source. According to NSA, SIGNIT process is represented as

shown in Figure1. [16] According to NSA views, SIGINT is flexible as it has the ability to re-target

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 465

SIGINT if the required expertise is available for the whole duration in comparison to HUMINT, the

barrier in redirecting its focus is that instant changes in emphasis requires the creation of new

information sources although support is provided by national intelligence in such situations.

Figure1. Signal Intelligence Process

SIGINT also provides the sense of reliability to the decision-makers as compared to HUMINT as

there might be the scenario that they do not match in their views. Signal in the form of electronic

intercepts considered to be more reliable than agents. But there are some potential problems with

SIGINT like significant amount of information cannot be analyzed due to the abundant supply of

information creates huge volume traffic, misinterpretation of signals [16].

3.2. Geospatial Intelligence (GEOINT):

It refers to the analysis of imagery which is then exploited to gain insight of the geographically

referenced activities online. According to De jure, Geospatial Intelligence definition is found in U.S.

Code Title 10: “The term “geospatial intelligence” means the exploitation and analysis of imagery

and geospatial information to describe, assess, and visually depict physical features and

geographically referenced activities on the earth. Geospatial intelligence consists of imagery,

imagery intelligence and geospatial information.” [27] It is considered to be multi-source of

information as it integrates the information gathered from other INTs into a specific context.

3.3. Human Intelligence (HUMINT):

It refers to the information gathered from human sources. According to the definition given by ICD

304, Clandestine collection considered to be closed source of information or classified information

refers to the secrecy in the sponsored or conducted activities. Overt collection considered being open

source of information or unclassified information refers to the eliciting information without

concealment and in a legal way [26]. In the HUMINT sphere, human factor is the key factor for

Customer Environment

Customer Target Source: <http://www.nsa.gov/sight/index.cfm>

Intelligence

Knowledge

Information

Data

Signal

Use It

Know It

Get It

Applied Knowledge

Facts/Relationships (Context)

Discrete Facts (Entities)

Bits/Streams (Receptacles)

Electrical Impulses (Sensors)

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 466

information gathering. This information source is crucial for gaining insight into the adversaries and

then allowing the corresponding action for achieving the aim of national security.

3.4. Measurement and Signature Intelligence (MASINT):

It refers to the intelligence gathered by equipment like radar that analyzes the different features of

data so that targets can be identified and tracked. [25] The accuracy in MASINT proves to be

advantageous in the military operations planning and deployments. When the collection of images is

coupled with spectral signature library then it can be used to survey the large areas and even able to

target the disturbed surfaces and camouflaged targets which is also one of the application of this

intelligence gathering discipline.

3.5. Open Source Intelligence (OSINT):

Open Source Intelligence is defined as according to U.S law and Intelligence Community as OSINT

is ‘‘intelligence that is produced from publicly available information and is collected, exploited, and

disseminated in a timely manner to an appropriate audience for the purpose of addressing a specific

intelligence requirement.’’ [6]

Intelligence Community Directive (ICD) 301 defined ‘‘Open Source Acquisition’’ as ‘‘The act of

gaining possession of, or access to, open source information synonymous with ‘open source collection.’ The preferred term is acquisition because by definition, open sources are collected and

disseminated by others open source exploiters acquire previously collected and publicly available

information second-hand’’ [8]. It is necessary to do this alternation in the terms from open source

collection to open source acquisition as the discipline OSINT means information elicitation from

open sources while the other discipline HUMINT refers to gathering information that may constitute

some questionnaires or involving public forums as information source.

Open Source Intelligence is generally considered as the gathering of information from publicly

available sources that are open and free to everyone but not all sources are open to public domain.

The Figure2 below represents the relation between open sources and intelligence. Examples include:

detail of publication which belongs to reputed journals which cannot be accessed otherwise without

having proper rights to access that, communication talks with the academicians, officials, etc.

Keeping this in mind, Stevan Dedijer listed four different types of information: [6]

white information (public information)

grey information

black information (classified information)

non-existing information

White information refers to that information which is completely available and accessible to public

domain without having property constraints and according to our estimate, around 90 percent of all

data in intelligence. Although it is publicly accessible but that does not mean it is always free.

‘‘Grey’’ information refers to the information which is gathered by using appropriate communication

channel and represents around 9 percent of all data in intelligence. The information exists but might

not get published or in process of publishing. Black information which is also termed as classified

information, gathered through covert activities which are not openly acknowledged, represents

around 0.9 percent of all in intelligence which is quite small. So grey information, refers to the

information gathered from legal or semi-legal sources, forms a part of what can be called the ‘‘grey’’

zone of OSINT. Such information can or cannot be reliable as the source of information is though

legal but everyone is not aware of its presence. Its subcategory, is generally called as grey literature

which refers to legal as well as ethical available information and it is acquired through proper

communication channels such as technical reports, documents, yearbooks, dissertations, working

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 467

papers, data sets, private companies reports and personal notes produced by non-profit organizations,

educational organizations, enterprises, government agencies, associations, etc.

Figure2. Relation between open sources and intelligence [29]

With advancement in technology, the other emerging intelligence gathering sources includes:

3.6. Market Intelligence:

It refers to the information which is gathered for a specific purpose like decision making in

marketing strategy. Market Intelligence plays a key role in determining strategies and success in the

market place. Customers, manufacturing and Research and Development are the core data sources in

Market Intelligence. The phase transition in markets and businesses characterizes the incorporation

of Market Intelligence and Business Intelligence in New Intelligence phase. It is believed that

involvement of supplier in market information collection activities leads to grater return on

investments. It does have a very positive impact if suppliers are involved in pre design activities in

marketing strategy. [5] [14]

3.7. Technical Intelligence (TECHINT):

It refers to gathering of information through the technology means. With the advancement in

technology, it has become easier to intrude into areas and locations that were earlier quite

challenging task. With the passage of time, there is more support for TECHINT in terms of financial

sources as compared to HUMINT. It should be realized that humans cannot be eliminated from any

intelligence sources. Thus it could be a better approach to fuse the intelligence so that it comes out to

be a powerful intelligence technology which complements each other and serves national security in

a best manner. [12]

3.8. Social Media Intelligence (SOCMINT):

It is the age of social media due to enormous growth of interest in social networking sites like

Facebook, Twitter, Linkedin and so on. It introduces a big change in people’s life in terms of

interactions, presenting views, etc. This change is happening on a very large scale. There are 250

million photos that are added per day, 200 million tweets on Twitter and 4 billion video views per

day. [18] People are sharing valuable information like it provides mental health status and the

indications by participants involved in school shootings such as the suspect TJ lane, involved in the

Ohio School Shooting, gives the impression of some indicative content as well as sensitive

information like your personal photos, working status. Even UN also observes that by analyzing the

information available on social networking sites can help in tacking crucial issues like job

insecurities, etc. There should be some framework within which it is to be used.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 468

3.9. Crowd Source Intelligence:

It refers to the intelligence gathered from the crowd i.e. humans source. As a whole, it purely

depends on human activity. Of the intelligence technologies, HUMINT and OSINT based on

gathering information sources by humans and then perform an analytic analysis of information which

can then act as a base for crowd sourcing. The other intelligence technologies do not have humans as

a core component for gathering information, therefore cannot be a part of crowd source intelligence.

[19] [20]

IV. OSINT PROCESS

The Open Source Intelligence process consists of the following generalized steps [28]:

4.1. Collect: It is the initial step in OSINT process. In this, when the request comes from some source like

employee in government agency, manager in a company or a CEO of enterprise for an intelligence

study, it just started from here. The core task of this step is to identify information sources and then

retrieve the relevant information.

4.2. Process: The next task of Open Source specialist in OSINT process is to process that collected information using Open Source Intelligence Tools. The exploitation of data is only possible when processing of

data occurs. After the completion of this step, exploited data is given to analyst.

4.3. Analyze: In this step, processed information is being analyzed by analyst who has a deep understanding of the

search request queries which is given by some person to the intelligence team and then makes a

report of the analyzed study. The Figure 3 is showing the OSINT process.

Figure3. The OSINT Process

4.4. Visualize: In this step, after the exploited data being analyzed by analyst, a written document report is prepared

that contains visuals in the form of images, maps, etc where required. Visualize the text data is very

Collect

Process

Analyze Visualize

Collaboration

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 469

effective in understanding the true context of conveying by the analyst which is corresponding to

search request query given by an individual or an intelligence team.

4.5. Collaboration: In this step, intelligence is moved from producers to consumers. The feedback is carried out in both

ways that is before and after the intelligence is received. It is a process that is carried out to address

the finished intelligence and becomes the deciding factor for further collection and analysis, if

required.

V. OSINT TOOLS

The various tools that are available in order to carry out the Intelligence study are as follows [7][21]:

5.1. Maltego:

It is OSINT tool that provides platform for extraction of data about the company, businesses or

enterprise like address of employees, designation of employees, IP address information, DNS

records, social networking activity of employees, frequent website searching, phone numbers and so

on and this all happen with the use of search engines on the Internet. This tool communicates to elicit

all the information and also represent the extracted data in a specific format which is easy to

understand and analyze.

5.2. Shodan:

It stands for Sentiment Hyper Optimized Data Access Network. The traditional each engines that

uses crawling technique to obtain the desired result and then used to display. It is available in both

versions-free as well as paid. With the free version, maximum limit is 50 results. For over and above,

you need to have paid version. The main purpose of this tool is to find vulnerable services in a web

server which plays key role in assessing the vulnerability. The various filters that can be applied to

this tool such as country, operating system and host names in order to further enhance the outcome.

5.3. Metagoofil:

It is also one of the Open Source Intelligence tools which are used to extract metadata from the target

destination. This OSINT information gathering tool can also extract MAC addresses from the files

having pdf, doc, xls and ppt extensions and thus provides an idea of network hardware resources on

the target machine. When enough information is extracted from the files then attacker can do brute

force attack. Automated software is used in brute force attack for the purpose of guessing the users

information like password, PIN (Personal Identification Number), etc. The metadata extracted with

this tool can also be used to extract the path information which finds its use in mapping a network.

The results are displayed in HTML format.

5.4. FOCA:

It stands for Fingerprinting Organizations with Collected Archives. It can also analyze metadata like

Metagoofil. Crawl option is provided in this tool for getting more detailed information about

insecure methods. It can also extract information about users, folders, emails, software used,

operating system and other useful information.

5.5 EXIF data viewers:

EXIF stands for Exchangeable Image File Format which is used in smart phones and digital cameras

that specifies image formats and sound formats for recording. The purpose of this tool is to provide

geo location which is stored for each image in the device. This can leak the sensitive information like

image location as GPS feature of smart phones is ON by default.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 470

VI. ADVANTAGES OF OSINT

Open Source Intelligence has come out to be a wide potential source of information elicitation. The

following are major advantages of open source intelligence: [2][10][11]

6.1. Less Expensive:

One of the biggest advantages of Open Source Intelligence as compared to the traditional intelligence

disciplines is that it is less expensive. Most of the information to be collected is easily approachable

using Internet. The organizations that have a tight intelligence budget can find this approach very

useful.

6.2. Accessing Information:

The information is available anytime and anywhere. Moreover, open sources are always up to date

with the information related to any topic. For example: Twitter is open and always accessible at

anytime. People use to tweet the latest information and thus one of the best source of up to dated

information.

6.3. Security:

It plays a key role in achieving national security as the information elicited from publicly available

sources like social networking sites, blogs, etc and hence becomes a great source of intelligence for attaining national security by analyzing the growing interest of people.

6.4. Business:

OSINT has become a core component for almost all businesses as it supports in decision making

process by planning long term strategies. The ability to understand, identify, classify and exploit this

publicly available information can be very helpful in tackling various business processes and

predicting crucial issues.

6.5. Social Media:

Now days, Social media like Face book, Twitter, Linkedin and so on are part of everyone’s life.

Everyone is active on these social networking sites. These social media outlets provide information

about people’s real life, regarding current trends, and other sensitive information like images and so

on. So, now it has become mandatory for an organization (profit or non-profit), commercial

enterprises due to potential for providing critical insights.

6.6. Updated data and Metadata:

Social networking information sources like Twitter usually provides the current up to date

information in comparison to other websites which have accumulated the content from various

sources. Even Metadata can be easily collected as each and every digital image contains metadata.

6.7. Semantic understanding:

Incorporation of semantic word to the open source intelligence allows removing the vague pieces of

information until a whole understandable context of information begins to emerge.

6.8. Applications:

Some of the examples where OSINT finds its applicability includes: Everyone is active now on

social media that helps organizations to protect the citizens especially children and women by

evaluating situations and also identify the critical cases by analyzing the growing trends for defining

strategies, Finds its usage in government agencies as well related to tax collection by analyzing the

target’s lifestyle, behavior, Applies text mining and hence finds its application in online

counterfeiting as well.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 471

VII. OBSTACLES OF OSINT

Open Source Intelligence has also the other side that depicts its weaknesses. There is no intelligence

gathering technique that does not have cons; open-source intelligence is also one of them. Following

are the obstacles in conducting open source intelligence study and needs to be tackled by intelligence

community. [3][10][15][22][23][24]

7.1. Reliable:

As open source information collected from various publicly and freely available sources, sometimes

it can be misleading as well. It needs to be checked against information that is available from closed

sources or classified information.

7.2. Information Overload:

The major concern is the information which is available freely and in huge amount, so it is quite

difficult to figure out the reliable sources. Sometimes, it is not even possible to process and analyze

such big amount of data.

7.3. Tools:

Without using OSINT tools, it is very cumbersome task to identify and search the right content

suited to your requirements. Choosing the right tool justifies the outcome of conducted intelligence study.

7.4. Analysis:

In the analysis, humans perform the analytical task so that correct or relevant information from the

misleading ones can be identified. Critical analysis is carried out by analyst and then analyzed report

is generated having visuals where required as it makes the interpretation of text in an easy manner.

7.5. Time-Consuming Activity:

In recent years, with the advancement of technology, huge amount of data or information is

available. So in order to carry out open source intelligence process for identifying, searching and

collecting processes is quite time consuming task.

7.6. Requirement Understanding:

In order to get the desired results in a timely and cost-effective manner one has to have the proper

understanding of the requirements. It is the initial stage in open source intelligence process and the

most critical one as well. The understanding of requirements forms the basis for further stages which

are carried out during the process. If initially it gets to the other direction then outcome will

definitely not move to the intended direction.

7.7. Business Intelligence:

OSINT combines with traditional intelligence information sources and becomes a powerful approach

that greatly empowers the decision making process of any business. In the new phase of Intelligence

study, Business Intelligence and Market Intelligence have a significant contribution in taking

decisions respective to the prevailing market trends.

7.8. Security Issues:

Whenever online research is conducted, the OSINT researcher must use VPN as it encounters

vulnerabilities in their own privacy and security. It is one of the major obstacles while doing

searching on Internet. If someone is not taking special precautions while doing deep web searching

then your sensitive information like IP address, etc will not be any more sensitive.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 472

7.9. Deep Web content:

There is a big difference in the searching mechanism used by surface web and deep web. Searching

that we usually done on the internet is surface web in which search engines create their indices by

crawling web pages/ surface web pages. The traditional search engines cannot able to search deep

web pages as they cannot go underneath the surface for retrieving the content. Deep web involves the

creation of dynamically created web pages in response to direct search request from searchable

databases. The standard search engines are not able to find most of the information as it is available

only in extending mode of search. Bright Planet’s search technology is the only search technology

that has the capability to handle direct queries simultaneously by using multiple-thread technology

and thus able to handle the process of identifying the content then qualify to make the data classify

and at last organize the content of deep web and surface web. Deep web is a storehouse of valuable

information.

Unlike, surface web that create indices while crawling the web pages, deep web pages searching

mechanism is bit different as it retrieves the content from searchable databases which are not

indexed. Dark Web deals with the sites that sell drugs, hacking software, etc which cannot be

otherwise done with the help of surface web. It is a subset of deep web. If you are looking for hidden

marketplace where you can carry out the above said activities then this is the solution that is dark net

websites with .onion domain. This is all possible with the TOR browser in TOR network.

VIII. FUTURE CHALLENGES OF OSINT

Following are the challenges that need to be addressed by intelligence community are as follows.

These are the predictable key challenges faced by intelligence community in conducting intelligence

study. [9][15]

8.1. Content Review: The major and foremost challenge before Intelligence Community is to provide users the desired

content of their area of interest. In order to provide relevant information to the intended user websites

and content providers have been studying users’ needs. This procedure is already being implemented

on one of the most popular social networking site that is Twitter where the users can indicate the

relevancy of content on the spot in the form of their valuable feedback. It is not easy to identify the

falsification of content. Therefore, some mechanism is to be adopted like data pattern mining that

identifies the consistent clustering pattern. Other measure is to have trust scores associated with each

social networking site that would prove to be the best solution.

8.2. Translation tools:

A good translator can further improve the output of automatic translation engine by using automatic

translation tools. The purpose of translating engines is to allow reading of texts in any language

without having language constraint. The constant improvement in these engines makes them almost

100% accurate with the passage of time.

8.3. Twitter:

The information before appearing in printed form is shared on the Twitter which is highly used by

journalists. They use to tweet their ongoing reports which includes variety of visual information.

8.4. Smart phones as information source:

Smart phones revolutionized the whole information industry. You can easily carry this small size of

information source, though information that you get is as big as you can think. Moreover, in this

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 473

world of information, by using advanced search engines one can retrieve the visual content in the

form of pictures, videos by just inputting the textual data.

8.5. Automated tools: With the Emerging automated tools, helps to identify and address the information preferences of the

individual users. The free availability of these types of tools would definitely have an adverse impact

on the conductance of intelligence study.

8.6. Professional Collaborations: Professional Collaboration would further enhance the filtering, processing and accessing of open

source information [9]. The sole purpose of collaboration is to provide collaboration between

producers and consumers so that producers can get valuable feedback from the customers who can

be very helpful for further processing and analysis of data.

IX. CONCLUSION

OSINT is now widely used information gathering source in an intelligent manner with the help of

available intelligence tools. Intelligence tools have a very significant impact in the exploitation of

data and it’s analyzing in such a critical manner that it fulfills the purpose of intended user. Earlier it

was not given much importance due to much consideration to the classified sources of information which were quite difficult to gather as they are secretive and complex in nature but with the passage

of time it has gained its own existence and also become an integral part now in almost every sector.

It rather works along with the traditional sources of information and has become the most powerful

intelligence discipline and the most dominating intelligence discipline.

REFERENCES I. Bjore, M. (2015, March 26). Osint overview. Retrieved April 17, 2018, from www.slideshare.net:

https://www.slideshare.net/mumlan/osint-overview-26-mar-2015-46413841

II. BrightPlanet. (2013, April 4). What is OSINT and how can your organization can use it. Retrieved April 15,

2018, from brightplanet.com: https://brightplanet.com/2013/04/what-is-osint-and-how-can-your-organization-

use-it/

III. BrightPlanet. (2013, January 3). Why you should tap into the deep web in 2013. Retrieved April 15, 2018, from

brightplanet.com: https://brightplanet.com/2013/01/why-you-should-tap-into-the-deep-web-in-2013/

IV. Christopher Andrew, R. J. (2009). Secret Intelligence: A Reader. London and New York: Routledge Taylor &

Francis Group.

V. Conway Lackman, K. S. (2000). The contribution of market intelligence to tactical and strategic business

decisions. Marketing Intelligence & Planning , 18 (1), 6-9.

VI. Gasper Hribar, I. P. (2014, May 12). OSINT: A "Grey Zone"? International Journal of Intelligence and

CounterIntelligence , pp. 529-549.

VII. Guurhart. (2016, September 8). Resource: OSINT tools and how you learn how to use them. Retrieved April 16,

2018, from www.peerlyst.com: https://www.peerlyst.com/posts/resource-osint-tools-and-how-you-learn-how-

to-use-them-guurhart

VIII. INTELLIGENCE COMMUNITY DIRECTIVE NUMBER 301. (2006, July 11). NATIONAL OPEN SOURCE

ENTERPRISE.

IX. Lapid, E. (2016). OSINT: A MAJOR SOURCE OF UP-TO-DATE INFORMATION. NATIONAL SECURITY

AND THE FUTURE , pp. 37-42.

X. Marco Varone, D. M. (n.d.). Advantages and disadvantages of open source intelligence. Retrieved from

www.expertsystem.com: http://www.expertsystem.com/advantages-disadvantages-open-source-intelligence/

XI. Marco Varone, D. M. (n.d.). The value of open source intelligence in the 21st century. Retrieved from

www.expertsystem.com: http://www.expertsystem.com/value-open-source-intelligence-21st-century/

XII. Matthew Crosston, F. V. (2017, January). Cyber, Intelligence, and Security. An Intelligence Civil War:

"HUMINT" vs. "TECHINT" , 1 (1) . The Institute for National Security Studies- Strategic, Innovative, Policy-

Oriented Research.

XIII. Michael Glassman, M. J. (2012). Intelligence in the internet age: The emergence and evolution of Open Source

Intelligence (OSINT). Computers in Human Behavior , 28 (2), 673-682.

International Journal of Recent Trends in Engineering & Research (IJRTER) Volume 04, Issue 04; April- 2018 [ISSN: 2455-1457]

@IJRTER-2018, All Rights Reserved 474

XIV. Michael Song, J. T. (2008). The Role of Suppliers in Market Intelligence Gathering for Radical and Incremental

Innovation. THE JOURNAL OF PRODUCT INNOVATION MANAGEMENT .

XV. Paulson, T. M. (2008). Intelligence Issues and Developments. New York: Nova Science Publishers,Inc.(2006).

XVI. In M. P. Peter Gill, Intelligence in an Insecure World (pp. 61-78). UK: Polity Press.

XVII. Quirine Eijkman, D. W. (2013, April). Open Source Intelligence and Privacy Dilemmas: Is it Time to Reassess

State Accountability? Security and Human Rights , p. 23.

XVIII. Sir David Omand, J. B. (2012, September 28). Introducing Social Media Intelligence. Intelligence and National

Security , pp. 801-823.

XIX. Steele, R. D. (2010). HUMAN INTELLIGENCE: ALL HUMANS, ALL MINDS, ALL THE TIME.

XX. Stottlemyre, S. A. (2015). HUMINT, OSINT, or Something New? Defining Crowdsourced Intelligence.

International Journal of Intelligence and Counter Intelligence , 578-589.

XXI. Target, T. (n.d.). Nine must-have OSINT tools/Nine OSINT tools every security researcher must have. Retrieved

April 16, 2018, from www.computerweekly.com:

https://www.computerweekly.com/photostory/2240160102/Nine-must-have-OSINT-tools/1/Nine-OSINT-tools-

every-security-researcher-must-have

XXII. Tarquin. (2018, April 13). How To Access Notorious Dark Web Anonymously (10 Step Guide). Retrieved April

21, 2018, from darkwebnews.com: https://darkwebnews.com/help-advice/access-dark-web/

XXIII. White Paper: The Deep Web: Surfacing Hidden Value. The journal of electronic publishing , 7 (1).

XXIV. Wright, A. (2008, October). Searching the Deep Web. 51 (10) , 14-15. Communications of the ACM.

XXV. Richelson, J. T. (2016). The U.S. Intelligence Community (Seventh ed.). Boulder: Westview Press. 1947, T. N.

(2009, July 9). INTELLIGENCE COMMUNITY DIRECTIVE NUMBER 304, HUMAN INTELLIGENCE.

XXVI. Todd S. Bacastow, D. B. (2009). Redefining Geospatial Intelligence. American Intelligence Journal , 27 (1), 38-

40.Best, C. (n.d.). Open Source Intelligence. (T.R.I.O), T. R. (2017). Background / OSINT. Retrieved April 23,

2018, from www.trioinvestigations.ca: https://www.trioinvestigations.ca/background-osint