80

UNIT 4: OSINT – OPEN SOURCE INTELLIGENCE

Structure:
4.0 Introduction
4.1 Learning Objectives
4.2 Open Source Intelligence (OSINT)
4.3 Uses of OSINT
4.4 Types of OSINT
4.5 Target Audience
4.6 Importance of OSNIT
4.7 OSNIT and Information Security
4.9 OSINT Tools
4.9.1 Maltego
4.9.2 the harvester
4.9.3 Shodan
4.9.4 SpiderFoot
4.9.5 Spyse
4.9.6 BuiltWith
4.9.7 Intelligence X
4.9.8 Metagoofil
4.9.9 Search code
4.10 Let Us Sum Up
4.11 Check Your Progress: The Key
4.12 References and Suggested Readings

4.0 INTRODUCTION

In the previous unit, we discuss the personal data protection Act. We shall
discuss Open Source Intelligence (OSINT) and its tools in the context of
Information Security. Let us first understand the meaning and concept of
OSINT before using it in the context of information security.

4.1 OBJECTIVES
 81

After studying this unit, you will be able to:

 describe (OSINT), and its uses for Information Security; and
 explain the importance of OSINT and its tools used for
Information Security

4.2 OPEN SOURCE INTELLIGENCE (OSINT)

Open-source intelligence (OSINT) gathers information via published or

openly accessible sources. In OSINT, advanced techniques are adapted to
locate or retrieve data buried inside the internet, which is not easy to
access directly or simply a regular user employs through search engines. It
is just like finding a needle in a haystack where intelligence/data is created
after analyzing the data related to a particular topic.
IT pros or malicious hackers generally employ highly advanced
techniques or state-sanctioned intelligence operatives to locate data and
create intelligence out of a particular topic or individual or accompany
while ignoring the data present over the surface via regular search engines.
Many techniques and tools are used worldwide for such operations, but
OSNIT is the most recognized. In other words, we can say that OSINT is
intelligence “extracted from publicly accessible material.” It means the
intelligence gathered is from the data intended for public consumption.
There is a significant difference between using OSINT and hacking or
using illegal means. Unlike hacking or stealing credits to illegally logging
inside someone's account, OSNIT is just collecting intelligence even while
browsing through someone’s social media profile. The information is
drawn and worked upon only through non-classified sources and while not
using covert actions.
OSINT-gathering activities are organized secretly to circumvent the
identity-revealing part of the process. So, it comes in handy if one knows
how to suppress its own digital identity and be anonymous online. In
OSNIT, data exchanges can also be done discreetly across unfriendly
environments so that the communication is anonymous and private.
We can learn about our digital footprint and retracing our digital traces
over the internet. To dig deep inside the vast internet space, one should at
 82

least have firsthand knowledge of operating various tools regarding

OSNIT.

4.3 USES OF OSINT

It is primarily used in national security, business intelligence, and law

enforcement. The details of its uses are presented below:

 Intelligence agencies use OSINT for so-called “targets of interest’.

They tend to keep a tab on various events, various types of chatter
on multiple platforms, equipment such as weapons systems, and
people.
 Hackers use it to identify technical vulnerabilities and human
targets to extract money by using phishing and other social
engineering strikes. To counter this, OSNIT helps security teams
arrange similar techniques to uncover and close down these
weaknesses.
 OSINT is used to counter the threat of attacks due to the clues that
entities leave exposed, compromising security. Similar to a
vulnerability scanner to discover flaws in a system, OSINT tools
gather complex data, like DOB, Identification numbers like Social
Security numbers, family relations, and even day-to-day routes and
hobbies that become the building blocks of compromising the
security of an account.

OSINT includes all publicly accessible sources of information. This

information can be found either online or offline.
Online: The Internet, which includes blogs, social networking sites,
video-sharing sites like YouTube.com, wikis, whose records of registered
domain names, metadata, and digital files, dark web resources, IP
addresses, people search engines, and anything that can be found online.
Offline: Traditional mass media: television, radio, newspapers, books,
magazines, specialized journals, academic publications, dissertations,
 83

conference proceedings, company profiles, annual reports, company news,

employee profiles, and résumés, annual reports, conference proceedings,
employee’s profile, press release, government publication, courts
proceedings, and informal agencies.

4.4 TARGET AUDIENCE

OSNIT analysis pulls together pieces of intelligence from public sources
to create a profile for the target. It is generally studied and employed by
the following types of users for data gathering:

 Government agencies, such as the military departments, are

significant clients of OSINT resources. More recently, major
technological advances and widespread use of the Internet
have made governments the primary users of OSINT
intelligence. These OSINT resources are needed for different
purposes like cyber-terrorism, counter-terrorism measures,
national security, understanding domestic and foreign public
opinion on various issues, and providing legislatures with
valuable information to influence their internal and external
policies.
 Law Enforcement Agencies: Different companies use OSINT
sources to protect citizens from abuse, identification theft, and
other crimes. It may be carried out by monitoring social media
channels for exciting key phrases and pix to help save you from
crimes earlier than they increase. Regulation enforcement uses
OSINT to screen and music a criminal’s networks throughout
distinct international locations. For instance, they use OSINT
approaches to compile information about human beings of interest
to create a complete profile for each one. They also use OSINT
resources for online counterfeiting and copyright violations.
 Business Corporations: Information is powerful, and companies
use OSINT resources to research new markets, monitor
 84

competitors' activities, plan marketing activities, and predict

anything that may affect their current performance and future
growth. In the past, the exploitation of OSINT resources was
limited to large businesses with reasonable intellectual budgets.
With the widespread use of the Internet, small budget companies
can use OSINT resources effectively and integrate the information
gained from their business plans.
 Penetration Testers and Black Hat Hackers/Criminal
Organizations: OSINT is used extensively by hackers and
penetration testers online to gather intelligence about a specific
target. It is also considered a valuable tool to aid in the conduct of
social engineering attacks. The first phase of any penetration
testing method begins with reconnaissance. Companies pay
penetration testers to break into internal networks to show where
vulnerabilities are and how to keep outsiders out. This is different
from black hat hackers who exploit these vulnerabilities to gain
unauthorized access to confidential data; However, both use the
same reconnaissance techniques and equipment to achieve their
task.
 Privacy-Conscious People: These are ordinary people who might
want to check how outsiders can break into their computing
devices and what their ISP knows about them. They also want to
recognize their online publicity stage to shut any protection gap
and delete any private facts that could have been posted
inadvertently. OSINT is a great device to look at how your digital
identity seems to the out of doors global, allowing you to hold your
privacy. People can also use OSINT to fight identification theft, for
instance, if someone is impersonating them. In this e-book, we can
educate you on different strategies to search for textual content,
photos, and video, as well as digital file metadata.

4.5 IMPORTANCE OF OSINT

 85

OSINT helps keep tabs on the information disarray or the bomb that
explodes now and then, with social media capturing the whole internet
scene altogether. There is so much fake information or boloney that must
be cleaned up to find the real story or information. Some of the common
reasons for organizations to follow OSINT are: (a) finding insecure
devices, (b) identification of unintentional leakage of sensitive data, (c) for
updating software, and (d) leakage of highly confidential data. The details
of the importance of OSINT are presented below:

Information Gathering Types

OSINT sources can be collected using three main methods: passive, semi-
passive, and active. One usage depends on the scenario in which the
gathering process operates in addition to the type of data. The three
gathering techniques generally describe how footprinting works, in other
words, acquiring technical information about the target IT infrastructure
(types of OS, network topology, server names, and so on).

Passive Collection: This is the most used type when collecting

OSINT intelligence. Indeed, all OSINT intelligence methods
should use passive collection because the main aim of OSINT
gathering is to collect information about the target via publicly
available resources. In this type, the target has no clue regarding
the intelligence-collecting activities. This kind of search is highly
anonymous. From a technical perspective, this type of gathering
reveals limited information about the target mainly because no
traffic (packets) is sent to the target server, directly or indirectly.
The primary resources gathered are generally limited to archive
information, unprotected files left on target servers, and content
present on the target website.

Semi-passive: From a technical view, this gathering sends limited

traffic to target servers to acquire general information. This traffic
resembles typical Internet traffic to avoid paying attention to
 86

reconnaissance activities. Instead of an in-depth investigation of

the target’s online resources, this only implements a light
investigation without initiating any alarm on the target’s side.
Although this gathering type is considered anonymous, the target
can know that surveillance is happening if they investigate the
issue.

Active Collection: In this type, there is a direct interaction with the

system to gather intelligence about it. The collecting information
will use advanced techniques to generate technical data on IT
infrastructure. For example, hackers may access open ports, scan
vulnerabilities (unpatched Windows systems), scan web server
applications, etc. This traffic will look suspicious or malicious
behavior and leave traces on the target’s intrusion detection system
(IDS) or intrusion prevention system (IPS). Conducting social
engineering attacks on the target is also considered a type of active
information gathering.

Advantages of OSINT
 Some of the benefits of the open-source gathering have been
underlined below:
 Sometimes, due to budget allocation, conventional information
collecting practices are not economically viable. So, OSNIT
plays an important role, as while employing OSINT for
intelligence gathering, the level of financial investment is
deficient.
 The information obtained during the process is not classified
and revealed at will, so getting any such information is an
excellent legal process.
 Users often share and update the information as the whole
process depends on public resources.
 87

 Business owners and decision-makers can also increase

perceptivity through OSINT data, which helps develop long-
term policies for diverse business goals.
 One of the main advantages is its invaluable role in national
security matters.

Disadvantages of OSINT
 One can easily understand that as quickly as one can muster
intelligence, a rival, on the contrary, can also employ it to
gather information about anything and use it for illegal
purposes.
 Just retrieving information does not have a substantial
advantage unless it is used meaningfully. The cleansing of junk
data from treasured details is one of the main challenging tasks
and depends on the volume of data we find.
 Sorting data is in itself very time-consuming.
 Data validation is another crucial factor that hinders the
usability of OSNIT, as one can even deliberately put false
information to mislead. Hence, the validation for making data
reliable is also a key factor.

4.8 OSNIT AND INFORMATION

SECURITY
The use of OSNIT is multifaced; on the one hand, it enables
individuals/corporations to keep tabs on their own publicly available
information over the internet that can help the hacker to get hold of and
manipulate various vital information. It also helps find measures for
reducing and eliminating such critical data to a large extent. Furthermore,
some of its objectives have been mentioned below:

 Public-Facing Assets: The most usual role is to help

recognize public-facing assets and then plot what kind of
information each holds that might be attributed to a potential
 88

attack surface later. Their purpose is not to find software

vulnerabilities or run penetration testing on them but to analyze
and document what information can be publicly discovered
about company assets or the company itself without hacking.
 Information Outside the Organization: Information about
an individual or a company doesn't need to be stored by
themselves via the appropriate platforms, but it can be outside
these premises like social media posts or at various domains or
just lying over the internet arbitrarily without the knowledge of
that individual or company. So just looking within doesn’t
serve the whole purpose of security. The tools must be
equipped to find relevant data about a subject, even outside the
premises.

 Assembling Discovered Information: Merely finding the

data or information won’t do any good. The tools must be
equipped to analyze the information gathered and convert it
into actionable intelligence that helps build strong defenses and
speculate the origins of future attacks.
 89

4.9 OSINT Tools

Check Your Progress 1

Note:

a. Space is given below for writing your answers.

b) Evaluate your answers with the answers given at the end.

i. What is Open-Source Intelligence?

ii. How can open-source intelligence assist my
organization in its investigations?

--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------

OSINT Framework

OSINT framework is a cyber security framework. OSINT tool helps in

data collection. Its’ tool for one’s organization can drastically expand
cyber security by aiding the discovery of information about the company,
its employees, IT assets, and various other classified/sensitive data that
can get into the wrong hands and thus be exploited by them.

Some of the tools used for OSINT are:

 Maltego

 the harvester
 Shodan
 SpiderFoot
 Spyse
 BuiltWith
 Intelligence X
 90

 Metagoofil
 Search code
 Google Dorks
 Recon-ng
 Babel X
 Mitaka

4.9.1 Maltego

The main focus of Maltego is to unearth various relationships

among individuals, enterprises, domains, and publicly accessible
information over the internet. It is beneficial in extracting
mammoth chunks of unearthed information and then thoroughly
strategizing it in charts and graphs. These graphs come in handy
for cultivating raw intelligence into actionable ones. Moreover,
every graph can hold up to thousands of data points.

It has been a visual interface that automates searching various

public data resources. A search plan is labeled as a “transform
action,” and many search plans, by default, include familiar
sources of general information. The program's main benefit is that
it uses public interfaces to accomplish its searches, making it
compatible with virtually several sources of information.
It maps the relationships between pieces of information. It uses
Entities, and Transforms, where Entities are “bits of information”
that we have obtained from a data source, and A Transform is “the
bit of code that generates some information based on a bit of
information we already have.
Transforms are generally written in Python, but there is no
restriction on which programming language can be used. Queries
that generate Transforms from Entities are XML-based, so the
Python code converts the questions and responses into Python
objects.
 91

These Transforms run on Transform servers that can interact with

data necessary for performing Transforms. A Maltego Desktop
Client doesn’t connect directly to this server, however. Maltego
uses servers called Transform Distribution Servers for making
connections:
When the client runs a Transform, the request is forwarded to
TDS. It is forwarded to the correct Transform server, and the
results are sent to users via their client.

Figure 4.1: Maltego

Source: docs:maltego.com

There are two different approaches to OSINT using Maltego:

Active: There is direct contact in real-time with the target;

it then provides more accurate data.

Passive: There is no direct contact with the target based on

historical or third-party records; it has a low risk of
detection. Here the information might be outdated.

One of the significant advantages of Maltego for OSINT is that

it is practically very private, even when more active OSINT
measures are being run. When a Transform is executed, the
Maltego Client interacts with a TDS, which, in turn, interacts with
 92

a Transform server that essentially handles the Transform. This

signifies that even if the target perceives that it is being scanned, it
would only be the Transform server detected. So, while the target
may know it has been scanned, it cannot deduce who observed it
first.

4.9.2 theharvester

It was designed as one of the most straightforward tools to gain

public information outside the organization’s network. It
effectively works as a reconnaissance phase before various
exercises like penetration testing.

Some popular search engines like Bing and Google, as well as

lesser-known ones like dogpile and DNS dumpster, are the primary
source of the harvester. It also gathers emails, names, subdomains,
IPs, and URLs.

It is a Python script that allows us to quickly and accurately catalog

e-mail addresses and subdomains directly related to our target.

It can search Google, Bing, and PGP servers for e-mails, hosts, and
subdomains. It also searches LinkedIn for user names. Most people
assume their e-mail address is benign, but one should be aware of
additional hazards. For example, during a reconnaissance, one
discovers an employee's e-mail address from your target
organization. Anyone can create a series of potential network
usernames by twisting and manipulating the information before the
“@” symbol.
 93

Figure 4.2: theharvester

Source: https://medium.com/hacker-toolbelt/the-harvester-
osint-reconnaissance-91a18a294a30

It is common for organizations to use the same user names and e-

mail addresses, with a handful of prospective usernames, so one
can easily attempt to attack via brute force into any services, like
SSH, VPNs, or FTP.
There are four different installation methods for theharvester.
These are:

 Kali Linux – already comes installed

 Docker
 From Source (without using Pipenv)
 From Source (with Pipenv)

4.9.3 Shodan

This tool is a devoted search engine for discovering intelligence

about devices, which are not often searchable over the top and are
hidden between layers but are present everywhere. We can also use
this tool to search for objects like open ports and liabilities on any
targeted systems.
 94

Figure 4.3: Shodan

The number of places that Shodan can monitor and search as part of an
OSINT effort is remarkable. Only tools like SHODAN can examine
“operational technology” (OT), such as the kind generally used in
industrial control systems at certain places like power plants and
manufacturing facilities.
It can gather information about all devices directly connected to the
Internet. If a device is instantly hooked up to the Internet, then Shodan can
easily query it for a variety of publicly-available information. The types of
devices that are indexed can vary enormously while ranging from small
desktops to nuclear power plants.
Some of the primary data that SHODAN indexes are taken from banners,
which are nothing but metadata regarding the software running on a
device. This comprises various information about the server software like
what options the service supports, a welcome message, or anything else
that the client would like to know before interacting with the server. For
example, the following is an FTP banner:
 95

220 kcg.cz FTP server (Version 6.00LS) ready.

It tells us the potential name of the server (kcg. cz), the type of FTP server,
and its version (6.00LS). For HTTP , a banner looks like this:

HTTP/1.0 200 OK
Date: Tue, 16 Feb 2010 10:03:04 GMT
Server: Apache/1.3.26 (Unix) AuthMySQL/2.20 PHP/4.1.2
mod_gzip/1.3.19.1a mod_ssl/2.8.9 OpenSSL/0.9.6g
Last-Modified: Tue, 19 oct 2021 08:51:04 GMT
ETag: "135074-61-3599f878"
Accept-Ranges: bytes
Content-Length: 97

Content-Type: text/html

4.9.4 Spiderfoot
It is a free OSINT reconnaissance tool that combines several data
sources to analyze IP addresses, domains, subdomains, email
addresses, mobile numbers, usernames, etc. It is readily accessible
via GitHub. Spiderfoot is available in both command-line
interfaces as well as in embedded web-server.

Figure 4.4: Spiderfoot

Source: www.spiderfoot.net

4.9.5 Spyse
 96

It is a cyber-security search engine to locate methodical info

regarding various internet entities and vulnerabilities. This is an
all-in-one program that does not use any additional tools for fast
and effortless investigation. It is also described as the “most
complete internet assets registry.”

Figure 4.5: Spyse

Source:https://thefutureofthings.com/14487-spyse-all-in-one-
cybersecurity-search-engine/

SPYSE engine implements a ready-to-use database with vast

volumes of internet data, which helps evade the scan to end while
creating its scanning infrastructure and being ano
anonymous
nymous while
gathering information.

The data it collects is publicly available on websites, inter-linked

servers, and IoT devices. It is then analyzed to locate any security
risks in the ties between these diverse entities.

4.9.6 Builtwith

This tool lets us find what any particular website comprises in

general. For instance, it can detect if a website employs Joomla,
WordPress, or Drupal as its CMS and provide s additional details.
 97

Figure 4.6: Builtwith

It can also generate an immaculate catalog of identified

JavaScript/CSS libraries that a website generally uses.
Furthermore, built With can also provide a list of plugins installed
on the websites, frameworks, server information, analytics and
tracking information, etc. We can also use it for reconnaissance
purposes.

4.9.7 Intelligence X

Intelligence X is one of the first-of-its-kind search engines and archival

facility which can preserve not only previous versions of web pages but
also the entire leaked data sets. These are generally removed from the web
because of their objectionable nature content or for some legal reasons.
 98

Figure 4.7: Intelligence X

Source: https://usersearch.org/updates/2022/04/10/top-16-open-source-
intelligence-tools-ever-made-osint/

Though this sounds similar to the Internet Archive’s Wayback Machine,

Intelligence X has some differences regarding the service's content on
safeguarding. No discrimination against the data, no matter how
controversial it is.

These searches are for the following selectors’ types:

 Email address
 Domain
 Phone Number
 URL
 MAC address
 Credit Card Number

4.9.8 Metagoofil

Metagoofil is another freely available tool on GitHub, which has

been optimized to extract metadata from public documents. It can
 99

investigate records of any kind that can be accessed via public

channels.

Searches return things like the usernames associated with

discovered documents and real names if available. It also maps the
paths of how to get to those documents, which in turn would
provide things like server names, shared resources, and directory
tree information about the host organization.

Everything that Metagoofil finds would be very useful for a

hacker, who could use it to do things like launch brute-force
password attacks or even phishing emails. Organizations that want
to protect themselves could instead take the same OSINT-gathered
information and protect or hide it before a malicious actor can take
the initiative.

4.9.9 Searchcode

For those who need to go deep into the complex matrix of OSINT
gathering, search code is a highly specialized search engine that
looks for valuable intelligence inside source code. This powerful
engine is surprisingly the work of a single developer.

Because a repository of code needs to be first added to the program

before becoming searchable, search code straddles the line
between an OSINT tool and one designed to find things other than
public information. However, it can still be considered an OSINT
tool because developers can use it to discover problems associated
with having sensitive information accessible inside code on
running apps or those still in development. In the latter case, those
problems could be fixed before deployment into a production
environment.

Check Your Progress 2

Note a) Space is given below for writing your answers.
b) Evaluate the answers with the ones provided at the end.
 100

(i) Which One of The Following Would Not Be Considered an OSINT

Tool?
a) WHOIS lookups
b) Google searches
c) Website perusal
d) Vulnerability scans

(ii) What is the process of evaluating a target to gather primary

information about systems, software, etc., without clearly involving
the target?

a) Active information gathering

b) Reconnaissance
c) Web searching
d) Passive information gathering

(iii) ______ is the free tool for the Windows OS lineage, which collects
information by scraping metadata from Microsoft Office documents.
a) Maltego
b) theharvester
c) FOCA
d) recon-ng

(iv) Which method of collecting open-source intelligence consists of the

collection of published documents, such as Microsoft Office??
a) File excavation
b) Metadata analysis
c) File scraping

4.10 LET US SUM UP

In this unit, we introduced you to the OSINT framework. In OSINT,

advanced techniques are introduced to locate or retrieve data buried inside
the internet which is not easy to access directly or by simple means which
a normal user employs through search engines. Techniques and tools used
for such kinds of operations are discussed. We can say that OSINT is
intelligence “extracted from publicly accessible material.” It means the
intelligence gathered is from the data intended for public consumption.
You came to know about OSINT, its uses, its importance of OSINT and
its tools used for Information Security.
 101

4.11 CHECK YOUR PROGRESS: THE KEY

1. (i) Open-source intelligence refers to the collection and analysis of

publicly available information for intelligence gathering or to further
another type of inquiry.

(ii) OSINT can drastically improve the quality of information

obtained and the speed at which data is received. Organizations have
also found that the cost of completing investigations can be
drastically reduced if OSINT is used.

2.
i. d.
ii. d
iii. c
iv. b

4.12 REFERENCES AND SUGGESTED

READINGS
https://osintframework.com/
https://github.com/lockfale/OSINT-Framework

Agate M. Ponder and Sutton (2016) Automating open source Intelligence,

Science Digest
Kim, R. (2022) Contribution of Open-Source Intelligence to Social Engineering
Cyber attacks. https://urn.fi/URN:NBN:fi:amk-2022061317577

S. Lee and T. Shon (2016) Open source intelligence base cyber threat
inspection framework for critical infrastructures," 2016 Future Technologies
Conference (FTC), 2016, pp. 1030-1033, doi: 10.1109/FTC.2016.7821730.

Nihad A. Hassan and Rami Hijazi, Open-Source Intelligence Methods and

Tools: A Practical Guide to Online Intelligence.
Open-Source Intelligence Techniques: Resources for Searching and Analyzing
Online Information April 2016
https://dl.acm.org/doi/book/10.5555/3033260.
Wenji Mao, Fei-Yue Wang (2012) New Advances in Intelligence and Security
Informatics, Science Digest