Module 4

1st Types of cyber threats

Cyber threats are dangers that can harm an organization’s computer systems, networks, or data. It is important to
know how fresh and useful the security data is to stop such threats. Even a small cyberattack can damage a
company’s reputation or business. Cybersecurity uses data analysis to check the company’s ability to detect
threats early, respond quickly, and recover fast. The main types of cyber threats are Malware, Phishing,
Eavesdropping, Denial-of-Service, and Insider threats.

Malware threats

Malware is harmful software that secretly enters systems, often when users click bad links or download infected
files. Malware can damage data, block access, or spy on users. Some malware, like Ransomware, locks the data
and demands money to unlock it. To stop malware, companies must scan all emails and files, use updated
antivirus tools, and share information about new malware with other users.

Phishing threats

Phishing happens when fake emails or messages try to trick people into sharing personal details like passwords
or bank information. These messages look real but are designed to steal data. Companies must regularly train
their employees to spot and avoid phishing attacks.

Eavesdropping threats

Eavesdropping happens when hackers secretly listen to communication between two parties, often through
unsafe public Wi-Fi. This allows them to steal private information. To prevent this, companies must use secure
networks, train staff on safe internet use, and test their systems for such risks.

Denial-of-service threats

Denial-of-Service (DoS) attacks flood company servers with too many requests, making the system slow or
unavailable for real users. Hackers often use many infected computers to launch such attacks. Companies should
monitor their systems, have backup services ready, and detect any drop in system performance quickly to avoid
service interruptions.

Insider threats

Not all threats come from outsiders; sometimes, company employees cause harm by stealing data or misusing
access. These insiders could be unhappy workers or spies. To prevent this, companies must monitor user
behavior carefully but fairly, protect sensitive data with extra checks, and control access to important resources.
Let me know if you want this in bullet points or table format for quicker revision!

2nd

Sure! Here's the simplified version of your provided content without changing the main headings:

2nd ## DEVELOPING CYBERSECURITY STRATEGIES

Cybersecurity strategies are like roadmaps that guide how to protect data and systems. These strategies use data,
analysis, and measurements. Cybersecurity data comes not only from inside the company but also from partners,
clients, and government agencies that share such information to improve security for everyone.

When building these strategies, data is collected, studied, and shown in clear visuals. Organizations use their own
data along with data from others to understand security risks.

A key part of this is using **metrics** — numbers that show how safe or risky things are. These metrics help
track the security of devices, data, tools, and even how people think about security.

Also, showing the data in simple charts or visuals helps people easily spot problems without exposing sensitive
information. Good data analysis can help reduce the chance of cyberattacks because it helps to find and fix
weaknesses early.

Some weak points (vulnerabilities) are well-known, but attackers may target rare or unnoticed ones. So, less
obvious risks should also get attention. **AI tools** can help decide which risks are most important to fix.

Lastly, fine and detailed analysis helps organizations balance between common and rare attacks.

---

## Organizing cybersecurity data and functions

Sure! Here's the table content rewritten in simple, easy words without changing the aspects:

Aspects Description (Easy Words)

This includes Network, Service, and Host areas. These are the main parts where
Domain
security data is collected.
This means where the sensor is placed in the network. It depends on the sensor’s
Vantage
position and how the network routes data.
What the sensor does with the data. For example: does it report the data, respond
Action
to it, control something, or save the data?
How strong or useful the collected data is. This also depends on how new the data
Validity
is — what is correct now may become wrong after some time, like an hour or a day.

Let me know if you want this in a Word or PDF format!

Cybersecurity work is done in layers — for data, processes, and people. Different tools help, like antivirus
software and file checkers. On devices, data is collected by placing sensors (small tools that gather information)
in the right places. For people, training on how to stay safe is also layered and continuous.

At the technology level, spam filters help block bad data. All this collected data is carefully checked to see if it
matches known threats like malware.

The table in the text (Table 9.1) lists the sources and ways security data is collected — like sensors in networks,
firewall logs, and network taps. But gathering a huge amount of data is not enough — the data must be useful
and clear, without too much repetition.

twork and speed of data transfer also affect how well this data is collected and checked.

---

## Cybersecurity data analytics

Figure 9.2 in the text shows an **Agile (step-by-step)** method of handling cyber threats. It includes immediate
and continuous efforts to protect systems.

Cybersecurity analytics help explore and predict threats. It usually starts with checking logs, then finding strange
patterns in data like unexpected IP addresses.

Simple graphs (like bar charts, histograms, or scatter plots) help understand this information. Depending on how
serious the risk is, deeper analysis may be needed.

Analytics tools help collect, study, and show this data properly.

## Physical security for cyber assets

People often forget that **physical safety** of computers and data centers is just as important as digital security.
Even if online security is strong, a person breaking into a building could cause big damage.

That’s why organizations use:

* Locks, fingerprint access, guards, and cameras

* Systems that detect and report any intrusions

Laws like **GDPR** and **HIPAA** require companies to make sure no one enters secure areas without
permission. These rules also guide how patient or customer data is handled.

---

## Cybersecurity analysis using business analysis capabilities

Cybersecurity analysis is not only about numbers and graphs. It also involves **Business Analysis (BA)** — a
way to understand the whole working system of the company.

BA helps ask the right questions, find the weak points, and build strong, lasting security plans. This is a
continuous job because new risks keep coming.

While data analytics shows the problems, business analysis helps in making the right judgments. It also watches
for mistakes, wrong data, or biases (like errors or misunderstandings) that could affect decisions.

So, both **analytics** (for data) and **analysis** (for decisions) are needed to make strong cybersecurity
strategies.

---

## Cybersecurity standards and frameworks

Many organizations follow popular cybersecurity standards like:

* **NIST Cybersecurity Framework**

* **ISO 2700x Series**

* **CARE Standard**

These help reduce risks by giving clear rules, tools, guidelines, training, and best practices.

Organizations also check how good their security is using **maturity models** — which measure how well-
prepared they are.
Another useful method is **CARVER**, which looks at:

* **Criticality**: How important is the target?

* **Accessibility**: How easy is it to reach?

* **Recuperability**: How easy is it to fix if damaged?

* **Vulnerability**: How easy is it to attack?

* **Effect**: What impact would it have?

* **Recognizability**: How easy is it to spot?

Originally used by the US military, CARVER helps businesses decide what to protect first.

---

Let me know if you want this in **even simpler language**, **summary form**, or **in points**!

3rd ## CYBERSECURITY INTELLIGENCE (CI)

---

Cybersecurity Intelligence (CI) means using **AI tools to improve cybersecurity work**. CI starts by setting
clear **security goals** and knowing how much risk is acceptable. The risks are arranged in order of importance
using standards and rules explained earlier.

CI depends on **data**, but also needs proper ways to measure and check progress. To do this well, companies
must balance **people, processes, technology, and money**. It’s also important to understand where the data
comes from. People are also a big part of this, because most security failures happen because of **human
mistakes**, like clicking on a harmful link.

Since AI systems are complex, **AI-powered cybersecurity tools** are needed to manage this. The choice of
tools depends on good measurement methods and understanding the possible problems.

CI also helps people build a **strong security mindset**. Training and guidance help in changing the way people
think about cybersecurity.

---

## Cybersecurity metrics and measurements in CI

CI uses **data to find weaknesses** (vulnerabilities). This data is collected and studied using tools. CI always
keeps the business needs in mind while using these tools.

Some **metrics** used are:

* **External** (outside events) like number of hacker attacks or breaches.

* **Internal** (inside company) like how fast the company spots a threat.
For example:

* Total number of malware cases,

* Percentage of attacks blocked.

CI tools learn from past attacks and get smarter with each one. This helps in focusing on important areas and
saving costs. It is also a part of the overall **business plan**.

**Transparency and honesty** are necessary when detecting and stopping attacks. Also, how safe customers feel
is as important as the actual safety — for example, if security is too strict, customers may find it hard to use the
service and feel frustrated.

CI works at two levels:

* **Tactical**: Daily checks like number of breaches, user login problems.

* **Strategic**: Big-picture view like tools used, competitor’s security, audits.

CI also plans what to do if a big data breach happens.

It learns not just from its own data but also from the whole industry.

---

## Levenshtein distance as a measure in CI

CI uses smart methods to spot dangers. One example is the **Levenshtein Distance** — which measures how
similar two words are. This helps in finding fake (phishing) websites.

For example:

“hello” vs “hallo” = distance 1 (only 1 letter difference).

Hackers might make fake websites like **“c0rp.adomain.com”** instead of **“corp.adomain.com”** to trick
people.

AI can quickly check lots of website names using this method to stop such attacks.

---

## Base rate fallacy in cybersecurity measure and the validity of positives and negatives in CI

CI also checks how accurate its alerts are. Sometimes, the system gives a **false positive** (warning when there
is no danger) or **false negative** (misses real danger).

For example, even if a tool is 99% accurate, it can still give many false warnings when checking thousands of
things. This is called the **Base Rate Fallacy**.

**Intrusion Detection Systems (IDS)** must think about these errors. Tools like **Bayes’ theorem** are used to
handle these problems. Even tools like **Face ID** can make such mistakes.

---

## Filtering algorithms for email phishing for CI

CI also protects emails from **phishing attacks** using AI tools. These` tools study emails to find suspicious
patterns.

But sometimes, filters wrongly send safe emails to the spam folder. So, these filters must be checked and updated
regularly.

CI also watches over **cloud-based emails**, which can be attacked if the mail server is not safe. Filters work
with many parts of the system like hardware, networks, and devices.

AI tools also search huge amounts of text to find hidden patterns or codes in emails. This helps in understanding
how hackers encrypt data and how to stop them.

Hackers also use **AI bots** to scan systems for weaknesses. So, companies need **AI tools** to fight back
fast and carefully. **Machine Learning (ML)** tools study past attacks and learn to spot unusual activity.

But AI has its own risks — for example, small changes to a picture can confuse AI systems into making mistakes.
Hackers can fool even the best AI tools sometimes.

---

## Tools for cybersecurity intelligence

CI needs **tools to help with analysis**. One type is **SIEM (Security Information and Event Management)**.
SIEM watches things like:

* Ports and protocols,

* IP addresses,

* Data size,

* Time and traffic details.

Some popular SIEM tools are:

* **SiLK Suite**,

* **Splunk**,

* **Webroot**,

* **SPAM Titan**,

* **Barracuda**.

These tools read and clean security data, find problems, and give alerts.

For example, **SiLK Suite** can check network traffic data (called **NetFlow**) to find unusual activity like
policy violations. It can show things like:

* Who connected to what,

* When and how often.

Data can be shown as numbers (quantitative) or ideas and pictures (qualitative).

Example:
* **Quantitative**: How many times users accessed a file in a week.

* **Qualitative**: Which user groups used which files.

---

Let me know if you want this as **bullet points**, **summary form**, or for **presentation slides**!

4th ## CYBERSECURITY ASPECTS IN BO

Cybersecurity is a very important part of digital businesses. In this section, cybersecurity is discussed as an
important part of **Business Optimization (BO)**. Cybersecurity does not only protect data but also helps
maintain the company’s image and customer trust. If a company’s system is not secure, it can lose customers and
face legal problems.

Good cybersecurity builds customer trust, protects the company’s money, and helps the business grow. A
complete and strong cybersecurity plan helps the company to handle problems, follow rules, and keep the
business running smoothly.

Cybersecurity is important in every part of AI used in BO — from customer touchpoints to analytics, cloud
storage, and user behavior. The four things — **handset, dataset, toolset, and mindset** — all need protection.
Devices, tools, processes, and people must be kept secure.

Rules and standards for cybersecurity help protect data and increase customer value. It is also important for
leaders, managers, and users to talk clearly about cybersecurity to avoid problems.

When a business uses data for decisions, it must also protect that data. Cybersecurity works along with
governance, risk, and quality to make sure the business runs well. This chapter explains the important parts of
cybersecurity in BO, keeping the goal of customer value in mind.

---

## Cybersecurity functions

Cybersecurity in BO has **two main jobs**:

1. **Protecting business processes** and data

2. **Using AI to find and stop security problems (Cybersecurity Intelligence - CI)**

Cybersecurity keeps data and business processes safe and uses analytics (descriptive, predictive, prescriptive) to
detect and stop threats.

Data goes through several steps — it comes from devices and users, moves through networks, is stored and
analyzed in the cloud, and results come back to the users. **Cybersecurity is needed at every step**, even when
the data is no longer needed.

Cybersecurity is not only about real threats but also about **perceived threats**. Even if there is no real risk,
customers may feel unsafe if they think the system is weak. This feeling can affect their decisions.

AI in cybersecurity is tricky because attackers also use AI. Hackers hide dangerous code using the same tools
that defenders use to protect. AI itself cannot always tell who is a friend and who is an enemy. This makes ethics
and trust very important in cybersecurity.
---

## Cybersecurity as a business decision

Cybersecurity problems cost businesses **trillions of dollars** worldwide. But these problems are not just about
money — they are also very complicated and can affect the whole business.

Using AI in business processes makes things even more complex. So, cybersecurity is not only a technical matter
— it is a **business decision**. It depends on people, processes, and technology.

Business needs, goals, costs, and process performance all decide the kind of cybersecurity that is needed.
Businesses have to **balance security with performance** — too much security can slow things down and make
customers unhappy.

Cybersecurity decisions should fit into the company’s business strategy. These decisions include goals,
technology, resources, and the market environment. Security must also follow rules and make sure customers and
staff trust the system.

Cybersecurity controls must be strong, clear, and easy to check.

---

## Cybersecurity and penalties

For digital businesses that depend on data, the rules and regulations are very strict. If companies do not control
their data properly, they can face **huge fines** and lose customer trust.

Cybersecurity is a part of business risk. Businesses must balance security costs with the value and benefits they
get. Heavy security increases costs and can reduce performance. So the customer’s experience is important when
deciding how much security is needed.

Good cybersecurity tools can find problems early and help stop them before they cause trouble.

---

## Cybersecurity challenges during BO

When AI is added to business processes, it changes the way things work. These changes create **new risks**.
AI-driven processes are complex and sometimes hard to understand ("black boxes"), which can make people feel
uncertain.

Cybersecurity has to protect both the company’s own data and other data it uses (like metadata from mobile
networks). New technologies like **blockchain and Ethereum** also bring security risks that no single business
can manage alone. This needs teamwork across industries.

Using **third-party services or AI development** does not remove the company’s risk. The main business is
still responsible for the final service given to the customer. Managing these risks with partners is difficult and
involves legal and business challenges.

---

## Cybersecurity vulnerabilities and impact

Companies that use data know there are risks. These risks make them targets for **cyberattacks from inside and
outside**. Because data and systems are so complex, there will always be some weaknesses.
A small security gap can lead to a **big problem**. For example, a former Australian Prime Minister’s flight
boarding pass was posted online, and it gave hackers enough information to access his personal details. This
shows how even a small mistake can cause serious issues.

Sometimes even **small changes in data** can confuse systems and lead to wrong results. Hackers use these
tricks to attack. For example, a **ransomware attack** may target just one important file and demand money to
unlock it.

Cybersecurity also includes making systems like **multi-factor authentication** common. But poor network and
system setups can still be a risk. Physical security (protecting the actual devices) is also important but often
forgotten.

---

## Cyber attacker’s psyche

Cybersecurity is not just about technology — it is also about understanding the **mind of the attacker**.
Attackers may have many reasons — revenge, fun, or money. They work hard and have lots of time, while
defenders are usually slower and have less freedom.

Attackers keep improving their skills and tools. They learn how defenders protect their systems and find ways
around them. They also hide their attacks using the same encryption as defenders.

Good **AI-based tools** can help defenders by spotting threats and risks early. But defenders also need enough
money, tools, and skills to fight off attackers.

Attackers pretend to be real users and send fake but convincing messages to trick people. They may take over
accounts, control browsers, read clipboard data, and scan applications. Attackers look for **weak, untrained
users** by sending test messages and planning bigger attacks later.

If attackers can break trust in a company’s data or processes, they don’t need to do much more damage — the
trust is already broken. That’s why understanding **attacker behavior** is very important when creating strong
cybersecurity plans.

---

Let me know if you want this in **bullet points**, **summary form**, or as a **presentation deck**!

5th write a short note on qulity and eisk

Here’s the **simplified version** of your provided content in **easy words** without changing the main
headings:

---

## **INTRODUCTION**

This chapter talks about the **quality and privacy of data** in digital business processes. The features of **Big
Data** (3+1 Vs) bring special challenges for maintaining quality.

Big Data is often **unstructured (no fixed format), large in size, fast in processing**, and **without a set design
(schema-less)**. These things make it hard to ensure data quality.
On social media, people share and create new data on their own. This data helps others, but it does not belong to
the company analyzing it. So, companies can’t be sure if this data is **true or trustworthy**.

Also, **AI systems** get data without knowing its full background (context). But understanding this
background is very important to ensure quality and make the right decisions. Simply testing AI systems is **not
enough**. The full picture of data must be checked to ensure total quality.

**Governance, Risk, and Compliance (GRC)** help make better decisions and reduce risks in data usage.

Good quality makes people trust **AI-based solutions**. Quality is not just technical but also gives real business
benefits. With good governance and quality, businesses can build their value.

If people don’t trust data or analytics, the size and speed of data won’t matter. Quality ensures trust by catching
errors early and fixing problems quickly.

---

## **Quality Initiatives**

Quality checks cover **data, intelligence (analytics), models, processes, and usability**.

Good quality efforts include:

* Testing data sources,

* Applying feedback regularly,

* Keeping track of all changes clearly.

Managing how data is **collected, stored, cleaned, used, and removed** is also part of quality.

Earlier, systems like databases and information systems worked **separately and were safe behind firewalls**.
But now, businesses need connected systems (like websites, call centers, banks).

This connection increases risks to both **quality and security**, which testing alone cannot fix.

Quality now includes checking **data, models, codes, processes, and people**. Quality management itself must
meet quality standards.

Training, selecting projects, and proper documentation are also important for quality. Quality is a **continuous,
improving process** that adds value to business efforts.

---

## **Testing and Assurance**

There are two parts to quality:

1. **Testing (Detection):**

This finds mistakes by comparing the system to a correct or ideal example. Proper data and models are needed to
judge the system.

2. **Assurance (Prevention):**

This avoids mistakes to ensure a **good user experience**. Assurance focuses on preventing errors in meaning
or appearance. The goal is a **perfect final product without defects**.
In business optimization (BO):

* Assurance includes setting data filters, managing data well, and using **agile development** methods.

* Testing includes checking algorithms, system performance, and fixing bugs.

---

## **Quality and Risks**

### **Direct and Indirect Impact of Bad Quality**

Bad data quality can harm business in many ways:

1. **Direct, Short-Term Effects:**

Example: Customers get faulty products or billing mistakes happen. These can be fixed with data cleaning and
testing.

2. **Direct, Long-Term Effects:**

Example: Choosing the wrong business partner. This is harder to fix because business processes and data
analysis methods also need to change.

3. **Indirect, Short-Term Effects:**

Example: Poor inventory handling or slow production hidden in processes. Solving this needs better data and
staff training.

4. **Indirect, Long-Term Effects:**

Example: Making the wrong product, bad marketing, legal troubles, or loss of company reputation. Fixing this
requires a complete improvement in data, knowledge, decisions, and skills of people involved.

---

## **Risks and Governance Policies**

**Risk management** and **quality efforts** go hand-in-hand. **GRC (Governance, Risk, and Compliance)**
provides rules to manage risks and follow regulations.

Good governance balances risk, cost, and service needs to keep the business running well. Governance must not
harm quality or business performance.

**GRC** also depends on understanding:

* The business type and size,

* Its abilities and limits,

* The tools available for governance.

Important rules like:

* **Sarbanes–Oxley Act (SOX)**

* **Health Insurance Portability and Accountability Act (HIPAA)**

require AI systems to explain and justify their decisions. These laws make it necessary to allow **audits and
accountability** in AI-based systems.

---

## **General Data Protection Regulation (GDPR)**

The **GDPR law** controls how personal data is collected and used. It also allows people to:

* Move their data to another company,

* See their data,

* Correct mistakes,

* Delete data if they want.

Though GDPR is a **European Union rule**, it affects companies worldwide. GDPR ensures that companies:

* Collect data only with clear permission,

* Tell users why and how their data is used,

* Give users a copy of their data when asked.

---

## **Quality and Ethics**

**AI systems cannot fully understand human values or situations** on their own. So, companies must check
AI’s decisions for **ethics, society, and legal effects**.

AI decisions should always be combined with **human intelligence, experience, and judgment** to make the
best and most useful decisions.

---

Let me know if you want a **summary in points** or a **Kannada version**!

5th

Here is your text rewritten in **simple and easy words** without changing the main headings:

---

5th ## **GOVERNANCE – RISK – COMPLIANCE AND DATA QUALITY**

Good governance happens when both **business leaders** and **technical experts** work together. Below this
team, there is a group that helps match the business’s abilities with its goals.

These groups make sure that the company’s **IT systems, services, and platforms** help the business meet its
aims.

When **GRC (Governance, Risk, Compliance)** is connected well to the business, it gives more value because
it does not just ensure rules are followed but also helps the company make a **profit (return on investment)**.
GRC makes sure that the business optimization (BO) process brings real benefits to the company.

GRC also helps meet rules from inside and outside the company—like legal, audit, and accounting standards.
GRC is important for AI investments too.

---

## **Business Compliance and Quality**

**Business compliance** means the company has to develop skills and ways to meet all the rules and
regulations. Following these rules improves quality and reduces risks.

Sometimes, because of new government laws, the business has to change the way it works. A flexible (agile)
business can react quickly to new laws.

For example:

* The **Sarbanes-Oxley (SOX) Act** protects shareholders from fraud. It makes the company’s CEO and CFO
responsible for keeping honest financial records.

* When laws about **carbon emissions** change, businesses must also change how they collect, check, control,
and report carbon data.

AI-powered businesses have to make changes in:

* Internal processes,

* Their IT systems (for correct and timely data collection),

* And the attitude of senior management.

GRC supports BO activities by offering a clear way to control how the company works. GRC also helps handle
**Big Data** as well as regular business data.

A formal GRC setup lets a company watch its activities, control risks, do audits, and prepare reports. This
increases the chance to stop fraud because the company can see everything clearly and control its business
processes.

So, it is important to talk about GRC when discussing **quality and value**.
---

## **GRC, Business, and Big Data**

Inside a business, **governance** means managing and guiding the company’s work. This needs knowing the
business goals and the abilities of the company.

With Big Data, **governance becomes more important** because the data comes from many sources and the
company has to work closely with partners.

**Risk management** helps governance by:

* Finding risks,

* Studying risks,

* Acting on them when needed.

Big Data creates fast-changing risks because data is always changing. Also, Big Data analysis itself can create
risks if the algorithms or tools are wrong. So, these algorithms must be tested carefully for:

* Correct meaning (semantics),

* Look and feel (aesthetics),

* Speed and performance.

**Compliance** means following all rules and standards (inside and outside the company). With Big Data, this
is very important because rules could be broken at any part of the system that uses data to make decisions.

Even when small employees make decisions using analytics, the **top company leaders (directors)** are still
responsible for the final result.

---

## **Quality of Service**

Analytics (data analysis) is provided to users as services. The quality of these services is maintained by
**GRC**. For example, **ITIL standards** help manage IT services properly.

Big Data services need good handling of service requests. These requests must be:

* Planned,

* Reviewed,

* Tested before being used.

Service requests in a company include:

* **Accounts and Permissions:**

Requests to create, close, or change user accounts and permissions.

* **Projects:**
Big tasks that need planning, teamwork, and more than five days to finish. These may have their own release
plans.

* **Enhancements:**

Small improvements or features that can be done in less than five days. These are simple, quick, and tested easily.

* **Defects:**

Problems or bugs that are found and fixed as part of incident handling or problem management. These may take
longer than five days and might need special project planning.

For any request to be accepted, the information must be:

* **Complete (all details given),**

* **Approved (by the right person),**

especially when working with vendors or external companies.

**Metrics and measurements** related to Big Data can help track performance, find risks, and see chances to
improve operations. These measures are useful for checking how well analytics and management are working.

---

If you want, I can also make a **short summary in points** or provide this in **Kannada**. Just let me know!

6th

Sure! Here is the **simplified version** of your text with **easy words**, keeping **main headings
unchanged**:

---

6th ## QUALITY ENVIRONMENT IN AI AND ML

### Assuring ML Quality

We can use data analysis to check the quality of data. This data and the algorithms (the methods used by AI) are
tested to make sure they work well.

For example, an AI system can check weather data to find unusual values like a 500°F change at the same place
within minutes. Finding such strange results helps us check if the algorithm is working properly and safely.

Quality checks for Machine Learning (ML) systems are done at three levels:

1. Testing how well and correctly the ML algorithm works.

2. Checking if the ML model gives unfair or biased results.

3. Making the working of the ML model more understandable.

Here, only the first point is explained. The other two are covered in Chapter 10.
ML algorithm testing starts by splitting data into **training data** (to teach the model) and **testing data** (to
check the model). Testing needs a smart approach that includes:

* **Black-box testing** (where we do not see the inside logic) and

* **White-box testing** (where we do see and check the internal logic).

Unstructured data is more difficult to test. So, it is good to make some sample data and test if the system's logic
works properly.

When machines find patterns by themselves (without humans knowing the questions), quality checking becomes
more about handling risks.

For Big Data systems, getting feedback for testing is very limited because we don’t know the complete structure
of the data at the beginning. Testing such systems requires checking both data and algorithms but sometimes also
involves guessing.

The quality of data and algorithms is never 100% perfect or complete. Testing and quality assurance in ML
includes checking:

* The test data,

* The training data,

* The performance of algorithms using new, open data.

Good management (GRC - Governance, Risk, Compliance) helps improve Business Optimization (BO). Leaders
define goals and also decide on quality rules. They need to balance between making wrong decisions and not
making decisions at all. If the data and algorithms are not checked properly, the risk becomes higher, especially
when people use them for decision-making.

The value and trust in Big Data also depend on how the users see and use the results. So, quality checking is not
only about technology but also about how the data is presented, explained, and experienced by people. All these
factors must be carefully checked so that business decisions are correct and valuable.

---

### Assuring Quality of Business Processes

Checking the quality of business processes includes first making a model (a plan) of how these processes work.
Then, this model is verified and improved just like software models are checked.

The quality of processes depends on how well the users follow them and what they want to achieve.

We create visual models (drawings or diagrams) to explain the processes and use techniques like walk-throughs,
inspections, and audits to check them. Business processes involve apps and analysis that help users reach their
goals. So, the quality also depends on how the users feel about what they achieved.

To improve quality, these visual models are shown to users, and their feedback is added repeatedly (this is called
**Agile method**).

Standards like **UML (Unified Modeling Language)** and **BPMN (Business Process Model and Notation)**
and related tools help improve these models.

Besides business processes, we also have processes like project management, business analysis, and solution
development, which also need quality checks.
Using well-planned activities and **Agile methods** makes the analytics more accurate, which helps improve
the entire business process.

---

### Developing the Quality Environment

When making a good quality environment for Big Data, some important points to consider are:

* **Identifying key business outcomes:**

First, define the goals of the business clearly so everyone knows what the Big Data project should achieve. The
more important the goals and the bigger the risks, the higher the need for good data quality.

* **Modeling the range or coverage of Big Data:**

Know where the data comes from, what types it is, where it is stored, and how much it costs. This helps to plan
the quality checks well, depending on the risks and costs of handling the data.

* **Extent of tool usage:**

Big Data needs special tools for quality checks. For example, NoSQL databases need tools to check if the right
data is being taken out and compared with reference data. Unstructured data is hard to check, so tools that can
test such data are required. The tools also need to work fast because Big Data comes very quickly and in
different forms.

* **Ensuring a balance between the rigors of quality and corresponding value:**

Quality teams and business leaders should work together to ensure that the effort spent on quality gives useful
results. There is no need to clean or check data that will not be used. Always keep the final use of the data and
the business goals in mind.

---

### Assurance Activities

Quality checking happens during all the main stages of data: entering, storing, cleaning, and retiring. Big Data
makes this harder because of its large size, fast speed, and many different types. Problems also come from data
management and risks. Big Data needs continuous cleaning and organizing.

Main activities are:

* Data coming into the system is filtered based on some assumptions.

* Data from social media, the crowd, or other systems needs different filters. These sources are not always in the
company’s control.

* Sometimes the purpose of the data is unclear, so guesses must be made before filtering.

* Because data arrives fast, tools are needed to filter it quickly.

* The complex analysis makes checking quality harder.

* Data is stored safely, often in the Cloud, so quality checks are needed there too.

* Different data types must stay separate as much as possible because mixed data types are harder to manage.
NoSQL databases need careful design before saving such data.
* Backup and copy strategies must be planned so that lost data can be restored easily.

* Data is checked often to remove errors or strange values before analysis.

* New data is monitored when added to old data.

* Old or unused data is removed or stored safely, even in systems like Hadoop.

---

### Developing the Testing Environment

Testing Big Data systems includes checking data, programs (scripts), algorithms, and tools. Important things to
do:

* **Testing of algorithms:**

Create special test environments to check algorithms that can’t run or test themselves.

* **Test scripts:**

Write scripts (small programs) based on real use cases and run them repeatedly in an Agile environment.

* **Repository of test cases:**

Keep a storage of test cases that can be used and improved over time.

* **Testing tools:**

Use standard tools to plan tests, create cases, and track results (like Silk tool). Some tools also check for security.

* **Test planning:**

Have a standard process for making test plans, scheduling tests, and deciding what resources are needed. Plans
should follow industry practices and Agile methods.

* **Test result tracking:**

Use tools to track test results and link them to requirements and product versions. Results analysis shows where
rework or retesting is needed.

* **Test approvals:**

Use tools to approve tests and track them till product release. Kanban boards can help in tracking.

* **Testing processes:**

The process of testing needs the right knowledge, training, and support. This should be a clear and active Agile
process.

New data used in analysis needs constant checking because:

* Data may change,

* New data is added,

* Old data may lose its importance over time.

New data is also linked with old data (structured or unstructured), from the company, third parties, or open
sources. This mixing of data needs careful planning, testing, and execution.

---

Let me know if you want this in **PDF, bullet points, or presentation format**!