lOMoARcPSD|37281518
DATA Warehousing Laboratory
data warehouse (Anna University)
Scan to open on Studocu
Studocu is not sponsored or endorsed by any college or university
Downloaded by SANGAVI M (
[email protected])
lOMoARcPSD|37281518
CCS341 DATA WAREHOUSING LABORATORY LTP C
2 0 2 3
COURSE OBJECTIVES:
To know the details of data warehouse Architecture
To understand the OLAP Technology
To understand the par琀椀琀椀oning strategy
To di昀昀eren琀椀ate various schema
To understand the roles of process manager & system manager
LIST OF EXPERIMENTS:
Note: The lab instructor is expected to design problems based on the topics
listed. The Examination shall not be restricted to the sample experiments
designed.
1. Data explora琀椀on and integra琀椀on with WEKA
2. Apply weka tool for data valida琀椀on
3. Plan the architecture for real 琀椀me applica琀椀on
4. Write the query for schema de昀椀ni琀椀on
5. Design data ware house for real 琀椀me applica琀椀ons
6. Analyse the dimensional Modeling
7. Case study using OLAP
8. Case study using OTLP
9. Implementa琀椀on of warehouse tes琀椀ng
TOTAL: 30 PERIODS
COURSE OUTCOMES:
Upon completion of the course, the students will be able to
CO1:Design data warehouse architecture for various Problems
CO2: Apply the OLAP Technology
CO3: Analyse the par琀椀琀椀oning strategy
CO4: Cri琀椀cally analyze the di昀昀eren琀椀a琀椀on of various schema for given problem
CO5: Frame roles of process manager & system manage
TEXT BOOKS:
1. 1. Alex Berson and Stephen J. Smith “Data Warehousing, Data Mining & OLAP”, Tata
McGraw – Hill Edi琀椀on, Thirteenth Reprint 2008.
2. 2. Ralph Kimball, “The Data Warehouse Toolkit: The Complete Guide to Dimensional
Modeling”, Third edi琀椀on, 2013.
lOMoARcPSD|37281518
REFERENCES:
1. Paul Raj Ponniah, “Data warehousing fundamentals for IT Professionals”, 2012.
2. K.P. Soman, Shyam Diwakar and V. Ajay “Insight into Data mining Theory and
Prac琀椀ce”, Easter Economy Edi琀椀on, Pren琀椀ce Hall of India, 2006.
List of Experiments
Ex.No Page Staff
Sl.No Date Title of the Experiments
. No. sign
1. 1 Data explora琀椀on and integra琀椀on with WEKA
2. 2 Apply weka tool for data valida琀椀on
3. 3 Plan the architecture for real 琀椀me applica琀椀on
4. 4 Write the query for schema de昀椀ni琀椀on
Design data ware house for real 琀椀me
5. 5
applica琀椀ons
6. 6 Analyse the dimensional Modeling
7. 7 Case study using OLAP
8. 8 Case study using OTLP
9. 9 Implementa琀椀on of warehouse tes琀椀ng.
lOMoARcPSD|37281518
EX.NO.:1 DATA EXPLORATION AND INTEGRATION WITH WEKA
AIM:
To exploring the data and performing integration with weka
PROCEDURE:
WEKA - an open-source software provides tools for data preprocessing, implementation
of several Machine Learning algorithms, and visualization tools so that you can develop machine
learning techniques and apply them to real-world data mining problems. What WEKA offers is
summarized in the following diagram:
lOMoARcPSD|37281518
WEKA Installation
To install WEKA on your machine, visit WEKA’s official website and download the installation
file. WEKA supports installation on Windows, Mac OS X and Linux. You just need to follow the
instructions on this page to install WEKA for your OS.
The WEKA GUI Chooser application will start and you would see the following screen
The GUI Chooser application allows you to run five different types of applications as listed
here:
Explorer
Experimenter
KnowledgeFlow
Workbench
Simple CLI
WEKA – Launching Explorer
let us look into various functionalities that the explorer provides for working with big data. When
you click on the Explorer button in the Applications selector, it opens the following screen:
lOMoARcPSD|37281518
On the top, you will see several tabs as listed here:
Preprocess
Classify
Cluster
Associate
Select Attributes
Visualize
Under these tabs, there are several pre-implemented machine learning algorithms. Let us
look into each of them in detail now.
Preprocess Tab
Initially as you open the explorer, only the Preprocess tab is enabled. The first step in machine
learning is to preprocess the data. Thus, in the Preprocess option, you will select the data file,
process it and make it fit for applying the various machine learning algorithms.
Classify Tab
The Classify tab provides you several machine learning algorithms for the classification of your
data. To list a few, you may apply algorithms such as Linear Regression, Logistic Regression,
Support Vector Machines, Decision Trees, RandomTree, RandomForest, NaiveBayes, and so on.
The list is very exhaustive and provides both supervised and unsupervised machine learning
algorithms.
Cluster Tab
lOMoARcPSD|37281518
Under the Cluster tab, there are several clustering algorithms provided - such as SimpleKMeans,
FilteredClusterer, HierarchicalClusterer, and so on.
Associate Tab
Under the Associate tab, you would find Apriori, FilteredAssociator and FPGrowth.
Select Attributes Tab
Select Attributes allows you feature selections based on several algorithms such as
ClassifierSubsetEval, PrinicipalComponents, etc.
Visualize Tab
Lastly, the Visualize option allows you to visualize your processed data for analysis. As you
noticed, WEKA provides several ready-to-use algorithms for testing and building your machine
learning applications. To use WEKA effectively, you must have a sound knowledge of these
algorithms, how they work, which one to choose under what circumstances, what to look for in
their processed output, and so on. In short, you must have a solid foundation in machine learning
to use WEKA effectively in building your apps.
Loading Data
The data can be loaded from the following sources:
Local file system
Web
Database
Loading Data from Local File System
There are three buttons
Open file
Open URL
Open DB
Click on the Open file ... button. A directory navigator window opens as shown in the following
screen
lOMoARcPSD|37281518
Loading Data from Web
Once you click on the Open URL button, you can see a window as follows:
lOMoARcPSD|37281518
We will open the file from a public URL Type the following URL in the popup box:
https://storm.cis.fordham.edu/~gweiss/data-mining/weka-data/weather.nominal.arff
You may specify any other URL where your data is stored. The Explorer will load the data from
the remote site into its environment.
Loading Data from DB
Once you click on the Open DB button, you can see a window as follows:
lOMoARcPSD|37281518
Set the connection string to your database, set up the query for data selection, process the query
and load the selected records in WEKA.
WEKA File Formats
WEKA supports a large number of file formats for the data. Here is the complete list:
arff
arff.gz
bsi
csv
dat
data
json
json.gz
libsvm
m
names
xrff
xrff.gz
The types of files that it supports are listed in the drop-down list box at the bottom of the screen.
This is shown in the screenshot given below.
lOMoARcPSD|37281518
As you would notice it supports several formats including CSV and JSON. The default file type
is Arff.
Arff Format
An Arff file contains two sections - header and data.
The header describes the attribute types.
The data section contains a comma separated list of data.
As an example for Arff format, the Weather data file loaded from the WEKA sample databases is
shown below:
lOMoARcPSD|37281518
From the screenshot, you can infer the following points:
The @relation tag defines the name of the database.
The @attribute tag defines the attributes.
The @data tag starts the list of data rows each containing the comma separated
fields.
The attributes can take nominal values as in the case of outlook shown here:
@attribute outlook (sunny, overcast, rainy)
The attributes can take real values as in this case:
@attribute temperature real
You can also set a Target or a Class variable called play as shown here:
@attribute play (yes, no)
The Target assumes two nominal values yes or no.
Understanding Data
Let us first look at the highlighted Current relation sub window. It shows the name of the
database that is currently loaded. You can infer two points from this sub window:
There are 14 instances - the number of rows in the table.
The table contains 5 attributes - the fields, which are discussed in the upcoming
sections.
On the left side, notice the Attributes sub window that displays the various fields in the
database.
lOMoARcPSD|37281518
The weather database contains five fields - outlook, temperature, humidity, windy and play.
when you select an attribute from this list by clicking on it, further details on the attribute itself
are displayed on the right hand side.
Let us select the temperature attribute first. When you click on it, you would see the following
screen:
In the Selected Attribute subwindow, you can observe the following:
The name and the type of the attribute are displayed.
The type for the temperature attribute is Nominal.
The number of Missing values is zero.
There are three distinct values with no unique value.
The table underneath this information shows the nominal values for this field as
hot, mild and cold.
It also shows the count and weight in terms of a percentage for each nominal
value.
At the bottom of the window, you see the visual representation of the class values
If you click on the Visualize All button, you will be able to see all features in one single window
as shown here:
lOMoARcPSD|37281518
Removing Attributes
Many a time, the data that you want to use for model building comes with many irrelevant fields.
For example, the customer database may contain his mobile number which is relevant in
analysing his credit rating
To remove Attribute/s select them and click on the Remove button at the bottom.
The selected attributes would be removed from the database. After you fully preprocess the data,
you can save it for model building.
Next, you will learn to preprocess the data by applying filters on this data.
lOMoARcPSD|37281518
Data Integration
Suppose you have 2 datasets as below and need to merge them together
Open a Command Line Interface
Run the following command replacing values as needed
java -cp weka.jar weka.core.Instances merge <path to file1> <path to file 2> >
<path to result file>
Example
java weka.core.Instances merge C:\Users\Ram\Downloads\file1.csv C:\Users\Ram\Downloads\
file2.csv > C:\Users\Ram\Downloads\results.csv
Finished redirecting output to 'C:\Users\Ram\Downloads\results.csv'.
Now you can see results.csv or results.csv file in your given location as below.
lOMoARcPSD|37281518
EX.NO.:2 APPLY WEKA TOOL FOR DATA VALIDATION
AIM:
To validate the data stored in data warehouse using Weka tool.
PROCEDURE:
Data validation is the process of verifying and validating data that is collected before it is
used. Any type of data handling task, whether it is gathering data, analyzing it, or structuring it
for presentation, must include data validation to ensure accurate results.
1. Data Sampling
After loading your dataset
Click on choose ( certain datasets in sample datasets does not allow this operation. I used
Brest-cancer dataset for this experiment )
Filters -> supervised -> Instance -> Re-sample
Click on the name of the algorithm to change parameters
Change biasToUniformClass to have a biased sample. If you set it to 1 resulting dataset
will have equal number of instances for each class. Ex:- Brest-cancer positive 20 negative
20.
Change noReplacement accordingly.
Change sampleSizePrecent accordingly. ( self explanatory )
lOMoARcPSD|37281518
2. Removing duplicates
Choose filters -> unsupervised -> instance -> RemoveDuplicates
Compare the results as below
2.2 Dealing with missing values
Open labor.arff file using weka. ( which has missing values )
Click on edit button on the top bar to get a view of dataset as blow. Here you can clearly
see the missing values in gray areas.
Filters -> unsupervised -> attribute -> replaceMissingValuesWithUserConstants
lOMoARcPSD|37281518
In attributes set the column number you want to replace values.
Set nominalStringReplacementValue to a suitable string if selected column is nominal.
Set numericReplacementValue to 0, -1 depending on your requirement if selected column
is numeric.
fill all replacement values, add “first-last” to attribute column to apply for all columns at
once.
ReplaceMissingValues filter which replace numeric values with mean and nominal values
with mode.
3. Data Reduction
PCA
Load iris dataset
Filters -> unsupervised -> attribute -> PrincipleComponents
lOMoARcPSD|37281518
Original iris dataset have 5 columns. ( 4 data + 1 class ). Lets reduce that to 3 columns ( 2
data + 1 class ).
maximumAttributes – No of attributes we need after reduction.
maximumAttributeNames – PCA algorithm calculates 4 principle components for this
dataset. Upon them we are selecting the 2 components which have the most variance
( PC1, PC2 ). Then we need to re-represent data again using these selected components
( reducing 4D plot to 2D plot ). In this process we can select how many principle
components we are using when re-generating values. See the final result below where you
can see new columns are created using 3 principle components multiplied by respective
bias values.
4. Data transformation
Normalization
Load iris dataset
Filters -> unsupervised -> attribute -> normalize
lOMoARcPSD|37281518
Normalization is important when you don’t know the distribution of data beforehand.
Scale is the length of number line and translation is the lower bound.
Ex :- scale 2 and translation -1 => -1 to 1, scale 4 and translation -2 => -2 to 2
This filter get applied to all numeric columns. You can’t selectively normalize.
Standardization
Load iris dataset.
Used when dataset known to be in Gaussian (bell curve) distribution.
Filters -> unsupervised -> attribute -> standardize
This filter get applied to all numeric columns. You can’t selectively standardize.
Discretization
Load diabetes dataset.
Discretization comes in handy when using decision trees.
Suppose you need to change weight column to two values like low and high.
lOMoARcPSD|37281518
Set column number 6 to AttributeIndices.
Set bins to 2 ( Low/ High)
When you set equal frequency to true there will be equal number of high and low entries
in the final column.
lOMoARcPSD|37281518
EX.NO.:3 PLAN THE ARCHITECTURE FOR REAL TIME APPLICATION
AIM:
To plan the architecture for real time application.
PROCEDURE:
DESIGN STEPS:
1. Gather Requirements: Aligning the business goals and needs of different departments
with the overall data warehouse project.
2. Set Up Environments: This step is about creating three environments for data warehouse
development, testing, and production, each running on separate servers
3. Data Modeling: Design the data warehouse schema, including the fact tables and
dimension tables, to support the business requirements.
4. Develop Your ETL Process: ETL stands for Extract, Transform, and Load. This process
is how data gets moved from its source into your warehouse.
5. OLAP Cube Design: Design OLAP cubes to support analysis and reporting
requirements.
6. Reporting & Analysis: Developing and deploying the reporting and analytics tools that
will be used to extract insights and knowledge from the data warehouse.
7. Optimize Queries: Optimizing queries ensures that the system can handle large amounts
of data and respond quickly to queries.
8. Establish a Rollout Plan: Determine how the data warehouse will be introduced to the
organization, which groups or individuals will have access to it, and how the data will be
presented to these users.
lOMoARcPSD|37281518
EX.NO.:4 QUERY FOR SCHEMA DEFINITION
AIM:
To Write a query for Star, Snowflake and Galaxy schema definitions.
PROCEDURE:
STAR SCHEMA
Each dimension in a star schema is represented with only one-dimension table.
This dimension table contains the set of attributes.
There is a fact table at the center. It contains the keys to each of four dimensions.
The fact table also contains the attributes
SNOWFLAKE SCHEMA
Some dimension tables in the Snowflake schema are normalized.
The normalization splits up the data into additional tables.
Unlike Star schema, the dimensions table in a snowflake schema are normalized.
FACT CONSTELLATION SCHEMA
A fact constellation has multiple fact tables. It is also known as galaxy schema.
The sales fact table is same as that in the star schema.
The sales fact table is same as that in the star schema.
The shipping fact table also contains two measures, namely dollars sold and units sold.
SYNTAX:
Cube Definition:
define cube < cube_name > [ < dimension-list > }: < measure_list >
Dimension Definition:
define dimension < dimension_name > as ( < attribute_or_dimension_list > )
SAMPLE PROGRAM:
STAR SCHEMA DEFINITION
define cube sales star [time, item, branch, location]:
dollars sold = sum(sales in dollars), units sold = count(*)
define dimension time as (time key, day, day of week, month, quarter, year)
define dimension item as (item key, item name, brand, type, supplier type)
lOMoARcPSD|37281518
define dimension branch as (branch key, branch name, branch type)
define dimension location as (location key, street, city, province or state, country)
SNOWFLAKE SCHEMA DEFINITION
define cube sales snowflake [time, item, branch, location]:
dollars sold = sum(sales in dollars), units sold = count(*)
define dimension time as (time key, day, day of week, month, quarter, year)
define dimension item as (item key, item name, brand, type, supplier (supplier key, supplier
type))
define dimension branch as (branch key, branch name, branch type)
define dimension location as (location key, street, city (city key, city, province or state, country))
FACT CONSTELLATION SCHEMA DEFINITION
define cube sales [time, item, branch, location]:
dollars sold = sum(sales in dollars), units sold = count(*)
define dimension time as (time key, day, day of week, month, quarter, year)
define dimension item as (item key, item name, brand, type, supplier type)
define dimension branch as (branch key, branch name, branch type)
define dimension location as (location key, street, city, province or state,country)
define cube shipping [time, item, shipper, from location, to location]:
dollars cost = sum(cost in dollars), units shipped = count(*)
define dimension time as time in cube sales
define dimension item as item in cube sales
define dimension shipper as (shipper key, shipper name, location as location in cube sales,
shipper type)
define dimension from location as location in cube sales
define dimension to location as location in cube sales
OUTPUT :
STAR SCHEMA
lOMoARcPSD|37281518
SNOWFLAKE SCHEMA
FACT CONSTELLATION SCHEMA
lOMoARcPSD|37281518
RESULT:
Thus the query for star, Snowflake and Galaxy schema was written Successfully.
EX.NO.:5 DESIGN DATA WARE HOUSE FOR REAL TIME APPLICATIONS
AIM:
To design a data warehouse for real time applications
PROCEDURE:
lOMoARcPSD|37281518
DATA CLEANSING AND TRANSFORMING
Dropping Tables
Since decision-making is concerned with the trends related to students’ history, behavior, and
academic
performance, tables “assets” and “item” are not needed; and therefore, they are discarded and
excluded from the
data warehouse.
DROP TABLE assets ;
DROP TABLE item ;
Merging Tables
Based on the design assumptions, the three tables “department”, “section”, and “course” do not
constitute
separately important parameters for extracting relevant patterns and discovering knowledge.
Therefore, they are
merged altogether with the “transcript_fact_table” table.
SELECT co_name FROM course, section, transcript
WHERE tr_id = n AND str_semester/year = se_semester/year AND tr_se_num = se_num AND
se_code =
co_code ;
ALTER TABLE transcript fact table ADD co_course TEXT ;
DROP TABLE department ;
DROP TABLE section ;
DROP TABLE course ;
lOMoARcPSD|37281518
Furthermore, table “Activities” is merged with table “RegistrationActivities” and a new table is
produced
called “RegisteredActivities”.
SELECT act_name FROM activities, registrationActivities
WHERE reg_act_id = act_id ;
New Columns
During transformation new columns can be added. In fact, tr_courseDifficulty is added to table
“transcript_fact_table” in order to increase the degree of knowledge and information.
ALTER TABLE transcript_fact_table ADD tr_courseDifficulty TEXT ;
Moreover a Boolean column is added to table “receipt” called re_paidOnDueDate
ALTER TABLE receipt (re_paidOnDueDate) ;
Removing Columns
Unnecessary columns can be removed too during the transformation process. Below is a list of
useless columns
that were discarded during the transformation process from tables “Account”, “Student”,
“Receipt” and
“Activities” respectively:
ALTER TABLE Receipt REMOVE re_dueDate
REMOVE re_dateOfPayment ;
ALTER TABLE Activities REMOVE ac_supervisor ;
ALTER TABLE Student REMOVE st_phone
REMOVE st_email ;
Conceptual Schema – The Snowflake Schema
The proposed data warehouse is a Snowflake type design with one center fact table and seven
dimensions
Output:
lOMoARcPSD|37281518
