Subject: DA&M By: Muhammad Imran Afzal
WEEK 9
Data Storage, Indexing, and Performance
Tuning Techniques
I. Data Storage in Databases
Efficient data storage is crucial for the scalability, reliability, and performance of a database
system. The way data is stored affects retrieval speed, disk space usage, and overall system
behavior.
1. Types of Data Storage
A. Heap Storage (Unordered Storage)
Data is stored without any specific order.
Suitable for small tables or when frequent inserts occur.
Slower for searching without indexes.
B. Clustered/Ordered Storage
Rows are stored based on a specific column's value (e.g., primary key).
Faster range queries and lookups on the clustered column.
C. Partitioned Storage
Large tables are divided into partitions based on ranges or keys (e.g., by date or
region).
Improves query performance and manageability.
D. Tablespaces (DBMS-specific)
Logical storage units that group related tables and indexes in physical files.
2. Storage Optimization Techniques
Normalization: Reduces redundancy, saves space.
Compression: Reduces storage footprint at the cost of CPU.
Data Archiving: Move old data to separate archive tables to improve active query
performance.
Blob/File Storage: Use external file systems for large files (e.g., images) when
necessary.
II. Indexing
Subject: DA&M By: Muhammad Imran Afzal
Indexing is a technique used to speed up data retrieval operations on a database table at the
cost of additional storage and write time.
1. What is an Index?
An index is a data structure (e.g., B-tree, hash table) that provides a fast lookup of rows in a
table based on the values of one or more columns.
2. Types of Indexes
A. Single-Column Index
Index on one column (e.g., employee_id)
B. Composite (Multi-column) Index
Index on multiple columns. Order matters.
sql
CopyEdit
CREATE INDEX idx_name_dept ON employees(name, department);
C. Unique Index
Enforces uniqueness (often used on primary key columns).
D. Full-Text Index
Used for searching large text fields.
E. Bitmap Index (for OLAP systems)
Efficient for columns with low cardinality (e.g., gender, status).
Physically orders the rows of a table by the indexed column.
F. Clustered vs. Non-Clustered Indexes There can be only one clustered index per table.
Think of the table itself being sorted
Clustered: Physically orders the table (only one allowed).
Non-Clustered: Separate structure that points to the table rows.
Separate structure from the actual table.
Stores the indexed column + a pointer to the actual row.
You can have many non-clustered indexes
3. Index Best Practices
Index columns used in WHERE, JOIN, ORDER BY, GROUP BY.
Avoid indexing columns with high update/delete frequency.
Don’t over-index – each index adds storage overhead and slows down writes.
Monitor index usage and drop unused indexes.
III. Performance Tuning Techniques
Subject: DA&M By: Muhammad Imran Afzal
Performance tuning involves analyzing and optimizing the performance of SQL queries and
database configurations.
1. Query Optimization
A. Use Proper Indexes
Use EXPLAIN or EXPLAIN PLAN to check if the query uses indexes.
B. **Avoid SELECT ***
Fetch only required columns to reduce I/O and memory usage.
sql
CopyEdit
-- Bad
SELECT * FROM orders;
-- Good
SELECT order_id, order_date FROM orders;
C. Filter Early
Use WHERE clauses to minimize result sets before joins or aggregations.
D. Use Joins Wisely
Prefer INNER JOIN over OUTER JOIN if applicable.
Ensure join columns are indexed.
E. Avoid Functions on Indexed Columns
Avoid wrapping indexed columns in functions in WHERE clauses.
sql
CopyEdit
-- Bad
WHERE UPPER(name) = 'JOHN';
-- Good
WHERE name = 'John';
F. Limit Use of Subqueries
Consider rewriting subqueries as JOINs or Common Table Expressions (CTEs).
2. Database Configuration Tuning
A. Memory Allocation
Subject: DA&M By: Muhammad Imran Afzal
Allocate sufficient memory to buffer pools (e.g., InnoDB buffer pool in MySQL).
Cache frequently accessed data.
B. Connection Pooling
Use connection pooling to reduce overhead from repeated connections.
C. Parallelism
Use parallel execution for large queries where supported.
3. Table and Index Maintenance
Analyze Table: Update statistics to help the optimizer.
Rebuild Indexes: Periodically rebuild fragmented indexes.
Vacuum (PostgreSQL): Clean up dead tuples.
4. Monitoring & Profiling Tools
EXPLAIN / EXPLAIN PLAN – Shows query execution path.
Query Profiler / Analyzer Tools (e.g., MySQL Workbench, pgAdmin).
Performance Schema (MySQL), Dynamic Management Views (SQL Server).
IV. Summary and Best Practices
Area Best Practice
Data Storage Normalize, compress, and archive data wisely
Indexing Use indexes on search/filter/join columns, avoid over-indexing
Query Tuning Use EXPLAIN, avoid SELECT *, optimize joins
Configuration Tune memory, cache, and connections
Monitoring Use database tools to track slow queries and deadlocks
Effective database performance relies on a balance between proper storage design, smart
indexing strategies, and continuous performance tuning. Always measure, analyze, and
iteratively improve based on your specific workload and environment.
Performance Tuning Techniques in Databases
Performance Tuning Techniques are strategies and methods used to optimize the speed,
efficiency, and responsiveness of a database system. The goal is to ensure the fastest
execution of queries, efficient use of system resources, and smooth user experience,
especially when handling large amounts of data or many simultaneous users.
Why Performance Tuning is Important
Reduce query response times.
Improve application performance.
Subject: DA&M By: Muhammad Imran Afzal
Reduce server load and resource usage.
Ensure scalability for growing data.
Avoid bottlenecks and system downtime.
Key Areas of Performance Tuning
1.Query Optimization
Improving how SQL queries are written and executed.
✅Techniques:
Avoid SELECT * – fetch only needed columns.
Use WHERE clauses to filter data early.
Use JOINs appropriately – index joined columns.
Rewrite subqueries as JOINs or CTEs (Common Table Expressions).
Avoid functions on indexed columns in WHERE clauses (e.g., UPPER(col)).
Use LIMIT to restrict returned rows.
Optimize use of GROUP BY, ORDER BY, and DISTINCT.
2.Indexing Strategies
Proper indexing drastically reduces the time it takes to retrieve data.
✅Techniques:
Create indexes on columns used in WHERE, JOIN, ORDER BY, and GROUP BY.
Use composite indexes when filtering on multiple columns.
Use bitmap indexes for low-cardinality columns (in OLAP systems).
Periodically rebuild or reorganize fragmented indexes.
Monitor and drop unused indexes to save resources.
3. Database Configuration Tuning
Tuning internal DBMS settings to use memory, CPU, and disk I/O efficiently.
✅Techniques:
Increase buffer pool/cache size (e.g., InnoDB Buffer Pool in MySQL).
Adjust query cache settings.
Tune sort area size, temp table size, parallelism, etc.
Enable connection pooling to reduce connection overhead.
Use workload management to prioritize critical queries.
4.Storage Optimization
Efficient use of disk and memory to speed up access to data.
Subject: DA&M By: Muhammad Imran Afzal
✅Techniques:
Use data compression where supported.
Use partitioning to divide large tables (by date, region, etc.).
Store frequently accessed data in memory (caching).
Archive old or unused data to reduce table size.
5. Monitoring and Analysis
Identifying and fixing slow queries and bottlenecks using diagnostic tools.
✅Tools & Methods:
Use EXPLAIN, EXPLAIN PLAN, or SHOW PLAN to see how queries run.
Monitor with tools like:
o MySQL Workbench (MySQL)
o pgAdmin (PostgreSQL)
o SQL Server Profiler
o Oracle AWR Reports
Track slow query logs, CPU usage, disk I/O, and memory usage.
Use performance dashboards or third-party tools like New Relic, Datadog, etc.
6.Application-Level Optimization
How the application interacts with the database affects performance too.
✅Techniques:
Use connection pooling to reduce DB overhead.
Batch operations instead of many single-row inserts/updates.
Use stored procedures to reduce round-trips between app and DB.
Avoid unnecessary queries – cache results in the app where possible.
✅Summary of Performance Tuning Techniques
Area Key Techniques
Query Optimization Use indexes, optimize joins, avoid SELECT *
Indexing Create useful indexes, avoid over-indexing
Configuration Tuning Adjust memory/cache sizes, connection pools
Storage Optimization Use partitioning, compression, and archiving
Monitoring & Analysis Use tools to find and fix bottlenecks
Application Optimization Use batching, caching, and stored procedures
Performance tuning is not one-time – it’s a continuous process. As data grows and usage
changes, ongoing analysis and optimization are key.
Subject: DA&M By: Muhammad Imran Afzal
