Transaction Log Logical Architecture

SQL Server 2005

Other Versions

The SQL Server 2005 transaction log operates logically as if the transaction log is a string of log records.
Each log record is identified by a log sequence number (LSN). Each new log record is written to the logical
end of the log with an LSN that is higher than the LSN of the record before it.

Log records are stored in a serial sequence as they are created. Each log record contains the ID of the
transaction that it belongs to. For each transaction, all log records associated with the transaction are
individually linked in a chain using backward pointers that speed the rollback of the transaction.

Log records for data modifications record either the logical operation performed or they record the
before and after images of the modified data. The before image is a copy of the data before the operation
is performed; the after image is a copy of the data after the operation has been performed.

The steps to recover an operation depend on the type of log record:

 Logical operation logged 

 To roll the logical operation forward, the operation is performed again.
 To roll the logical operation back, the reverse logical operation is performed.
 Before and after image logged 
 To roll the operation forward, the after image is applied.
 To roll the operation back, the before image is applied.

Many types of operations are recorded in the transaction log. These operations include:

 The start and end of each transaction.

 Every data modification (insert, update, or delete). This includes changes by system stored
procedures or data definition language (DDL) statements to any table, including system tables.
 Every extent and page allocation or deallocation.
 Creating or dropping a table or index.

Rollback operations are also logged. Each transaction reserves space on the transaction log to make sure
that enough log space exists to support a rollback that is caused by either an explicit rollback statement
or if an error is encountered. The amount of space reserved depends on the operations performed in the
transaction, but generally is equal to the amount of space used to log each operation. This reserved space
is freed when the transaction is completed.

The section of the log file from the first log record that must be present for a successful database-wide
rollback to the last-written log record is called the active part of the log, or the active log. This is the
section of the log required to do a full recovery of the database. No part of the active log can ever be
truncated.

Working with Transaction Log Backups

SQL Server 2005
Other Versions

Updated: 17 July 2006

 The topic is relevant only for databases that are using the full or bulk-logged recovery models.

This topic presents concepts about how to back up and restore (apply) transaction logs. Under the full and
bulk-logged recovery models, taking routine backups of transaction logs (log backups) is necessary for
recovering data. With log backups, you can recover the database to the point of failure or to a specific
point in time. We recommend taking frequent enough log backups to support your business
requirements, specifically your tolerance for work loss such as might be caused by a damaged log drive.
The appropriate frequency for taking log backups depends on your tolerance for work-loss exposure
balanced by how many log backups you can store, manage, and, potentially, restore. Taking a log backup
every 15 to 30 minutes might be enough. If your business requires that you minimize work-loss exposure,
consider taking log backups more frequently. More frequent log backups have the added advantage of
increasing the frequency of log truncation, resulting in smaller log file or files.

Before you can create the first log backup, you must create a full backup, such as a database backup or
the first in a set of file backups. Restoring a database by using only file backups can become complex.
Therefore, we recommend that you start with a full database backup when you can. Thereafter, backing
up the transaction log regularly is necessary. This not only minimizes work-loss exposure but also enables
truncation of the transaction log. Typically, the transaction log is truncated after every conventional log
backup. However, log truncation can be delayed. For more information, see Factors That Can Delay Log
Truncation.

In SQL Server 2005, you can back up the log while any full backup is running.

The Log Chain

A continuous sequence of log backups is called a log chain. A log chain starts with a full backup of the
database. Usually, a new log chain is only started when the database is backed up for the first time, or
after the recovery model is switched from simple recovery to full or bulk-logged recovery.

To restore a database up to the point of failure, the log chain must be intact. That is, an unbroken
sequence of transaction log backups must extend up to the point of failure. Where this sequence of log
must start depends on the type of data backups you are restoring: database, partial, or file. For a database
or partial backup, the sequence of log backups must extend from the end of a database or partial backup.
For a set of file backups, the sequence of log backups must extend from the start of a full set of file
backups.

If you are using only file backups, you must back up the log from the beginning of the first full file backup.
You can start taking log backups immediately after the first full file backup. We recommend starting then
because the first log backup can take a long time. While the log is being backed up, you back up the
other files. To restore the database from only file backups, the set of full file backups must be augmented
with one or more log backups that cover the interval between the first and last file backup.

Note:
To identify the backup that starts the log chain in a set of backups, query the begins_log_chain column of the b
HEADERONLY on the backup device to see the BeginsLogChain column in the results set.

Taking regular transaction log backups is necessary. In addition to letting you restore the backed-up
transactions, a log backup truncates the log to remove the backed up log records from the log file. If you
do not back up the log frequently enough, the log files can fill up. For information about how to handle a
full transaction log, see Troubleshooting a Full Transaction Log (Error 9002).

Important:
If a log backup becomes missing or damaged, start a new log chain by creating a full or differential database ba
transaction log to start a new log chain. We recommend that you retain transaction logs backups that come befo
you ever want to restore the database to a point in time within those backups. For information about how to hel
see Security Considerations for Backup and Restore.

For information about how to create log backups, see Creating Transaction Log Backups.

How Are Log Backups Used?

Restoring a log backup rolls forward the changes that were recorded in the transaction log to re-create
the exact state of the database at the time the log backup operation started. When you restore a
database, you will have to restore the log backups that were created after the full database backup that
you restore, or from the start of the first file backup that you restore. Typically, after you restore the most
recent data or differential backup, you must restore a series of log backups until you reach your recovery
point. Then, you recover the database. This rolls back all transactions that were incomplete when the
recovery started and brings the database online. After the database has been recovered, you cannot
restore any more backups.

Important:
To help prevent work loss before an offline restore or after a failure, we recommend that you back up the tail o
 that are not yet backed up. For more information, see Tail-Log Backups.

Applying Transaction Log Backups

SQL Server 2005
Other Versions

 The topic is relevant only for the full recovery model or bulk-logged recovery model.

This topic describes applying transaction log backups as part of restoring a SQL Server database. To apply
a transaction log backup, the following requirements must be met:

 The immediately previous full database backup or differential database backup must be restored
first.
 All transaction logs that are created after that full or differential database backup must be
restored in chronological order. If a transaction log backup in this log chain is lost or damaged, you
can restore only transaction logs before the missing transaction log. 
 The database has not yet been recovered. The database cannot be recovered until after the final
transaction log has been applied. If you recover the database after restoring one of the
intermediate transaction log backups, that before the end of the log chain, you cannot restore the
database past that point without restarting the complete restore sequence, starting with the full
database backup. 

Recovery and Transaction Logs

When you finish the restore operation and recover the database, recovery rolls back all incomplete
transactions. This is known as the undo phase. Rolling back is required to restore the integrity of the
database. After rollback, the database goes online, and no more transaction log backups can be applied
to the database.

For example, a series of transaction log backups contain a long-running transaction. The start of the
transaction is recorded in the first transaction log backup, but the end of the transaction is recorded in the
second transaction log backup. There is no record of a commit or rollback operation in the first
transaction log backup. If a recovery operation runs when the first transaction log backup is applied, the
long-running transaction is treated as incomplete, and data modifications recorded in the first transaction
log backup for the transaction are rolled back. SQL Server does not allow for the second transaction log
backup to be applied after this point.

Note:
 In some circumstances, you can explicitly add a file during log restore. For more information, see Controlling C
Roll Forward.

Having Enough Log Backups for a Restore Sequence

You must have enough log records backed up to complete a restore sequence. The necessary log
backups, including the tail-log backup where required, must be available before the start of the restore
sequence.

Using Log Backups to Restore to the Point of Failure

Assume the following sequence of events.

Time Event
8:00 A.M. Back up database to create a full database backup.
Noon Back up transaction log.
4:00 P.M. Back up transaction log.
6:00 P.M. Back up database to create a full database backup.
8:00 P.M. Back up transaction log.
9:45 P.M. Failure occurs.

Note:
For an explanation of this example sequence of backups, see Creating Transaction Log Backups.

To restore the database to its state at 9:45 P.M. (the point of failure), either of the following alternative
procedures can be used:

Alternative 1: Restore the database by using the most recent full database backup

1. Create a tail-log backup of the currently active transaction log as of the point of failure.
2. Do not restore the 8:00 A.M. full database backup. Instead, restore the more recent 6:00 P.M. full
database backup, and then apply the 8:00 P.M. log backup and the tail-log backup. 
This occurs again for the 9:45 P.M. transaction tail-log backup. 

Alternative 2: Restore the database by using an earlier full database backup

Note:
This alternative process is useful if a problem prevents you from using the 6:00 P.M. full database backup. Thi
 restoring from the 6:00 P.M. full database backup.

1. Create a tail-log backup of the currently active transaction log as of the point of failure.
2. Restore the 8:00 A.M. full database backup, and then restore all four transaction log backups in
sequence. This rolls forward all completed transactions up to 9:45 P.M. 
This alternative points out the redundant security offered by maintaining a chain of transaction
log backups across a series of full database backups. 

Using Log Backups to Restore to a Specific Point in Time

In some cases, you can also use transaction logs to restore a database to a specific point in time. For more
information, see Restoring a Database to a Point Within a Backup.

Appling Transaction Log Backups

A best practice is to restore all the log backups by using WITH NORECOVERY:

RESTORE LOG database_name FROM <backup_device> WITH NORECOVERY

Then, after restoring the last log backup, recover the database in a separate operation:

RESTORE DATABASE database_name WITH RECOVERY

To apply a transaction log backup

 How to: Apply a Transaction Log Backup (Transact-SQL) 

 How to: Restore a Transaction Log Backup (SQL Server Management Studio)  

To restore to your recovery point

 How to: Restore to the Point of Failure (Transact-SQL) 

 How to: Restore to a Point in Time (SQL Server Management Studio)  
 SqlRestore (SMO)
Transaction Log Physical Architecture
The transaction log is used to guarantee the data integrity of the database and for data recovery. The topics
in this section provide the information about the physical architecture of the transaction log. Understanding
the physical architecture can improve your effectiveness in managing transaction logs.

The transaction log in a database maps over one or more physical files. Conceptually, the log file is a string
of log records. Physically, the sequence of log records is stored efficiently in the set of physical files that
implement the transaction log.

The SQL Server Database Engine divides each physical log file internally into a number of virtual log files.
Virtual log files have no fixed size, and there is no fixed number of virtual log files for a physical log file. The
Database Engine chooses the size of the virtual log files dynamically while it is creating or extending log
files. The Database Engine tries to maintain a small number of virtual files. The size of the virtual files after
a log file has been extended is the sum of the size of the existing log and the size of the new file increment.
The size or number of virtual log files cannot be configured or set by administrators.

The only time virtual log files affect system performance is if the log files are defined by
small size and growth_increment values. If these log files grow to a large size because of many small
increments, they will have lots of virtual log files. This can slow down database startup and also log backup
and restore operations. We recommend that you assign log files a size value close to the final size required,
and also have a relatively largegrowth_increment value
The transaction log is a wrap-around file. For example, consider a database with one physical log file divided
into four virtual log files. When the database is created, the logical log file begins at the start of the physical
log file. New log records are added at the end of the logical log and expand toward the end of the physical
log. Log truncation frees any virtual logs whose records all appear in front of the minimum recovery log
sequence number (MinLSN). The MinLSN is the log sequence number of the oldest log record that is required
for a successful database-wide rollback. The transaction log in the example database would look similar to
the one in the following illustration.

When the end of the logical log reaches the end of the physical log file, the new log records wrap around to
the start of the physical log file.

This cycle repeats endlessly, as long as the end of the logical log never reaches the beginning of the logical
log. If the old log records are truncated frequently enough to always leave sufficient room for all the new log
records created through the next checkpoint, the log never fills. However, if the end of the logical log does
reach the start of the logical log, one of two things occurs:

 If the FILEGROWTH setting is enabled for the log and space is available on the disk, the file is
extended by the amount specified ingrowth_increment and the new log records are added to the
extension. For more information about the FILEGROWTH setting, see ALTER DATABASE (Transact-
SQL).

 If the FILEGROWTH setting is not enabled, or the disk that is holding the log file has less free space
than the amount specified ingrowth_increment, an 9002 error is generated.

If the log contains multiple physical log files, the logical log will move through all the physical log files before
it wraps back to the start of the first physical log file.
Introduction to Transaction Logs
Every SQL Server 2005 database has a transaction log that records all transactions and the database
modifications made by each transaction. The transaction log is a critical component of the database and, if
there is a system failure, the transaction log might be required to bring your database back to a consistent
state. The transaction log should never be deleted or moved unless you fully understand the ramifications of
doing this.

  Operations Supported by the Transaction Log

The transaction log supports the following operations:

 Recovery of individual transactions.

 Recovery of all incomplete transactions when SQL Server is started.
 Rolling a restored database, file, filegroup, or page forward to the point of failure.
 Supporting transactional replication.
 Supporting standby-server solutions.

Recovery of Individual Transactions

If an application issues a ROLLBACK statement, or if the Database Engine detects an error such as the loss
of communication with a client, the log records are used to roll back the modifications made by an
incomplete transaction.

Recovery of all Incomplete Transactions When SQL Server Is Started

If a server that is running SQL Server fails, the databases may be left in a state where some modifications
were never written from the buffer cache to the data files, and there may be some modifications from
incomplete transactions in the data files. When an instance of SQL Server is started, it runs a recovery of
each database. Every modification recorded in the log which may not have been written to the data files is
rolled forward. Every incomplete transaction found in the transaction log is then rolled back to make sure
the integrity of the database is preserved.

Rolling a Restored Database, File, Filegroup, or Page Forward to the Point of Failure
After a hardware loss or disk failure affecting the database files, you can restore the database to the point of
failure. You first restore the last full database backup and the last differential database backup, and then
restore the subsequent sequence of the transaction log backups to the point of failure. As you restore each
log backup, the Database Engine reapplies all the modifications recorded in the log to roll forward all the
transactions. When the last log backup is restored, the Database Engine then uses the log information to roll
back all transactions that were not complete at that point.

Supporting Transactional Replication

The Log Reader Agent monitors the transaction log of each database configured for transactional replication
and copies the transactions marked for replication from the transaction log into the distribution database.
For more information, see How Transactional Replication Works.

Supporting Standby-Server Solutions

The standby-server solutions, database mirroring, and log shipping, rely heavily on the transaction log. In a
log shipping scenario, the primary server sends the active transaction log of the primary database to one or
more destinations. Each secondary server restores the log to its local secondary database. For more
information, see Log Shipping Overview.

In a database mirroring scenario, every update to a database, the principal database, is immediately
reproduced in a separate, full copy of the database, the mirror database. The principal server instance sends
each log record immediately to the mirror server instance which applies the incoming log records to the
mirror database, continually rolling it forward. For more information, see Database Mirroring Overview.

  Transaction Log Characteristics

Following are the characteristics of the SQL Server Database Engine transaction log:

 The transaction log is implemented as a separate file or set of files in the database. The log cache is
managed separately from the buffer cache for data pages, which results in simple, fast, and robust
code within the Database Engine.
  The format of log records and pages is not constrained to follow the format of data pages.
 The transaction log can be implemented in several files. The files can be defined to expand
automatically by setting the FILEGROWTH value for the log. This reduces the potential of running
out of space in the transaction log, while at the same time reducing administrative overhead. For
more information, see ALTER DATABASE (Transact-SQL).
 The mechanism to reuse the space within the log files is quick and has minimal effect on transaction
throughput.
 Working with Transaction Log Backups
  The topic is relevant only for databases that are using the full or bulk-logged recovery models.
 This topic presents concepts about how to back up and restore (apply) transaction logs. Under the
full and bulk-logged recovery models, taking routine backups of transaction logs (log backups) is
necessary for recovering data. In SQL Server 2005 and later versions, you can back up the log while
any full backup is running.
 Before you can create the first log backup, you must create a full backup, such as a database
backup or the first in a set of file backups. Restoring a database by using only file backups can
become complex. Therefore, we recommend that you start with a full database backup when you
can. Thereafter, backing up the transaction log regularly is necessary. This not only minimizes
work-loss exposure but also enables truncation of the transaction log. Typically, the transaction log
is truncated after every conventional log backup. However, log truncation can be delayed. For more
information, see Factors That Can Delay Log Truncation.
 We recommend taking frequent enough log backups to support your business requirements,
specifically your tolerance for work loss such as might be caused by a damaged log drive. The
appropriate frequency for taking log backups depends on your tolerance for work-loss exposure
balanced by how many log backups you can store, manage, and, potentially, restore. Taking a log
backup every 15 to 30 minutes might be enough. If your business requires that you minimize work-
loss exposure, consider taking log backups more frequently. More frequent log backups have the
added advantage of increasing the frequency of log truncation, resulting in smaller log files.
 To limit the number of log backups that you need to restore, it is essential to routinely back up your
data. For example, you might schedule a weekly full database backup and daily differential database
backups.

Note

By default, every successful backup operation adds an entry in the SQL Server error log and in the system event log. If you
very frequently, these success messages accumulate quickly, resulting in huge error logs that can make finding other messa
such cases, you can suppress these log entries by using trace flag 3226, if none of your scripts depend on those entries. For
information, see Trace Flags (Transact-SQL).

   The Log Chain

 A continuous sequence of log backups is called a log chain. A log chain starts with a full backup of
the database. Usually, a new log chain is only started when the database is backed up for the first
time or after the recovery model is switched from simple recovery to full or bulk-logged recovery.
 Unless you choose to overwrite existing backup sets when creating a full database backup, the
existing log chain remains intact. With the log chain intact, you can restore your database from any
full database backup in the media set, followed by all subsequent log backups up through your
recovery point. The recovery point could be the end of the last log backup or a specific recovery
point in any of the log backups.
 To restore a database up to the point of failure, the log chain must be intact. That is, an unbroken
sequence of transaction log backups must extend up to the point of failure. Where this sequence of
log must start depends on the type of data backups you are restoring: database, partial, or file. For
a database or partial backup, the sequence of log backups must extend from the end of a database
or partial backup. For a set of file backups, the sequence of log backups must extend from the start
of a full set of file backups.
 If you are using only file backups, you must back up the log from the beginning of the first full file
backup. You can start taking log backups immediately after the first full file backup. We recommend
starting then because the first log backup can take a long time. While the log is being backed up,
you back up the other files. To restore the database from only file backups, the set of full file
backups must be augmented with one or more log backups that cover the interval between the first
and last file backup.
 Note

To identify the backup that starts the log chain in a set of backups, query the begins_log_chain column of the backupset t
runRESTORE HEADERONLY on the backup device to see the BeginsLogChain column in the results set.

 Taking regular transaction log backups is necessary. In addition to letting you restore the backed-up
transactions, a log backup truncates the log to remove the backed up log records from the log file.
If you do not back up the log frequently enough, the log files can fill up. For information about how
to handle a full transaction log, see Troubleshooting a Full Transaction Log (Error 9002).

Important

If a log backup becomes missing or damaged, start a new log chain by creating a full or differential database backup and the
the transaction log to start a new log chain. We recommend that you retain transaction logs backups that come before a mis
backup, in case you ever want to restore the database to a point in time within those backups. For information about how to
your backups, see Security Considerations for Backup and Restore.

 For information about how to create log backups, see Creating Transaction Log Backups and Tail-
Log Backups.
   How Are Log Backups Used?
 Restoring a log backup rolls forward the changes that were recorded in the transaction log to re-
create the exact state of the database at the time the log backup operation started. When you
restore a database, you will have to restore the log backups that were created after the full
database backup that you restore, or from the start of the first file backup that you restore.
Typically, after you restore the most recent data or differential backup, you must restore a series of
log backups until you reach your recovery point. Then, you recover the database. This rolls back all
transactions that were incomplete when the recovery started and brings the database online. After
the database has been recovered, you cannot restore any more backups.

Transaction Log Logical Architecture

The SQL Server transaction log operates logically as if the transaction log is a string of log records. Each log
record is identified by a log sequence number (LSN). Each new log record is written to the logical end of the
log with an LSN that is higher than the LSN of the record before it.

Log records are stored in a serial sequence as they are created. Each log record contains the ID of the
transaction that it belongs to. For each transaction, all log records associated with the transaction are
individually linked in a chain using backward pointers that speed the rollback of the transaction.

Log records for data modifications record either the logical operation performed or they record the before
and after images of the modified data. The before image is a copy of the data before the operation is
performed; the after image is a copy of the data after the operation has been performed.

The steps to recover an operation depend on the type of log record:

 Logical operation logged

 To roll the logical operation forward, the operation is performed again.

 To roll the logical operation back, the reverse logical operation is performed.

 Before and after image logged

 To roll the operation forward, the after image is applied.

 To roll the operation back, the before image is applied.

Many types of operations are recorded in the transaction log. These operations include:

 The start and end of each transaction.

  Every data modification (insert, update, or delete). This includes changes by system stored
procedures or data definition language (DDL) statements to any table, including system tables.

 Every extent and page allocation or deallocation.

 Creating or dropping a table or index.

Rollback operations are also logged. Each transaction reserves space on the transaction log to make sure
that enough log space exists to support a rollback that is caused by either an explicit rollback statement or if
an error is encountered. The amount of space reserved depends on the operations performed in the
transaction, but generally is equal to the amount of space used to log each operation. This reserved space is
freed when the transaction is completed.

The section of the log file from the first log record that must be present for a successful database-wide
rollback to the last-written log record is called the active part of the log, or the active log. This is the section
of the log required to do a full recovery of the database. No part of the active log can ever be truncated.