<soapbox>

I’m in the fortunate position of living in a region of the world that’s relatively free of natural disasters. We’re reasonably geologically stable, the nearest fault lines are the Great Rift Valley and somewhere in the Antarctic ocean. We don’t get tornadoes, we’re a long way from the ocean (and Madagascar partially protects the coast from typhoons and tsunamis)

Given that, and looking at the recent events in Japan and Southern USA, local IT managers might be grateful that their critical infrastructure is here, not there. But that is no reason for complacency, no reason to ignore putting a disaster recovery plan in place.

Major huge national disasters, while they attract a whole lot of attention (and rightly so) are probably not the main cause of IT disasters. IT disasters, for the most part, are likely to be caused more by smaller events like these¹

• A drive in the RAID 5 array fails, and the SAN admin switches out the wrong drive.
• A SAN controller fails and overwrites a LUN or 2 with binary garbage.
• The server room floor collapses dumping the SAN 2 floors down into a garage, with the server on top of it.
• A water leak a floor above the server room results in the UPS getting a shower, and the resultant power surge fries the servers’ main boards
• A developer with far too many permissions truncates an important table on the production server, thinking he was working on the development environment.
• The mains power fails but the generators don’t come online because their fuel was drained a day earlier in preparation for maintenance.

Events like those (or probably even more mundane events) are the ones that we need to plan for. Relatively minor disaster that can leave business without critical infrastructure or critical data for hours or days.

You need to plan for the small disasters as well as the large ones. Plan for the dropped table. Plan for two drives failing in the RAID 5 array. Plan for the server’s power supply failing. Plan for the big disasters too, just don’t think that they’re the only thing that endangers your data and your business.

(1) I’ve personally seen 3 of those events happen, I’ve heard from people who have seen two more and I know of a company that’s at risk of one. They’re not just made-up improbably occurrences.

One that seems to come up now and again is the COPY_ONLY Option on backups. Typically the comment is something like ‘Use the COPY_ONLY option when taking ad-hoc full backups so that you don’t impact the log backups.’ Now we know from the blog posts linked above that full backups don’t ever break the log chain (and I’m not going to run yet more tests to prove it) so what is the copy only option there for?

Books Online states the following regarding the COPY_ONLY option for backups – “Specifies that the backup is a copy-only backup, which does not affect the normal sequence of backups. A copy-only backup is created independently of your regularly scheduled, conventional backups. A copy-only backup does not affect your overall backup and restore procedures for the database.”

Well, that doesn’t clear things up much. It does however go on to say this: “When used with BACKUP DATABASE, the COPY_ONLY option creates a full backup that cannot serve as a differential base. The differential bitmap is not updated, and differential backups behave as if the copy-only backup does not exist. Subsequent differential backups use the most recent conventional full backup as their base.”

So it’s not the log chain that copy only is there to not affect, it’s the differential base. Let’s test and see how it works.

CREATE DATABASE TestingBackups
GO

USE TestingBackups
GO

CREATE TABLE Testing (
ID INT IDENTITY PRIMARY KEY,
SomeValue CHAR(4)
);
GO

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Full1.bak'
GO

INSERT INTO Testing (SomeValue)
VALUES ('abc')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Diff1.bak' WITH DIFFERENTIAL
GO

INSERT INTO Testing (SomeValue)
VALUES ('def')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Full2.bak'
GO

INSERT INTO Testing (SomeValue)
VALUES ('ghi')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH DIFFERENTIAL
GO

Right, so that’s a database with two full backups and two differential backups and a couple data changes in between. Let’s drop the database and test some restore strategies. The goal is to restore the database to the point that it was at the time the second differential backup was taken.

First the obvious route. Full backup 2 and differential backup 2.

RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Full2.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups WITH RECOVERY

That works. That’s the most obvious and likely the one that will be used the most. What happens though if we don’t have full backup 2? What if it was a full backup that an evil developer took without asking, used to restore a development/test database somewhere and then deleted the backup file?

There are two options there that might work:

• Full backup 1 and differential backup 2 (differential backups are cumulative aren’t they?)
• Full backup 1, differential backup 1 and then differential backup 2.

Option 1

RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Full1.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups WITH RECOVERY

Well that didn’t work…

Msg 3136, Level 16, State 1, Line 1
This differential backup cannot be restored because the database has not been restored to the correct earlier state.
Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

Option 2

RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Full1.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Diff1.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups WITH RECOVERY

That didn’t work either.

Msg 3136, Level 16, State 1, Line 1
This differential backup cannot be restored because the database has not been restored to the correct earlier state.
Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.

Differential backup 1 restored fine, differential backup 2 didn’t. The reason is that, unlike log backups, differential backups are affected by full backups. Specifically, a differential backup is based on the last full backup that ran before it. If an ad-hoc full backup is taken, any differential run after that is based on the ad-hoc full backup and not the one run as part of the standard backup plan. Not fun if a developer runs one then deletes the backup file when it’s no longer needed.

This is what copy-only affects when specified on a full backup. A full backup run with copy-only does not change the differential base and does not reset the list of extents changed since the last full backup. Hence an ad-hoc full database backup, if specified with the COPY_ONLY option, won’t cause administrators unpleasant surprises when test restores are done or a disaster occurs and a full restore is necessary.

Let’s drop that test database and recreate it using COPY_ONLY on the second full backup.

CREATE DATABASE TestingBackups
GO

USE TestingBackups
GO

CREATE TABLE Testing (
ID INT IDENTITY PRIMARY KEY,
SomeValue CHAR(4)
);
GO

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Full1.bak'
GO

INSERT INTO Testing (SomeValue)
VALUES ('abc')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Diff1.bak' WITH DIFFERENTIAL
GO

INSERT INTO Testing (SomeValue)
VALUES ('def')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Full2.bak' WITH COPY_ONLY
GO

INSERT INTO Testing (SomeValue)
VALUES ('ghi')

BACKUP DATABASE TestingBackups TO DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH DIFFERENTIAL
GO

Now the restore path using full backup 1 and differential backup 2 does work.

RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Full1.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups FROM DISK = 'C:\Databases\Backup\TestingBackups_Diff2.bak' WITH NORECOVERY
GO
RESTORE DATABASE TestingBackups WITH RECOVERY

Processed 168 pages for database ‘TestingBackups’, file ‘TestingBackups’ on file 1.
Processed 5 pages for database ‘TestingBackups’, file ‘TestingBackups_log’ on file 1.
RESTORE DATABASE successfully processed 173 pages in 0.215 seconds (6.286 MB/sec).
Processed 64 pages for database ‘TestingBackups’, file ‘TestingBackups’ on file 1.
Processed 1 pages for database ‘TestingBackups’, file ‘TestingBackups_log’ on file 1.
RESTORE DATABASE successfully processed 65 pages in 0.103 seconds (4.873 MB/sec).
RESTORE DATABASE successfully processed 0 pages in 0.371 seconds (0.000 MB/sec).

With that out of the way, the question still remains as to whether COPY_ONLY has anything to do with log backups or the log chain. Books online also says this about them: “When used with BACKUP LOG, the COPY_ONLY option creates a copy-only log backup, which does not truncate the transaction log. The copy-only log backup has no effect on the log chain, and other log backups behave as if the copy-only backup does not exist.”

So COPY_ONLY changes the effect a backup has on the log chain only when specified on a transaction log backup, not on full databases backups. Useful if you need an ad-hoc log backup for something.

When running CheckDB on an in-use database, sometimes messages like the following will appear in the SQL error log

1 transactions rolled back in database ‘Testing’ (8). This is an informational message only. No user action is required.
DBCC CHECKDB (Testing) WITH no_infomsgs executed by theAdmin found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 31 seconds.

What’s going on here? Why is checkDB rolling transactions back?

Well, the truth is, it’s not rolling anything back.

When CheckDB runs it has to be able to get a consistent view of the database, one without half-done transactions, without data modifications in progress. On SQL 2000 (and earlier), CheckDB used the transaction log to get that consistent view. It would read through the transaction log to see what had been done to the various structures in the database. In SQL 2005 this was changed and CheckDB now uses the database snapshot feature to get its consistent, point-in-time view of the database.

Database Snapshots

A database snapshot is a read-only, point-in-time copy of a database. It uses a combination of sparse file and copy-on-write technology to reduce space usage and, as a result, it’s usually much smaller than the database it was based on.

When a snapshot is created, any open transactions are rolled back within the context of the database snapshot to generate the consistent, point-in-time copy. This generates messages in the SQL error log stating how many transactions were rolled back

CheckDB

The first thing that CheckDB has to do is to create an internal, hidden database snapshot to get the consistent view of the DB that it needs. It will do this unless it’s run with the TABLOCK option. When the snapshot is created, any open transactions will be rolled back but only in the context of the snapshot, not in the context of the actual database. CheckDB will then run against the snapshot and will drop it once complete.

So the answer to the question of why checkDB is rolling back transactions is a simple one. It isn’t.

Something that I’ve started to see recently is the idea that Backup Log … With Truncate Only, which was deprecated in SQL 2005 and gone in SQL 2008 can be simulated by backing the log up to the NUL device (backup log … to disk = ‘Nul’). Is that true? If so, is that a good idea?

To answer those questions, first we need to explore what the two statements do.

Backup Log With Truncate_Only

When this statement is run against a database in full or bulk-logged recovery, SQL discards the inactive log records for that database. From that point until a full database backup is done, the database is in pseudo-simple recovery, the inactive log records get discarded when a checkpoint runs and any attempt to back the log up results in an error message.

Msg 4214, Level 16, State 1, Line 1
BACKUP LOG cannot be performed because there is no current database backup.

So once the log has been truncated it’s exceedingly clear that the log chain is broken.

Backup Log to disk = ‘Nul’

To understand what this is doing, first we need to understand what Nul is, in the context of a file. It is not just a misspelling of NULL.

DOS, since the earliest days, has had a number of devices that can be treated like files, eg LPT1, COM1 and CON. NUL is another of those devices (virtual files, if you like) and is equivalent to the unix \dev\nul, a ‘file’ that deletes any data that’s written to it. The bitbucket, if you will.

So anything that’s written to NUL is discarded. So when a transaction log backup is made with ‘nul’ as the file destination, SQL dutifully reads over the inactive log records, formats them as for a transaction log backup and hands them off to the operating system which promptly discards the data and sends back an acknowledgement that the data has been written.

So SQL thinks that the log chain is intact. It discards the log records that were sent to Nul as it would after a normal log backup, because it thinks that they were backed up to disk. It does not switch the DB into a pseudo-simple recovery model. Inactive log records are not discarded at a checkpoint and future log backups will succeed.

That’s the main difference. Log records will still be retained after a backup to NUL. Log backups will succeed after a backup to NUL. They’re useless for restoring the DB because there’s a log backup missing, but they’ll succeed.

Replacement?

So, is backup to nul a replacement for backup with truncate_only? No, it’s not. The replacement for Backup Log with Truncate_Only is well documented. It’s switching the database to simple recovery. If the inactive log records in the log file do not need to be retained for recoverability, this is the way to tell SQL to discard them. If log backups need to be made after this, the DB can be switched back to full and a full DB backup taken.

Backup Log to Disk = ‘Nul’ is the same as backing up the log to a file and then deleting the backup file. Nothing more.

It does not. The execution stats available through sys.dm_exec_query_stats are only retained while the plan for the query is in cache. As soon as the plan is removed from cache (for whatever reason), the query stats for that query will be discarded. There are also a couple of reasons for a query’s plan not going into the cache at all.

Let’s try a quick example. I’m going to use the AdventureWorks database (SQL 2008), because it’s convenient. I’m going to create 4 procedures, run them several times, along with a couple other commands and then compare what a trace shows and what a query of the plan cache shows.

Create Procedure GetLatestPriorWeeksTransactions (@WeekEnds DATETIME)
AS

SELECT p.Name, TransactionDate, TransactionType, Quantity, ActualCost
FROM Production.TransactionHistory th inner join Production.Product p on th.ProductID = p.ProductID
WHERE TransactionDate BETWEEN @WeekEnds AND DATEADD(ww,-1,@WeekEnds)

GO

CREATE Procedure ArchiveTransactionHistory (@ArchiveEndDate DATETIME)
AS
Insert Into Production.TransactionHistoryArchive (TransactionID, ProductID, ReferenceOrderID, ReferenceOrderLineID, TransactionDate, TransactionType, Quantity, ActualCost, ModifiedDate)
SELECT TransactionID, ProductID, ReferenceOrderID, ReferenceOrderLineID, TransactionDate, TransactionType, Quantity, ActualCost, ModifiedDate
  FROM Production.TransactionHistory
  WHERE TransactionDate < @ArchiveEndDate

DELETE FROM Production.TransactionHistory WHERE TransactionDate < @ArchiveEndDate
GO

CREATE PROCEDURE SalesToday (@ProductID int)
AS
SELECT p.name, sum(sod.lineTotal) AS TotalSales, SUM(OrderQty) AS TotalSold
  FROM Sales.SalesOrderHeader soh
    INNER JOIN Sales.SalesOrderDetail sod on soh.SalesOrderID = sod.SalesOrderID
    INNER JOIN Production.Product p on sod.ProductID = p.ProductID
  WHERE DATEADD(dd,dateDiff(dd,0,soh.OrderDate),0) = DATEADD(dd,dateDiff(dd,0,GETDATE()),0)
GO

CREATE PROCEDURE PersonSearch (
  @Surname Varchar(50) = NULL,
  @FirstName Varchar(50) = NULL
)
WITH RECOMPILE
AS

SELECT Firstname, middlename, LastName
  FROM Person.Person
  WHERE (LastName like @surname + '%' OR @Surname IS NULL)
    AND (FirstName like @FirstName + '%' OR @FirstName IS NULL)
GO

I’m going to use the same trace definition that I detailed in the article that I wrote for Simple-Talk, and use the same technique as is in the article to load the trace up and get aggregated totals.

When I query the plan cache, this is what I get back from it.

And this is what I got from the trace (with the queries relating to the trace removed)

So why the discrepancy?

Let’s start with the entries returned in the query of the trace, not the procedure cache.

The ALTER PROCEDURE wouldn’t be visible in a query of the procedure cache, because DDL statements don’t have execution plans. There’s only one way to run an ALTER PROCEDURE, so there’s no need for a plan, hence there’s nothing to cache and nothing for the query stats to hook off.

According to the procedure cache, the PersonSearch procedure was never run, but the trace records 35 executions. There’s no entry for this procedure in the cache, because it’s marked WITH RECOMPILE. Since a new plan has to be compiled on each execution, the plan is never cached and hence it will never be visible in a query of the plan cache and there will never be saved execution statistics for it.

Now, how about the two with very different execution counts? In each of those cases, something happened to throw the procedure’s existing plan out of cache part way through the test. When the plan gets thrown from cache the query stats get discarded as well.

In the case of the SalesToday procedure, the procedure was altered part way through the test. Altering a procedure or any object that the procedure depends upon will invalidate the cached plan and cause it to be discarded

The other procedure, GetLatestPriorWeeksTransactions, had it’s plan invalidated due to a statistics update. The archiving of TransactionHistory changed enough rows in the table that the next time a query ran it triggered a statistics update. That statistics update would have invalidated any plans that used those statistics. In general, this doesn’t have to be stats update on a permanent table. If a procedure uses a temp table and adds rows to that temp table, there’s a good chance of getting statistics-based recompiles of statements within that procedure, thereby skewing the results of queries against the plan cache.

In conclusion, while the DMVs exposing the plan cache are very useful in seeing how queries are running, they cannot be seen as a complete replacement for SQL Trace because of the possibility of certain types of commands not getting cached at all, and the possibility of plans getting invalidated and dropped from cache for various reasons.

This is another things that I’ve seen a few things recently on various forums. A DB’s transaction log fills the disk, probably due to a lack of log backups, and then either SQL is stopped and the log file deleted or the database is detached and the log file deleted. Either way it’s not a particularly good thing to do.

No, I’ll be blunt, it’s a appallingly bad thing to do. The transaction log is not an optional piece of the database. It’s not like an installation log or activity log where the entries are there just in case anyone’s interested. The transaction log is what SQL uses to ensure, at all times, that the data in the database is transactionally consistent. It’s what gives SQL databases the consistency and durability properties that are required from a relational database engine.

The transaction log has a number of uses within SQL.

1. In SQL 2000 and before, the inserted and deleted tables were materialised from the transaction log. This was changed in SQL 2005 and they’re now materialised from the row version store in TempDB.
2. A transaction rollback uses the transaction log to determine what needs to be undone.
3. In transactional replication, the log reader uses the transaction log to determine what changes need to be replicated
4. Used by Change Data Capture in SQL 2008 to extract changes made to registered tables
5. During restart-recovery to ensure that transactions that had committed when the service stopped but whose changes had not been written to the data file are replayed and to ensure that transactions that hadn’t completed are rolled back.

For now, I’m just going to consider the last one.

When the SQL Service is stopped or a database detached, SQL will try to shut the database down cleanly. That means running a checkpoint and writing all dirty data pages to disk. The checkpoint has to write into the log that it has run. If there’s no more space in the log, then the checkpoint cannot run and the database cannot be cleanly shut down. This is not usually a problem. When the database is attached or the service restarted SQL will run restart-recovery on it, the transaction log will be used to bring the database back to a transactionally consistent state. That’s great, but what happens if the transaction log’s not there?

To see exactly what will happen, I’m going to set up a database that’s got a very small max size for the log file (because I don’t want to have to fill the drive to get results), set it to full recovery and take a backup. Taking a backup will start the log chain and ensure that the log will grow until it runs out of space. Reuse of the log will not happen as I’m not taking any log backups. Once the database is backed up, I’m going to run inserts until the log fills up.

CREATE DATABASE TestingLogDeletion
ON (NAME = Test_dat,
FILENAME = ‘D:\Develop\Databases\TestingLogDeletion.mdf’,
SIZE = 10,
MAXSIZE = UNLIMITED,
FILEGROWTH = 5 )
LOG ON ( NAME = Test_log,
FILENAME = ‘D:\Develop\Databases\TestingLogDeletion.ldf’,
SIZE = 5MB,
MAXSIZE = 10MB);
GO

ALTER DATABASE TestingLogDeletion SET RECOVERY FULL;
GO

USE TestingLogDeletion
GO

CREATE TABLE Filler (
ID INT IDENTITY PRIMARY KEY,
FillerStr CHAR(300)
)
GO

BACKUP DATABASE TestingLogDeletion TO DISK = ‘D:\Develop\Databases\Backups\TestingLogDeletion.bak’

Right, so that’s the setup all done. Now to fill the log up.

DECLARE @i INT;
SET @i = 0;

WHILE (@i < 10000)
BEGIN
INSERT INTO Filler (FillerStr) VALUES (CAST(@i AS VARCHAR(10)))
SET @i = @i+1;
END
GO

Doesn’t take long either. After just 4778 inserts, this popped up.

Msg 9002, Level 17, State 2, Line 6
The transaction log for database ‘TestingLogDeletion’ is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases

Trying to run a checkpoint gives the same error. So there’s not enough space in the log to do a clean shutdown. First I’m going to try shutting SQL down, deleting the log and then restarting SQL. See how the database handles that. Note, the highest identity value in that table was 4778 before the shutdown.

Once SQL restarts, I’m going to query that table, see what’s in there.

select MAX(ID) from TestingLogDeletion.dbo.Filler

Msg 945, Level 14, State 2, Line 1
Database ‘TestingLogDeletion’ cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

That’s not good. The state of the database, according to sys.databases, is RECOVERY_PENDING. Restart recovery has not run and cannot run because the log file’s missing. The error log says the following

Error: 17207, Severity: 16, State: 1.

FileMgr::StartLogFiles: Operating system error 2(The system cannot find the file specified.) occurred while creating or opening file ‘D:\Develop\Databases\TestingLogDeletion.ldf’. Diagnose and correct the operating system error, and retry the operation.

File activation failure. The physical file name “D:\Develop\Databases\TestingLogDeletion.ldf” may be incorrect.

The log cannot be rebuilt because the database was not cleanly shut down.

There are two ways to fix this. Restore from backup or attempt a log rebuild and repair. If there is a current database backup, the first option is by far the best one. The repair should be a last resort, for when there is no backup.

In this case, for demonstration purposes, I’m going to pretend there’s no backup and go for a repair. This technique is given in more detail on Paul Randal’s blog – Using EMERGENCY mode to access a RECOVERY PENDING or SUSPECT database

Step 1 – Set the database into Emergency mode

Fortunately in 2005 and later, using Emergency mode is documented and supported and anyone with sysadmin permissions can set a database into Emergency mode.

ALTER DATABASE TestingLogDeletion SET EMERGENCY

Now that the DB is in Emergency mode I can access it again. So what’s in that table that I was populating? Remember that the highest identity value before the shutdown and log deletion was 4778.

I’ve lost almost 3300 records. Imagine if this was a sales system, or stock market trading or something similar.

Because the DB was not shut down cleanly, all the changed data pages were not written to disk. Normally those changes would be replayed from the transaction log during the restart-recovery but because the log was deleted and restart-recovery can’t run, those changes have been lost, and there is no way to get them back.

Step 2 – Emergency mode repair

To bring the DB back online I need to run a repair. This will fix any structural problems caused by the missing log and bring the database online.

ALTER DATABASE TestingLogDeletion SET SINGLE_USER
DBCC CHECKDB(TestingLogDeletion, REPAIR_ALLOW_DATA_LOSS)

CheckDB complained about the missing log then went and rebuilt the log with a whole bunch of extra warnings

File activation failure. The physical file name “D:\Develop\Databases\TestingLogDeletion.ldf” may be incorrect.

The log cannot be rebuilt because the database was not cleanly shut down.

Warning: The log for database ‘TestingLogDeletion’ has been rebuilt. Transactional consistency has been lost. The RESTORE chain was broken, and the server no longer has context on the previous log files, so you will need to know what they were. You should run DBCC CHECKDB to validate physical consistency. The database has been put in dbo-only mode. When you are ready to make the database available for use, you will need to reset database options and delete any extra log files.

DBCC results for ‘TestingLogDeletion’.
CHECKDB found 0 allocation errors and 0 consistency errors in database ‘TestingLogDeletion’.

In this case there was no corruption. That will not always be the case. Because recovery acts on both user data and the DB’s structure there may be physical corruption that has to be repaired. It’s even possible that there’s irreparable corruption to system tables or allocation pages and if Emergency mode repair fails there’s no other way to bring the DB back online

After running repair if I check sys.databases the state of the database is Online, so I can bring it out of restricted user mode and it’s usable again.

Now that was bad enough. Lost data and a database that was unavailable for however long CheckDB takes to repair, and on big databases that could be many hours. Let’s see how bad things are when I detach the database and then delete the log file.

Same setup, but instead of shutting SQL down, I’ll detach the database.

Interesting point, if I do the detach through Object Explorer the detach dialog comes back with an error saying that detach failed because the transaction log is full. However the detach actually succeeds, the database is not longer there.

So, DB detached. Now I’m going to delete the log and use the CREATE DATABASE syntax to reattach it.

CREATE DATABASE TestingLogDeletion
ON (FILENAME = ‘D:\Develop\Databases\TestingLogDeletion.mdf’)
FOR ATTACH_REBUILD_LOG

That should work fine, right. Right?

Wrong.

File activation failure. The physical file name “D:\Develop\Databases\TestingLogDeletion.ldf” may be incorrect.
The log cannot be rebuilt because the database was not cleanly shut down.
Msg 1813, Level 16, State 2, Line 1
Could not open new database ‘TestingLogDeletion’. CREATE DATABASE is aborted.

Much the same as the error that we got when stopping SQL, deleting the log file and restarting SQL but with one very big difference. I can’t set the DB into emergency mode and run repair. The attach failed, the database is not there.

So this is a bit of a catch-22. I can’t rebuild the log because the database isn’t attached and I can’t attach because the log is missing. Nasty situation. Same as in the first case, the recommended approach here is to restore backups (and we all have backups, right?). If there are no backups, there is a way to hack the DB back into the server, but it isn’t pretty.

To get that DB back into SQL I need to create a new database with the same number of files and the same (as much as possible) size of files.

CREATE DATABASE TestingLogDeletion_2
ON (NAME = Test_dat,
FILENAME = ‘D:\Develop\Databases\TestingLogDeletion_2.mdf’,
SIZE = 10,
MAXSIZE = UNLIMITED,
FILEGROWTH = 5 )
LOG ON ( NAME = Test_log,
FILENAME = ‘D:\Develop\Databases\TestingLogDeletion_2.ldf’,
SIZE = 5MB,
MAXSIZE = 5MB);
GO

Then stop SQL, delete both of the files of the newly created database and rename the mdf file of the database I detached earlier to match the mdf file of this DB. Once that’s done, restart SQL. When SQL starts it will find that the log file is missing and the DB not shut down cleanly and it will put the DB into the RECOVERY_PENDING state. From here the steps to get the DB back online are the same as in the first case. Set it into Emergency mode and run CheckDB with a repair option. Again there will be downtime while checkDB repairs and rebuilds the log, there may be some data loss and there’s always the chance that emergency mode repair will fail.

So, in conclusion. Don’t delete the transaction log. Sometimes SQL can rebuild it without too many problems, sometimes it can’t.

1. No backups
2. No corruption checks

Backups

Having no backups is pretty inexcusable, whether its because no backups were scheduled or whether it’s because the backups were failing. Paul’s written about the importance of the right backups, and I’m not going to repeat him here.

Just having backups is not sufficient. The backups have to be restorable. A backup that’s not restorable is, in my opinion, worse than no backup at all. That means that the jobs have to be checked to ensure that they did run without error and the backups have to be restored somewhere to test them. Ultimately that’s the only way to be 100%, absolutely sure that a backup is restorable.

In SQL 2005 and higher, if the database has checksum page protection and the backup is done with the checksum option then damaged pages will be detected at backup time and damage to the backup set will be detected by RESTORE VERIFYONLY. This may not be the case if the backup was not made with the checksum option.

The other thing that has to be considered is the integrity of the backup media. I’ve seen a case where an IO problem not only damaged the database but also damaged the backup file to the point where it would not restore, despite the backup being on a different LUN of the SAN with no drives shared between it and the data LUN

So, to conclude this section, do you have backups and are you sure your database backups can be restored? Have you tried recently?

Corruption checks

If corruption checks aren’t done regularly, there’s a good chance that any corruption that occurs will remain hidden for quite some time. If this does happen, and the database backups are not made with the CHECKSUM option, then the corrupt pages will get backed up. This causes obvious problems when trying to fix corruption by restoring from a backup.

I’ve seen people say that if they have the Page Verifications et to checksum, there’s no need to run CheckDB. That is not true. Other than backups (when taken with the CHECKSUM option) there is no other operation that reads and checks all of the page checksums in the database. Just having the page checksums in place doesn’t help if there’s never a check made to see that all of them are still correct.

A good database maintenance routine must include running CheckDB regularly or, if  there isn’t time to run checkDB, a mixture of CheckAlloc, CheckCatalog and CheckTable on all tables.

How often? Often enough that, should corruption be detected, restoring from a clean backup is an option.

In Conclusion

I would consider the protection of data to be a DBA’s top priority. So backups and integrity checks should be top of the list of things to check and double check, because slipping in this area could have very nasty consequences for the business.

Do you have backups of all your critical production systems?
Are they restorable?
Are you sure?
Do you have corruption in any of your databases?
Are you sure?

A backup solution is a one that creates separate copies of data that can be used to restore the original in the case of data loss. A high availability solution is one designed to keep a system usable and available in the face of hardware (or similar) failures

Solutions like RAID, database mirroring, clustering and SAN replication are forms of high availability. They are not backup solutions and cannot replace good database backups. Having the database on a RAID array will not help if the DB becomes corrupt. SAN replication or database mirroring is of no use if a user somehow deletes critical records.

On the other hand, a backup strategy, however good, is not going to help much when a critical piece of hardware burns out and the business will be losing millions if the app’s not up within 10 minutes. If the app is critical, both backups and an appropriate high availability solution have to be considered. Picking the right high availability solution is complex. Entire books have been written on it.

The only time having no backups at all is an acceptable option is if it’s a system where the data can be completely recreated from another source with no difficulties (example here would be a reporting database that’s replicated directly from a different server)

Here’s another take on backups – http://www.sqlskills.com/BLOGS/PAUL/post/Importance-of-having-the-right-backups.aspx

On SQL 2000, the only way to know was to to go back through the history of the checkDB jobs (everyone is running checkDB regularly, right?) and see when the last successful run was.

On SQL 2005 and higher, when checkDB runs successfully, ie without finding any errors, it writes the date of the last run into the database’s boot page.  Whenever SQL Server then starts that database up, be it during a server start, a database restore or bringing the database online, SQL prints a message into the error log.

CHECKDB for database ‘Testing’ finished without errors on 2008-12-22 10:20:06.007 (local time).

This makes it very easy to see when the database was last known to be clean (without any corruption). Note that SQLL is not running checkDB when it prints that. It’s simply printing out the date that CheckDB last completed without finding any problems. The information can also be read out of the boot page using an undocumented DBCC command.

Use Testing
GO

DBCC TRACEON(3604)
DBCC DBINFO
DBCC TRACEOFF(3604)

This prints out a whole bunch of information, all found in the database’s boot page. The info of interest here is in the second block (on SQL 2008 RTM)

dbi_dbccLastKnownGood = 2008-12-22 10:20:06.007

If the date listed is 1900-01-01 00:00:00, it means that CheckDB has never run successfully on that database. If that is the case, there will be no entries in the error log when the database is started.

Anyone who’s done any research at all on database corruption and IO problems has probably seen an example of the 823 and 824 errors.

Error 823 is a physical IO error. It means that, for some reason, the OS was incapable of completing the requested IO. It may mean that the disk has failed, the disk is missing, a filter driver is misbehaving, or a number of other things. On SQL 2005 it looks something like this:

Error: 823, Severity: 24, State: 2.
The operating system returned error 21(The device is not ready.) to SQL Server during a read at offset 0x0000d2364e2000 in file ‘D:\Data\SomeDB.mdf’. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

Error 823 is classified as a severity 24, the second highest severity error that SQL has. As the error message says, it’s a very serious condition.

Error 824 is a logical IO error. It means that the page requested was read from disk successfully, but there was something wrong with the page. The page header may have invalid values in it, the checksum may be incorrect, the torn page bits may be incorrect, etc. On SQL 2005, it looks something like this:

Msg 824, Level 24, State 2, Line 1.
SQL Server detected a logical consistency-based I/O error: incorrect checksum (expected: 0x7532c420; actual: 0x320e4240). It occurred during a read of page (1:342) in database ID 15 at offset 0x00000020e24000 in file ‘D:\Data\SomeDB.mdf’. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

Error 824 is also classified a severity 24, which is appropriate, seeing as it’s saying that some pages within the data file are absolute garbage.

So far, all well and good. They’re severe errors with very high severities and will be picked up by agent alerts. But there’s another IO error that’s less well known and harder to notice, though in my opinion, no less severe.

Error 825 is called a read-retry error. What it means is that an IO was requested and either the IO failed or the resulting page was corrupt. SQL then tried a couple more times and on either the 2nd, 3rd or 4th try, the IO succeeded.

The error looks something like this:

Msg 825, Level 10, State 2, Line 1.
A read of the file ‘D:\Data\SomeDB.mdf’ at offset 0x00000020e24000 succeeded after failing 1 time(s) with error: incorrect checksum (expected: 0x7532c420; actual: 0x320e4240). Additional messages in the SQL Server error log and system event log may provide more detail. This error condition threatens database integrity and must be corrected. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

The big thing to note about this is that it’s a severity 10. An informational message. That means that agent alerts set for the higher severity errors will completely miss this, and intermittent IO errors is perhaps not something that DBAs want to be ignorant of. The read may have eventually succeeded this time, but there’s no guarantee that it will eventually succeed the next time it’s tried.