Previously I took a look at the CacheHit and CacheMiss events to see how they behave and how to identify what’s been searched for in the cache. in this follow up, I want to take a similar look at the CacheInsert and CacheRemove events, see when they fire and how to identify the objects that they relate to.

Again, a word of caution, these can be frequently occurring events on busy servers and so traces should be kept short and to a minimum of events and columns. That said, these should occur a lot less often than the CacheHit and CacheMiss events. If they are occurring very frequently it may indicate that the SQL Server is not reusing plans efficiently.

CacheInsert

The CacheInsert event fires after a CacheMiss. The search for a matching plan in the cache failed, firing a CacheMiss event. Since there’s no plan, the optimiser is invoked to generate one and then that plan is inserted into the plan cache before the Query Execution engine begins execution.

The event is fairly simple, though of course there are a few surprises (what in SQL doesn’t have?).

Exec FireCacheEvents
GO

SELECT ID, SomeDate, Status
FROM TestingCacheEvents
WHERE Status =  'C'

Three CacheInsert events for two batches. The first is simple enough, it’s the insert of the plan for the stored procedure. The procedure name is in the TextData column and the procedure’s ID is in the ObjectID column. The ObjectName column is not populated for this event.

The second and third Cacheinsert events are the interesting ones. The second one shows a parameterised version of the ad-hoc SQL statement, while the third shows an unparameterised version of the same ad-hoc SQL statement. Clearly this query was simple enough to qualify for auto-parameterisation. So, are there two plans for this been inserted into cache? To answer that one, I need to switch over to SSMS and query the plan cache.

select usecounts, size_in_bytes, cacheobjtype, objtype, st.text,  qp.query_plan
 from sys.dm_exec_cached_plans cp
 cross apply  sys.dm_exec_sql_text(cp.plan_handle) st
 cross apply  sys.dm_exec_query_plan(cp.plan_handle) qp
 where st.text not like  '%sys.dm_exec%'

Well, there are three entries and the Cache Object Type is listed as compiled plan for all three, but there’s something different between them. Take a look at the size in bytes for the ad-hoc and prepared statements. The ad-hoc is less than half the size of the prepared. That’s not the only thing different. If I open up the execution plans, the exec plan for the prepared statement looks normal. There’s a clustered index scan (there are no nonclustered indexes on this table yet) and a select operator. The plan for the adhoc statement however…

There’s something missing here, like the rest of the plan. Ok, so the entry with the ad-hoc type is not a complete plan, so what is it?

There doesn’t seem to be anything on MSDN on this (at least nothing I could find), but two of Kalen’s books¹. (Inside Microsoft SQL Server 2005: Query Tuning and Optimisation, page 283; Microsoft SQL Server 2008 Internals, page 533) mention this. These ad-hoc entries are shell queries, cached just to make the parameterised form of the query easier to find. All the execution plan contains for these shell queries is a pointer to the plan for the parameterised version of the query.

For the ad-hoc query that gets inserted as a shell plan and the parameterised query, the objectID is, as with the CacheHit and CacheMiss events, a hash of the query text and does not match to any object in the database. The full text of the batch is then given in the TextData column.

I think that’s enough for CacheInsert. On to it’s opposite.

CacheRemove

The CacheRemove event fires, as one would expect, when a plan is removed from cache. The interesting question is, when does that occur?

Is it every event that could possibly result in an inefficient or invalid plan, or just memory pressure and explicit cache flushes that trigger the CacheRemove event?

Let’s see…

As an aside, I’m going to add a nonclustered index to the table that I’m using, so that the update of statistics will have an effect. Otherwise the only execution path possible is a table scan, and changing statistics won’t affect that.

CREATE NONCLUSTERED INDEX idx_TestingCacheEvents_Status
 ON  TestingCacheEvents (Status)
GO

Right, onto the testing.

Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO
DBCC FREEPROCCACHE -- Gone.  All gone
WAITFOR DELAY '00:00:05'
GO

Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO
DBCC  FREEPROCCACHE(0x0500060063A9355540614285000000000000000000000000) -- Just one removed, the proc
WAITFOR DELAY '00:00:05'
GO

Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO
exec sp_recompile  'TestingCacheEvents' -- recompile on the table, both proc and ah-hoc plans  invalidated.
WAITFOR DELAY '00:00:05'
GO

Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO
UPDATE TOP(1)  TestingCacheEvents
 SET Status = 'C'
 WHERE Status = 'B'
UPDATE STATISTICS TestingCacheEvents WITH FULLSCAN
WAITFOR DELAY  '00:00:05'
GO

Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO
ALTER TABLE  TestingCacheEvents ALTER COLUMN FILLER CHAR(325)
WAITFOR DELAY '00:00:05'
GO
Exec FireCacheEvents
GO
SELECT ID, SomeDate, Status
FROM  TestingCacheEvents
WHERE Status = 'C'
GO

Well, this is interesting…

The only things that caused the CacheRemove event to fire were the two DBCC FreeProcCache, the first that, as can be seen from the EventSubClass, cleared the entire plan cache, and the second that cleared a single plan.

If I go back and look at the whitepaper Plan Caching in SQL Server 2008, it differentiates between removing a plan from cache (which memory pressure and cache flushes do) and invalidating the plan (which recompiles, schema changes, stats changes and the like do). Based on that whitepaper and these results, I would conclude that the CacheRemove event only fires when the plan is actually removed from cache, not when it’s just invalidated. The invalidation of a plan simply results in a recompile next time the plan is needed.

Now that the question of when it appears is answered, let’s finish with what it looks like.

There are two possible formats for this event, and it’s the EventSubClass that shows which of the two the particular event is.

The first possibility is when all of the plan cache has been flushed, identified by a EventSubClass of “2 – ProcCacheFlush”. This will be a result of a DBCC FREEPROCCACHE and some server reconfiguration events. In this case, there will be no ObjectID and the text data will simply state that the cache has been flushed.

The second possibility is more interesting, especially if monitoring plans being aged out of memory. An EventSubClass of “1 – Compplan Remove” identifies a single plan being removed from cache. These are fired when either a single plan or part of the plan cache is cleared. Examples here are DBCC FREEPROCCACHE with a plan or SQL handle passed to it, DBCC FLUSHPROCINDB, database reconfigurations (restore, detach, offline)

With the one, the ObjectID does have a value and it’s the ObjectID of the pbject whose plan is being thrown out of cache. As with all the other cache events, for ad-hoc queries the ObjectID is just a hash of the query text and only really useful for matching to CacheMiss, CacheHit and CacheInsert events and the object name or ad-hoc batch appears in the TextData

That’s more than I planned to write, but CacheRemove turned out to be a little more complex than I expected. Next up, the last part of this sort series – the recompile events.

For people interested in the details of how SQL is using and reusing execution plans, there are some useful events in profiler for watching this in detail, under the Stored procedure group:

• SP:CacheMiss
• SP:CacheInsert
• SP:CacheHit
• SP:CacheRemove
• SP:Recompile
• SP:StmtRecompile

Additionally there’s the SQL:StmtRecompile event under the TSQL group.

For now, I just want to look briefly at the CacheMiss and CacheHit events.

One word of caution early on, these are frequently occurring events and it may not be a good idea to trace these on busy production servers. If you do need to, keep the duration of the trace short and the columns to a minimum.

CacheMiss

The cache miss event fires any time SQL looks for the execution plans for an object or ad-hoc batch and does not find it in the plan cache.

For an object (scalar function, multi-statement table-valued function, stored procedure or trigger) the match is done on the object ID (along with some of the connection’s SET options and possibly the database user and c couple other factors¹). For an ad-hoc batch, the match is done on a hash of the text of the batch (along with some of the connection’s SET options and possibly the database user)

When testing stored procedures from Management Studio (or another SQL querying tool), two CacheMiss events will appear in the trace.

What’s going on here?

Let’s start from the bottom and work up. The last event there, the SP:Completed records the completion of the stored procedure and lists both the ObjectID and ObjectName and, if I check in the database, ObjectID 1301579675 does indeed belong to the stored procedure FireCacheEvents.

The second event (the second cache miss) has the same ObjectID and ObjectName. So this is the failed cache lookup for the stored procedure (failed because this is the first time I ran it and the plan was, as a result, not in cache).

The first entry is the one that’s curious. The ObjectName is not populated and the ObjectID doesn’t match anything in the database. So what is the cache lookup trying to find?

The TextData column give a hint. What it’s trying to find is a cached plan for the ad-hoc batch containing the whole of the text submitted to SQL (In this case just ‘EXEC FireCacheEvents’). This too could contain queries (though it doesn’t in this case) and needs a plan lookup. With just an EXEC in the batch, it won’t have a plan and hence won’t be found in cache, but there’s still a lookup for it.

If there was any SELECT, INSERT, UPDATE or DELETE (or MERGE on SQL 2008) statement within that ad-hoc batch, the batch would also be cached and future cache lookups would succeed but, since it’s just an EXEC, there’s no plan to cache.

This additional cache lookup won’t occur if the execution is via RPC (e.g. A .Net call with the SQLCommand.CommandType = CommandType.StoredProcedure) but it will any time there is an ad-hoc SQL batch submitted (e.g. from Management Studio or .Net if the SQLCommand.CommandType is set to CommandType.Text)

That’s ad-hoc batches and stored procs. Let’s try the third possibility – a parameterised query.

This looks semi-familiar. The first CacheMiss is again for the entire of the ad-hoc batch (consisting of the DECLARE and the EXEC in this case). The second looks similar, something in TextData and no ObjectName. This is the parameterised query, identifiable as parameterised by the (@Status Char(1)) parameter definition right at the beginning. Again here the ObjectID is a hash of the text and doesn’t match to a real object in the database.

It is worth noting that, if both the CacheMiss and CacheInsert events are traced, the CacheMiss events will only appear if there is no subsequent CacheInsert for the same ObjectID. If I ran the exact same script as above, but had the SP:Completed, SP:CacheMiss and SP:CacheInsert events in the trace, there would still be only three events recorded, the cache miss for the ad-hoc batch (as that’s not cached there’s no matching CacheInsert event), the CacheInsert for the stored proc and then the SP:Completed. The CacheMiss for the procedure wouldn’t appear, though it’s presence can be intuited from the presence of the CacheInsert.

That should about cover it for the CacheMiss. If the failed cache lookup is for is an ad-hoc batch or parameterised query, the TextData column will be populated with the contents of the batch or query and the ObjectID will be a hash of the text (and shouldn’t match any object in the database). If the failed cache lookup is for a procedure (or function or trigger), the object name column is usually populated (not always) with the name of the object, TextData is blank and the ObjectID matches the ObjectID of the object in the database

CacheHit

Onto the CacheHit event. This, as its name implies, is the opposite to the CacheMiss. The CacheMiss indicates that a lookup to the plan cache failed to find a matching plan. The CacheHit indicates that a lookup to the plan cache succeeded in finding a matching plan (based on object id, set options, maybe user, and various other conditions).

It’s not certain, even if the cache lookup succeeds, that the plan will indeed be used for the execution of the query/batch/procedure as there are a number of stability and optimality related checks that will be done before the plan is used.

So let’s see how this event looks.

EXEC FireCacheEvents
GO

SELECT ID, SomeDate, Status
FROM dbo.TestingCacheEvents
WHERE Status = 'B'
GO 

Two ad-hoc batches, first with just a stored procedure call, second with just an ad-hoc SQL statement.

No big surprises here, not after looking at the CacheMiss events. The first is the CacheMiss for the first of the ad-hoc batches, the one with just the EXEC in it. Since that is not cached, there will be a CacheMiss every time that executes.

The first of the CacheHit events is for the stored procedure. As with the CacheMiss, this CacheHit for the stored procedure has an ObjectID that matches the ObjectID for that procedure in the database, and the ObjectName column is populated with (surprise) the object name while the TextData is blank.

The second CacheHit is for the ad-hoc batch with the SELECT statement. As is probably expected by this point, the ObjectID there is just a hash of the text, the ObjectName is blank and the TextData is populated with the full text of the batch.

In Conclusion

If you’re monitoring cache usage with the CacheMiss and CacheHit events, there are two different ways to identify what the lookup was looking for.

If the lookup was for the plan of a stored procedure, trigger or function, the ObjectID column contains a value that matches the ObjectID for that object in the database. The TextData column is blank and the ObjectName column is (usually) populated with the name of the object. I did encounter a couple of cases where the ObjectName was blank for a CacheMiss event for a stored procedure, not quite sure why. More investigation is necessary.

If the lookup was for the plan of an ad-hoc batch or parameterised query, the ObjectID contains a meaningless value, the ObjectName column is blank and the TextData contains the entire of the batch/query.

It may also be worth mentioning that the DatabaseID column is populated for all CacheMiss and CacheHit events, regardless of what the lookup is looking for. Additionally the DatabaseName column is populated for all CacheHit events (but is not an available column for the CacheMiss)

(1) For anyone who wants more information, there are two excellent resources available from Microsoft:

• Batch Compilation, Recompilation, and Plan Caching Issues in SQL Server 2005
• Plan Caching in SQL Server 2008

Both go extremely deep into caching, what influences matching of plans, plan reuse, recompilations and the plan cache itself.

Reproduction code:

CREATE TABLE TestingCacheEvents (
 ID INT IDENTITY PRIMARY KEY,
 SomeDate DATETIME,
 Status CHAR(1),
 Filler CHAR(300) DEFAULT ' '
);
GO

INSERT INTO TestingCacheEvents (SomeDate, Status)
SELECT TOP (10000)
 DATEADD(dd,FLOOR(RAND(a.object_id+b.column_id*5000)*500),'2000/01/01'),
 CHAR(65+FLOOR(RAND(b.object_id+a.column_id*5000)*10))
 FROM master.sys.columns a CROSS JOIN master.sys.columns b;
GO

CREATE PROCEDURE FireCacheEvents
AS
 SELECT ID, SomeDate, Status
 FROM TestingCacheEvents
 WHERE Status = 'G'

GO

I keep seeing this come up on various forums when people ask what makes a database go suspect or what could have caused their database to be marked suspect. I can almost guarantee when that question comes up, one or more people will answer ‘missing files’. That may have been true on earlier versions (I don’t have a SQL 2000 or earlier instance to play with), but it is not true in the current versions of SQL. A missing file may result in the database being inaccessible (depending what file), but it will not result in a suspect database.

Let’s prove it.

I’m going to create a database with three files, two of which are in the primary filegroup, with one user table on each filegroup. (T-SQL code at the end) Before each test I’ll begin a transaction, modify data in both tables, shut SQL down so that there’s an uncommitted transaction in the log (database cannot be cleanly shut down), then rename a file before restarting SQL and see what happens.

I’m not going to play with the transaction log. That I’ve done before. In SQL 2005/2008, if the transaction log is missing and the database was cleanly shut down, SQL will recreate it. If the the transaction log is missing and the database was not cleanly shut down, the database goes into the RECOVERY_PENDING state, so no SUSPECT here.

Let’s try the file in the secondary filegroup first.

BEGIN TRANSACTION
  UPDATE dbo.Table1 SET SomeColumn = 'aaa'
  UPDATE dbo.Table2 SET SomeColumn = 'aaa'

Shut SQL down (SHUTDOWN WITH NOWAIT), then off to explorer and rename ‘Secondary.ndf’ to ‘Secondary.missing.ndf’ and restart the SQL instance.

It’s definitely not online, the plus sign is missing.

According to sys.databases (the state_desc column), the database state is ‘RECOVERY_PENDING’.

Due to the uncommitted transaction that was running at the time SQL was shut down, SQL needs to run restart recovery on this database before bringing it online. It cannot do that because of the missing file. Hence the database state is recovery pending, recovery has not started. A look at the SQL error logs gives a clear reason why.

Unable to open the physical file “D:\Develop\Databases\Secondary.ndf”. Operating system error 2: “2(failed to retrieve text for this error. Reason: 15105)”.

Operating system error 2 is “File not found”

The important thing to note about the RECOVERY_PENDING state is that it is not fatal. If I fix the underlying reason, (in this case the renamed file) I can restart the database and it will come online. To do that I’m simply going to take it offline, rename the file back to what it should be, then bring the DB back online.

ALTER DATABASE TestingSuspect SET OFFLINE

Rename the secondary data file back to what it should be.

ALTER DATABASE TestingSuspect SET ONLINE

Back online and fully usable.

Next up, the second file in the primary filegroup. Again, begin transaction, do updates, stop SQL, rename ‘Primary2.ndf’ to ‘Primary2.missing.ndf’, restart SQL.

Again the database is inaccessible and again the state is RECOVERY_PENDING. Again taking the database offline, fixing the filename and bringing it online makes it fully available with no problems.

The final test, the primary file itself, ‘Primary.mdf’. Same process as before. Just as in the earlier two cases, the database comes up in the RECOVERY_PENDING state, not SUSPECT and the same process as before allows the DB to be brought online without any problems.

So in conclusion…

For a database to go suspect after a SQL startup, one or of the database files must be actually damaged. Just having the files missing will not result in the database being marked suspect. Missing or inaccessible files results in the database being marked recovery_pending only

Database code:

CREATE DATABASE [TestingSuspect]
ON PRIMARY
 (NAME = N'Primary',  FILENAME = N'D:\Develop\Databases\Primary.mdf'),
 (NAME = N'Primary2',  FILENAME = N'D:\Develop\Databases\Primary2.ndf'),
FILEGROUP [Secondary]
 (NAME = N'Secondary', FILENAME = N'D:\Develop\Databases\Secondary.ndf')
LOG ON
 (NAME = N'TestingSuspect_log', FILENAME =  N'D:\Develop\Databases\TestingSuspect_log.ldf')
GO
USE [TestingSuspect]
GO

CREATE TABLE Table1 (
 ID INT IDENTITY,
 SomeColumn CHAR(200)  -- filler
)
ON [Primary]

CREATE TABLE Table2 (
 ID INT IDENTITY,
 SomeColumn CHAR(200)  -- filler
)
ON [Secondary]

INSERT INTO Table1 (SomeColumn)
SELECT TOP 50000 ' '
 FROM  master.sys.columns a CROSS JOIN master.sys.columns b
INSERT INTO Table2  (SomeColumn)
SELECT TOP 50000 ' '
 FROM master.sys.columns a CROSS  JOIN master.sys.columns b

On the second day of DBA hell, the server gave to me
Two databases with widespread corruption, no backups (1, 2)

On the third day of DBA hell, the server gave to me
Three suspect databases, no backups (1,2,3)

On the fourth day of DBA hell, the server gave me nothing, cause I didn’t have a job any longer…

How does one end up with a critical production database that has no backups? I could kinda understand if the backups were damaged, if the corruption went undetected for long enough that it was in the backups as well, but to have no backups at all? Of an important database?

The only excuse for having no backups is if the database can be trivially and completely recreated from another source with minimal impact to the users. This is not the normal scenario.

There’s an immense amount of information available on backup and restore strategies.

• “Introduction to backup and restore strategies” on MSDN
• Paul Randal’s Technet article “Understanding SQL Server backups“
• Paul Randal’s presentation to the SQLPass DBA Virtual Chapter “Building the Right Backup Strategy” (not available for download at this point, check back in a few days)

That’s just a quick list, there’s far more information available than that. Enough that there’s really no good excuse to not have backups when they’re needed.

As Steve Jones (blog|twitter) is fond of saying “Good backup, good resume. You only need one”

Be Explicit and detailed

This is not a competition won by luck. There are no dice rolled, no coins tossed, no numbers drawn from a hat.

As an entrant, you need to convince the judges that you (or the person you are nominating) are the best of the best. The only thing that you can use to do that are the answers on the entry form.

The more the better (within reason). To give an idea, last year the answers to one question (What’s the hallmarks of an exceptional DBA?) ranged from one word to half a page. Which of those two do you think the judges rated higher?

If you can, get a colleague to read over your answers before submitting them. Ask them for their opinion, ask them if there are any pieces that they’d change or add to show you (or the person you are nominating) in their very best light.

Spell check

Please, please, please run a spell check and grammar check over your entries before submitting. This goes double if English (or American) is not your first language. There is nothing that makes an entry look bad more than por speeling thet teh jugdes mast spand tyme desifering.

No, not all of us speak English fluently, but there are enough grammar and spell checkers available (hint Firefox includes one if you download the dictionary) that not bothering shows a lack of interest and professionalism. Besides, if the judges can’t work out what you’re saying, they’re not going to rate your entry highly.

On this point, watch the l33t speak and SMS/twitter style word-shortening. They’re harder to read that fully written out words, and space is not at a premium for these entries. Again, you should be trying to show that you are a professional, much like you would when writing up a CV.

Watch the humour

What’s funny for one person may be annoying or offensive to another. A joke about ‘cleaning up after those incompetent developers’ may not be funny to a judge who is a developer or comes from a development background. Again, keep it professional, imagine that these answers are going to be seen by the CIO/owner/MD of the company you work for.

Along the same lines, funny answers aren’t. One entry last year, for the question “Why do you deserve to win?”, gave as an answer “No idea ”

Well dude, if you don’t know why you should win, I sure as hell don’t.

In Conclusion

If you’re planning to enter this competition, you have one chance to make an impression with the judges – your answers. Make it the best impression that you can.

Previous parts of this series can be found:

• Exists vs In
• In vs Inner Join
• Not Exists vs Not In
• Not Exists vs Left Outer Join … Is Null

In this roundup post, I’m going to do multiple tests on the 6 query forms, with different numbers of rows, indexes, no indexes and, for the negative forms (NOT IN, NOT EXISTS), nullable and non-nullable join columns.

In the individual tests, I used 250000 rows in the first table and around 3000 rows in the secondary table. In this roundup, I’m going to use 3 different row counts, 1000000 rows, 50000 rows and 2500 rows. That should give a reasonable idea for performance at various table sizes. (Not much point in going smaller than 2500 rows. Everything’s fast on 100 rows)

Some notes on the tests.

• The version of SQL is SQL Server 2008 SP1 x64 Developer Edition.
• The tests were run on a laptop. Core-2 Duo, 3 GB memory. SQL limited to 1 processor, so no parallelism possible.
• Each query will be run 10 times, reads, cpu and duration measured by profiler and averaged.
• Each query will be run once before the tests start to ensure that the data is in cache and the execution plans are generated and cached.
• Reproduction scripts will be available for download.

Exists vs. In vs. Inner Join

First, no indexes on the join columns

Now with indexes on the join columns

Not Exists vs. Not In vs. Left Outer Join … Is Null

First test with the columns join columns nullable, no indexes

Then with join columns nullable with indexes

Now, let’s make the join columns not nullable. Again, no indexes to start with.

and finally, join columns not nullable, with indexes

These results seem to pretty much confirm the earlier conclusions.

Exists and IN perform much the same, whether there are indexes on the join column or not. When there are indexes on the join columns, the INNER JOIN is slightly (very slightly) slower, which is more noticeable on the large tables, much less on the medium or small ones. (Note I’m mostly looking at CPU time, as the duration is also affected by sending of results to client, in this case, lots and lots of results)

When it comes to NOT In and NOT Exists they perform much the same when the columns involved are not nullable. If the columns are nullable, Not In is significantly slower because it has a different behaviour when nulls are present.

The join is slightly slower than Not Exists (or Not In on non-nullable columns), again only noticeable on the large table, probably because the optimiser has to do a full join with a secondary filter rather than the anti-semi join that it can use for Not Exists and Not In.

My conclusion from earlier posts stands. If all you are doing is looking for matching or non-matching rows and you don’t need any columns from the second table, use IN or Exists (or their negations), as appropriate for the situation. Only when you need columns from the second table should Join be used.

I think (and hope) that this adequately concludes the discussion on the Exists and In and joins, both behaviour and performance.

Reproduction scripts

So what’s the solution? Well, throwing hardware at the problem is an old favourite. There are few workloads that a nice 256-processor Itanium with a terabyte or two of memory won’t handle, but servers like that are a little on the expensive side and lots of money spent on expensive hardware means less that can be spent on annual bonuses.

There is another option, a hidden, undocumented option that can improve query performance, maybe a little, maybe substantially.

First thing that you need to do to get this one is to enable the hidden options in sp_configure. This is done much the same way as the advanced options.

exec sp_configure 'show hidden options', 1
RECONFIGURE WITH EXTREME OVERRIDE

Once that’s done, the undocumented option can be enabled.

exec sp_configure 'run queries faster', 101010
 RECONFIGURE WITH EXTREME OVERRIDE

How much improvement this will give depends on the kind of queries being run. OLTP systems usually see a greater improvement than decision-support, unless there’s full text search or spatial queries, in which case there will likely be substantially less of a gain.

Now, there are a few things to consider.

1. This is obviously undocumented and that means unsupported.
2. It may not work on the next version of SQL.
3. If you call support, disable the option first and don’t tell them you were running it!

.

Happy April Fools’ day.

.

..

…

….

Seriously now, there’s no options that, when enabled, makes SQL run queries faster. There is no silver bullet for performance problems, there is no one-size-fits-all fix.

Fixing performance problems involves finding the current bottleneck and removing it, then repeating that operation until performance is acceptable. It’s a complex area and there’s a lot to it. Simply throwing hardware at the problem may not produce much, if any, performance gain, especially if the hardware wasn’t the bottleneck.

If you have a query performance problem and don’t know where to start, ask on one of the SQL forums (like SQLServerCentral) if it’s not an urgent problem. If it is, or if there are serious problems, consider getting a consultant in to help out. One of the quickest ways to learn is to learn from someone who knows what they are doing.

For previous parts, see

• In vs Exists
• In vs Inner Join
• Not in vs Not Exists

I’m looking at NOT EXISTS and LEFT OUTER JOIN, as opposed to NOT IN and LEFT OUTER JOIN, because, as shown in the previous part of this series, NOT IN behaves badly in the presence of NULLs. Specifically, if there are any NULLs in the result set, NOT IN returns 0 matches.

The LEFT OUTER JOIN, like the NOT EXISTS can handle NULLs in the second result set without automatically returning no matches. It behaves the same regardless of whether the join columns are nullable or not. Seeing as NULL does not equal anything, any rows in the second result set that have NULL for the join column are eliminated by the join and have no further effect on the query.

It is important, when using the LEFT OUTER JOIN … IS NULL, to carefully pick the column used for the IS NULL check. It should either be a non-nullable column (the primary key is a somewhat classical choice) or the join column (as nulls in that will be eliminated by the join)

Onto the tests

The usual test tables…

CREATE TABLE BigTable (
id INT IDENTITY PRIMARY KEY,
SomeColumn char(4) NOT NULL,
Filler CHAR(100)
)

CREATE TABLE SmallerTable (
id INT IDENTITY PRIMARY KEY,
LookupColumn char(4) NOT NULL,
SomeArbDate Datetime default getdate()
)

INSERT INTO BigTable (SomeColumn)
SELECT top 250000
char(65+FLOOR(RAND(a.column_id *5645 + b.object_id)*10)) + char(65+FLOOR(RAND(b.column_id *3784 + b.object_id)*12)) +
char(65+FLOOR(RAND(b.column_id *6841 + a.object_id)*12)) + char(65+FLOOR(RAND(a.column_id *7544 + b.object_id)*8))
from master.sys.columns a cross join master.sys.columns b

INSERT INTO SmallerTable (LookupColumn)
SELECT DISTINCT SomeColumn
FROM BigTable TABLESAMPLE (25 PERCENT)
-- (3918 row(s) affected)

First without indexes

-- Query 1
SELECT BigTable.ID, SomeColumn
	FROM BigTable LEFT OUTER JOIN SmallerTable ON BigTable.SomeColumn = SmallerTable.LookupColumn
	WHERE LookupColumn IS NULL

-- Query 2
SELECT ID, SomeColumn FROM BigTable
WHERE NOT EXISTS (SELECT LookupColumn FROM SmallerTable WHERE SmallerTable.LookupColumn = BigTable.SomeColumn)

Let’s take a look at the execution plans

The plans are almost the same. There’s an extra filter in the JOIN and the logical join types are different. Why the different joins?

If we look at the execution plan for the NOT EXISTS, the join type is Right Anti-Semi join (a bit of a mouthful). This is a special join type used by the NOT EXISTS and NOT IN and it’s the opposite of the semi-join that I discussed back when I looked at IN and INNER JOIN

An anti-semi join is a partial join. It does not actually join rows in from the second table, it simply checks for, in this case, the absence of matches. That’s why it’s an anti-semi join. A semi-join checks for matches, an anti-semi join does the opposite and checks for the absence of matches.

The extra filter in the LEFT OUTER JOIN query is because the join in that execution plan is a complete right join, i.e. it’s returned matching rows (and possibly duplicates) from the second table. The filter operator is doing the IS NULL filter.

That’s the major difference between these two. When using the LEFT OUTER JOIN … IS NULL technique, SQL can’t tell that you’re only doing a check for nonexistance. Optimiser’s not smart enough (yet). Hence it does the complete join and then filters. The NOT EXISTS filters as part of the join.

Technical discussion done, now how did they actually perform?

– Query 1: LEFT OUTER JOIN
Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 15, physical reads 0.

SQL Server Execution Times:
CPU time = 157 ms,  elapsed time = 486 ms.

– Query 2: NOT EXISTS
Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 15, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 358 ms.

Can’t make a big deal out of that.

Now, index on the join columns

CREATE INDEX idx_BigTable_SomeColumn
ON BigTable (SomeColumn)

CREATE INDEX idx_SmallerTable_LookupColumn
ON SmallerTable (LookupColumn)

and the same queries

With indexes added, the execution plans are even more different. The LEFT OUTER JOIN is still doing the complete outer join with a filter afterwards. It’s interesting to note that it’s still a hash join, even though both inputs are sorted in the order of the join keys.

The Not Exists now has a stream aggregate (because duplicate values are irrelevant for an EXISTS/NOT EXISTS) and an anti-semi join. The join here is no longer hash, it’s now a merge join.

This echoes what I found when looking at IN vs Inner join. When the columns were indexed, the inner join still went for a hash join but the IN changed to a merge join. At the time, I thought it to be a fluke, I’m not so sure any longer. More tests on this are required…

The costing of the plans indicates that the optimiser believes that the LEFT OUTER JOIN form is more expensive. Do the execution stats carry the same conclusion?

– Query 1: LEFT OUTER JOIN
Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 172 ms,  elapsed time = 686 ms.

– Query 2: NOT EXISTS
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 78 ms,  elapsed time = 388 ms.

Well, yes, they do.

The reads (ignoring the existence of the worktable for the hash join) are the same. That’s to be expected, both queries executed with a single scan of each index.

The CPU time figures are not. The CPU time of the LEFT OUTER JOIN form is almost twice that of the NOT EXISTS.

In conclusion…

If you need to find rows that don’t have a match in a second table, and the columns are nullable, use NOT EXISTS. If you need to find rows that don’t have a match in a second table, and the columns are not nullable, use NOT EXISTS or NOT IN.

The LEFT OUTER JOIN … IS NULL method is slower when the columns are indexed and it’s perhaps not as clear what’s happening. It’s reasonably clear what a NOT EXISTS predicate does, with LEFT OUTER JOIN it’s not immediately clear that it’s a check for non-matching rows, especially if there are several where clause predicates.

I think that’s about that for this series. I’m going to do one more post summarising all the findings, probably in a week or two.

There have been a fair few forums questions in recent months asking for help in removing index scans, loop joins, sorts or other, presumed, slow query operators. There have been just as many asking how to replace a subquery with a join or a join with a subquery or similar aspects of a query usually for performance reasons.

The first question that I have to ask when looking at requests like that is “Why?”

Why is removing a particular query operator the goal? Why is changing a where clause predicate the goal? If it’s to make the query faster, has the query been examined and has it been confirmed that query operator or predicate really is the problem?

The title of this post refers to a comment I’ve seen again and again in blogs or articles about front-end development. “Premature optimisation is the root of all evils.” It’s true in the database field as well.

While optimisation is very important in database development, trying to optimise queries without any idea where the problem with the query is, or even if the query is a problem at all is about as effective in fixing a database performance problem as using a shotgun from 100 meters is in killing mosquitoes. If you hit the problem, it’s by shear luck and nothing else.

There’s two sides to this problem.

The first aspect of this is, during development, spending time on optimising a query (or stored procedure) without any idea whether or not the query is inefficient and no idea whether or not the changes made make any improvement or not.

Firstly this is a waste of time that could be better spent developing other queries. Second it creates an incorrect impression that the queries have been optimised when in fact nothing of the sort has been done.

The second aspect when, with a production database that is performing badly, queries are modified almost at random in an attempt to fix the performance problem quickly.

This almost never works. It wastes time fixing stuff that very likely isn’t broken in the first place all the while the database performance deteriorates and management curses SQL Server as ‘nonscalable’

So, what is the right approach for the above two scenarios?

1. Don’t optimise queries without knowing if they need it.
2. Don’t optimise queries without knowing if they need it. ¹

New development

When writing queries and stored procedures they need to be tested against a representative data set on a server with representative workload and their performance characteristics evaluated to see if they are acceptable. If the query’s performance characteristics are acceptable, then that query requires no optimisation²

This doesn’t mean write bad code and push it to production. It means write good, solid code, following accepted coding standards, ensure that it runs acceptably against production-volumes of data, and do not spend hours or days trying to get it running a couple of milliseconds faster.

And if the query doesn’t perform acceptable, identify the problematic portion and fix that, don’t flail around rewriting bits of the query in the hope that the problem will magically go away.

The execution plan is the primary tool here, along with the output of Statistics IO.

Fixing existing code

When evaluating existing databases with know performance problems, limit the performance tuning to queries that really are performing badly and need optimisation. It’s often true that fixing the top 5-10 worst performing queries will have massive effects in overall system performance, far more than tuning twice that number of queries that aren’t really a problem.

The best tool for finding which queries really are the worst offenders is SQL Trace.

When looking at queries that are a problem, identify the portions that are inefficient and target attempts at optimisation towards those problems.

In conclusion

Measure Twice.
Optimise if necessary.

(1) No, that wasn’t a typo.

(2) At that time. Later changes to schema or data volume may require existing queries to be revised.

For more details on exactly how to identify problematic queries, refer to the series I wrote at Simple Talk last year.

Just one note before diving into that. The examples I’m using are fairly simplistic and that’s intentional. I’m trying to find what, if any, are the performance differences in a benchmark-style setup. I’ll have some comments on more complex examples in a later post.

The most important thing to note about NOT EXISTS and NOT IN is that, unlike EXISTS and IN,  they are not equivalent in all cases. Specifically, when NULLs are involved they will return different results. To be totally specific, when the subquery returns even one null, NOT IN will not match any rows.

The reason for this can be found by looking at the details of what the NOT IN operation actually means.

Let’s say, for illustration purposes that there are 4 rows in the table called t, there’s a column called ID with values 1..4

WHERE SomeValue NOT IN (SELECT AVal FROM t)
is equivalent to
WHERE (
SomeValue != (SELECT AVal FROM t WHERE ID=1)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=2)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=3)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=4)
)

Let’s further say that AVal is NULL where ID = 4. Hence that != comparison returns UNKNOWN. The logical truth table for AND states that UNKNOWN and TRUE is UNKNOWN, UNKNOWN and FALSE is FALSE. There is no value that can be AND’d with UNKNOWN to produce the result TRUE

Hence, if any row of that subquery returns NULL, the entire NOT IN operator will evaluate to either FALSE or NULL and no records will be returned

So what about EXISTS?

Exists cannot return NULL. It’s checking solely for the presence or absence of a row in the subquery and, hence, it can only return true or false. Since it cannot return NULL, there’s no possibility of a single NULL resulting in the entire expression evaluating to UNKNOWN.

Hence, when the column in the subquery that’s used for comparison with the outer table can have nulls in it, consider carefully which of Not Exists or Not in you want to use.

Ok, but say there are no nulls in the column. How do they compare speed-wise. I’m going to do two tests, one where the columns involved in the comparison are defined as NULL and one where they are defined as NOT NULL. There will be no null values in the columns in either case. In both cases, the join columns will be indexed. After all, we all index our join columns, right?

So, first test, non-nullable columns. First some setup

CREATE TABLE BigTable (
id INT IDENTITY PRIMARY KEY,
SomeColumn char(4) NOT NULL,
Filler CHAR(100)
)

CREATE TABLE SmallerTable (
id INT IDENTITY PRIMARY KEY,
LookupColumn char(4) NOT NULL,
SomeArbDate Datetime default getdate()
)

INSERT INTO BigTable (SomeColumn)
SELECT top 250000
char(65+FLOOR(RAND(a.column_id *5645 + b.object_id)*10)) + char(65+FLOOR(RAND(b.column_id *3784 + b.object_id)*12)) +
char(65+FLOOR(RAND(b.column_id *6841 + a.object_id)*12)) + char(65+FLOOR(RAND(a.column_id *7544 + b.object_id)*8))
from master.sys.columns a cross join master.sys.columns b

INSERT INTO SmallerTable (LookupColumn)
SELECT DISTINCT SomeColumn
FROM BigTable TABLESAMPLE (25 PERCENT)
-- (3898 row(s) affected)

CREATE INDEX idx_BigTable_SomeColumn
ON BigTable (SomeColumn)
CREATE INDEX idx_SmallerTable_LookupColumn
ON SmallerTable (LookupColumn)

Then the queries

-- Query 1
SELECT ID, SomeColumn FROM BigTable
WHERE SomeColumn NOT IN (SELECT LookupColumn FROM SmallerTable)

-- Query 2
SELECT ID, SomeColumn FROM BigTable
WHERE NOT EXISTS (SELECT LookupColumn FROM SmallerTable WHERE SmallerTable.LookupColumn = BigTable.SomeColumn)

The first thing to note is that the execution plans are identical.

The execution characteristics are also identical.

Query 1
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 221 ms.

Query 2
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 247 ms.

So, at least for the case where the columns are defined as NOT NULL, these two perform the same.

What about the case where the columns are defined as nullable? I’m going to simply alter the two columns involved without changing anything else, then test out the two queries again.

ALTER TABLE BigTable
 ALTER COLUMN SomeColumn char(4) NULL

ALTER TABLE SmallerTable
 ALTER COLUMN LookupColumn char(4) NULL

And the same two queries

-- Query 1
<pre>SELECT ID, SomeColumn FROM BigTable
WHERE SomeColumn NOT IN (SELECT LookupColumn FROM SmallerTable)

-- Query 2
SELECT ID, SomeColumn FROM BigTable
WHERE NOT EXISTS (SELECT LookupColumn FROM SmallerTable WHERE SmallerTable.LookupColumn = BigTable.SomeColumn)

And as for their performance…

Query 1
Table ‘SmallerTable’. Scan count 3, logical reads 500011, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 437, physical reads 0.

SQL Server Execution Times:
CPU time = 827 ms,  elapsed time = 825 ms.

Query 2
Table ‘BigTable’. Scan count 1, logical reads 437, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 9, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 228 ms.

Radically different execution plans, radically different performance characteristics. The NOT IN took over 5 times longer to execute and did thousands of times more reads.

Why is that complex execution plan required when there may be nulls in the column? I can’t answer that one, probably only one of the query optimiser developer can, however the results are obvious. When the columns allow nulls but has none, the NOT IN performs significantly worse than NOT EXISTS.

So, take-aways from this?

Most importantly, NOT EXISTS and NOT IN do not have the same behaviour when there are NULLs involved. Chose carefully which you want.

Columns that will never contain NULL values should be defined as NOT NULL so that SQL knows there will never be NULL values in them and so that it doesn’t have to produce complex plans to handle potential nulls.

On non-nullable columns, the behaviour and performance of NOT IN and NOT EXISTS are the same, so use whichever one works better for the specific situation.