There have been a fair few forums questions in recent months asking for help in removing index scans, loop joins, sorts or other, presumed, slow query operators. There have been just as many asking how to replace a subquery with a join or a join with a subquery or similar aspects of a query usually for performance reasons.

The first question that I have to ask when looking at requests like that is “Why?”

Why is removing a particular query operator the goal? Why is changing a where clause predicate the goal? If it’s to make the query faster, has the query been examined and has it been confirmed that query operator or predicate really is the problem?

The title of this post refers to a comment I’ve seen again and again in blogs or articles about front-end development. “Premature optimisation is the root of all evils.” It’s true in the database field as well.

While optimisation is very important in database development, trying to optimise queries without any idea where the problem with the query is, or even if the query is a problem at all is about as effective in fixing a database performance problem as using a shotgun from 100 meters is in killing mosquitoes. If you hit the problem, it’s by shear luck and nothing else.

There’s two sides to this problem.

The first aspect of this is, during development, spending time on optimising a query (or stored procedure) without any idea whether or not the query is inefficient and no idea whether or not the changes made make any improvement or not.

Firstly this is a waste of time that could be better spent developing other queries. Second it creates an incorrect impression that the queries have been optimised when in fact nothing of the sort has been done.

The second aspect when, with a production database that is performing badly, queries are modified almost at random in an attempt to fix the performance problem quickly.

This almost never works. It wastes time fixing stuff that very likely isn’t broken in the first place all the while the database performance deteriorates and management curses SQL Server as ‘nonscalable’

So, what is the right approach for the above two scenarios?

1. Don’t optimise queries without knowing if they need it.
2. Don’t optimise queries without knowing if they need it. ¹

New development

When writing queries and stored procedures they need to be tested against a representative data set on a server with representative workload and their performance characteristics evaluated to see if they are acceptable. If the query’s performance characteristics are acceptable, then that query requires no optimisation²

This doesn’t mean write bad code and push it to production. It means write good, solid code, following accepted coding standards, ensure that it runs acceptably against production-volumes of data, and do not spend hours or days trying to get it running a couple of milliseconds faster.

And if the query doesn’t perform acceptable, identify the problematic portion and fix that, don’t flail around rewriting bits of the query in the hope that the problem will magically go away.

The execution plan is the primary tool here, along with the output of Statistics IO.

Fixing existing code

When evaluating existing databases with know performance problems, limit the performance tuning to queries that really are performing badly and need optimisation. It’s often true that fixing the top 5-10 worst performing queries will have massive effects in overall system performance, far more than tuning twice that number of queries that aren’t really a problem.

The best tool for finding which queries really are the worst offenders is SQL Trace.

When looking at queries that are a problem, identify the portions that are inefficient and target attempts at optimisation towards those problems.

In conclusion

Measure Twice.
Optimise if necessary.

(1) No, that wasn’t a typo.

(2) At that time. Later changes to schema or data volume may require existing queries to be revised.

For more details on exactly how to identify problematic queries, refer to the series I wrote at Simple Talk last year.

Just one note before diving into that. The examples I’m using are fairly simplistic and that’s intentional. I’m trying to find what, if any, are the performance differences in a benchmark-style setup. I’ll have some comments on more complex examples in a later post.

The most important thing to note about NOT EXISTS and NOT IN is that, unlike EXISTS and IN,  they are not equivalent in all cases. Specifically, when NULLs are involved they will return different results. To be totally specific, when the subquery returns even one null, NOT IN will not match any rows.

The reason for this can be found by looking at the details of what the NOT IN operation actually means.

Let’s say, for illustration purposes that there are 4 rows in the table called t, there’s a column called ID with values 1..4

WHERE SomeValue NOT IN (SELECT AVal FROM t)
is equivalent to
WHERE (
SomeValue != (SELECT AVal FROM t WHERE ID=1)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=2)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=3)
AND
SomeValue != (SELECT AVal FROM t WHERE ID=4)
)

Let’s further say that AVal is NULL where ID = 4. Hence that != comparison returns UNKNOWN. The logical truth table for AND states that UNKNOWN and TRUE is UNKNOWN, UNKNOWN and FALSE is FALSE. There is no value that can be AND’d with UNKNOWN to produce the result TRUE

Hence, if any row of that subquery returns NULL, the entire NOT IN operator will evaluate to either FALSE or NULL and no records will be returned

So what about EXISTS?

Exists cannot return NULL. It’s checking solely for the presence or absence of a row in the subquery and, hence, it can only return true or false. Since it cannot return NULL, there’s no possibility of a single NULL resulting in the entire expression evaluating to UNKNOWN.

Hence, when the column in the subquery that’s used for comparison with the outer table can have nulls in it, consider carefully which of Not Exists or Not in you want to use.

Ok, but say there are no nulls in the column. How do they compare speed-wise. I’m going to do two tests, one where the columns involved in the comparison are defined as NULL and one where they are defined as NOT NULL. There will be no null values in the columns in either case. In both cases, the join columns will be indexed. After all, we all index our join columns, right?

So, first test, non-nullable columns. First some setup

CREATE TABLE BigTable (
id INT IDENTITY PRIMARY KEY,
SomeColumn char(4) NOT NULL,
Filler CHAR(100)
)

CREATE TABLE SmallerTable (
id INT IDENTITY PRIMARY KEY,
LookupColumn char(4) NOT NULL,
SomeArbDate Datetime default getdate()
)

INSERT INTO BigTable (SomeColumn)
SELECT top 250000
char(65+FLOOR(RAND(a.column_id *5645 + b.object_id)*10)) + char(65+FLOOR(RAND(b.column_id *3784 + b.object_id)*12)) +
char(65+FLOOR(RAND(b.column_id *6841 + a.object_id)*12)) + char(65+FLOOR(RAND(a.column_id *7544 + b.object_id)*8))
from master.sys.columns a cross join master.sys.columns b

INSERT INTO SmallerTable (LookupColumn)
SELECT DISTINCT SomeColumn
FROM BigTable TABLESAMPLE (25 PERCENT)
-- (3898 row(s) affected)

CREATE INDEX idx_BigTable_SomeColumn
ON BigTable (SomeColumn)
CREATE INDEX idx_SmallerTable_LookupColumn
ON SmallerTable (LookupColumn)

Then the queries

-- Query 1
SELECT ID, SomeColumn FROM BigTable
WHERE SomeColumn NOT IN (SELECT LookupColumn FROM SmallerTable)

-- Query 2
SELECT ID, SomeColumn FROM BigTable
WHERE NOT EXISTS (SELECT LookupColumn FROM SmallerTable WHERE SmallerTable.LookupColumn = BigTable.SomeColumn)

The first thing to note is that the execution plans are identical.

The execution characteristics are also identical.

Query 1
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 221 ms.

Query 2
Table ‘BigTable’. Scan count 1, logical reads 342, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 8, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 247 ms.

So, at least for the case where the columns are defined as NOT NULL, these two perform the same.

What about the case where the columns are defined as nullable? I’m going to simply alter the two columns involved without changing anything else, then test out the two queries again.

ALTER TABLE BigTable
 ALTER COLUMN SomeColumn char(4) NULL

ALTER TABLE SmallerTable
 ALTER COLUMN LookupColumn char(4) NULL

And the same two queries

-- Query 1
<pre>SELECT ID, SomeColumn FROM BigTable
WHERE SomeColumn NOT IN (SELECT LookupColumn FROM SmallerTable)

-- Query 2
SELECT ID, SomeColumn FROM BigTable
WHERE NOT EXISTS (SELECT LookupColumn FROM SmallerTable WHERE SmallerTable.LookupColumn = BigTable.SomeColumn)

And as for their performance…

Query 1
Table ‘SmallerTable’. Scan count 3, logical reads 500011, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 437, physical reads 0.

SQL Server Execution Times:
CPU time = 827 ms,  elapsed time = 825 ms.

Query 2
Table ‘BigTable’. Scan count 1, logical reads 437, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 9, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 228 ms.

Radically different execution plans, radically different performance characteristics. The NOT IN took over 5 times longer to execute and did thousands of times more reads.

Why is that complex execution plan required when there may be nulls in the column? I can’t answer that one, probably only one of the query optimiser developer can, however the results are obvious. When the columns allow nulls but has none, the NOT IN performs significantly worse than NOT EXISTS.

So, take-aways from this?

Most importantly, NOT EXISTS and NOT IN do not have the same behaviour when there are NULLs involved. Chose carefully which you want.

Columns that will never contain NULL values should be defined as NOT NULL so that SQL knows there will never be NULL values in them and so that it doesn’t have to produce complex plans to handle potential nulls.

On non-nullable columns, the behaviour and performance of NOT IN and NOT EXISTS are the same, so use whichever one works better for the specific situation.

This month, Richard Sweetnam, one of Microsoft SA’s Premier Field Engineers will be presenting on Tips and Tricks for Management Studio.

Hope to see you all there.

What are Genetic Algorithms?

Genetic algorithms are a form of evolutionary computation, a branch of artificial intelligence that focuses on evolving effective or optimal solutions to difficult problems, based on the biological theory of evolution.

Genetic algorithms are, at their core, a search/optimisation technique. They are a way of finding maximum/minimum solutions to problems and, can be effective when there is no algorithmic solution to the problem. An example here would be the ‘Travelling Salesman’ problem.

Genetic algorithms work by taking an initial population of potential solutions (referred to as individuals), selecting a subset of the population that has the highest fitness then using that subset to generate a second generation. From the second generation again a subset with the highest fitness is selected and used to generate a third generation. This repeats until either the ‘fittest’ individual is considered a good enough solution, or until a certain number of generations have passed.

There are advantage to using genetic algorithms to solve problems over more traditional methods like hill climbing.

• Genetic algorithms can quickly produce good solutions though they may take a lot of time to find the best solution. This is a benefit when the problem is such that the absolute best solution is not necessary, just one that is ‘good enough’
• They are not susceptible to getting trapped by local maxima.
• They do not work on the entire search space one potential solution at a time, but rather work on populations of potential solutions, focusing towards more optimal areas of the search space.

A genetic algorithm will almost always find an optimal solution, given enough time. The main downside is that they may take a lot of time to find that optimal solution.

Components of a Genetic Algorithm

There are two main critical parts in setting up a genetic algorithm for a problem.

• The encoding of the potential solutions into a form where they can be operated on.
• The fitness function which defines which individuals are better than others, which are closer to the maximum that is being searched for.

Most of the design work when using genetic algorithms goes into those two problems.

Encoding

Encoding is the process of taking all the values that make up a potential solution and turning them into a form that the genetic algorithm can operate on.

The selection of an encoding is of utmost importance to the effectiveness of the entire process and a poor representation can make the entire problem much harder than it should. Unfortunately there has been little academic work done on the process of designing representations.

Often for genetic algorithms, the end result of the encoding will be a binary string. There are other variations of evolutionary computation that use other representations, from the arrays of real numbers used by evolutionary strategies to the code trees used by genetic programming.

Fitness function

Depending on the problem, the fitness function can be trivial to write or near-impossible. The design of the fitness function is completely based on the problem that is being solved.

There are two important considerations for a fitness function.

• It must be deterministic.
• It must be fast

If the fitness of an individual is assessed twice, it must come to the same value¹. If the fitness function could return different values for the same individual, then it is of no use in determining the fittest individuals in the population and hence the genetic algorithm will not be able to identify the best solution to the problem.

The fitness of each individual is assessed at least once in each generation. The calculation of the fitness function is usually the most time consuming part of the entire process and the longer the fitness function takes to run, the longer the entire process is run

(1) There are cases where the fitness of an individual may depend on external factors which change over time. Hence a fitness function may give different values for one individual if calculated at different times. Genetic algorithms in a changing environment are a little beyond the scope of this entry.

Evolution Process

In order to create a new generation, the fittest individuals from the previous generation are taken and used to generate the next generation. There are two main operators that are used to generate a generation from the previous one. Crossover and mutation.

Crossover

Crossover involved taking two individuals, splitting each one’s encoded string and swapping parts to generate two new individuals.

Say we had two individuals with the following encoded strings (spaces added for clarity)
0000 0001 1111 1110
0101 1010 1100 0011
and we chose the splitting point for the crossover after the 4th bit, the resulting strings after the crossover will be
0000 1010 1100 0011
0101 0001 1111 1110

In genetic algorithms crossover is the primary operator used. What I described here was a single crossover. There are a number of other variations that can be used.

Mutation

Mutation is an operator applied to a single individual. It’s usually applied after crossover has generated new individuals. Mutation involves flipping a single bit somewhere in the encoded string.

Let’s take the two individuals that were generates by the crossover earlier and apply a random mutation to each
0000 1010 1100 0011
0101 0001 1111 1110
After
0000 1010 1101 0011
0101 0001 1011 1110

In genetic algorithms mutation is very seldom applied and only a small percentage of individuals in a generation will be affected by the mutation operator.

Example

As a quick example let’s manually evolve a simple function to see how the whole thing works.

Let’s say I have an array of 4 numbers (call it num) between 0 and 15. I want to know what values give me the best value for the following.

num[1]-num[2]-num[3]+num[4]

I know, that’s simple enough that we could work out the optimal solution just by eye. Not the point. This is enough to do a quick and effective demo with.

I’m going to encode that by simply converting the numbers in the array to binary and concatenating the binary representations of the 4 numbers (spaces just added for clarity). The fitness function is already defined. I’m going to start with an initial population of eight individuals.

1111 1111 1100 1110 – fitness = (15-15-12+14) = 2
0101 1010 1100 0011 – fitness = (9-10-12+3) = –10
1011 0111 0011 1111 – fitness = (11-7-3+15) = 16
1111 1001 1010 0011 – fitness = (15-9-10+3) = -1
1010 1010 1010 1010 – fitness = (10-10-10+10) = 0
1000 0010 0111 0110 – fitness = (8-2-7+6) = 5
0000 0001 1111 1110 – fitness = (0-1-15+14) = -2
1010 0101 0010 0101 – fitness = (10-5-2+5) = 8

From this I’m going to take the 4 individuals with the highest fitness, use crossover operations (with the crossover point exactly in the middle) between them until I have 8 individuals for the 2nd generation and then apply a single bit mutation to one of the individuals (detailed steps left as an exercise for the reader)

1111 1111 0011 1111 – fitness = (15-15-3+15) = 12
1011 0111 1100 1110 – fitness = (11-7-12+14) = 6
1111 1011 0010 0101 – fitness = (15-11-2+5) = 7
1010 0101 1100 1110 – fitness = (10-5-7+6) = 4
1011 0111 0111 0110 – fitness = (11-7-7+6) = 3
1000 0010 0011 1111 – fitness = (8-2-3+15) = 18
1010 0101 0111 0110 – fitness = (10-5-7+6) = 4
1000 0010 0010 0101 – fitness = (8-2-2+5) = 9

We can already see an improvement. The average and maximum fitness is much higher than for the first generation. I’ll do one more generation in this example, again taking the 4 fittest individuals, crossing over to generate 8 new individuals and then applying a single bit mutation to two individuals. This time however, the crossover point will between the 4th and 5th bit.

1000 1111 0011 1111 – fitness = (8-15-3+15) = 5
1111 0010 0011 1111 – fitness = (15-2-3+15) = 25
1000 1011 0010 0101 – fitness = (8-11-2+5) = 0
1111 0010 0011 1111 – fitness = (15-2-3+15) = 25
1111 0010 1010 0101 – fitness = (15-2-10+5) = 8
1000 0111 0011 1111 – fitness = (8-7-3+15) = 13
1111 0010 0010 0101 – fitness = (15-2-2+5) = 16
1000 1011 0010 0101 – fitness = (8-11-2+5) = 0

I think that’s enough for this example. We’re getting fairly close to the best possible solution (15,0,0,15), close enough to see how this works. The population size was very low, that’s why there are duplicates appearing in the results of the crossover. With a larger search space there would be a lot more diversity.

I hope that anyone still reading found this brief diversion into the realms of AI interesting. The regular SQL-related posts will return soon.

I could easily think of two events that dramatically influenced where I am today, finding a third with as major an impact was difficult. I think the third one qualifies as an important enough event, while it didn’t really affect my career, it did influence my community involvement.

I canna take it anymore

I grew up surrounded by two things, computers and science fiction.

My father was a computer programmer in those days (today to runs a software company) and there were computers around from the earliest I remember. From the Sharp that I played Asteroids and The Valley on, to the NCR with it’s beeping keyboard where I first started programming (in a variant of basic), to the 80286 that my father gave me when he bought himself something faster. I’ve always had computers around that I could use. Despite that, I never had any intention of going into IT as a career.

My mother is a trekkie (classic Star Trek only please) so I grew up watching (and reading) lots of Science Fiction. From Star Trek to Dr Who to Battlestar Galactica to the entire science fiction collection at the local library I watched and read everything I could get my hands on, and it wasn’t long before I started reading Science fact as well as Science fiction. By the time I got to high school my career plans were leaning in the direction of Physics and Astronomy. Placing very high in the national Science Olympiad and almost winning a trip to Space Camp just strengthened those intentions.  I enjoyed playing with computers, but that was more a hobby (and, by that point, a place to play games)

I entered university with the intention to major in Physics, take a related subject as my second major and then get an Honours degree¹ in Physics and find a job in astronomy or physics research. I took Computer Science as my second major because it was one of the few subjects that I was interested in that didn’t conflict with the other subjects I had to take (Chemistry 1 and Maths 1) I spend most of my spare time in my first two years in the Physics department library. I reckon that I must have read easily a third of that library in those two years

Just two problems with that intention. Firstly, there’s almost no demand in this country for physicists other than the universities and the national observatory. Secondly, by the time I got to 3rd year physics, I couldn’t handle the Maths involved. It was part way through the course on Quantum Physics (which contained more maths than some of the 3rd year maths courses did) that I realised that if I couldn’t handle the maths at this point, there was no way I’d ever be able to get a post-grad degree in physics.

I finished the Bachelors degree majoring in Physics and Computer Science and then applied for the honours degree in the Computer Science department

(1) In South Africa the Honours degree is a one year post-grad degree that sits between the Bachelors degree and the Masters degree.

Don’t hold back, tell me what you really think.

Fast forward about five and a half years.

I’d been doing assorted development work since leaving university. Starting with Oracle Reports, moving into MS Access and Visual Basic and finally into the web development area. I loved web dev. It was complex enough to be a challenge but not so complex that it wasn’t fun, and the whole thing just made sense to me.

I was working at the home loans division of a major bank doing a mixture of new development, maintenance of existing code and reports. When the main application started having performance problems, the company got a consultant in to fix it, and I got assigned to help out because I knew SQL better than any of the other devs (translate, I could write a select statement across multiple tables). I’ve spoken before about what I learnt from that.  After that the consultant got involved in other projects and I went on with other work, but there were a lot of problems in the department and so I was looking for somewhere else to work.

I told the consultant that I was planning to resign and I was looking for job in development, preferably web development. His reply was rather blunt, something along the lines of “Don’t be stupid, that’s a waste of your time and talents.” He then spoke of the type of work that he and his colleague did, enterprise server stuff, Biztalk, SQL, Exchange, etc. It was a surprise just how much of that work there was.

I won’t say that I immediately took his advice and swore off web dev forever. I didn’t. It did however get me thinking and reading and realising just how much there was to that kind of work, how much there was to IT that wasn’t front-end development and wasn’t system administration.

When I took another job with a different bank a couple months later, one of the conditions that I asked for was to have to option of moving from the DB developer role back into web development if I asked. They agreed. I never took the option up.

Can I ask your opinion?

Fast forward another couple of years. It was 2006 and I was attending my very first TechEd South Africa. By this point I was doing performance tuning full time at the company I worked at, and I knew a fair bit about SQL Server and I posted on the forums occasionally. I’d attended the PASS European Summit the year before and the competency and knowledge of some of the speakers there had stunned me.

Microsoft had managed to get a few really good international speakers for that TechEd, including one who presented on SQL Server (who, incidentally had also presented at PASS Europe the year before). After the conference was over, just before the closing keynote, I saw the SQL Server speaker in the passage heading, like I was, for the keynote. So I went over. said ‘hi’ and all that, and asked one question (well, two if you count ‘Can I help you with your bags?’)

“How do I get to where you are?”

The answer took the entire walk to the keynote and some more time. None of the advice was implemented immediately. Some (this blog) took almost a year, some I never implemented, but it got me thinking and it got me contributing to the community, not just been a passive recipient of information.

There’s other things I could talk about, like the six months of writing Oracle reports and what that taught me about writing select statements or the invitation to join a roleplaying group that helped get over my problems with speaking in public, but I think this is about enough.

Everyone I know seems already tagged, so we’ll just leave it there….

Lies, damned lies and statistics (DBA-388-S)

This session went very well. I was comfortable with the material, it’s a topic I really like and in general it felt, to me at least, like a good session. The ratings seem to agree with that.

If I make Very Poor = 1 and Excellent = 5 then, averaging all the scores over all the questions, overall that session rated at 4.82/5

Not bad at all.

Edit: The overall PASS Summit session ratings are out and this session came in at 7th overall (all sessions including pre/post cons, all tracks) and 5th in the DBA track, behind only Buck Woody, Kimberly Tripp and Paul Randal I am extremely surprised to have come in that high at a conference like the PASS Summit.

Insight into Indexes (DBA-315)

This session was a whole different story. It did not go well at all, and I didn’t need the ratings to tell me that.

I wasn’t overly comfortable with the material. This is not to say that I didn’t know it, I did, but I wasn’t comfortable with it. In retrospect, I should have scrapped the entire presentation and done it over from scratch in a different way, even if that meant doing it the night before. Lesson learnt there.

To add to that, I broke my own rules for presentations. Usually I’m at the session room at least 5 minutes before the previous session finishes, with my laptop booted, the presentation loaded, management studio (and profiler if necessary) open and any pre-demo scripts already run. That way, as soon as the speaker who’s presenting in the session before mine finishes, I can get on stage, plug the laptop in, get the projector online and then relax.

In this case, I was late. The previous speaker had already left and my laptop was still switched off. Hence I rushed to get everything loaded and ready, and Windows, sensing the urgency, promptly crashed hard.

Cue 2 minutes of frantically trying to reboot laptop (it was ignoring all shut down requests) and load presentation onto the desktop in case my laptop didn’t reboot. All while the AV guy’s trying to get the audio on and the recording started.

Let’s just say it went downhill from there.

So, ratings for that one.

If I do the same averaging as for the first one, that comes out at 4.55. Not the worst I’ve ever had, though not by much. Lessons learnt.

One very important thing to note right off is that they are not equivalent in all cases.

An inner join between two tables does a complete join, it checks for matches and returns rows. This means, if there are multiple matching rows in the second table, multiple rows will be returned. Also, when two tables are joined, columns can be returned from either.  As a quick example (definition of BigTable towards the end of the post)

DECLARE @SomeTable (IntCol int)
Insert into @SomeTable (IntCol) Values (1)
Insert into @SomeTable (IntCol) Values (2)
Insert into @SomeTable (IntCol) Values (2)
Insert into @SomeTable (IntCol) Values (3)
Insert into @SomeTable (IntCol) Values (4)
Insert into @SomeTable (IntCol) Values (5)
Insert into @SomeTable (IntCol) Values (5)

SELECT *
 FROM BigTable b INNER JOIN @SomeTable  s ON b.SomeColumn IN s.IntCol

This returns 7 rows and returns columns from both tables. Because the values in @SomeTable are duplicated, the matching rows from BigTable are returned twice.

With an IN, what is done is a semi-join, a join that checks for matches but does not return rows. This means if there are multiple matching tables in the resultset used for the IN, it doesn’t matter. Only one row from the first table will be returned. Also, because the rows are not returned, columns from the table referenced in the IN cannot be returned. As a quick example

DECLARE @SomeTable (IntCol int)
 Insert into @SomeTable (IntCol) Values (1)
 Insert into @SomeTable (IntCol) Values (2)
 Insert into @SomeTable (IntCol) Values (2)
 Insert into @SomeTable (IntCol) Values (3)
Insert into @SomeTable (IntCol) Values (4)
Insert into @SomeTable (IntCol) Values (5)
 Insert into @SomeTable (IntCol) Values (5)

SELECT *
 FROM BigTable
 WHERE SomeColumn IN (Select IntCol FROM @SomeTable)

This returns 5 rows and only columns from BigTable.

So, that said, how does the performance of the two differ for the cases where the results are identical (no duplicates in the second table, no columns needed from the second table)? For that, I’m going to need larger tables to play with.

DROP TABLE dbo.BigTable
DROP TABLE dbo.SmallerTable

CREATE TABLE BigTable (
id INT IDENTITY PRIMARY KEY,
SomeColumn CHAR(4),
Filler CHAR(100)
)

CREATE TABLE SmallerTable (
id INT IDENTITY PRIMARY KEY,
LookupColumn CHAR(4),
SomeArbDate DATETIME DEFAULT GETDATE()
)

INSERT INTO BigTable (SomeColumn)
SELECT top 250000 char(65+FLOOR(RAND(a.column_id *5645 + b.object_id)*10)) + char(65+FLOOR(RAND(b.column_id *3784 + b.object_id)*12)) + char(65+FLOOR(RAND(b.column_id *6841 + a.object_id)*12)) + char(65+FLOOR(RAND(a.column_id *7544 + b.object_id)*8))
FROM master.sys.columns a CROSS JOIN master.sys.columns b

INSERT INTO SmallerTable (LookupColumn)
SELECT DISTINCT SomeColumn
FROM BigTable TABLESAMPLE (25 PERCENT)
-- (3819 row(s) affected)

That’s the setup done, now for the two test cases. Let’s first try without indexes and see how the INNER JOIN and IN compare. I’m selecting from just the first table to ensure that the two queries are logically identical. The DISTINCT used to populate the smaller table ensures that there are no duplicate rows in the smaller table.

SELECT BigTable.ID, SomeColumn
FROM BigTable
WHERE SomeColumn IN (SELECT LookupColumn FROM dbo.SmallerTable)

SELECT BigTable.ID, SomeColumn
FROM BigTable
INNER JOIN SmallerTable ON dbo.BigTable.SomeColumn = dbo.SmallerTable.LookupColumn

Something of interest straight away, the execution plans are almost identical. Not completely identical, but the only difference is that the hash join for the IN shows a Hash Match (Right Semi Join) and the hash join for the INNER JOIN shows a Hash Match (Inner Join)

The IOs are the same and the durations are extremely similar. Here’s the IO results and durations for five tests.

IN

Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 14, physical reads 0.

SQL Server Execution Times:
CPU time = 156 ms,  elapsed time = 2502 ms.
CPU time = 157 ms,  elapsed time = 2323 ms.
CPU time = 156 ms,  elapsed time = 2555 ms.
CPU time = 188 ms,  elapsed time = 2381 ms.
CPU time = 203 ms,  elapsed time = 2312 ms.

INNER JOIN

Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 14, physical reads 0.

SQL Server Execution Times:
CPU time = 125 ms,  elapsed time = 2922 ms.
CPU time = 140 ms,  elapsed time = 2372 ms.
CPU time = 188 ms,  elapsed time = 2530 ms.
CPU time = 203 ms,  elapsed time = 2323 ms.
CPU time = 187 ms,  elapsed time = 2512 ms.

Now let’s try with some indexes on the join columns.

CREATE INDEX idx_BigTable_SomeColumn ON BigTable (SomeColumn)
CREATE INDEX idx_SmallerTable_LookupColumn ON SmallerTable (LookupColumn)

Now when I run the two queries, the execution plans are different, and the costs of the two are no longer 50%. Both do a single index scan on each table, but the IN has a Merge Join (Inner Join) and the INNER JOIN has a Hash Match (Inner Join)

The IOs are still identical, other than the WorkTable that only appears for the Hash Join.

IN

Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 14, physical reads 0.

INNER JOIN

Table ‘Worktable’. Scan count 0, logical reads 0, physical reads 0.
Table ‘BigTable’. Scan count 1, logical reads 3639, physical reads 0.
Table ‘SmallerTable’. Scan count 1, logical reads 14, physical reads 0.

So what about the durations? Honestly it’s hard to say anything completely conclusive, the durations of both queries are quite small and they are very close. To see if there is any measurable different, I’m going to run each one 100 times, use Profiler to log the duration and CPU and then average the results over the 100 executions. While running this, I’m also going to close/disable everything else I can on the computer, to try and get reasonably accurate times.

IN

Average CPU: 130.
Avg duration: 2.78 seconds

INNER JOIN

Average CPU: 161.
Avg duration: 2.93 seconds

Now is that enough to be significant? I’m not sure. However, looking at those results along with the IO and execution plans, I do have a recommendation for In vs Inner Join

If all you need is to check for matching rows in the other table but don’t need any columns from that table, use IN. If you do need columns from the second table, use Inner Join.

I still intend to go over NOT IN and NOT EXISTS and, after this one, I also want to take a look at the LEFT JOIN with IS NULL check vs NOT IN for when you want rows from Table1 that don’t have a match in Table 2.

I did fairly well with the goals that I set back in October. The experiment design for my thesis is not documented and the WPF book is not finished, but the rest is all done and the exam was easily passed. So not too bad there. Better than I managed in the first half of the year.

And now for next year…

The biggest thing is that I’m going to take a near-complete break from the SQL community for at least the first half of the year so that I can focus on my thesis. No writing articles, no contributing to books, no giving or writing presentations. I’ll still blog, though likely only once a month on SQL-related stuff, though there may well be some AI stuff appearing from time to time. I’ll still be around on the SSC forums, though not as much as I currently am. I’ll also still be involved in the local usergroup. I can’t abandon that.

So, with that out of the way, the goals for the next six months:

• Read one SQL book
• Read at least two AI books
• Get the experiment for my thesis designed and coded.
• Write at least two chapters of the thesis
• Get back into computer graphics and get two images done

To add to that, strange as it may seem, I’m going to ensure that I take time to read, relax and exercise away from the computer. This year I’ve spend too much time ‘busy’. I have a stack of books almost a metre high that I’ve bought but not opened. I have a similar stack of movies and games (though not quite as high) and I had a nasty bout of burnout Oct/Nov and I really don’t want  that again.

Mine would have to be two very closely related things. Procrastination and short attention span. (why do you think this post is 4 days late?)

I tend to put stuff off until the absolute last minute (and sometimes beyond then) and then work frantically to get it finished. And because I’m also a bit of a perfectionist, I’m not satisfied with half-done work or a hack job so I’ll be working late into the night (or over a weekend) to get it done properly.

To add to that, unless I’m doing something I really enjoy, my attention span tends not to be very long. 15-20 minutes is good, it’s usually less. After that it’s either fight to stay focused or take a short break. In and of itself, that’s not so bad, the problem is that the breaks are often not so short, if I get caught up in whatever else I’m doing, or distracted by something else (and something else and something else).

So what am I doing about this?

Well, firstly, I have no solutions. There are things that I’m doing that are helping, but I won’t say they are general solutions.

Procrastination

I’m splitting big tasks up into small (very small) chunks and setting deadlines for those chunks. Sometimes more than one a day. That way, everything should be done at the last minute, so I can’t put things off, much.

This works fairly well if I have a lot of things to do. It doesn’t tend to work well for me when I’ve got relatively few things that need doing and lots of time to do them in. Still, it is helping, as long as I take the time to break work down and list small tasks. I do need to apply a bit more discipline here to make sure that I do break work down and I do adhere to the tasks deadlines.

Attention span

This one’s harder. While I can push through when I start losing focus, it’s difficult and gets more difficult as time goes on.

Because I know that I can’t focus on one thing straight through for hours, I’ll usually have two thing that I’m busy with that I can switch between. For example I’ll have an SSIS package that I’m developing and a profile trace that I’m analysing and I can switch between them.

What I have to be careful about is time-consuming non-work related distractions. Which, seeing as I work from home with a big shelf of books behind me and an xbox in the lounge can be difficult. But that comes down to discipline. No games during the day. No more than an hour of gaming over supper when there’s work that needs doing.

This is also why, even though I’ve joined the Twitter madness, the Twitter client is closed for most of the day. It’s also why I’ve started closing Outlook for most of the day. If the clients want me urgently, they’ll phone. Mail can always wait a couple of hours.

Well, I think that’s enough about me. I’m curious to see what Steve Jones and Paul Randal have to say (if anything)

A trivial plan is essentially a plan for a query where a specific plan will always be the most optimal way of executing it. If we consider something like SELECT * FROM SomeTable then there’s only one real way to execute it, a scan of the cluster/heap.

The trivial plan is somewhat of a query optimiser optimisation. If the query qualifies for a trivial plan (and there are lots of restrictions) then the full optimisation process doesn’t need to be started and so the query’s execution plan can be generated quicker and with less overhead. The fact that a query has a trivial plan at one point doesn’t necessarily mean that it will always have a trivial plan, indexes may be added that make the selection of plan less of a sure thing and so the query must go for full optimisation, rather than getting a trivial plan

Nice theory, but how does one tell if a particular query has a trivial execution plan? The information is found within the execution plan, the properties of the highest-level operator has an entry ‘Optimisation level’ For a trivial plan this will read ‘TRIVIAL’

It’s also, for the brave people who like XML, found within the xml form of the plan

&amp;lt;StmtSimple StatementCompId=&amp;quot;1&amp;quot; StatementEstRows=&amp;quot;11&amp;quot; StatementId=&amp;quot;1&amp;quot; StatementOptmLevel=&amp;quot;TRIVIAL&amp;quot; StatementSubTreeCost=&amp;quot;0.0032941&amp;quot; StatementText=&amp;quot;SELECT * FROM forums&amp;quot; StatementType=&amp;quot;SELECT&amp;quot; QueryHash=&amp;quot;0xB38EBF594006422E&amp;quot; QueryPlanHash=&amp;quot;0xCC9AB99E7081C81D&amp;quot;&amp;gt;    <br />&amp;lt;!-- most of the rest of the plan here --&amp;gt;     <br />&amp;lt;/StmtSimple&amp;gt;

So, are they cached? The way to find that out is to run a variety of queries and see what’s sitting in the cache afterwards.

I’m going to clear the procedure cache then run a couple queries against the AdventureWorks database, checking the graphical execution plan for each one. After they’ve all been run, I’ll query the plan cache and check the optimisation level of the cached plans,

Query 1:

SELECT FirstName, LastName
    FROM Person.Person
    WHERE BusinessEntityID = 42

According to the graphical plan, this query has a trivial plan.

Query 2:

SELECT TOP (10) Name FROM Production.Product

According to the graphical plan, this query also has a trivial plan.

Query 3:

SELECT * FROM Sales.SalesOrderHeader sh
    INNER JOIN sales.SalesOrderDetail sd
        ON sh.SalesOrderID = sd.SalesOrderID
    WHERE sh.ShipDate > '2008/05/25'

According to the graphical plan, this query has a non-trivial plan, the optimisation level is listed as full.

SELECT st.text, qp.query_plan,
     qp.query_plan.value('
       declare default element
       namespace "http://schemas.microsoft.com/sqlserver/2004/07/showplan";
       (//StmtSimple/@StatementOptmLevel)[1]','varchar(20)') AS OptimisationLevel
    FROM sys.dm_exec_cached_plans cp
        CROSS APPLY sys.dm_exec_query_plan(cp.plan_handle) qp
        CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) st
    WHERE text not like '%sys.dm_exec_cached_plans%'

This returns 4 rows. All three of the queries that I ran against AdventureWorks, two with optimisation levels of Trivial, one with an optimisation level of Full, and the unparameterised ‘shell’ of the first of the queries.

So it appears that some trivial plans are indeed cached.