So what’s the solution? Well, throwing hardware at the problem is an old favourite. There are few workloads that a nice 256-processor Itanium with a terabyte or two of memory won’t handle, but servers like that are a little on the expensive side and lots of money spent on expensive hardware means less that can be spent on annual bonuses.

There is another option, a hidden, undocumented option that can improve query performance, maybe a little, maybe substantially.

First thing that you need to do to get this one is to enable the hidden options in sp_configure. This is done much the same way as the advanced options.

exec sp_configure 'show hidden options', 1
RECONFIGURE WITH EXTREME OVERRIDE

Once that’s done, the undocumented option can be enabled.

exec sp_configure 'run queries faster', 101010
 RECONFIGURE WITH EXTREME OVERRIDE

How much improvement this will give depends on the kind of queries being run. OLTP systems usually see a greater improvement than decision-support, unless there’s full text search or spatial queries, in which case there will likely be substantially less of a gain.

Now, there are a few things to consider.

1. This is obviously undocumented and that means unsupported.
2. It may not work on the next version of SQL.
3. If you call support, disable the option first and don’t tell them you were running it!

.

Happy April Fools’ day.

.

..

…

….

Seriously now, there’s no options that, when enabled, makes SQL run queries faster. There is no silver bullet for performance problems, there is no one-size-fits-all fix.

Fixing performance problems involves finding the current bottleneck and removing it, then repeating that operation until performance is acceptable. It’s a complex area and there’s a lot to it. Simply throwing hardware at the problem may not produce much, if any, performance gain, especially if the hardware wasn’t the bottleneck.

If you have a query performance problem and don’t know where to start, ask on one of the SQL forums (like SQLServerCentral) if it’s not an urgent problem. If it is, or if there are serious problems, consider getting a consultant in to help out. One of the quickest ways to learn is to learn from someone who knows what they are doing.

I could easily think of two events that dramatically influenced where I am today, finding a third with as major an impact was difficult. I think the third one qualifies as an important enough event, while it didn’t really affect my career, it did influence my community involvement.

I canna take it anymore

I grew up surrounded by two things, computers and science fiction.

My father was a computer programmer in those days (today to runs a software company) and there were computers around from the earliest I remember. From the Sharp that I played Asteroids and The Valley on, to the NCR with it’s beeping keyboard where I first started programming (in a variant of basic), to the 80286 that my father gave me when he bought himself something faster. I’ve always had computers around that I could use. Despite that, I never had any intention of going into IT as a career.

My mother is a trekkie (classic Star Trek only please) so I grew up watching (and reading) lots of Science Fiction. From Star Trek to Dr Who to Battlestar Galactica to the entire science fiction collection at the local library I watched and read everything I could get my hands on, and it wasn’t long before I started reading Science fact as well as Science fiction. By the time I got to high school my career plans were leaning in the direction of Physics and Astronomy. Placing very high in the national Science Olympiad and almost winning a trip to Space Camp just strengthened those intentions.  I enjoyed playing with computers, but that was more a hobby (and, by that point, a place to play games)

I entered university with the intention to major in Physics, take a related subject as my second major and then get an Honours degree¹ in Physics and find a job in astronomy or physics research. I took Computer Science as my second major because it was one of the few subjects that I was interested in that didn’t conflict with the other subjects I had to take (Chemistry 1 and Maths 1) I spend most of my spare time in my first two years in the Physics department library. I reckon that I must have read easily a third of that library in those two years

Just two problems with that intention. Firstly, there’s almost no demand in this country for physicists other than the universities and the national observatory. Secondly, by the time I got to 3rd year physics, I couldn’t handle the Maths involved. It was part way through the course on Quantum Physics (which contained more maths than some of the 3rd year maths courses did) that I realised that if I couldn’t handle the maths at this point, there was no way I’d ever be able to get a post-grad degree in physics.

I finished the Bachelors degree majoring in Physics and Computer Science and then applied for the honours degree in the Computer Science department

(1) In South Africa the Honours degree is a one year post-grad degree that sits between the Bachelors degree and the Masters degree.

Don’t hold back, tell me what you really think.

Fast forward about five and a half years.

I’d been doing assorted development work since leaving university. Starting with Oracle Reports, moving into MS Access and Visual Basic and finally into the web development area. I loved web dev. It was complex enough to be a challenge but not so complex that it wasn’t fun, and the whole thing just made sense to me.

I was working at the home loans division of a major bank doing a mixture of new development, maintenance of existing code and reports. When the main application started having performance problems, the company got a consultant in to fix it, and I got assigned to help out because I knew SQL better than any of the other devs (translate, I could write a select statement across multiple tables). I’ve spoken before about what I learnt from that.  After that the consultant got involved in other projects and I went on with other work, but there were a lot of problems in the department and so I was looking for somewhere else to work.

I told the consultant that I was planning to resign and I was looking for job in development, preferably web development. His reply was rather blunt, something along the lines of “Don’t be stupid, that’s a waste of your time and talents.” He then spoke of the type of work that he and his colleague did, enterprise server stuff, Biztalk, SQL, Exchange, etc. It was a surprise just how much of that work there was.

I won’t say that I immediately took his advice and swore off web dev forever. I didn’t. It did however get me thinking and reading and realising just how much there was to that kind of work, how much there was to IT that wasn’t front-end development and wasn’t system administration.

When I took another job with a different bank a couple months later, one of the conditions that I asked for was to have to option of moving from the DB developer role back into web development if I asked. They agreed. I never took the option up.

Can I ask your opinion?

Fast forward another couple of years. It was 2006 and I was attending my very first TechEd South Africa. By this point I was doing performance tuning full time at the company I worked at, and I knew a fair bit about SQL Server and I posted on the forums occasionally. I’d attended the PASS European Summit the year before and the competency and knowledge of some of the speakers there had stunned me.

Microsoft had managed to get a few really good international speakers for that TechEd, including one who presented on SQL Server (who, incidentally had also presented at PASS Europe the year before). After the conference was over, just before the closing keynote, I saw the SQL Server speaker in the passage heading, like I was, for the keynote. So I went over. said ‘hi’ and all that, and asked one question (well, two if you count ‘Can I help you with your bags?’)

“How do I get to where you are?”

The answer took the entire walk to the keynote and some more time. None of the advice was implemented immediately. Some (this blog) took almost a year, some I never implemented, but it got me thinking and it got me contributing to the community, not just been a passive recipient of information.

There’s other things I could talk about, like the six months of writing Oracle reports and what that taught me about writing select statements or the invitation to join a roleplaying group that helped get over my problems with speaking in public, but I think this is about enough.

Everyone I know seems already tagged, so we’ll just leave it there….

Mine would have to be two very closely related things. Procrastination and short attention span. (why do you think this post is 4 days late?)

I tend to put stuff off until the absolute last minute (and sometimes beyond then) and then work frantically to get it finished. And because I’m also a bit of a perfectionist, I’m not satisfied with half-done work or a hack job so I’ll be working late into the night (or over a weekend) to get it done properly.

To add to that, unless I’m doing something I really enjoy, my attention span tends not to be very long. 15-20 minutes is good, it’s usually less. After that it’s either fight to stay focused or take a short break. In and of itself, that’s not so bad, the problem is that the breaks are often not so short, if I get caught up in whatever else I’m doing, or distracted by something else (and something else and something else).

So what am I doing about this?

Well, firstly, I have no solutions. There are things that I’m doing that are helping, but I won’t say they are general solutions.

Procrastination

I’m splitting big tasks up into small (very small) chunks and setting deadlines for those chunks. Sometimes more than one a day. That way, everything should be done at the last minute, so I can’t put things off, much.

This works fairly well if I have a lot of things to do. It doesn’t tend to work well for me when I’ve got relatively few things that need doing and lots of time to do them in. Still, it is helping, as long as I take the time to break work down and list small tasks. I do need to apply a bit more discipline here to make sure that I do break work down and I do adhere to the tasks deadlines.

Attention span

This one’s harder. While I can push through when I start losing focus, it’s difficult and gets more difficult as time goes on.

Because I know that I can’t focus on one thing straight through for hours, I’ll usually have two thing that I’m busy with that I can switch between. For example I’ll have an SSIS package that I’m developing and a profile trace that I’m analysing and I can switch between them.

What I have to be careful about is time-consuming non-work related distractions. Which, seeing as I work from home with a big shelf of books behind me and an xbox in the lounge can be difficult. But that comes down to discipline. No games during the day. No more than an hour of gaming over supper when there’s work that needs doing.

This is also why, even though I’ve joined the Twitter madness, the Twitter client is closed for most of the day. It’s also why I’ve started closing Outlook for most of the day. If the clients want me urgently, they’ll phone. Mail can always wait a couple of hours.

Well, I think that’s enough about me. I’m curious to see what Steve Jones and Paul Randal have to say (if anything)

Who has been a great leader in your career and what made them a great leader?

I think the person who stands out the most is my previous manager. I’m not going to name him, and I’m not going to drone on for pages as to why I think he qualified, I’m going to give a couple of examples of things that happened.

A couple years back I realised that I was so busy keeping up with my work load that I wasn’t finding time to keep up with new developments, new ways of doing things, new releases, etc.

I went to my manager and explained to him that I was supposed, as part of my job reponsibilities, to keep up with new developments and share that information with the rest of the team, if it was applicable. However the pile of optimisation work was overwhelming. Would he mind if I put aside half an hour a day to just read books, blogs, support articles, etc, even though I had so much other work piled up that it would take a couple months to finish it all. His immediate reply, “Are you sure half an hour’s enough?”

Another time I was sitting in his office summarising the cause of a DB outage. It had been a deployment problem if I recall, it shouldn’t have happened. One of the high level business managers stormed in and demanded to know who he should shout at for this. Manager’s reply, “You shout at me.”

We all got shouted at later, but by our manager, not by the business person and it was more a case of “This should not have happened, it won’t happen again, will it?”

When I decided to leave that company, the hardest part was telling my manager that I was leaving. It turned out that he resigned two months after I left, for very much the same reasons.

Hmmm, now, who to tag?

I think Paul Randal and Christopher Stobbs

1) “I don’t know” is not a sign of weakness

When I started in IT, I knew everything (at least I thought I did). Furthermore I thought I had to know everything. That lead to a number of problems, from ignoring offers for help, to tackling things beyond my abilities, right up to embarrassing myself in front of management after it became apparent I didn’t really know what I was talking about.

2) Beware of self-proclaimed experts

Early on I worked with a database guru. At least, he said he was. Looking back now, older and (perhaps) wiser, many of his ‘best practices’ were nothing of the sort.

Anyone can go around claiming to be an expert. The thing is to see what other people say about them

3) Setbacks are not disasters

I was retrenched from the first company I worked for when they hit financial difficulties. I spent 6 months struggling to make ends meet. At the time I thought it was the worst thing possible. However it lead to a job that changed my focus to databases, leading to where I am now.

It also taught me what some of my major weaknesses are – administration (finding work, chasing payments, etc)

4) Usergroups are not a waste of time

I thought that when I started out (see point 1). Now I run one. Go figure

I think just about everyone’s answered this already, so no tags.

There’s also a video and an article linked in the show’s notes which are referred to several times in the podcast.

Top things I learned from that:

• There are 25 counters that all admins should be monitoring, regardless of what the server does
• Some counters are no longer useful with changes to the way disks are attached
• It is possible, and relatively easy, to have a blackbox perfmon trace permanently running and automatically starting on server start.
• It’s normal on Server 2008 to see some CPUs high and others not, because the cost of balancing the CPUs grows with the number of cores.

1) Welcome to the deep end

The first challenge that I’m going to write about took place around five years ago. At the time I was a web developer working at a bank. I was also the person there who knew the most about SQL Server, though that’s not saying much. Basically I could write queries better than the other devs.

The main system that we were working on was slow. Actually, slow’s a compliment. It crawled during busy periods. There were no DBAs and none of the devs knew enough about SQL to even begin to troubleshoot. The company hired an external consultant to come in and fix the problem. It’s probably the best thing they could have done.

The consultant required that one of the in-house staff be assigned to work with him, to both assist and learn so that the next time there was a performance problem, the in-house staff could handle it. Because of my knowledge of SQL, I was the one chosen.

I thought the assignment would be a piece of cake. I thought I knew a fair bit of SQL. I was wrong.

The consultant was an expert in every sense of the word. He had forgotten more about SQL than I had ever known, he could glance over a query and list the problems with it and he had no tolerance for pretence or excuses. If I didn’t know something I was expected to ask or research it. If I said something would be done by a specific date, it had to be done by that date.

Looking back, I think that was the hardest I’d ever worked at that company. I spent the days at the office working on the performance problems, either with the consultant or alone based on his recommendations and guidance and I spent the evenings reading through SQL docs and resources he’d provided.

By the time that project was over, I had an exceedingly good idea just how little of SQL I knew and how much there was still to learn.

It was all good. That was the beginning of my shift from just another web developer to the database field that I enjoy so much now.

2) A whole new way of working

The second challenge is one I’m still going through now, so there’s no resolution.

Up until August this year, I was a database developer/DBA for a large bank. It was a good job, it paid well, but I was starting to get seriously bored. SQL forums provided more challenge than the job did. I decided, after much thought, to move to the consulting role.

To say it’s a challenge is an understatement.

Firstly I have to juggle work for different clients to ensure that everything gets done by when it was promised. There’s no manager to complain to when there’s too much work.

Instead of a regular 8-5 at an office, I’m working whatever hours are necessary from home. I’m easily distracted and there are a lot of distractions at home, from the TV and XBox in the lounge, to the bookshelf full of Sci-fi and fantasy in the study, to the cats, etc, etc. Then I get angry with myself if I haven’t got as much done as I think I should.

What I’m trying is to set small tasks and specific deadlines for them, even if they’re only going to take a couple of hours. That way I can stay focused on them and there’s visible progress.

So far it seems to be working fairly well. We’ll see how the next few months go.

Oh, and the consultant from challenge 1. He’s my colleague now.

I think most people have been tagged with this already, so I’m only going to pass it to one person. Let’s see if Kalen has anything to say on this subject

1) The nagging feeling of something not being right

The first major mistake occurred during one of the first projects that I did from scratch after university. I was working for a small software house that specialised (at the time) in small MS Access apps. When I say small, I’m talking 5-10 users or single user desktop apps. It’s the kind of thing that Access is quite good at.

I was writing a budgetting app for a large insurance company. They wanted to be able to send something out to all of the branch managers, all of the regional managers and all of the provincial managers, so that they could capture their budget info for the next year. The main restriction that I had was that the app must not need to connect to the head office. The managers should be able to copy the app onto laptops and work from home with no internet/WAN access of any form (this was 1999). In addition, the various branch/regional/provincial offices must only have the budget data for themselves and any subsidiary offices.

So I created a second mdb file to be a template data file. It had only the budget line item data (stationary, rent, furniture, etc), copied from the main data file, but none of the offices’ actual budget values. Then, I wrote some rather complex DAO/VBA code that would create one mdb file per office by making a copy of the tempate database (because DAO doesn’t allow a file that’s in use to be copied), insert that office’s data into it, and then place that data file in a specific location so that it could be copied down to the branches.

I tested and I tested and the procedure worked perfectly. Finally the insurance company’s IT people called and said they’d finished making the required changes to the budget line items and percentages for the year and could I go in and babysit the process, since it was the first time. No problem! I went in, ran the process, checked that all the files were there (they were) and then went home. (Anyone see the mistake yet?)

All the way home I had a strange feeling that something was wrong, but I couldn’t place it.

Two days later the calls started. Branch managers complaining that their databases were missing certain line items or had incorrect percentages. I checked over the code but I couldn’t find a thing wrong. There was no way that the code could have caused those problems. I wrote up patches that would correct the data and sent those out. Still couldn’t find the cause. Then, on the third day of calls, the manager from the Botswana office called. He dabbled in IT in his spare time and he knew a fair bit about Access. While we were chatting he mentioned that a none of the changes that head office had told him had been done were in the database he had.

At that point, the other shoe dropped. Head office had made changes to the line item details in their master data file. I hadn’t transfered those changes to the template database, so when I created all of the branch databases, they had the old version of the line items.

Later that day I got to explain to 178 branch managers across three countries why they were going to have to completely redo any budgets that they’d done. To say I wasn’t popular is a rather large understatement

2) Test all assumptions

The second mistake was a fair bit more recent (3 or so years ago) and does involve SQL Server.

I was rewriting the core stored procs for a rather large banking system. The number of users and data were growing and the stored proc that was called for all data access was slow, (Actually, slow’s a compliment) and was very convoluted. Tables were being queried multiple times for the same data in different places and if statements were nested so deep it was impossible to figure out what was actually going to run. The procedure was like a tower built on sand. Very shaky and likely to fall over if touched.

So, for performance reasons, and so that we could make planned enhancements, I rewrote the proc, from scratch. Because of the critical nature of this proc (core proc for a system that ran the investment and corporate divisions of the bank) I was exceedingly careful and I tested and checked all assumptions, except one.

During the course of the rewrite/optimisation I ran across a temp table with a column called FilterGuid, type varchar(50). From where the table was used, I could tell that it contained values of primary keys for various tables, for use in IN statements. The system standards required that every table have a uniqueidentifier for the primary key. The varchar type was causing performance problems due to the implicit conversions and resultant index scans, and so I changed the data type from varchar(50) to uniqueidentifier.

Two weeks of extensive testing found no problems, so the rewritten proc, along with a bunch of subprocedures it called, went into production on a thursday evening.

Next morning within 5 min of me getting to my desk, there were several people there. A small portion of the system that controlled the interbank payments was not working, some payments were already almost two hours late, and the error was “Error converting varchar to uniqueidentifier” (For anyone not familiar with banking, there are strict time limits on the interbank payments and harsh fines if those time limits are exceeded)

It took very little investigation to reveal that the payments portion of the system was breaking the application standards. While the devs for that section claimed that they used guids, they actually used strings that looked like guids, if you didn’t look too closely that it. An example would be “18e21k37-2966-4d75-8jf6-1g08b828a143″.

It took about 2 hours to find, fix, test and deploy a fix. I’m just glad the bank didn’t take the penalties out of my salary, or I wouldn’t have been paid a cent for the rest of the time I worked there.

Moral of that story – test assumptions, don’t trust standards, and don’t believe developers when they say that they follow the standards.

Passing the baton on

So, I’m passing this one to Tim Mitchell, because he said he missed out on it, and, several steps up the ladder, to Paul Randal