SQL in the Wild » Personal

2011 Book review

Gail — Sun, 01 Jan 2012 14:30:00 +0000

Another year over and much as I did last year, I’m going to briefly go over the books I read this last year.

I will freely admit, very few of these could be considered ‘classic literature’, most is a mix of sci-fi, fantasy or adventure fiction. That’s just what I like to read.

Book total this year was 53, up from the 45 I managed in 2010 and above the 50 that I aimed for. Part of this is that I travelled more (and hence had time with nothing to do but read), part is because I took a couple of small vacations (and spent time reading) and part is due to getting an iPad and loading a couple of book apps on there.

The iPad is never going to replace real, physical, paper books for me. I love the smell of new books, the feel of the book (and you can’t read an iPad in the bath without significant risk). That said, it is convenient when travelling and for carrying a few books easily. It’s especially nice when getting books from Amazon. 6 week shipping vs immediate delivery. No contest there.

My choice for best books of the year:

First Lord’s Fury (Codex Alera) by Jim Butcher. This is the climax of the Alera series and definitely the best of the bunch. Fast moving, tense, full of action and altogether an excellent ending for an excellent series. One thing I really like about this one: It doesn’t end with ‘happily ever after’.
Elantris by Brandon Sanderson. This is a bit of a surprise. I got this on sale without too much in the way of expectations. Not to say I don’t like Brandon Sanderson, I’ve enjoyed everything of his that I’ve read, but this was his first published book and so I was willing to give it a little leeway. Not necessary. Good characters (though I’m sure I recognise that headstrong princess from a few places), good plot without too many holes and an intriguing mystery that all comes together logically in a way that leaves you saying ‘But, of course that’s the problem’. Definitely recommend and looking forward to more of his work.
Star Trek: Destiny (trilogy). Yes, I’m recommending Star Trek novels. The world has not ended. I find most Star Trek novels are quickly churned out, mediocre novels. Average writing, average plotting (at best) and usually a reset button to return the universe to the way it was at the end. This trilogy is none of those. The plot works, it’s intertwined over three books and about four time-periods and the crew of at least four ships, and it works. It also leaves the universe dramatically changed (in a way that I did not foresee coming). Finally it’s one of the few time travel tales I’ve read that doesn’t leave me cringing.

Sooo… books per month.

You can almost see from that which months I was travelling or on holiday. June – trip to UK and a few days at leisure. Oct – trip to Pass and lots of time to read while travelling. Nov – Week away in the middle of nowhere.

Lastly, books per genre. Yes, I read a lot of fantasy. (note, these links go to the library pages on this blog, there’s a link to the Amazon page from there)

Science Fiction

The Long Night of Centauri Prime (Babylon 5: Legions of Fire, Book 1) by Peter David
The Light of Other Days by Arthur C. Clarke
Star Trek: Destiny #3: Lost Souls by David Mack
Star Trek: Destiny #2: Mere Mortals by David Mack
A Confederation of Valor (omnibus) by Tanya Huff
Star Trek: Destiny #1: Gods of Night by David Mack
Earthfall (Homecoming) by Orson Scott Card
Deathstalker by Simon R. Green
Childhood’s End by Arthur C. Clarke
Songs of Distant Earth by Arthur C. Clarke
The Time Machine (SF Masterworks) by H. G. Wells
The Call of Earth by Orson Scott Card

Fantasy

Other Fiction

Non-fiction

Tourist time, Amsterdam and Vancouver

Gail — Sun, 27 Feb 2011 17:47:58 +0000

Nothing fancy here, just some of the photos that I took in Amsterdam and Vancouver. Taken with my phone, as I forgot to check the camera battery before I left, and I forgot to pack the charger.

Amsterdam

Amsterdam Central train station

One of the canals downtown Amsterdam

Vancouver

The steam-powered clock

One of the parks around the city.

Entrance to Chinatown

Taken from near the ScienceWorld

Half-frozen fountains Friday afternoon

And snow on Saturday.

And now for a completely inappropriate use of SQL Server

Gail — Tue, 22 Feb 2011 14:30:41 +0000

A while back I wrote up a short introductory overview of Genetic Algorithms. Just for the shear, absolute fun of it, I thought I’d implement a basic genetic algorithm within SQL Server and use it to solve a form of the knapsack problem.

Now first a few comments on this. As the title states, this is not really an appropriate use of SQL Server. Genetic algorithms are generally implemented in languages like Java, C++, C#, etc; languages that are good at complex mathematics, string manipulation and have complex data types. I’m not necessarily using efficient, well-performing methods here, UDFa abound. This is not an article on best practices and well-performing code. I’m also doing no error handling, which I would if this were a real system (in a more suitable language)

Still, doing just for the sake of seeing if it’s possible is all sorts of fun. So, without further ado, the knapsack problem, an approximate solution with genetic algorithms in SQL Server. (For the record, this is a multi-constrained, bounded knapsack problem)

The scenario

There’s a bag that has a maximum volume that it can hold and a maximum mass that it can hold (and we assume that we can pack perfectly with no wasted space). There are eight items, all with different masses, different volumes and different values. The goal here is to maximise the total value of all the items contained within the bag.

CREATE TABLE ObjectStatistics (
  ObjectNumber TINYINT NOT NULL,
  Mass NUMERIC(4,2) NOT NULL,
  Volume NUMERIC(4,2) NOT NULL,
  Value NUMERIC(4,2) NOT NULL,
  NumberAvailable TINYINT NOT NULL,
  CONSTRAINT pk_ObjectStats PRIMARY KEY CLUSTERED (ObjectNumber)
);

CREATE TABLE BagStatistics (
  MaxMass NUMERIC(5,2),
  MaxVolume NUMERIC(5,2)
);

INSERT INTO dbo.ObjectStatistics (ObjectNumber, Mass, Volume, Value, NumberAvailable)
VALUES
  (1,0.5,0.45,2,50),
  (2,1.5,0.25,20,10),
  (3,0.1,1.25,8,150),
  (4,0.25,0.1,1,250),
  (5,0.67,0.3,6,100),
  (6,0.34,0.75,18,5),
  (7,1.25,0.25,5,40),
  (8,1.1,0.8,10,25);

INSERT INTO dbo.BagStatistics (MaxMass, MaxVolume)
VALUES  (100, 75);

Those two tables set up the constraints for the scenario, the maximum mass and volume for the bag and the mass, volume, value and maximum number available for each of the items.

The setup

Next we need a way to store the potential solutions (referred to as ‘individuals’). I decided on two tables (pretending to be doing proper DB design here). Keeping everything in one table would also have worked, it would just have required a different implementation later on.

CREATE TABLE Individuals (
  Generation SMALLINT,
  IndividualID SMALLINT,
  CONSTRAINT pk_Individuals PRIMARY KEY CLUSTERED (Generation, IndividualID)
);

CREATE TABLE IndividualDetails (
  Generation SMALLINT,
  IndividualID SMALLINT,
  ObjectNumber TINYINT,
  Quantity TINYINT,
  CONSTRAINT pk_IndividualDetails PRIMARY KEY CLUSTERED (Generation, IndividualID, ObjectNumber),
  CONSTRAINT fk_IndividualDetails_Individuals FOREIGN KEY (Generation, IndividualID) REFERENCES dbo.Individuals (Generation, IndividualID)
);

I’m going to add two more tables, one to make the processing easier (it’ll become clear later where this is used) and one to store some results. Also a view to get around the ‘cannot use RAND inside a UDF’ limitation.

CREATE TABLE Numbers (
  Value TINYINT PRIMARY KEY,
  BinaryRepresentation CHAR(8) UNIQUE
);

CREATE TABLE GenerationStatistics (
  Generation SMALLINT,
  MaxFitness NUMERIC(8,2),
  AvgFitness NUMERIC(8,2)
);
GO

CREATE VIEW RandomNumber
AS
  SELECT RAND(CHECKSUM(NEWID())) AS Random;
GO

That’s the basics out of the way. There are two important things I need to decide on straightaway.

The first is an encoding of the individual’s properties (in this case object quantities) so that the genetic algorithm can manipulate them. I’m going to use a binary representation – convert the quantity for each object into binary and concatenate the strings together. 8 bits per object.

The second is a fitness function, a way to tell which individuals are the best. In this case it’ll be a simple sum of the values of all the items, taking into account the max mass, max volume and max quantity for each.

I decided to implement both of these as scalar UDFs (yes, shock and horror, I’m using data-accessing scalar UDFs) and make them calculated columns of the Individuals table.

The integer-to-binary uses a slightly modified version of a function written by Mitch Wheat, available on his blog – http://mitch-wheat.blogspot.com/2006/10/t-sql-way-converting-integers-to.html. Thanks Mitch

-- Originally from http://mitch-wheat.blogspot.com/2006/10/t-sql-way-converting-integers-to.html
CREATE FUNCTION IntegerToBinary(@intval int)
RETURNS char(8)
AS
BEGIN
 DECLARE @bincode char(64);

 SET @bincode = '0000000100100011010001010110011110001001101010111100110111101111';

 RETURN (
   SUBSTRING( @bincode, ((@IntVal / 16) & 15) * 4 + 1 , 4 ) +
   SUBSTRING( @bincode, (@IntVal & 15) * 4 + 1 , 4 )
 );
END
GO

CREATE FUNCTION dbo.ComputeBinaryString(@Generation SMALLINT, @Individual SMALLINT)
RETURNS CHAR(64)
AS
BEGIN
 DECLARE @BinaryString VARCHAR(64) = '';

 SELECT @BinaryString = @BinaryString + dbo.IntegerToBinary(Quantity)
   FROM dbo.IndividualDetails
   WHERE Generation = @Generation AND IndividualID = @Individual
   ORDER BY ObjectNumber;

 RETURN @BinaryString;
END;
GO

CREATE FUNCTION dbo.Fitness(@Generation SMALLINT, @Individual SMALLINT)
RETURNS NUMERIC(6,2)
AS
BEGIN
 DECLARE @MaxMass NUMERIC(5,2), @MaxVolume NUMERIC(5,2);
 SELECT @MaxMass = MaxMass, @MaxVolume = MaxVolume
   FROM dbo.BagStatistics;

 DECLARE @TotalValue NUMERIC(6,2) = 0;

 SELECT @TotalValue = SUM(id.Quantity*os.Value)
   FROM dbo.IndividualDetails id
   INNER JOIN dbo.ObjectStatistics os ON id.ObjectNumber = os.ObjectNumber
   WHERE Generation = @Generation AND IndividualID = @Individual
   HAVING SUM(id.Quantity*os.Mass) <= @MaxMass
     AND SUM(id.Quantity*os.Volume) <= @MaxVolume
     AND SUM(CASE WHEN id.Quantity>os.NumberAvailable THEN 1 ELSE 0 END) = 0;

 RETURN @TotalValue;
END;
GO

ALTER TABLE dbo.Individuals
ADD BinaryString AS dbo.ComputeBinaryString (Generation, IndividualID),
    Fitness AS dbo.Fitness(Generation, IndividualID);
GO

At this point I have a way to store the individual in each generation, a way to represent them in order for the genetic algorithm to process them and a way to calculate the fitness. Good to go.

The framework

I now need to implement the genetic algorithm itself. The main components that I need to implement are the crossover and the mutation and then an evolution procedure to generate new individuals from the current ones.

I’m going to again use functions for the crossover and mutation, for convenience and easy coding. A multi-statement table-valued UDF for the crossover and a scalar UDF for the mutation. (Performance? What performance?)

The crossover’s fairly simple. Take 2 binary strings as parameters. Pick a random point along the string and cut the two strings at that point. Create two new strings from the first part of the first input string concatenated with the second part of the second input string and from the first part of the second input string concatenated with the second part of the first input string.

If I started with ’00000000′ and ’11111111′ and picked 5 as the crossover point, afterwards I’d get ’00000111′ and ’11111000′

CREATE FUNCTION dbo.Crossover(@BinaryString1 CHAR(64), @BinaryString2 CHAR(64))
  RETURNS @NewIndividuals TABLE (CrossoverPoint TINYINT, BinaryString CHAR(64))
AS
BEGIN

 -- where in the string is the crossover
 DECLARE @CrossoverPoint TINYINT;
 SELECT @CrossoverPoint = FLOOR(Random*64)+1
   FROM RandomNumber;

 INSERT INTO @NewIndividuals
 SELECT @CrossoverPoint, SUBSTRING(@BinaryString1,0,@CrossoverPoint) + SUBSTRING(@BinaryString2,@CrossoverPoint, 64-@CrossoverPoint+1)
 UNION ALL
 SELECT @CrossoverPoint, SUBSTRING(@BinaryString2,0,@CrossoverPoint) + SUBSTRING(@BinaryString1,@CrossoverPoint, 64-@CrossoverPoint+1);

 RETURN;

END
GO

The mutation’s not much more complex. Take a single binary string as input. Pick a random position along the string. Flip that bit (0->1; 1->0). Return the modified string

CREATE FUNCTION dbo.ApplyMutation(@BinaryString CHAR(64))
RETURNS CHAR(64)
AS
BEGIN

DECLARE @MutationLocation TINYINT, @SwappedBit CHAR(1);

-- Which bit are we toggling?
SELECT @MutationLocation = FLOOR(Random*64)+1
  FROM RandomNumber;

-- simple toggle for the chosen bit
SET @SwappedBit = 1 - CAST(SUBSTRING(@BinaryString, @MutationLocation,1) AS BIT);

-- and stuff it back in
SET @BinaryString = STUFF(@BinaryString,@MutationLocation,1,@SwappedBit);

RETURN @BinaryString;
END
GO

One thing to note at this point is that neither the crossover nor the mutation has any dependency on the details of the problem being solved. This is intentional, and is generally how genetic algorithms work. With the details encoded properly (in this case into a binary string), the crossover and mutation should be able to work just on the encoded string without needing any knowledge of the actual problem.

Last thing needed here is something that will take the existing individuals, apply the crossover and mutation and return new individuals. This time I think a stored procedure would be best.

CREATE PROCEDURE Evolve (@CurrentGeneration INT)
AS
BEGIN

CREATE TABLE #SelectedIndividuals (
  BinaryString CHAR(64),
  CrossoverGroup TINYINT
);

-- use the fittest half of the current population to evolve the next generation
-- ensures that the low fitness individuals are eliminated (survival of the fittest FTW)
INSERT INTO #SelectedIndividuals (BinaryString, CrossoverGroup)
SELECT BinaryString, NTILE(2) OVER (ORDER BY (SELECT NEWID())) AS CrossoverGroup
  FROM (
    SELECT TOP (50) PERCENT BinaryString
      FROM Individuals
      ORDER BY Fitness DESC
  ) sub;

-- Join so that I have the two binary strings I'm crossing over in the same row.
-- The function then does the cross over and the mutation applies to the result
SELECT @CurrentGeneration AS Generation, ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS IndividualID, dbo.ApplyMutation(co.BinaryString) AS NewIndividual
 FROM (
    SELECT BinaryString, ROW_NUMBER() OVER (ORDER BY (SELECT NEWID())) AS PartnerNo
      FROM #SelectedIndividuals
      WHERE CrossoverGroup = 1
    ) AS Group1
  INNER JOIN (
    SELECT BinaryString, ROW_NUMBER() OVER (ORDER BY (SELECT NEWID())) AS PartnerNo
      FROM #SelectedIndividuals
      WHERE CrossoverGroup = 2
    ) AS Group2
    ON Group1.PartnerNo = Group2.PartnerNo
   CROSS APPLY dbo.CrossOver(Group1.BinaryString, Group2.BinaryString) co;
END;
GO

That’s all that’s needed for the GA framework.

The solution

What’s needed for the solution is a loop to run through the generations one by one, a way to decode the individual returned by the Evolve procedure and some housekeeping.

First off, I need to fill the Numbers table.

INSERT INTO dbo.Numbers (Value, BinaryRepresentation)
SELECT Value, dbo.IntegerToBinary(Value)
  FROM (
    SELECT TOP (256) Row_Number() OVER (ORDER BY column_id)-1 AS Value FROM sys.columns
  ) sub;
GO

The setup for the execution loop involves declaring a few variable, a temp table and setting up the initial generation (randomly).

SET NOCOUNT ON
DECLARE @CurrentGeneration SMALLINT = 0;
DECLARE @TotalGenerations SMALLINT = 50;
DECLARE @GenerationSize SMALLINT = 64; -- keep to powers of 2, and everything will work...

-- Create initial generation
-- First the individuals
INSERT INTO Individuals (Generation, IndividualID)
SELECT TOP(@GenerationSize) @CurrentGeneration AS Generation, ROW_NUMBER() OVER (ORDER BY column_id) AS IndividualID
  FROM sys.columns;

-- Now the details
INSERT INTO IndividualDetails (Generation, IndividualID, ObjectNumber, Quantity)
SELECT Generation, IndividualID, ObjectNumber, CEILING(RAND(CHECKSUM(NEWID()))*NumberAvailable/4)
  FROM Individuals
    CROSS JOIN ObjectStatistics;

-- Initial generation statistics
INSERT INTO GenerationStatistics (Generation, MaxFitness, AvgFitness)
SELECT @CurrentGeneration, MAX(Fitness), AVG(Fitness)
FROM Individuals;

-- placeholder table for new generations
CREATE TABLE #NewIndividuals (
  Generation SMALLINT,
  IndividualID SmallINT,
  BinaryString CHAR(64)
);

The processing loop itself is nothing complex. Insert the results of the Evolve procedure into a temp table. From the temp table insert into the Individuals, decode the binary string and insert into IndividualDetails, then delete the lower-fitness individuals so that the population remains at a constant size.

In this case I’ve decided just to run for a fixed number of iterations. Often genetic algorithms would terminate after a certain number of iterations or if the max fitness didn’t improve by more than a certain threshold. For simplicity I’ll leave that out. It’s trivial to implement if anyone wants to experiment further.

WHILE (@CurrentGeneration<@TotalGenerations)
 BEGIN

   SET @CurrentGeneration+=1;

 -- Generate the new generation
   INSERT INTO #NewIndividuals (Generation, IndividualID, BinaryString)
   EXEC Evolve @CurrentGeneration;

 -- store the new individuals
   INSERT INTO Individuals (Generation, IndividualID)
   SELECT Generation, IndividualID
     FROM #NewIndividuals;

 -- chop binary string, insert into details table.
   INSERT INTO IndividualDetails (Generation, IndividualID, ObjectNumber, Quantity)
   SELECT Generation, IndividualID, Allele, Value
     FROM (
       SELECT Generation, IndividualID, Value+1 AS Allele, SUBSTRING(BinaryString, 1 + (Value * 8), 8 ) AS BinaryString
         FROM #NewIndividuals
           CROSS JOIN Numbers
         WHERE Value < 8
     ) sub
       INNER JOIN Numbers ON sub.BinaryString = Numbers.BinaryRepresentation;

 -- delete all but the top (@GenerationSize) fittest individuals. Keep fixed population size.
   DELETE FROM IndividualDetails
     FROM IndividualDetails LEFT OUTER JOIN (
       SELECT TOP (@GenerationSize) Generation, IndividualID
         FROM Individuals
         ORDER BY fitness DESC
     ) Chosen ON IndividualDetails.Generation = Chosen.Generation AND IndividualDetails.IndividualID = Chosen.IndividualID
     WHERE Chosen.IndividualID IS NULL;

   DELETE FROM Individuals WHERE BinaryString = ''; -- only possible when there are no details

 -- Compute current max fitness and current avg fitness
   INSERT INTO GenerationStatistics (Generation, MaxFitness, AvgFitness)
   SELECT @CurrentGeneration, MAX(Fitness), AVG(Fitness)
     FROM Individuals;

 -- cleanup
 TRUNCATE TABLE #NewIndividuals;
 END -- while
DROP TABLE #NewIndividuals

And that’s pretty much all there is to it. All that’s left is to see how well the genetic algorithm did and how good the best solution it found really is.

Please note that this entire solution is driven by random numbers, if you run it you will get different results to what I did.

SELECT TOP (1) i.Generation, i.IndividualID, TotalMass, TotalVolume, TotalValue
  FROM dbo.Individuals AS i
    INNER JOIN (
      SELECT Generation, IndividualID, SUM(Quantity*Value) AS TotalValue,  SUM(Quantity*Mass) AS TotalMass, SUM(Quantity*Volume) AS TotalVolume
        FROM dbo.IndividualDetails d INNER JOIN dbo.ObjectStatistics AS os ON d.ObjectNumber = os.ObjectNumber
        GROUP BY Generation, IndividualID
    ) id ON i.Generation = id.Generation AND i.IndividualID = id.IndividualID
  ORDER BY Fitness DESC;

Not half bad.Is it the absolute best? Probably not. Genetic algorithms do not guarantee that they will find the best solution, just that they will find good ones

If we look at how the fitness (total value) improved over the generations it shows a massive initial increase in fitness then slow refinements. I could have run only 10 generations and still had a reasonably good result.

I think that’s enough playing with AI for now. While I doubt there’s any real practical use, I hope it was at least entertaining.

Full code: GeneticAlgorithms

The books of the year

Gail — Fri, 31 Dec 2010 23:35:02 +0000

At the beginning of the year I set myself a goal of 50 books. I managed 45, which isn’t bad considering there were a couple of months where I was re-reading old favourites only.

I’m not going to go into the level of detail Paul Randal did, rather I’m just list the books read, the genre and give my top 3 of the year.

Top 3:

Princeps’ Fury: Book Five of the Codex Alera by Jim Butcher I’m adoring the entire Codex Alara series. Well written, strong character, interesting form of magic and looks like a climactic ending. The last book should be in my post box in a couple of weeks.
Sabriel (Abhorsen) by Garth Nix Actually the entire series belongs here. They’re apparently teenage books, but they’re complex and deal with adult themes.
In Search of Schrödinger’s Cat: Quantum Physics and Reality by John Gribbin Exceptional take on the birth of quantum physics, minimal to no maths skills required. It’s written for the layman and is part history, part physics.

Complete list:

Fantasy:

Science Fiction

Other fiction

The Templar Legacy by Steve Berry
Stray by A.N. Wilson
Rainbow Six by Tom Clancy
The Teeth Of The Tiger (Jack Ryan) by Tom Clancy
Wolf of the Plains by Conn Iggulden
Night Probe! by Clive Cussler
Valhalla Rising (Dirk Pitt Adventure) by Clive Cussler

Non-fiction

Goal for 2011… 50 books.

Looking back

Gail — Fri, 31 Dec 2010 22:46:12 +0000

2010 has been an ‘interesting’ year. Interesting in the Chinese curse sense of the word mostly.

I set a bunch of goals back at the end of 2009, but to be quite honest I’m not even going to look at them. If I achieved even one it was more chance than anything. Still there have been a couple of positive aspects to the year.

Enthusiasm for SQL-related stuff (and pretty much everything else) was finally relocated. I wrote 3 blog posts this last week, and it didn’t feel like a chore for the first time in many months.
For the first time since I moved to consulting, my income comfortably exceeds my expenses. Moving to consulting at the start of an economic downturn was probably a stupid thing to do, but I never do things the easy way.
Got the hardest part of my Masters thesis out of the way, all that’s really left is a bit more data collection and writing up.

So, some goals for the new year…

Finish the thesis. This isn’t optional. I’ve committed to handing it in by June.
Aikido at least once a week and archery at least one Saturday a month. I’ve skipped far too many classes and practices these last few months.
Attempt the first MCM test. This probably won’t be until the latter part of the year. The Master’s thesis comes first
Get back into computer graphics. Old hobby that’s been mostly neglected, but that I still enjoy

No blog-related goals or article quotas. I intend to write a fair bit, but I don’t want to set requirements.

One last thing, for anyone wondering about the computer graphics, these are some work-in-progress or finished pieces.

On burnout and recovery

Gail — Thu, 09 Dec 2010 14:00:07 +0000

I suspect for a fair number burnout is something that they believe happens to others. People with less fulfilling jobs, people who don’t love their work, people in poorly managed companies. Other people.

i certainly didn’t consider myself at risk for burnout. I loved my work, I had a great manager prior to moving to consulting and was pretty much my own manager afterwards. I had a hobby (or 5), practiced a sport, took time to play games and relax. At risk? Never!

At least that’s what I thought until sometime around May last year when I realised I had absolutely no enthusiasm for anything related to IT. Blogging was a chore, books held no interest and many a day I opened management studio intending to do some development or investigation and then several hours later I’d close it without a line of code having being written.

The biggest mistake (I think) was in how I handled it. Instead of taking a break (and work was quiet around that time so I could), I took on more commitments, thinking that it would help motivate. Exceptional DBA Awards. SSC Articles. Tech-Ed presentations. PASS Summit presentations. SQL Exams. Hell I even started discussing and planning to write a book.

Bad idea.

Far from acting as motivation, the long list of things to do just made things worse. Far worse. I now had hard deadlines, people chasing me and still no motivation or enthusiasm. Naturally everything came to a head at the worst possible time and place – the PASS summit in Seattle that year. It’s really hard to write and deliver speeches, prep and deliver presentations and generally act friendly and enthusiastic when the only thing that you want to do is get on the next flight and go home. Fortunately I think only a couple of people noticed.

It’s only in the last couple of months (October/November 2010) that the enthusiasm for writing, blogging, researching (and in fact anything other than sitting and watching the world go by) has partially returned. Is the burnout past? No, definitely not, I still have days (weeks) where I can’t summon the enthusiasm to care, but it is getting better.

But this isn’t just a post on history or a poor attempt at sympathy. The point is how I handled it (badly). Looking back, what I should have done was

Take a break (work was quiet at the time). Not a weekend crashed in front of the TV, but a proper break – out of town for a week or so.
Get some help from friends, rather than pushing them away and pretending everything was fine.
Try a new technology rather than piling on SQL stuff. WCF, Ruby, F#, anything as long as it was different.

I’m far from qualified to offer advice on this issue, but I would suggest to anyone feeling the same way, don’t ignore it, don’t try to work through it and don’t hide it. Few problems go away by themselves, burnout certainly doesn’t.

Genetic Algorithms

Gail — Thu, 28 Jan 2010 14:00:59 +0000

Warning. This post has absolutely nothing to do with SQL Server.

What are Genetic Algorithms?

Genetic algorithms are a form of evolutionary computation, a branch of artificial intelligence that focuses on evolving effective or optimal solutions to difficult problems, based on the biological theory of evolution.

Genetic algorithms are, at their core, a search/optimisation technique. They are a way of finding maximum/minimum solutions to problems and, can be effective when there is no algorithmic solution to the problem. An example here would be the ‘Travelling Salesman’ problem.

Genetic algorithms work by taking an initial population of potential solutions (referred to as individuals), selecting a subset of the population that has the highest fitness then using that subset to generate a second generation. From the second generation again a subset with the highest fitness is selected and used to generate a third generation. This repeats until either the ‘fittest’ individual is considered a good enough solution, or until a certain number of generations have passed.

There are advantage to using genetic algorithms to solve problems over more traditional methods like hill climbing.

Genetic algorithms can quickly produce good solutions though they may take a lot of time to find the best solution. This is a benefit when the problem is such that the absolute best solution is not necessary, just one that is ‘good enough’
They are not susceptible to getting trapped by local maxima.
They do not work on the entire search space one potential solution at a time, but rather work on populations of potential solutions, focusing towards more optimal areas of the search space.

A genetic algorithm will almost always find an optimal solution, given enough time. The main downside is that they may take a lot of time to find that optimal solution.

Components of a Genetic Algorithm

There are two main critical parts in setting up a genetic algorithm for a problem.

The encoding of the potential solutions into a form where they can be operated on.
The fitness function which defines which individuals are better than others, which are closer to the maximum that is being searched for.

Most of the design work when using genetic algorithms goes into those two problems.

Encoding

Encoding is the process of taking all the values that make up a potential solution and turning them into a form that the genetic algorithm can operate on.

The selection of an encoding is of utmost importance to the effectiveness of the entire process and a poor representation can make the entire problem much harder than it should. Unfortunately there has been little academic work done on the process of designing representations.

Often for genetic algorithms, the end result of the encoding will be a binary string. There are other variations of evolutionary computation that use other representations, from the arrays of real numbers used by evolutionary strategies to the code trees used by genetic programming.

Fitness function

Depending on the problem, the fitness function can be trivial to write or near-impossible. The design of the fitness function is completely based on the problem that is being solved.

There are two important considerations for a fitness function.

It must be deterministic.
It must be fast

If the fitness of an individual is assessed twice, it must come to the same value¹. If the fitness function could return different values for the same individual, then it is of no use in determining the fittest individuals in the population and hence the genetic algorithm will not be able to identify the best solution to the problem.

The fitness of each individual is assessed at least once in each generation. The calculation of the fitness function is usually the most time consuming part of the entire process and the longer the fitness function takes to run, the longer the entire process is run

(1) There are cases where the fitness of an individual may depend on external factors which change over time. Hence a fitness function may give different values for one individual if calculated at different times. Genetic algorithms in a changing environment are a little beyond the scope of this entry.

Evolution Process

In order to create a new generation, the fittest individuals from the previous generation are taken and used to generate the next generation. There are two main operators that are used to generate a generation from the previous one. Crossover and mutation.

Crossover

Crossover involved taking two individuals, splitting each one’s encoded string and swapping parts to generate two new individuals.

Say we had two individuals with the following encoded strings (spaces added for clarity)
0000 0001 1111 1110
0101 1010 1100 0011
and we chose the splitting point for the crossover after the 4th bit, the resulting strings after the crossover will be
0000 1010 1100 0011
0101 0001 1111 1110

In genetic algorithms crossover is the primary operator used. What I described here was a single crossover. There are a number of other variations that can be used.

Mutation

Mutation is an operator applied to a single individual. It’s usually applied after crossover has generated new individuals. Mutation involves flipping a single bit somewhere in the encoded string.

Let’s take the two individuals that were generates by the crossover earlier and apply a random mutation to each
0000 1010 1100 0011
0101 0001 1111 1110
After
0000 1010 1101 0011
0101 0001 1011 1110

In genetic algorithms mutation is very seldom applied and only a small percentage of individuals in a generation will be affected by the mutation operator.

Example

As a quick example let’s manually evolve a simple function to see how the whole thing works.

Let’s say I have an array of 4 numbers (call it num) between 0 and 15. I want to know what values give me the best value for the following.

num[1]-num[2]-num[3]+num[4]

I know, that’s simple enough that we could work out the optimal solution just by eye. Not the point. This is enough to do a quick and effective demo with.

I’m going to encode that by simply converting the numbers in the array to binary and concatenating the binary representations of the 4 numbers (spaces just added for clarity). The fitness function is already defined. I’m going to start with an initial population of eight individuals.

1111 1111 1100 1110 – fitness = (15-15-12+14) = 2
0101 1010 1100 0011 – fitness = (9-10-12+3) = –10
1011 0111 0011 1111 – fitness = (11-7-3+15) = 16
1111 1001 1010 0011 – fitness = (15-9-10+3) = -1
1010 1010 1010 1010 – fitness = (10-10-10+10) = 0
1000 0010 0111 0110 – fitness = (8-2-7+6) = 5
0000 0001 1111 1110 – fitness = (0-1-15+14) = -2
1010 0101 0010 0101 – fitness = (10-5-2+5) = 8

From this I’m going to take the 4 individuals with the highest fitness, use crossover operations (with the crossover point exactly in the middle) between them until I have 8 individuals for the 2nd generation and then apply a single bit mutation to one of the individuals (detailed steps left as an exercise for the reader)

1111 1111 0011 1111 – fitness = (15-15-3+15) = 12
1011 0111 1100 1110 – fitness = (11-7-12+14) = 6
1111 1011 0010 0101 – fitness = (15-11-2+5) = 7
1010 0101 1100 1110 – fitness = (10-5-7+6) = 4
1011 0111 0111 0110 – fitness = (11-7-7+6) = 3
1000 0010 0011 1111 – fitness = (8-2-3+15) = 18
1010 0101 0111 0110 – fitness = (10-5-7+6) = 4
1000 0010 0010 0101 – fitness = (8-2-2+5) = 9

We can already see an improvement. The average and maximum fitness is much higher than for the first generation. I’ll do one more generation in this example, again taking the 4 fittest individuals, crossing over to generate 8 new individuals and then applying a single bit mutation to two individuals. This time however, the crossover point will between the 4th and 5th bit.

1000 1111 0011 1111 – fitness = (8-15-3+15) = 5
1111 0010 0011 1111 – fitness = (15-2-3+15) = 25
1000 1011 0010 0101 – fitness = (8-11-2+5) = 0
1111 0010 0011 1111 – fitness = (15-2-3+15) = 25
1111 0010 1010 0101 – fitness = (15-2-10+5) = 8
1000 0111 0011 1111 – fitness = (8-7-3+15) = 13
1111 0010 0010 0101 – fitness = (15-2-2+5) = 16
1000 1011 0010 0101 – fitness = (8-11-2+5) = 0

I think that’s enough for this example. We’re getting fairly close to the best possible solution (15,0,0,15), close enough to see how this works. The population size was very low, that’s why there are duplicates appearing in the results of the crossover. With a larger search space there would be a lot more diversity.

I hope that anyone still reading found this brief diversion into the realms of AI interesting. The regular SQL-related posts will return soon.

Great start to the new year. Not!

Gail — Sun, 04 Jan 2009 10:32:20 +0000

My flat got broken into last night. Cell phone, laptop, camera and wallet all gone.

Yay. Not!

Edit: And my iPod. And an ornamental knife I bought in Scotland a few year back.

Stepping down

Gail — Tue, 03 Jun 2008 21:35:55 +0000

Sunday saw the last game (for now) of the d20 Modern campaign I’m running. I’m going to miss DMing, but it was for the best. I’m struggling with a bit of burnout and a lack of free time.

The campaign went quite well. The players seemed to enjoy it and that’s the only real measure of any importance. We didn’t get quite as far as I had hoped, but a lot of hints and clues were dropped during the various adventures and I think that all the players have a better idea of the ‘big picture’. They’ve also managed to disrupt the big bad guy’s plans often enough to be irritating.

For the next few months at least, we’ll be playing high-fantasy D&D as we return to the world of Per-rune. Currently our characters are about to disembark ship after a rather exciting trip to the Elven kingdom. All we need is to do some shopping for winter gear and then head inland back to the small town of Kurat, where the whole thing started. Easy, right?

Birthday musings

Gail — Fri, 07 Dec 2007 19:45:11 +0000

Another year gone, another year older. It’s been an interesting year.

Professionally, I’m very happy with the past year. I’ve achieved several goals, got to know some interesting people in the SQL community and, of course, started this blog. I’ve also made some fairly important decisions about the future.

Academically, I’m not so happy. My masters thesis is behind where it should be. (a fact I’m sure my supervisor would agree with.) Most of the initial reading is done and I know what I need to do. I need to sit down and get busy. This is going to have to take a higher priority in the next few months.

On more personal matters, a good friend moved up to JHB. It’s very good to see her again. The roleplaying campaign is going well and from the feedback I’ve been having, the players are all enjoying themselves.

Overall, it’s been a good year. Here’s to the next one.