Something that come to mind is the new threads added in InnoDB. To bad they show up first in 5.6.2+. We would have needed them now.. But isn’t that always the case? We are facing the trouble of grooming a table from loads of historical data. It’s running InnoDB, so we all know it’s kinda expensive to do massive deletes from it. Especially if you’re not doing it on your PK. We were debating how to setup a groomer job on a table like this, and no matter how we do it, the only feasible way is to delete by PK. Massive data deletes will still cause for performance degradation when the pages become to dirty. This has of course changed in the new additions to InnoDB in 5.6.2+. Deleting by PK is still the fastest way, and will be faster when we one day switch to 5.6.x. Read more about the new InnoDB stuff here: InnoDB Page Cleaner Thread

It was for sure a very rewarding conference, but one may debate the absence of MySQL itself (Oracle). Normally there is a lot going on, and a lot of announcements from MySQL. In this case Oracle owns MySQL. I don’t know if they are trying to make a statement, not showing up and providing information to the community. Or are they to self absorbed, and want to pull Open database people to join on Oracle World? I don’t know, and I really hope it’s not the latter. They seem to put a lot of money and effort into continued development of MySQL and InnoDB.

Monty did a bald move in his keynote, and later had to appologize in a public blog post: monty says. Me and Tobias Asplund had a decent chat with him at the SkySQL dinner, and there is a lot of new interesting stuff happening in MariaDB. A lot of effort has been put into subquery optimization and the execution plans are now better than ever. Hopefully Oracle will not be too scared to adopt those patches. We will have to try MariaDB here at Marin to see if we’ll get any performance improvements running subqueries. It’s not a production test just to calm those worried souls.

I enjoyed Yoshinori Matsunobo’s Tutorial. There was a lot of refreshing material, but also some new things I have not been able to test before. Good stuff! Check out Matsunobos blog

I also enjoyed many of the facebook talks. It’s very giving to see how they attack some of the problems others never see. The amount of servers and data those guys are handling is insane. Their data drift spotting stuff was awesome as well.

I am surprised by the amount of people looking for DBA’s or data architects. Almost all talks or keynotes said they were hiring.

All in all, it was a great conference. I am almost looking forward to the next one.

Scenario:
You have a huge table with constant activity, containing terabytes of data. You need to update/delete roughly a million random rows. Random selects from the table keep some indexes you could use hot. Running a big update, locking rows/table is not an option since you have approx. 40-50 inserts/sec to the table. So you need to run smaller batches when updating table. LIMIT would be nice, but is not supported with OFFSET when using UPDATE or DELETE.

Updating using IN and subquery together with limit is not supported in 5.0.x (usure about 5.1.x and forward), Else this would have been a viable solution.

Selects with specific where clauses might take longer due to some indexes/keys being too big as well

Solution:
Selecting all Unique/Primary keys to a temp table with an auto increment primary key and then update the production table using this temporary table as a reference table to which rows to update, with a BETWEEN clause on the temporary table’s auto_increment field.
May also enclose everything in a transaction and delete the same ‘between’ on the temporary reference table. Will help you keep track of what you actually did in case you need to abort this.

Example
Production table:

CREATE TABLE `huge_table` (
`request_id` bigint(20) NOT NULL auto_increment,
`customer_id` int(11) default NULL,
`data_id` bigint(20) default NULL,
`group_id` int(11) default NULL,
`user_id` int(11) default NULL,
`action_id` int(11) default NULL,
`external_id` int(11) default NULL,
`user_data` varchar(2048) default NULL,
`entry` text,
`extra_term` varchar(255) default NULL,
`transaction` text,
`receive_time` datetime default NULL,
`from_id` varchar(32) default NULL,
`url` varchar(2048) default NULL,
`from_ip` varchar(2048) default NULL,
`useragent` varchar(2048) default NULL,
`tz` int(11) default NULL,
`cvalue` varchar(255) default NULL,
`rvalue` varchar(255) default NULL,
`uid` varchar(40) default NULL,
PRIMARY KEY (`request_id`),
KEY `k1` (`receive_time`,`action_id`,`url`,`from_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1047142423 DEFAULT CHARSET=utf8

You need to update action_id = 100 on approx 1.5m rows which have receive_time between ’2010-08-01 00:00:00′ and ’2010-08-31 23:59:59′ where from_id equals ‘q7b4x5aa0303erer’.

temporary table:

CREATE TABLE `tmp_ids` (
`id` int(11) NOT NULL auto_increment,
`prod_id` bigint(20) default NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=15424160 DEFAULT CHARSET=utf8

Fill the temporary table with the id’s you need to update in the production table:

INSERT INTO tmp_ids SELECT * FROM huge_table USE INDEX (k1) WHERE from_id=’q7b4x5aa0303erer’ AND receive_time BETWEEN ’2010-08-01 00:00:00′ AND ’2010-08-31 23:59:59′;

Then just iterate this in a bash script or our favorite scripting language (increasing the between values of course):

UPDATE huge_table ht JOIN tmp_ids ti ON ht.request_id=ti.prod_id SET ht.action_id=100 WHERE ti.id BETWEEN x AND x;

example bash script:

#!/bin/bash

# Config params
user=”username”
pass=”password”
host=”hostname”
db=”schema_name”
tmptbl=”tmp_table”
livetbl=”live_table”
tmpcol=”live_id”
livecol=”request_id”
tmpkey=”id”
upcol=”from_id=1″
waittime=”0.5″

# Print usage function
function usage() {
echo “$(basename $0) ”
echo “rows – number of rows in one go.”
exit 0
}

# Since we need atleast one variable to continue, check so first variable is supplied
[ -e $1 ] && usage && exit

# get that variable
rows=$1
# Check which is the first key we will use in temp table
s=`mysql -N -u$user -p$pass -h$host $db -e”select min(id) from $db.$tmptbl;”`
# Set latter between value
let b=$s+$rows;
# Count how many rows we have to update so we know how long to loop
count=`mysql -N -u$user -p$pass -h$host $db -e”select count(*) from $db.$tmptbl;”|head`;

# Print starting line
echo “Starting up with updates from id: $s to $b, performing $rows at one go..”

# loop till we are done
while [ 1 ];
do

# are we done? If so exit
if (($count <= 0));
then
exit
fi

# Run actual db updates
echo -n "updating $livetbl between $s and $b.."
mysql -N -u$user -p$pass -h$host $db -e"update $livetbl td join $tmptbl i on td.$livecol=i.$tmpcol set $setval where i.$tmpkey between $s and $b"
echo -n " deleting in $tmptbl.."
mysql -N -u$user -p$pass -h$host $db -e"delete from $db.$tmptbl where $tmpkey between $s and $b"
echo -n " Done!"

# set between rows
let s=$s+$rows
let b=$b+$rows
let count=$count-$rows

# wait if there are more records to update
if (($count > 0));
then
echo ” ..waiting.. still $count records to update”;
sleep $waittime
fi
done

# All done
echo “Script done. No more records to update”

As you may already know i will be holding a session together with Anders Karlsson from MySQL/SUN about MySQL HA. We will walk through different solutions and mention pros and cons of them. I will also present how we solved the HA situation at bwin games ab.

Those of you who still have not booked their tickets and entry to MySQL UC, i have a surprise for you! Book your entry now and use this promotion code “mys09fsp” and you will get 20% off your total cost! Isn’t that awesome? Just enter the code at the signup/booking page and you will receive the discount.

Looking forward to see you guys at MySQL UC in Santa Clara. Write me a email if you are going and we’ll try to hook up. Probably there will be a “nordic night” as last year. If you are from around the Nordic/Scandinavian region, make sure to join us. We’ll probaly be going out and eat something nice and just talk random stuff.

So, what should you do now if you didn’t already do it? Go to http://www.mysqlconf.com and register for this years MySQL User Conference in Santa Clara / California.

Cheers

We will do a walkthrough of technologies that can be used, pros, cons, common pitfalls and how we solved it at bwin games ab (company i work at).

I will post more information about this later. Just wanted to share my excitement.

The whole California visit was not only work. I managed to squeeze in a week of vacation to it. Traveled around NoCal, did some shopping and met some friends.

Edit, i found a recording of Dick Hardts presentation:http://blip.tv/file/981411

Will try to keep something here updated on the way…