Something that come to mind is the new threads added in InnoDB. To bad they show up first in 5.6.2+. We would have needed them now.. But isn’t that always the case? We are facing the trouble of grooming a table from loads of historical data. It’s running InnoDB, so we all know it’s kinda expensive to do massive deletes from it. Especially if you’re not doing it on your PK. We were debating how to setup a groomer job on a table like this, and no matter how we do it, the only feasible way is to delete by PK. Massive data deletes will still cause for performance degradation when the pages become to dirty. This has of course changed in the new additions to InnoDB in 5.6.2+. Deleting by PK is still the fastest way, and will be faster when we one day switch to 5.6.x. Read more about the new InnoDB stuff here: InnoDB Page Cleaner Thread

It was for sure a very rewarding conference, but one may debate the absence of MySQL itself (Oracle). Normally there is a lot going on, and a lot of announcements from MySQL. In this case Oracle owns MySQL. I don’t know if they are trying to make a statement, not showing up and providing information to the community. Or are they to self absorbed, and want to pull Open database people to join on Oracle World? I don’t know, and I really hope it’s not the latter. They seem to put a lot of money and effort into continued development of MySQL and InnoDB.

Monty did a bald move in his keynote, and later had to appologize in a public blog post: monty says. Me and Tobias Asplund had a decent chat with him at the SkySQL dinner, and there is a lot of new interesting stuff happening in MariaDB. A lot of effort has been put into subquery optimization and the execution plans are now better than ever. Hopefully Oracle will not be too scared to adopt those patches. We will have to try MariaDB here at Marin to see if we’ll get any performance improvements running subqueries. It’s not a production test just to calm those worried souls.

I enjoyed Yoshinori Matsunobo’s Tutorial. There was a lot of refreshing material, but also some new things I have not been able to test before. Good stuff! Check out Matsunobos blog

I also enjoyed many of the facebook talks. It’s very giving to see how they attack some of the problems others never see. The amount of servers and data those guys are handling is insane. Their data drift spotting stuff was awesome as well.

I am surprised by the amount of people looking for DBA’s or data architects. Almost all talks or keynotes said they were hiring.

All in all, it was a great conference. I am almost looking forward to the next one.

Scenario:
You have a huge table with constant activity, containing terabytes of data. You need to update/delete roughly a million random rows. Random selects from the table keep some indexes you could use hot. Running a big update, locking rows/table is not an option since you have approx. 40-50 inserts/sec to the table. So you need to run smaller batches when updating table. LIMIT would be nice, but is not supported with OFFSET when using UPDATE or DELETE.

Updating using IN and subquery together with limit is not supported in 5.0.x (usure about 5.1.x and forward), Else this would have been a viable solution.

Selects with specific where clauses might take longer due to some indexes/keys being too big as well

Solution:
Selecting all Unique/Primary keys to a temp table with an auto increment primary key and then update the production table using this temporary table as a reference table to which rows to update, with a BETWEEN clause on the temporary table’s auto_increment field.
May also enclose everything in a transaction and delete the same ‘between’ on the temporary reference table. Will help you keep track of what you actually did in case you need to abort this.

Example
Production table:

CREATE TABLE `huge_table` (
`request_id` bigint(20) NOT NULL auto_increment,
`customer_id` int(11) default NULL,
`data_id` bigint(20) default NULL,
`group_id` int(11) default NULL,
`user_id` int(11) default NULL,
`action_id` int(11) default NULL,
`external_id` int(11) default NULL,
`user_data` varchar(2048) default NULL,
`entry` text,
`extra_term` varchar(255) default NULL,
`transaction` text,
`receive_time` datetime default NULL,
`from_id` varchar(32) default NULL,
`url` varchar(2048) default NULL,
`from_ip` varchar(2048) default NULL,
`useragent` varchar(2048) default NULL,
`tz` int(11) default NULL,
`cvalue` varchar(255) default NULL,
`rvalue` varchar(255) default NULL,
`uid` varchar(40) default NULL,
PRIMARY KEY (`request_id`),
KEY `k1` (`receive_time`,`action_id`,`url`,`from_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1047142423 DEFAULT CHARSET=utf8

You need to update action_id = 100 on approx 1.5m rows which have receive_time between ’2010-08-01 00:00:00′ and ’2010-08-31 23:59:59′ where from_id equals ‘q7b4x5aa0303erer’.

temporary table:

CREATE TABLE `tmp_ids` (
`id` int(11) NOT NULL auto_increment,
`prod_id` bigint(20) default NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=15424160 DEFAULT CHARSET=utf8

Fill the temporary table with the id’s you need to update in the production table:

INSERT INTO tmp_ids SELECT * FROM huge_table USE INDEX (k1) WHERE from_id=’q7b4x5aa0303erer’ AND receive_time BETWEEN ’2010-08-01 00:00:00′ AND ’2010-08-31 23:59:59′;

Then just iterate this in a bash script or our favorite scripting language (increasing the between values of course):

UPDATE huge_table ht JOIN tmp_ids ti ON ht.request_id=ti.prod_id SET ht.action_id=100 WHERE ti.id BETWEEN x AND x;

example bash script:

#!/bin/bash

# Config params
user=”username”
pass=”password”
host=”hostname”
db=”schema_name”
tmptbl=”tmp_table”
livetbl=”live_table”
tmpcol=”live_id”
livecol=”request_id”
tmpkey=”id”
upcol=”from_id=1″
waittime=”0.5″

# Print usage function
function usage() {
echo “$(basename $0) ”
echo “rows – number of rows in one go.”
exit 0
}

# Since we need atleast one variable to continue, check so first variable is supplied
[ -e $1 ] && usage && exit

# get that variable
rows=$1
# Check which is the first key we will use in temp table
s=`mysql -N -u$user -p$pass -h$host $db -e”select min(id) from $db.$tmptbl;”`
# Set latter between value
let b=$s+$rows;
# Count how many rows we have to update so we know how long to loop
count=`mysql -N -u$user -p$pass -h$host $db -e”select count(*) from $db.$tmptbl;”|head`;

# Print starting line
echo “Starting up with updates from id: $s to $b, performing $rows at one go..”

# loop till we are done
while [ 1 ];
do

# are we done? If so exit
if (($count <= 0));
then
exit
fi

# Run actual db updates
echo -n "updating $livetbl between $s and $b.."
mysql -N -u$user -p$pass -h$host $db -e"update $livetbl td join $tmptbl i on td.$livecol=i.$tmpcol set $setval where i.$tmpkey between $s and $b"
echo -n " deleting in $tmptbl.."
mysql -N -u$user -p$pass -h$host $db -e"delete from $db.$tmptbl where $tmpkey between $s and $b"
echo -n " Done!"

# set between rows
let s=$s+$rows
let b=$b+$rows
let count=$count-$rows

# wait if there are more records to update
if (($count > 0));
then
echo ” ..waiting.. still $count records to update”;
sleep $waittime
fi
done

# All done
echo “Script done. No more records to update”

I have been using cron since I started using linux, ages back. I have never needed to use a “%” character within the command line before, since I have been encapsulating everything in scripts. Yesterday I wanted to make one of the scripts here at marin software a tad more generic, and added the line:

0 8 * * * ~/scripts/script.sh "`date +'%F %H:%M:%S' -d yesterday`"

to crontab. Now, this did not work.

I did not know that the “%” character denotes a new line in crontab. Anyway, this is useful if you want to run a script with some percent characters in the input or if you want to run a mysql query containing % in your crontab line.

I guess you learn something new every day, even though this might be a simple thing (you may already know), I just wanted to share it

We will do a walkthrough of technologies that can be used, pros, cons, common pitfalls and how we solved it at bwin games ab (company i work at).

I will post more information about this later. Just wanted to share my excitement.

You can read more about bwin AG at http://www.bwin.org and you can play at http://www.bwin.com

For those who does not know what bwin/betandwin does, they are the biggest online sportsbetting, casino and poker company in Europe. The Swedish branch (where i work) is the division who develops, maintains and operates the poker platform (old ongame e-solutions ab). Our office is located in the center of Stockholm, in the main central station building.

Same company, new name, same work new work.. :S

My position changed a bit until it was time for me to actually begin (from the time i signed the contract) so i was a little surprised when i got to work.

So, i enter the office, and there is kaos all over. Terrible incident had raged the night…

I was not much of help this day.. I just got there and felt confused because i had no computer no nothing.. just sitting there looking like a goof..