NetApp file size limitation. What not to do..

So, I’ve had the pleasure (not so much) to witness first hand what happens when you go about your day and think everything is fine and dandy with your MySQL servers and Boom – one of your tables has grown to 16T and now is saying it’s full. Ok, don’t flame me too much, I know one 16T table isn’t amazing, but unfortunately the app in questions is a legacy app and this particular table is used as an audit log of changes. Not a big deal if the application would have a sane retention policy for this data and clean it up when it’s done with it. After all, developers think about data lifecycle when they write applications *grin*.

So, we run some of our database environments on top of NFS storage (yes, not optimal, but this thing was here before I got here and has been hard to kill). The NFS storage in question is NetApp and if you are going to run NFS as database storage – NetApp would be the way to go.

My team does not manage the storage and we rely on our storage team to manage and maintain/monitor the storage, which they do very well.

Unfortunately, my team nor the storage team was aware that NetApp has a single file size limit on NFS, and that limit is 16TB. All good you would think – who has a 16TB file anyway. Well, for starters – lets say you would not be running innodb_file_per_table, you would store all your data in ib_dataX, and who knows how your have that setup (most have it autogrow) which means you would be limited to a total innodb database size (all schemas and all tables) of 16TB (including indexes).

Further investigation show that there’s also a single LUN size limit on NetApp (that is if you figure you can just run iSCSI on NetApp to get around the file size limit). That size limit is *drum roll* you guessed it – 16TB. This is because *hold your breath* NetApp creates a file in a NetApp volume, that it presents as a LUN..

Ok, not a big issue; create multiple LUN’s in the same NetApp volume (again, NetApp puts everything in Volumes). Then use LVM, assign each LUN to a PV, then those to a VG and finally present them to the OS as an LV and you now can have as big files as you want with no limitation of size and still use NetApps snapshots with full consistency for MySQL backups.

Or – switch to storage with low latency, direct attached or SAN, and never look back.

Leave a comment

Your comment