Confessions of a Database Administrator
- 1 of
-
Confessions of a Database Administrator
Between growing volumes of data, complex infrastructures and the fact that more DBAs are managing multiple database platforms, the job isn’t getting any easier. -
Finger-Pointing Can Get Ugly
When there’s a database performance problem, it always seems to come down to the DBAs and developers going back and forth trying to determine if it’s a system-related issue or poorly written code. Sometimes the finger pointing can get ugly. -
Monitoring Reveals Poor Performance
It’s always a kick to the gut when you take over for a DBA who left the firm and you install a database performance monitoring tool that shows you just how poor a job the previous DBA did—and all the work you need to do to get things running well. -
SQL Is Like a Chemistry Lab
Dealing with an SQL database is like dealing with nitroglycerin: If you make the wrong move, you could have a corrupted database on your hands in a flash. -
Running Checks Your Database Can’t Cash
We had a database sitting on the cloud. The dev team forgot to turn off the diagnostic logging on the deployment code, which was writing logs every second to the database. We got slammed by the cloud provider due to excessive I/O, and the database filled up without truncation of the log database, crashing the application. -
When High Availability Isn’t Available
High-availability architectures aren’t always as “available” as advertised due to unknown dependencies on particular databases referencing one another. This is usually known only to the development team—not the DBAs—and is only brought to light when a failover occurs and an app crashes. -
Colas and CoLos Don’t Mix
One Friday, our team uncovered severe disk block stickiness (literally) that took down a client’s databases. The root cause was a vendor in a neighboring cage at the colocation facility who had spilled a Big Gulp onto our SAN. The resolution? More stringent enforcement of the “no food or drink” policy, promoting a SAN from development to production, a weekend of restoring a dozen databases from tape, and informal plans to beat the vendor’s tech senseless. -
Blame the Network
Here's another nightmare scenario: The network guys take network core switch A offline for updates, but then reboot core switch B, too, resulting in all paths to storage—primary and secondary—failing. Yeah, it happened, and it was the perfect recipe for a really fun night for us DBAs who were left to mop up the mess.
“The days of simple database management, if there ever was such a thing, are long gone. I’m finding trials around every corner. So much depends on the peak performance of databases—the very heart of IT—that the pressure is almost unbearable. I must confess the challenges I face every day."
OK, so most database administrators (DBAs) probably don’t spend much time in server room confessionals offering up such heartfelt ruminations, but, if they did, we imagine at least one might sound a little like the above monologue. Between exponentially growing volumes of data, increasingly complex infrastructures that rely on peak database performance and the fact that more DBAs are being asked to manage multiple database platforms, the job definitely isn’t getting any easier. So, at SolarWinds, we asked our thwack community of IT professionals what challenges they would like to confess, and this slideshow is based on their responses. By following best practices, peak-performing databases are within your grasp. First, document consistent processes for ensuring integrity and security, such as backup and restore, encryption, detection of anomalies and security events in logs. Then save on costs by ensuring team members can spend enough time proactively optimizing the databases. Be sure to have a common set of goals, metrics and service-level agreements across all databases, ideally based on application response times, not just uptimes. Finally, establish a road map for moving to the cloud (or not) and for reducing workload costs by moving databases to lower-license-cost versions or open-source alternatives.