Building a New Data Center
I was part of the new data center build project at my work, which had started in the Summer of 2010. Last week, we had successfully migrated our production site to this new data center in Atlanta, GA. First of all, unless you’ve been part of this kind of project before, you have no clue how much work and effort is involved from the initial planning to execution and the day when the light goes green. Our team took one day to smoothly transition production served traffic from our data center in Sunnyvale to a new data center in Atlanta. Once the transition was successful and everything was validated – it was a great feeling of accomplishing something we’ve worked hard on over the last six months, and I wanted to reflect my thoughts on the key points that i think made it a successful transition.
Building a data center project requires a thorough planning. You need to be able to understand all the steps in the project and their dependencies on each other. Once that part is clear, it is critical to come up with the set of tasks and their due dates. Even though we had to modify tasks/dates all the time, having a clear plan from the beginning is crucial. We’ve used Microsoft Project software for this.
As we went through the project we’ve tried to document everything on our team internal Wiki page. We all know that engineers rarely go back and update the documentation once everything is up and running, so we made an effort to document everything as we go through the building stage.
Building for Scalability
Our old data center was built years ago when server infrastructure was at least 10X smaller in size and we managed to grow our traffic with the minimum investment. At some point it became clear that if we want to be able to scale and support our business growth – we had to implement major changes to our infrastructure, all while serving production traffic with the 99% uptime SLA. This was definitely an option, but due to other reasons, primarily cost, we went on with building a new data center in a different geographical location. Given that, this was our best chance to come in and build a new data center to be able to scale our existing traffic at least 10X or 20X.
This project was a team effort. There were a lot of people involved to make it happen, including local data center rack/stack team, operations team including DBAs, system/network engineers, project management team, electrical contractors and different hardware/software vendors/resellers. Each team had a specific role/tasks to achieve to ensure a successful project delivery.
Communication is an important part in any project. In building a new data center where you are forced to work not only with local resources but remote also, communications becomes even more integral to guarantee success. It is much more preferred to over-communicate rather then risk a chance of slipping the project dates due to miscommunication. Teams must realize that communication is a key to success.
Having a project manager for such a complex project as building a new data center is pretty much a must. We were lucky enough to have one with many lots of experience in the field. Project manager was keeping track of the tasks to complete, dates to beat and so on. This must be a technically strong person with exceptional organization skills.
Focus and Priorities
No matter how soon you have to build a new data center and how well you need to focus on it, supporting your existing data center that serves customer’s traffic is your priority number one, as any minute of downtime is translated into dollar amount of revenue loss. You, as an engineer, have to understand that and toggle those priorities on a daily or hourly basis. Unless your manager is micro-managing you, these are your calls to make.
Managing vendors and making sure they meet your deadlines and pricing was an essential part of this project. We had to interact with at least a dozen of vendors from electrical contractors to hardware resellers. The key there was to have clear requirements of what we needed or what we tried to accomplish and then going out there and finding the best solution at the best price.