The paradigm has been shifting to moving new sources of data onto the data warehouse platform and figuring out how to integrate and use them later on. To a data warehouse architect who’s been in the business for more than a year or two, working under these prerogatives is akin to hearing fingernails being raked across a chalkboard. The holy mantram of data warehousing has always been that the integration is the message. We’ve been dedicating our careers to eliminating stovepipe analytical reporting databases and providing the business with an integrated solution to their business intelligence needs. We’ve debunked off-the-shelf packaged data warehouses as not providing significant competitive advantage. We’ve talked about the risks of federated data warehouses. Stovepipe data marts? Blasphemy!  I don’t know about anyone else, but every data warehousing book I’ve read, and every seminar I’ve ever attended has pounded home the point that this approach is a surefire recipe for disaster. And yet here we are, being asked by the full chain of command, from project management to the CIO, to essentially do things we’ve been trained to fight against — to get data loaded and then figure out how to integrate it and how the business might use it later on. I reckon that too many years of a perceived “ready, aim, aim, aim” approach has swung the pendulum to the “ready, fire, aim” end of the spectrum at many organizations.

So after you’ve taken your dings for expressing your concerns about resources and timelines, and after you’ve pulled out a more than a few hairs trying to figure out how to match up legacy keys coming from five different transaction systems (while still waiting for a CDI implementation that’s been years in the works already) you grab a triple latte, and sit down and roll up your sleeves and try to figure out where you can bend the lines on what you know are data warehouse architecture best practices.

Some of the notable changes you may see on such a project (aside from the frighteningly compressed timeline, and absence of business people at requirements sessions) — and yes I am being just a touch cynical in my portrayal:

• Mega design workshops where the IT business analysts, data architects, DBA’s and ETL developers are all in the same room at the same time to save the effort of translating business requirements into logical models and then physical designs.
• The project leader is in attendance at a lot of meetings, simply to urge the group to “move on” when discussion of a difficult issue goes on for more than ten minutes.
• Standards tend to become guidelines, and things like mis-named objects are quickly written off as “sweating the small stuff”. The old rule about commenting everything that goes into the database changes to commenting when you know what something is without having to call someone up or send out an email and wait for a reply.
• The project manager tends to talk about fixing design anomalies in the “semantic layer”. Making sense of the design and communicating it to the users becomes a metadata issue.

Some things you absolutely should not budge on:

• The need to have a common business model forming the architectural framework. This entails constructing conceptual and logical and physical data models prior to the first table being built or the first line of ETL (or in this case ELT) code being written. Without such a framework, you most surely are destined to have your name and reputation attached to what is known in the biz as a “data outhouse”.
• The tendency of the DBA team to “move on” and modify the physical database, upon learning things from the source data — and not feeding this back into the conceptual and logical data models. When the data models become throw-away design artifacts and the DBMS catalogue and DDL scripts become the only up-to-date design metadata — be very afraid — afraid of managing change in the future!
• Sometimes you just have to push to have the schedule relaxed when it is just too unrealistic. It might risk having you branded as a pessimist — a negative thinker, but in the long run you must go on the record at the very least in communicating your issues and concerns to project management. Once you are on the record, they know that they will have to share the burden of blame if the end result is a useless bucket-o-data. The risk of not identifying the issues is that your reputation as a competent data architect can be badly tarnished.

One final bit of advice: Don’t let the pressures get to you. I’ve had to learn to accept that one of your responsibilities as data architect is to present issues and concerns about both the overall architecture as well as the specifics of the project to management. Ultimately, management makes the calls and you have to do the best you can with what you’ve got. As long as they understand the implications of first assembling that ‘ bucket-o-data’, and then trying to organize and integrate it later on, and agree to that approach at the outset, there’s no point in getting overly stressed out over it.  Trust me, I’ve been down that road. Go home and play with the kids or the dog, get out on the sportbike and carve some twisting backroads, go for a jog, play your guitar, go out for a nice dinner with your better half. Remind yourself that it’s great to have a job with so many intellectual challenges. Thinking of the work as “bringing the operational data stores onto the data warehouse platform” can help to rationalize it. And then there’s always that miraculous semantic layer and wonderous metadata repository to fix everything up right as rain.