New Wine or Technology In Old Bottles!

"Neither do men put new wine into old bottles: else the bottles break, and the wine runneth out, and the bottles perish: but they put new wine into new bottles, and both are preserved." [ref]

New technology used in old ways also breaks them and leads to pathological systems. The best summary and critique of this pathology, as it exists in Architecture (not Software or Enterprise Architecture, but real, putting-up-buildings Architecture), is this passage, about the Parthenon:

"The famous flutings on the famous columns---what are they there for? To hide the joints in wood--when columns were made of wood, only these aren't, they're marble. The triglyphs, what are they? Wood. Wooden beams, the way they had to be laid when people began to build wooden shacks. Your Greeks took marble and they made copies of their wooden structures out of it, because others had done it that way. Then your masters of the Renaissance came along and made copies in plaster of copies in marble of copies in wood. Now here we are making copies in steel and concrete of copies in plaster of copies in marble of copies in wood. Why?"

Unfortunately, this happens all too often in our world of technology. Consider, as an example, one great Bloated Behemoth Enterprise, whose technology needs were well in place and large-scale even thirty or forty years ago, in the days when Mainframes bearing Tape-drives ruled the earth. 

A computer-historical perspective is helpful here, and luckily TheHackerCIO has spent some time both talking to the old-timers (one goes to our local Users Group) and reading about the bad-old-days. In those ancient times, programs were structured around the tape based file system. A typical program would read, as input, the Customer Master tape, which contained an entry for every customer, and a second tape input -- let's say a New Orders tape -- containing a row for each new order needing processing, and already sorted by customer Id. 

The program, then would read a customer order, then advance the Customer Master file until it located this customer, pull out the data it needed to complete the order, and then record the finalized purchases in yet another tape. Note that when building a new system in this kind of ecosystem, one must always source things from existing files, stored on tapes. At the end of job-run, a new tape has been produced, which is input to the next job. And these jobs are all run on a particular schedule, carefully contrived, and supervised by the operators. 

Now enter the Relational Database. The point of this technological innovation, and the internal genius of its principles,  was to have a Master Database for the enterprise. Instead of files, all data would be stored in tables. Now, updates could be made transactionally to the system of record in real time, at the same time as others were querying that same data to determine how things were changing. To take our example, as new orders were placed, it was now possible to obtain the necessary data from the CUSTOMER Table, and create a row in a NEW_ORDERS Table to handle it. As the RDBMS evangelists put things, this tool allowed for:

• reduced data redundancy
• increased data availability
• increased data security

And there was a whole methodology for properly "normalizing" the data, and consequently eliminating the double-update problem, and eliminating concern with determining the system-of-record, by replacing it with one universal system. 

Unfortunately, the Bloated Behemoth Enterprise dealt with Databases differently. They saw them as Yet-Another-Tape-Based-File System. 

And so, each new development project, at its commencement, began life by creating it's own database, just as they would have defined a new Tape-File layout. They sourced them by creating unload jobs, just as they would have unloaded a Tape and loaded the data into their new Tape File layout. They put new wine into old bottles.

And after thirty years of such accretions, they now have tens of thousands of databases, in every dialect of SQL possible, scattered across myriad platforms, with a tangled web of sourcing one database from the unloaded output of another, all kept in several "AutoSys" style batch-job schedulers, so that the proper, and necessary order of loading can take place. 

Which is to say -- for those unacquainted with this kind of BBE -- that at a particular time in the evening (typically midnight) the online systems are brought down, the databases are all quiesced, backups are taken, then crucial batch jobs run, many of which consist of unload jobs to extract data from one database, just as if it were a Tape, and load up another. 

As time has progressed, this batch window grows longer and longer, to progressively consume the evening -- not to mention the disk space available. 

This is a perfect example of the pathology of putting new technology into old bottles. 

And, en passant, it is an example of why Architecture must never adopt the "timeless (and thoughtless) way of building" that merely tinkers with using new things in the same old way, without spending the time to ensure that a proper understanding of the new way is properly adopted and promulgated. 

Consider this a Cautionary Tale! Always seek to know and find the inner logic of a new technology. Always seek to ensure that New wine get's the proper new bottle it needs. Otherwise, you'll want to get drunk when you see the results. 

I Remain,

TheHackerCIO