Straddling the legacy/next-gen divide
In the 1980’s, the computing world faced a major problem. Many applications, including a large number crucial to the operation of entire companies, had been written to run on older mainframe systems, such as Burroughs’s large systems. These applications accounted for millions of lines of code and could not be readily replaced by newly written software designed for modern systems. For instance, many of the older, larger systems were designed around custom integrated database systems; the newer systems being developed and shipped in the 1980’s, however, used SQL databases. When a company decided to replace an older system with a new one, they were forced to choose between keeping the old systems running until the software could be rewritten—a process that was projected to take decades—or to find some other way to replace the hardware and backends without rewriting the legacy applications.
To solve this problem, companies turned to a wide array of middleware products and projects. The basic idea behind the middleware craze was to build a layer of software that could, in the ultimate case, accept data from any system and transform it into a format any system could use. In a typical workflow, an application written for a desktop computer could form an SQL query which would be sent to a middle server. The middleware server would, in turn, use some form of mapping database to determine where the requested information could be found in an older application running on a Burroughs B6500 (or IBM S390, or some other similar system), and then use some interface to retrieve the data. The data would then be transformed into what appeared to be an SQL response, and sent back to the requesting application running on the desktop system.
The promise made on the behalf of middleware packages, as with all things in information technology, quickly became astounding. Middleware applications could not only transform data, they could act as a “data broker” of sorts. An SQL join might be passed to the middleware application that required reading an SQL database on one server and information out of an older proprietary database on a large system. Or perhaps one table resided on a local drive, while the other needed to be drawn from a proprietary database on a large system. Whatever the location of the data, middleware could get at it and transform it into a useable format.
One crucial question was: how was the middleware meant to interact with these older large systems? Since mainframes manufacturers were not known for their open stance on sharing data, there were no real open interfaces into these systems. Further, by the time middleware became popular, a few of these mainframe vendors had actually gone out of business, making it apparently impossible to retrieve the data through anything other than their native terminal interface.
To get at this data, these middleware packages often included screen scraping capabilities. A developer could define what keystrokes the middleware package should use to reach a specific screen, and then where on the screen each field would be located. The middleware application would then read the screen, pulling the data into the correct format for the requesting application. This, of course, built an entirely new layer of legacy on top of the old legacy systems, often compounding complexity.
Whatever happened to middleware? As a result of competition, many of the mainframe companies were forced to open up interfaces directly into the data, and to provide interfaces into the data that allowed SQL and other common formats to be used in working with it. Thus, some parts of middleware moved into the large systems it was designed to support. This often enabled these applications to be completely rewritten without losing data—although in some cases, large amounts of data was simply abandoned in these older systems. The trend to either abandon or rewrite older applications was accelerated by the big Y2K panic—a topic for another post at another time.
The meaning of middleware today has morphed to any software that sits between the operating system and an application. It is primarily used to support input/output functions, file operations, and video operations. This modern middleware serves the same basic purpose, but without the big mainframes and without screen scraping.