System z Batch Optimization: Another Pipes Option?

Over the last 20 years or so I have encountered many sites looking for solutions to streamline their batch processing, only to find that sometimes they are their own worst enemy, because their cautious Change Management approach means they will not change or even recompile COBOL application source, unless absolutely forced to do so.  Sometimes VSAM file tuning is the answer, sometimes identifying the batch critical path, and on occasion, finding that key file or database that is processed on several or more occasions, which might benefit from parallelism is the answer.

BatchPipes was first introduced with MVS/ESA, allowing for data (E.g. BSAM, QSAM) to be piped between several jobs, allowing concurrent job processing, reducing the combined elapsed time of the associated job stream.  BatchPipes maintains a queue of records that are passed between a writer and reader.  The writer adds records to the back of a pipe queue and the reader processes them from the front.  This record level processing approach avoids any potential data set serialization issues when attempting to concurrently write and read records from the same physical data set.

The IBM BatchPipes feature has evolved somewhat and BMC have offered similar functionality with their initial Data Accelerator and Batch Accelerator offering, subsequently superseded by MainView Batch Optimizer Job Optimizer Pipes.  It seems patently obvious that to derive the parallelism benefit offered by BatchPipes, the reader and writer jobs need to be processed together.  For many, such a consideration has been an issue that has eliminated any notion of BatchPipes implementation.  Other considerations include a job failure in the BatchPipes process, where restart and recovery might include several jobs, as opposed to one.  Therefore widespread usage of BatchPipes has been seemingly limited.

The first step for any BatchPipes consideration is identifying whether there is any benefit.  IBM provide a BatchPipes SMF analysis tool to determine the estimated time savings and benefits that can be achieved with BatchPipes.  This tool reads SMF record types 14, 15 and 30 (Subtypes 1, 4 and 5) to analyse data set read and write activity, reconciling with the associated processing job.  As an observation, sometimes a data source might have a different data set name, be both permanent and temporary, while consuming significant I/O and CPU resource for processing.  Such data source reconciliation can easily be achieved, as the record and associated I/O count for such a data source is the same, for entire data set processing operations.  The analysis tool will identify the heavy I/O jobs and be a great starting point for any analysis activities.

UNIX users will be very familiar with the concept of pipes, where a UNIX pipeline is a sequence of processes chained together by their standard streams, where the output of each process (stdout) feeds directly as input (stdin) to the next one.  Wouldn’t it be good if there was a hybrid approach to BatchPipes, using a combination of standard z/OS and extended UNIX Systems Services (USS)?

With z/OS 2.2, JES2 introduced new functions to facilitate the scheduling of dependent batch jobs.  These functions comprise Job Execution Control (JEC) and can be utilized by making use of the new JOBGROUP and related Job Control Language (JCL) statements.  The primary goal of JEC is to provide an easy-to-use control mechanism, allowing complex batch jobs to be processed in inter-related constituent pieces.  Presuming that these constituent pieces can be run in parallel, improved throughput can be achieved by exploiting the concurrency functions provided by JEC.

UNIX named pipes can be used to pass data between simultaneously executing jobs, where the UNIX pipe can either be temporary or permanent.  One or more processes can connect to a UNIX named pipe, write to it, and read from it, as and when required.  Unlike most types of z/OS UNIX files, data written to a named pipe is always appended to existing data rather than replacing existing data.  Therefore, the STOR command is equivalent to the APPE command when UNIXFILETYPE=FIFO is configured.  This UNIX pipe facility, managed by the JES2 JEC functions can be leveraged to provide benefit for multiple step job processing and concurrent job processing, with the overall benefit of a reduction in overall batch stream elapsed time.

In conclusion, the advancement in JES2 JEC processing simplifies the batch scheduling and restart configuration processing, while the usage of UNIX named pipes leverages from existing z/OS USS functionality, safeguarding good performance using a tried and tested process.

Finally, returning full circle to my initial observation of Change Management considerations when performing batch optimization initiatives; recently I worked with a customer I visited in 2001, where they considered and dismissed BatchPipes Version 2.  We piloted this new UNIX pipe facility in Q4 2016, in readiness for their Year End processing, where they finally delivered a much needed ~2 Hour reduction in their ~9 Hour Critical Path Year End batch process.  Sometimes patience is a virtue, assisted by a slight implementation tweak…

zAPI: System z Deployment Into The API Economy

Having been in the IT industry for 35+ years, I have always fully embraced and learned new technologies, to find strategic solutions for business challenges.  Obviously, starting in 1980, my heritage is IBM Mainframe, supplemented by UNIX, Wintel and Linux along the way.  Each and every platform has its merits, and during this 35+ year period, I have attended many conferences, for all platforms.  What I have noticed during this period is the attendance of many IBM Mainframe CIO, CTO or Chief Architect individuals at non-IBM Mainframe conferences, but very few, if any, equivalent Distributed Systems personnel at IBM Mainframe conferences.

I’m always surprised and disappointed to hear about organizations talking about decommissioning the IBM Mainframe platform, with tenuous reasons, based on Distributed Systems FUD messaging, as opposed to their own business requirements.  Thankfully these scenarios are decreasing over the years.  Presumably if an organization decides to migrate from one Distributed Systems platform to another or perhaps the Cloud, they do at least attend the relevant platform conferences to make an informed decision.

Over the last 25 years or so, IBM themselves compete with differing divisions and options, whether UNIX (AIX), System z and in recent years, Linux on z Systems, most notably with the LinuxONE launch at LinuxCon 2015.  One would hope that the world’s key IT decision makers might attend LinuxCon with an open mind and learn more about the System z Mainframe?

A ridiculous notion might be that one server platform technology can satisfy a 21st Century organizations IT infrastructure for their mission critical services.  Clearly that has not been the case since the advent of Client Server and today’s emerging Digital business requires an infrastructure of multiple layers, where the underlying server technology is somewhat arbitrary, and arguably a commodity resource.  Conversely the underlying data and associated applications differentiate one business from another, delivering business value and competitive edge.

Let’s take some time to consider this IT architecture design, which very quickly dismisses any notion that one server technology delivers all business requirements:

Such an architecture diagram does not impose any technology decisions.  Conversely it explores the “data journey” from access or creation, via Systems of Engagement (SoE) to eventual storage within Systems of Record (SOR) data repositories (I.E. Database).  Some might say it was forever thus, with the exception of the Multi-Channel SDK’s & API’s layer, where the savvy organizations will embrace DevOps, Hybrid Cloud and connectivity (I.E. API, SDK) solutions, seamlessly integrating modern agile applications, with that most valuable business asset, Systems of Record (SoR) data.

Today’s Application Developer doesn’t need to concern themselves as to the platform used for their DevOps application processes, the Transaction Server or indeed the Database Server.  Sure, several decades ago, maybe even a decade ago, application code was deeply associated if not confined to a specific CPU server architecture.  Clearly that is no longer the case.  Any organization that still thinks in this legacy manner, is behind the times, and this is unfortunate.  Associating such outdated thinking with the System z Mainframe is arguably careless, and not a reason for dismissing an incumbent System z platform, or not considering a System z platform in the future.

Arguably the greatest strengths of today’s System z IBM Mainframe, currently packaged as the z13 or LinuxONE, are as a Database Server (E.g. DB2), Transaction Server (E.g. CICS, WebSphere Application Server) and Security Server (E.g. ACF2, RACF, Top Secret).  From a LinuxONE viewpoint, it’s just another server, capable of processing all of the latest strategic Open Source and Commercial Off The Shelf (COTS) Cloud, Database and Application solutions, while benefitting from the unparalleled System z Quality of Service (QoS) attributes.

However, for those organizations already deploying a System z Mainframe, its greatest perceived issue is TCO.  Without doubt the convoluted and intricate Workload Licence Charges (WLC) are unnecessarily complicated and perceived as being very expensive.  Optimizing these costs requires a modicum of expertise, safeguarding that the best contractual conditions are negotiated.  However, I encounter the same complexities with Distributed Systems platforms, where software license costs can spiral out of control for significant CPU capacity deployments.  Whatever platform is deployed, System z Mainframe or Distributed System, unless the business has the requisite skills in place, technical and commercial, to safeguard the lowest cost possible, commercial ISV suppliers will take advantage of such an oversight.

I’m not advocating any server technology, System z Mainframe, Distributed System or Cloud, as each resource has its merits, depending on the business requirement.  However, today’s 21st Century organization must enable new business channels by leveraging from and arguably enable new business channels by monetizing their Systems of Record (SoR) enterprise data.

Today, organizations need to consider an API Economy, where they expose their internal digital business assets or services in the form of Web API services to external 3rd party partners and consumers, with an overall objective of unlocking increased business value via the creation of new assets.  Such an API Economy will require integration of Transaction and Data resources, specifically:

  • Centrally manage the consumption of enterprise wide business logic, for both Systems of Record (SoR) & Systems of Engagement (SoE)
  • Extend business (E.g. Product, Brand) reach from Systems of Record (SoR), incorporation Systems of Engagement (SoE)

Previously I wrote about How to Connect Mobile Workloads to System z, detailing the conceptual steps required to expose existing SoR data assets with SoE transaction services, via z/OS Connect.  For a fully integrated end-to-end integrated solution, we must also consider the Application Programming Interfaces (API), being the digital glue that seamlessly links applications, services and systems together.

IBM API Connect is a solution that manages the API lifecycle for both On-Premises and Cloud environments.  IBM API Connect delivers capabilities to Create, Run, Manage & Secure API resources and Microservices.  It also enables you to rapidly deploy and simplify API administration, across the organization.

API Connect can be deployed On-Premises via Linux on z Systems, in the cloud (E.g. Bluemix), as well as all other popular Distributed Systems.  Once again, the main message is that the chosen server is arbitrary, System z Mainframe, Distributed System or Cloud.  The server should be considered as a commodity resource, leveraging from existing business logic (I.E. SoE) and data (I.E. SoR), while evolving existing Application Lifecycle Management (E.g. Agile, API Economy, DevOps) is the key.

My final observation is the Mainframe Baby Boomer (E.g. Born ~1960) versus the Millennial (E.g. Born ~1995) technical personnel resource.  Without doubt, there are significant differences in their approach to application programming, but only one resource, namely the Baby Boomer knows the business really well.  I think these folks have the ability to learn another 21st Century programming language, as well as COBOL, but perhaps their best attribute is an analytical role, especially for the integration of SoE and SoR layers.  Working very closely with Millennial technical resources, delivering the new Application (I.E. App, API) resources, the Mainframe Baby Boomer still has something valuable to offer in their final employment years.  For the avoidance of doubt, still delivering value from an analytical viewpoint, while transferring their skills and knowledge to their successors, namely the Millennial.

In conclusion, dismissing any server technology for Fear, Uncertainty or Doubt (FUD) reasons, is an unproductive and ridiculous notion.  More importantly, what might your business lose in opportunity, spending several years or more, migrating from one platform to another, while your competitors are embracing the Digital Age with an API Economy approach, delivering more value from their existing business SoE (transactions) and SoR (data) assets?

The IBM Mainframe: Just Another Node On The IP Network!

With the introduction of MVS/ESA Version 4.3 in 1993, the IBM Mainframe included the major foundations for meaningful Distributed Systems connectivity, including the first steps of POSIX compliance via OpenEdition functionality.  However, even before this timeframe, the TCP/IP protocol was available in the first release of MVS/ESA Version 4 (4.1), although in a very limited fashion.  In this instance, MVS was benefitting from the path already trodden by the VM Operating System and the TCP for VM software product.  Put another way, even when TCP/IP was in its early stages, being deployed and evolved in universities and scientific laboratories (E.g. CERN), its foundation was being embedded into the IBM Mainframe.

Early IBM Mainframe TCP/IP usage allowed for RS/6000 (AIX) connectivity, LAN integration via Novell NetWare, typically via the 3172 Interconnect Controller, Sockets Interface (E.g. CICS), et al.  In 1994, IBM introduced the Open Systems Adapter (OSA) processor feature for S/390 Parallel Enterprise Servers.  The OSA provided native Open Systems connectivity to the Local Area Network (LAN), directly via the Mainframe processor.  The OSA feature supported the Fiber Distributed Data Interface (FDDI), Token-Ring & Ethernet LANs, arguably making the 3172 controller obsolete.

So, since the early-mid 1990’s, even before pervasive usage of the Internet, the Mainframe was already a fully functioning and efficient user of IP networking.

How is the TCP/IP function being utilized by the IBM Mainframe today?

TCP/IP on z/OS supports all of the well-known server and client applications.  The TCP/IP started task is the engine that drives all IP-based activity on z/OS.  Even though z/OS is an EBCDIC host, communication with ASCII-based IP applications is seamless.

IP applications running on z/OS use a resolver configuration file for environmental values.  Locating a resolver configuration file is somewhat complicated by the dual operating system nature of z/OS (UNIX and MVS).  Nearly each and every z/OS customer deploys the following core TCP/IP services:

TCP/IP Daemon: The single entity that handles, and is required for, all IP-based communications in a z/OS environment is the TCP/IP daemon itself.  The TCP/IP daemon implements the IP protocol stack and runs a huge number of IP applications to the same specifications as any other operating system.

TCP/IP Profile: Is loaded by TCP/IP when started.  If a change needs to be made to the TCP/IP configuration after it has been started, TCP/IP can be made to reload the profile dynamically (or read a new profile altogether).

FTP Server: Like some other IP applications, FTP is actually a z/OS UNIX System Services (USS) application.  It can be started within an MVS environment, but it does not remain active in z/OS.  It immediately forks itself into the z/OS UNIX environment and tells the parent task to kill itself.

Telnet Daemon: There are two telnet servers available in the z/OS operating environment.  One is the TN3270 server, which supports line mode telnet, but it is seldom used for just that.  Instead, it is primarily used to support the TN3270 Enhanced protocol. The other telnet server is a line mode server only, referred to as the z/OS UNIX Telnet server (otelnetd).

Many IBM and ISV software products exploit IP and USS functionality, most typically WebSphere (MQ).

Whether UNIX System Services (USS) or TCP/IP usage, the convergence of the IBM Mainframe and UNIX technologies arguably became mandatory with the deployment of TCP/IP on the IBM Mainframe.  Obviously the technical personnel that support these different platforms have their own viewpoint as to which platform might be the best, but that is somewhat of an arbitrary point.  However, what is absolutely certain is recognition of how data is stored and secured in a UNIX environment and indeed the z/OS (MVS) specific environment, originally named MVS OpenEdition, but now commonly referred to as OMVS.

There are fundamental differences too numerous to mention when comparing the User and File management policies and processes, when comparing the security and data access lifecycle intricacies of z/OS and UNIX.  So what you might say!  This might be a cursory and lax attitude, as business critical data is probably being stored in OMVS file systems, if only for FTP purposes, but more than likely for other more pervasive and user based access (E.g. Database, Messaging, Data Mining, Data Exchange, et al).

So, which technical party is managing the security of Unix System Services (USS) file systems for the OMVS Mainframe deployment?  Is it the Mainframe Systems Programmer, the Unix System Admin or the Mainframe Security Team, or somebody else?  To date, some people might have thought it didn’t matter, but of course, seasoned security professionals knew that this was never the case.  However, the migration to z/OS 2.1 is a tangible juncture for each and every IBM Mainframe installation to review their USS and thus OMVS security deployment.  Why?

The BPX.DEFAULT.USER facility was introduced with OS/390 2.4 and was a commonly used process for implementing USS (OMVS) security.  However, with z/OS 2.1, the BPX.DEFAULT.USER facility is withdrawn, meaning that the Mainframe user must perform some migration actions.  IBM provide some generic assistance with this challenge via APAR OA42554 and APAR OA37164.  However, maybe this is an ideal juncture to perform a thorough review of USS (OMVS) security, vis-à-vis a comprehensive and dispassionate audit, highlighting issues, implementing standards and securing exposures.  For example, use of UID(0) must be eradicated and certainly no human being should be allocated such privileges.

There are some useful guidelines available from security specialists such as Vanguard, where the process can be simplified using their Identity & Access Management (IAM) toolset.  Similarly, recent user conferences have included presentations on this subject matter.

In conclusion, the IBM Mainframe can be classified as just another node on the IP (TCP/IP) network.  However, as always, no matter how secure the Mainframe platform might be, the biggest threat is typically the human being, and for USS, the migration to z/OS 2.1 forces us to review OMVS security settings.  Therefore, let’s do a good job and eradicate any security exposures we might have inadvertently implemented over the years.  As we all know, passing an external security audit process doesn’t necessarily mean our IT systems and processes are secure, while sometimes the internal security people are better qualified or more knowledgeable than external auditors.  Arguably most external auditors will do a good job of auditing UNIX platforms, yet their Mainframe knowledge and abilities are typically limited.  It is therefore somewhat of a paradox that in this particular area of z/OS USS, the typical UNIX exposures are not highlighted in the typical Mainframe security audit process…

One must draw one’s own conclusions as to the merits of engaging 3rd Mainframe security specialists to perform such an audit, coinciding with this z/OS 2.1 migration activity, safeguarding that OMVS security and processes are as good and secure as they can be.  Put another way, why wouldn’t a Mainframe organization go that extra mile to safeguard their most valuable of assets, namely business critical data, engaging a 3rd party specialist to review and provide guidance on this subject matter.