All Flash & Substance – Is The System z Microsecond The New Millisecond?

Is 2016 the year of the All Flash disk array?  Seemingly from a System z perspective, 2016 has seen improvement in the All Flash disk array offerings from the major disk suppliers, namely EMC, HDS and IBM.  From a usability perspective, managing latency might be the overall challenge, where these ultra-fast SSD systems are delivering I/O performance response times measured in the ~250-500 Microsecond (μs) range, potentially consigning the traditional Millisecond (ms) measurement to history!

Whatever the speeds and feeds might be, as of 2016, the benchmark for a System z All Flash Disk Array is seemingly measured @ <500 Microsecond μs response time, supporting ~n PB of capacity and delivering ~nnn GB/S throughput for mixed read/write workloads.  Of course, strong encryption, typically full disk Data @ Rest Encryption (D@RE) based and full seamless data replication interoperability are also mandatory.

Historically we evolved from Data Processing to Information Technology, not only automating the capture and processing of data, but gradually evolving our processes, using this data for business advantage.  In recent years, the information explosion has dictated that each and every business must be a cognitive business, using intelligent analytics to gain insight and faster decision-making from the business data collected.

Currently the Internet of Things (IoT) supplements the medium-term Cloud, Analytics, Mobile, Social & Security (CAMSS) initiative, being the processes and associated solutions required by cognitive businesses to make timely and informed decisions, capturing deeper customer insight, ultimately delivering competitive advantage.  Therefore the 21st century business generates a significant requirement for storage capacity and performance to fully realize the benefits of this truly business aligned cognitive approach.

The largest global organizations from all verticals leverage from the power and true 24*7*365 availability and reliability of the System z Mainframe to power enormous relational databases, processing millions of customer transactions on an hourly basis.  These always-on, mission critical business environments demand the performance, reliability, TCO and System z platform integration delivered by the associated DASD (3390) subsystem.

Each and every System z user will have their IHV of choice for delivering disk storage, in alphabetical order, EMC (E.g. VMAX AFA/All Flash Array), HDS (HAF/Hitachi Accelerated Flash) or IBM (E.g. DS8888).  The choice of disk storage was forever thus, reviewing the market place and choosing the best option for your business.  What might require reflection is how the DASD I/O subsystem is managed and the associated interaction with said IHV supplier.  Systems Management solutions such as Easy Disk Analyze Mainframe (EADM) and IntelliMagic Vision (Disk Magic) will certainly simplify the analysis and presentation of DASD subsystem performance data.

However the emphasis of the actual System z DASD subsystem for an All Flash array might move from being an internal consideration, to a direct and timely communication with the IHV supplier.  Put very simply, in an environment where Mission Critical systems rely upon ultra fast processing of massive amounts of data, any flash memory issues, whether capacity or defect related will need IHV interaction ASAP, arguably “Before The Event”.  As with the System z Server itself, where we’re used to On/Off Capacity on Demand (OOCoD) processes, maybe we need to consider a similar approach with our All Flash System z DASD arrays.  For the avoidance of doubt, as opposed to waiting for an issue to impact our business, maybe we need to work smarter with our IHV, to safeguard that sufficient flash memory is available, to proactively resolve capacity or defect issues…

Aligning this with our traditional 3390 DASD I/O subsystem analysis, which might have been on a daily basis, from the rich RMF/CMF data resource, we must fully automate this process to minimize or eliminate the Mean Time To Resolution (MTTR)!  The ultimate benefit will be the delivery of meaningful messages that incorporate our 3rd party IHV supplier, who potentially with Remote Support Facility (RSF) type processes, deploys the “Golden Screwdriver” to seamlessly safeguard the performance profiles of our Mission Critical business applications, leveraging from the latest All Flash disk array.

In conclusion, as always, technology can deliver business benefits, with substance, and this includes All Flash disk arrays.  As always, what might need to evolve are the associated Systems Management processes.  Therefore asking yet another potential rhetorical question, what is more important, the System z Server or timely data access?  The diplomatic answer is that they’re equally important and if so, let’s safeguard the availability of All Flash memory for our DASD subsystems, with the requisite levels of meaningful proactive reporting and IHV supplier interaction.

System z: Optimizing DASD I/O Subsystem Performance

Historically there was a very simple synergy between the IBM S/370 Mainframe and its supporting disk I/O (DASD) subsystem, allowing for Mainframe host to physical and logical disk device (I.E. 3390) connectivity. The analysis and tuning of this I/O subsystem has always been and continues to be supported by the SMF Type 7n records via IBM RMF and the BMC CMF alternative. However, over the years, major advances in DASD subsystems and the System z Mainframe server have delivered many layers of technology resources (E.g. Cache, Memory, FICON Channels, RAID Storage, Proprietary Microcode, et al) and this has introduced complexities into highlighting DASD I/O subsystem performance problems.

The focus of technology based metrics (E.g. I/O Rate Response Time, I/O MB/S Bandwidth, et al) have also been complemented with more meaningful business focussed Service Level Agreements (SLA). Therefore today’s System z I/O Performance Analyst must gather and act upon proactive meaningful information from the ever-increasing amounts of performance data available. Put another way, too much data can deliver not enough information! As previously stated, it was forever thus, RMF and CMF have always collected the requisite performance data available and arguably no other data source is required (E.g. OMEGAMON/TMON/SYSVIEW Performance Monitor, SAS/MXG/MICS/WPS Performance Database). RMF/CMF is the ideal data source for thorough and timely System z I/O performance management, where intelligent analytics and expert knowledge are required to present this “Golden Record”.

However, today’s System z Support Teams need simple and timely presentation of the data, highlighting potential challenges, graphically presented for their Management, allowing for simple tracking of SLA agreements and technology changes (I.E. Software/Hardware Upgrades).

Additionally, Workload Manager (WLM) can control non-paging queued DASD I/O requests, based upon device busy conditional processing. Therefore the z/OS system can manage I/O priorities in a Sysplex, based on WLM service class goals. WLM dynamically adjusts the I/O priority based on service class goal performance and whether a DASD device can influence the overall performance objectives. For obvious reasons, this WLM function does not micro-manage I/O priorities, only changing a service class period’s I/O priority infrequently. WLM is deployed by many System z users to assist in the automated management of system resources (E.g. CPU, Memory, I/O, et al), based upon Service Level goals.

From a DASD subsystem technology viewpoint, there is no longer an obvious one-one direct connection between the Mainframe host and DASD device. An increasing number of technological advances, both microcode and hardware (E.g. Memory, Fibre Channel, Function Assist Processing, et al) have diminished the requirement for data access directly from the physical device. Put another way, in today’s world of System z servers with multiple cache level CPU chips (I.E. Relative Nest Intensity), massive and multiple processor memory resources (I.E. z13 @ 10 TB Memory), high bandwidth Fibre Channel (I.E. FICON, zHPF) subsystem and a hierarchy of DASD memory (I.E. SSD/Flash, Cache), it’s not uncommon to consider an I/O that requires physical device access as a problem! Finally and most importantly, from a DASD subsystem viewpoint, each of the recognized System z DASD providers, EMC (Symmetrix VMAX), HDS (VSP G1000) and IBM (DS8870) have highly proprietary DASD subsystems that provide z/OS plug compatibility, but deliver overall I/O performance using their own unique architecture and internal algorithms.

Of course, an over configured hardware environment will deliver a poor TCO, while an under configured environment will manifest in SLA issues and bad user experiences, where the middle-ground always delivers the optimal environment. Resource optimization always demands proactive day-to-day management, from an internal and indeed external communication viewpoint. With the highly proprietary design features of the IHV DASD subsystems, whether EMC, HDS or IBM, having the right information and identifying the precise problem, simplifies the communication process with the IHV. Such communication might highlight a resource under provision (E.g. Memory Capacity), a subsystem setting tweak requirement, either host or subsystem based, or indeed a hardware failure. In today’s world, these issues need to be fixed in minutes or hours, not days or weeks.

Therefore, where does today’s System z I/O Performance Analyst start to collect the required information to safeguard that their DASD subsystem is optimized, both from a capacity and performance viewpoint?

A simplistic viewpoint of an I/O health-check should consider the following:

  • Service Level Agreements (SLA): Are overall objectives being delivered or missed?
  • User Experience: Are users (customers) complaining of poor service or response times?
  • I/O Metric Performance: Are there obvious signs of abnormal performance statistics?

Several decades ago, an overall I/O health check might have been a periodic (E.g. Weekly or longer) activity, whereas today it’s undoubtedly a Business As Usual (BAU) and 24*7 activity. Therefore a fully automated solution is required, built upon the tried and tested System z performance fundamentals, namely RMF or CMF. The ideal solution will perform analytics based data reduction, presenting the right information, at the right time, allowing for intelligent business based communication, both internally, to customers and end users from an SLA viewpoint, and externally, with IHV DASD suppliers, safeguarding optimal performance and TCO.

EADM (Easy Analyze DASD Mainframe) is a solution from Technical Storage that performs automated performance analysis of the z/OS I/O subsystem, delivering predictive analytics for better storage capacity planning and performance measurement. The Technical Storage EADM architects have in excess of 40 years IBM Mainframe experience, specializing in the I/O subsystem, and so it’s no surprise that EADM delivers expert and timely knowledge via an easy-to-use solution.

EADM is an easy-to-install and easy-to-use plug-and-play solution that has no proprietary considerations, requiring no additional System z resource (E.g. CPU, Memory, DASD, et al) requirements. Installed on Microsoft server platforms, EADM is easily virtualized via VMware, Hyper-V, et al, requiring no target database for performance data storage. EADM performs a daily health check of the entire System z disk subsystem. EADM works around the clock, delivering customized and automatic user friendly GUI type reports. For today’s System z technician, the open and IP architecture base of EADM allows for secure remote access via Mobile, Tablet or Laptop devices, as and when required.

Operations and performance teams are alerted as soon as performance variances occur, typically in minutes, assisting in the identification of underlying root problems, causing changes in system behaviour. Incorporating intelligent and meaningful I/O performance indicators, with drill-down and zoom-in ability, storage technicians can determine if the problem is temporary, permanent, local or global. By simplifying the data reduction process (E.g. RMF/CMF data from numerous LPAR/Sysplex environments), EADM safeguards that the internal technical team can efficiently manage their ever increasingly complex and large DASD environment, for intelligent and timely communications with internal business teams and external suppliers alike.

EADM simplifies the System z I/O subsystem capacity and performance management process, delivering expert reports and timely historical analysis, for example:

  • Automatic daily (24 Hour) analysis of Sysplex wide workload (On-Line TP & Batch) I/O response times
  • Systematic intelligent alerts of early performance variances with exact occurrence time indicators
  • Identification of I/O performance hot-spots with DASD volume and data set level granularity
  • Performance trending at DFSMS Storage Group, Subsystem LCU and DASD volume level
  • DR (E.g. PPRC) simulations to prevent data loss and forecast Data Centre failover scenarios
  • I/O subsystem WLM indicators to determine exactly what impacts performance objectives
  • Full FICON channels and zHPF analysis, incorporating typical I/O throughput indicators
  • HyperPAV and associated LCU indicators to easily balance volumes, optimizing PAV alias allocation
  • Performance monitoring and balancing via intelligent LCU, SSID and I/O analytics
  • DASD capacity usage via DCOLLECT data, comparing assigned vs. allocated vs. actual disk utilization
  • EADM supports entry-level several LPAR and complex multiple CPC/LPAR System z configurations

A well provisioned and performing System z I/O subsystem is of vital importance for safeguarding today’s ever increasing storage requirements of mission critical business applications. A poorly performing I/O subsystem will generate unnecessary and extra CPU overhead, with potential and tangible TCO impact, in conjunction with potential business impact. Although the advances of the System z server and underlying DASD I/O subsystem can compensate for many application code or data placement issues, the fundamental concepts of analysing and tuning the I/O subsystem remain.

Therefore the savvy and proactive System z customer will safeguard that they find a solution to deliver optimal DASD I/O performance. Without doubt, such an analysis could be performed by a highly-skilled individual, but today’s 21st Century world demands a hybrid of technical and commercial skills. Therefore a solution that incorporates the diagnostic knowledge of the most highly trained technician, performs intelligent analytics on a plethora of Sysplex wide performance data sources and presents the information required, is one that will deliver benefit each and every day. EADM is an example of such a solution, delivering demonstrable System z TCO optimization benefits, while safeguarding a short-term ROI, with simple deployment and resource utilization attributes.

Apple Style Meets IBM Substance

It was the early 1980’s when IBM first announced the Personal Computer (PC), a major breakthrough for delivering affordable and practical computing into the home.  One of the primary features of this computing evolution was the “open architecture” of the PC, built from off-the-shelf and commodity components.  Of course, we all know that around this time, DOS became MS-DOS via Bill Gates and Microsoft, where the rest as they say, is history!

At this time the IBM Mainframe (1964) had nearly 2 decades longevity and was already proving a scalable, secure and reliable platform.  So here we are, some 3 decades later, where Apple and IBM have announced a Global Partnership to Transform Enterprise Mobility.

Whatever your opinion of Apple technology, in the last decade or so they have undoubtedly delivered slick design and style for mobile devices, namely the smartphone and tablet.  Therefore whether the Enterprise accept the premise or not, Bring Your Own Device (BYOD) is inevitable, where employees expect to use their personal devices in the workplace.

IBM have continued to be a dominant force in the Enterprise market, whether with Mainframe technology or not, while establishing a credible presence in the Cloud market space.  As always the world of IT is constantly changing and even though IBM sold its PC business to Lenovo in 2004; some 10 years later, as part of the exclusive IBM MobileFirst for iOS agreement, IBM will sell iPhones and iPads with industry-specific solutions to business clients worldwide.

So what role if any will the IBM zSeries platform play in this Apple deal?  As always, the zSeries platform will deliver enterprise scalability and strength for Security, Database and Messaging integration, but beyond these features, I’m not so sure.  Of course, from a data presentation viewpoint, nothing changes, iOS integration and the ability to present Mainframe originated data remains forever thus for Apple and indeed all other mobile devices.  Similarly from a business transaction viewpoint, the zSeries platform participates in the delivery of mobile support, where from an IBM technology viewpoint, the Worklight solution is one example of an end-to-end integrated development studio software product.

Despite the obvious benefits for Apple, gaining access to the Enterprise via IBM technology and their customer base, and for IBM, delivering the market leading mobile technology into their customer base, what does this mean for the Enterprise?

Business as usual mostly, but Identity & Access Management (IAM) would appear to be a significant challenge.  Firstly, rightly or wrongly, most people don’t consider Apple software to have any security exposures, as the market place for iOS security solutions (E.g. Anti-Virus, Malware, zero day exploits, et al) is limited?  However, one might ponder why the Windows Operating System became such a target for the hacker.  Said hacker might be an opportunist, just because they can, or something more sinister, trying to gain government or business secrets.  So, if the Apple smartphone and tablet devices become ubiquitous if not de facto in the Enterprise, how long will it be before security exposures for iOS and related apps become common place?

I’m open-minded about BYOD (or am I)?  My heart tells me, yes, let the workers use their own device in the workplace, but my head tells me, no way!  Generally for technology decisions, my head always wins.  In this instance, I don’t think my head has a chance; overwhelming company worker desire to use their own mobile device in the workplace, whether iOS, Android, Java ME, Windows Phone, Blackberry, et al, will win out.  If this is the case, this is perhaps where the maturity and reliability of the IBM zSeries Mainframe can assist.

Therefore, at least for Identity & Access Management (IAM), secure access to the most valuable resource within an organization, the data itself via the zSeries server makes sense.  Whether this is via two if not several factor authentication remains to be seen.  However, I’m much more comfortable with an IAM solution that leverages from a Mainframe External Security Manager (ESM), namely ACF2, RACF or TopSecret, as opposed to a universal log-in via a Social Media web site, such as Facebook.  Just because you can log into an Enterprise and arguably mission critical CRM application, such as Salesforce via Facebook Authentication, doesn’t necessarily mean you should…

IBM Mainframe – Enterprise Software License Agreements Pros & Cons

An often quoted phrase in the Mainframe user base is “why are our Mainframe software costs so high”?  Sometimes we might have to look closer to home when finding the answers to our questions…

Over the years, Mainframe software portfolios in the customer environment might have become unwieldy, with duplication of software function, unused software, unsupported software products, and so on.  Typically this scenario occurs due to Merger & Acquisition (M&A) activity, where in an ideal world, a standard LPAR (image) with an optimally configured software portfolio would be deployed, which inevitably will generate the requirement for a modicum of migration activity, from one software product to another.  The complexity of software migration can change dramatically from a simple change, generally associated with Systems Management (E.g. Monitors) products to enormously complex, generally involving Database Subsystems (E.g. Adabas, DB2, IDMS, et al) and Programming Languages (E.g. COBOL, PLI, et al) while there is some middle ground with some Systems Management products (E.g. Security, Storage, Scheduling) that maintain metadata (policy data).  Therefore only the truly committed Mainframe user will adopt and fully commit to this standard LPAR methodology, benefitting to some extent from lower software costs.

Similarly over the last 20 years or so, the perceived requirement for Enterprise Software License Agreements has increased, where the fundamental premise is that such agreements make life easier for both the customer and ISV alike.  An interesting notion indeed, and one must draw one’s own conclusions as to whether such a utopia can exist; therefore as always, the caveat emptor (let the buyer beware) term must apply!

However, with such fully encompassing requirements and associated pricing mechanisms, the need for each and every major ISV to have a fully rounded software portfolio has ensued.  Therefore we have witnessed a lot of M&A activity in the Mainframe ISV market place, where several dominant players have emerged, in no particular order, BMC (Advantage), CA (FlexSelect, MLP, OLP) and IBM (ESSO, ELA), while some might say ASG should be included in this list.  Generally it seems to be the norm that each and every Mainframe customer will have at least one Enterprise Software License Agreement in place, typically with IBM because of the need to deploy the z/OS (z/VM, z/VSE, zLinux) operating system, generally in conjunction one other, whether ASG, BMC or CA.

The advantages of an Enterprise Software License Agreement are primarily:

  • Simplified license management via many products from one supplier
  • A several (3-5) year license agreement, only requiring periodic review and negotiation
  • Perceived cost benefit, with discount based upon volume, both in terms of software and CPU power
  • Perceived deployment benefit, treating Distributed and Mainframe platforms equally
  • Simplified support, as each and every software product should have the same look and feel

However, for a balanced review, we must identify the potential disadvantages, for example:

  • Is each and every software product from this single supplier the best for our business?
  • How do we renegotiate this agreement, because our business requirements have unexpectedly changed?
  • How do we exit this agreement, because our relationship with this supplier has failed?
  • How do we calculate a tangible cost and value for each and every product we deploy?

As always, the devil is in the detail, and although most pros and cons seem fairly innocuous at first glance, the considerations generated regarding contract termination or renegotiation are significant.  For example, if the Mainframe user chooses a 3 year Enterprise Software License Agreement, do they need to decide at least 18 Months before contract expiration that they must migrate to alternative software products, to terminate their relationship with a supplier?  So at first glance, volume discount and simplification look good, but how expensive and disruptive will contract termination be?

In real-life human terms, this is somewhat analogous to Marriage, a long-term relationship between two parties that choose to declare significant commitment to one another, but perhaps, the realm of possibility exists that said relationship will fail, and of course, in the absence of a bulletproof pre-nuptial, complications occur, and exit from the relationship is both financially expensive and disruptive.  Hmmm, so where is the equivalent of a pre-nuptial for the Enterprise Software License Agreement?  In an ideal world, the commercially savvy customer will have planned for such a possibility, but whether they have or have not, the supplier will have been paid for their software, and the customer may not have any choice but to renew or extend their agreement!  So which party is the winner and which one is the loser in such a scenario?  Does one party benefit from a heads we win and tails you lose proposition?

How does the Mainframe customer choose the best software product for their business requirement?  In an ideal world, they document their business requirement, collect information on market place offerings, review pricing options, generate a shortlist of suitable products, and eventually choose the “best-of-breed” product.  How is such a structured and balanced approach possible when deploying the Enterprise Software License Agreement?  The first thought must be cost based, as software has already been paid for, so if there’s a product in the portfolio we could use, we need to use it, whether it’s the best product or not.

If a Mainframe user is using an internal chargeback system for computing use, how can they fairly cost the pricing metric, if they don’t know the price of software products used?  Equally, how can the Mainframe user attempt to identify single product pricing when Enterprise Software License Agreements detail no granularity of pricing information?  Perhaps a modicum of research might help, where some global Government regulations dictate that contract details must be published for public scrutiny.  Therefore ISV Mainframe software list pricing details can be identified, for example, IBM and BMC.

One must draw one’s own conclusions, where some Mainframe customers may perform a structured review of the market place, and even though the technical recommendation might be for a product not covered by an Enterprise Software License Agreement, typically from a smaller ISV, the product chosen is one already paid for, or at least available from the Enterprise Software License Agreement.  This generates several issues, including but not limited to, alienating the smaller ISV community, having used them for expediency, and not delivering the best solution for your business…

So does the self-fulfilling prophecy ensue, where the Mainframe customer questions the cost of Mainframe software, but perhaps implicitly or unknowingly, said Mainframe user has contributed to such an environment, where a limited number Mainframe ISV’s control the Mainframe software market?

Isn’t it somewhat of a paradox that in The UK, the monopolies commission would review the merits of an M&A between two major grocery supermarket or energy supplier companies, and yet whether in The UK or globally, there are several major ISV’s (E.g. ASG, BMC, CA, IBM) dominating the Mainframe software market, primarily via Enterprise Software License Agreements?  Can this really be a good thing for the Mainframe user, limited supplier choice and therefore a lack of healthy competition?

Perhaps it is the responsibility of the Mainframe user to actually choose software impartially, and from time-to-time choose the best product, regardless of ISV.  This might generate a more active market place for software choice, while it was forever thus, the larger ISV is so big that they can easily acquire the smaller ISV who has developed and sold a good product, but at least the Mainframe ISV market place continues to evolve.  In this case, it seems somewhat logical that the Mainframe user is in control of their destiny, but only by safeguarding that their default option is not the Enterprise Software License Agreement.  They encourage an active and impartial ISV software market by dispassionately reviewing the open market and choosing the best Mainframe software product for their business!

Lewis Carroll once said “integrity is doing the right thing, even when no one is watching”!  When was the last time a major ISV declared an open book policy for your business, offering you flexible options to benefit from their Enterprise Software License Agreement, while allowing you to choose a best-of-breed software product, but not from their software portfolio, giving you a discount (credit note) for their software product that didn’t match your business requirement?