Simplified Business Facing IBM Z Mainframe DevOps APM Problem Determination

Increasingly IBM Z Mainframe stakeholders are becoming cognizant that traditional processes for handling Information Technology operations are becoming obsolete, hence the emergence of DevOps (DevSecOps) frameworks.  Driven by digital transformation & the perpetually increasing demand for new digital services, consuming vast unparalleled amounts of data, Data Centres are becoming increasingly pressurized to deliver & maintain these mission-critical services.  A major challenge is the availability of these services, where transaction & throughput workloads can be unpredictable, often ad-hoc demand driven (E.g. Consumer) & not the typical periodic planned peaks (E.g. Monthly, Annual, et al).

Today’s inward facing, dispassionate & honest CIO knows their organization can spend inordinate amounts of time, being reactive to business application impact incidents, often finding they spend too long reacting to incidents & all too often they don’t have enough bandwidth to be proactive & prevent the incident from occurring in the first place.  It’s widely accepted that for the majority of Global 1000 companies, deploying an IBM Z Mainframe platform provides them with the de facto System Of Record (SOR) data platform, with associated Database (E.g. Db2) & Transaction (E.g. CICS, IMS) subsystems.  Therefore playing such a central & integral part of today’s 21st century digital application infrastructure, business performance issues can affect the entire application, dictating that early detection & resolution of performance issues are business critical, with the ultimate goal of eliminating such issues altogether.

Technologies such as z/OS Connect, provide a simple & intuitive API based method for the IBM Z Mainframe to become an interconnected platform, with all other Distributed Platforms.  This dictates the evolution in Operations Management processes, considering the business application from a non-technical viewpoint, treating management from a holistic viewpoint with end-to-end monitoring, regardless of the underlying hardware & software platforms.

Today’s 21st Century digital economy dictates that central Operation teams don’t have inordinate amounts of time & indeed the requisite Subject Matter Expert (SME) skills for problem investigation activities.  A more proactive & automated response would be the deployment of simplified, lean & cost-efficient automated monitoring processes, allowing Operations teams to detect potential problems & their associated failure reason in near real-time.

Distributed tracing provides a methodology for interpreting how applications function across processes, services & networks.  Tracing uses the associated activity log trail from requests processed, capturing tracing information accordingly, as they move from interconnected system to system.  Therefore with Distributed tracing, organisations can monitor applications with Event Streams, helping them to understand the size & shape of the generated traffic, assisting them in the identification & related causes of potential business application problems.  It comes as no surprise that Distributed tracing has become a pivotal cornerstone of the DevOps toolbox, leveraging from the pervasive Kafka Open-Source Software architecture technology for distributed systems.  Simply, Kafka provides meaningful context from messaging & logging generated by IT platforms various, delivering data flow visualizations, simplifying identification & prioritization of business application performance anomalies.  Put simply, Kafka Distributed tracing pinpoints where failures occur & what causes poor performance (I.E. X marks the spot)!

From a business & therefore non-technical viewpoint, the utopia is to understand the user experiences delivered & associated business impacts; ideally positive, therefore eliminating the negative.  Traditionally from a technical viewpoint, experts have focussed on MELT (Metrics, Events, Logs, Traces) data collection, allowing for potential future problem determination & resolution.  Historically when this was the only data available, it therefore follows, manual & time consuming technical processes ensued.  As we have explored, DevOps is about simplification, optimization, automation & ultimately delivering the best business service!  If only there was a better way…

OpenTelemetry is a collection of tools, APIs & SDKs, utilized to instrument, generate, collect & export telemetry data (Metrics, Events, Logs, Traces) to assist software performance behavioural analysis.  Put simply, OpenTelemetry is an Open-Source Software vendor agnostic standard for application telemetry & supporting infrastructures & services data collection:

  • APIs: Code instrumentation deployment for telemetry data trace generation
  • SDKs: Collect the telemetry data for the rest of the telemetry data processing
  • In-Process Exporters: Translate telemetry data into custom formats for Back-End processing or
  • Out-Of-Process Exporters: Translate telemetry data into custom formats for Back-End processing

In conclusion, from a big picture viewpoint, the IBM Z Mainframe is just another IP node on the network, seamlessly interconnecting with Distributed Systems platforms for 21st century digital business application processing.  Regardless of technical platform, DevOps is not a technical discipline, it’s a business orientated user experience process & as such, requires automated issue detection & rapid resolution.  Open-Source Software (OSS) frameworks such as OpenTelemetry & Distributed Tracing allow for the simplified low cost collection & visualization of instrumentation data.  How can the IBM Z Mainframe organization incorporate a DevOps facing solution to aggregate this log data, providing an optimal cost, resource friendly Application Performance Management (APM) solution for simplified business application performance identification?

z/IRIS (Integrable Real-Time Information Streaming) integrates the IBM Z Mainframe platform into commonplace pervasive enterprise wide Application Performance Monitoring (APM) solutions, allowing DevOps resources to gain the insights they need to better understand Mainframe utilization & potential issues for mission critical business services.

z/IRIS incorporates OpenTelemetry observability for IBM Z Mainframe systems & applications, enriching traces (E.g. Db2 Accounting, Db2 Deadlock, zOS Connect, JES2, OMVS, STC, TSO) with attributes to facilitate searching, filtering & analysis of traces in today’s 3rd party enterprise wide APM tools (E.g. AppDynamics, Datadog, Dynatrace, IBM Instana, Jaeger, New Relic, Splunk, Sumo Logic).

Capturing metrics & creating associated charts has been an integral part of performance monitoring for several decades or more.  z/IRIS seamlessly integrates with APM tools such as Instana & data visualization tools such as Grafana to supply zero maintenance automated dashboards for commonplace day-to-day usage.  Of course, each & every business requires their own perspectives, hence z/IRIS incorporates easy-to-use customizable dashboards for such requirements. Because APM & data visualization tools collect data metrics from a variety of information sources, tracing every request from cradle (E.g. Client Browser) to Grave (E.g. Host Server), the z/IRIS Mainframe data combinations for your digital dashboards are potentially infinite, where the data presented is always accurate & in real time.

z/IRIS is simple to use & simple to install, incorporating many tried & tested industry standard Open-Source Software components, optimizing costs & simplifying product support.  Wherever possible, using Java based applications, from an IBM Z Mainframe viewpoint, CPU utilization is minimized, utilizing zIIP processing cycles whenever available.  z/IRIS delivers a lightweight, resource & cost efficient z/OS APM solution to provide an end-to-end performance analysis of today’s 21st Century digital solutions.  Because z/IRIS leverages from industry standard Open-Source frameworks deployed by commonplace Distributed Systems APM solutions, the instrumentation captured & interpreted by z/IRIS enriches dynamically as APM functionality increases.  For example, Datadog Watchdog Insights can identify increased latency from a downstream z/OS Connect application, just by processing new analytics, from existing telemetry data.  The data had already been captured, as APM functionality evolves, new meaningful business insights are gained.  z/IRIS can deliver the following example benefits for any typical IBM Z Mainframe DevOps environment:

  • Automated IBM Z Mainframe Observability: Automate the collection of end-to-end data tracing information.
  • Real Time Impact Notification: Intelligent data processing to present meaningful DevOps dashboard notifications of business applications service status & variances.
  • Universal Access & Ease Of Use: Facilitate end-to-end Application Performance Monitoring (APM) for all IT teams, not just IBM Z Mainframe Subject Matter Experts (SME).
  • Reduce MTTD & MTTR For Optimized User Services: Reduce Mean Time To Detect (MTTD) & ideally eradicate the Mean Time To Repair (MTTR), the typical Key Performance Indicators (KPIs), with intelligent root cause analysis.

IBM Z Mainframe Pre-Production Testing: Spring Into Stress Testing via zBuRST

For those of us in the Northern Hemisphere it’s been another long & cold Winter & for many, a time of pandemic lockdown.  As we enter Spring, we often associate this annual season with hope & new life & perhaps opportunity.  Henry Wadsworth Longfellow once wrote ”If Spring came but once in a century, instead of once a year, or burst forth with the sound of an earthquake, and not in silence, what wonder and expectation would there be in all hearts to behold the miraculous change”!  Let’s not carried away, but I have recently worked with an IBM Z customer to finally perform a Pre-Production full workload test via the IBM Z Business Resiliency Stress Test (zBuRST) solution…

In an ideal world, zBuRST would offer a much needed solution for all IBM Z Mainframe users with limited resource or budget to perform Pre-Production full workload testing activities.  However, in reality, there are some significant qualification caveats, primarily a minimum of 10,000 MIPS workload capacity & the need for latest generation z14 or newer Mainframe servers.  As with anything in business or indeed life, if you don’t ask, you will never know & there is some flexibility from an installed MIPS viewpoint via your local IBM account team.

IBM Z Business Resiliency Stress Test (zBuRST) is a solution that enables the use of spare IBM Z server physical resources to stress test changes at Production workload scale, allowing qualitative & quantitative validation of any Production change to safeguard the performance & resilience profile of IBM Z mission critical workloads.  For the avoidance of doubt, a Pre-Production test can be verified with a minimal data subset for qualitative purposes, but only a 100%+ data quantitative stress test will verify the SLA & KPI metrics required for a mission critical workload.  zBuRST only supports Pre-Production (DevTest) environments, which could include a GDPS internal environment, or a 3rd party DR supplier.  However, zBuRST cannot be used for any DR activity, testing or real-life invocation.  Hopefully most IBM Z mainframe users are savvy & have included some flexibility in their 3rd party DR provision contracts, allowing for periodic use of such facilities, not solely DR based.  This is not an unusual requirement & if you rely upon a 3rd party provider for IBM Z resilience, work with them to evolve your IBM Z resource provision service contracts accordingly.

From a big picture viewpoint, zBuRST reduces change risk, safeguarding business resiliency by enabling the detection and resolution of abnormalities and defects in a Pre-Production environment, which inevitably manifest business outages, disruptions, or slowdowns:

  • For IBM Z users with matching (identical) hardware in a standalone test or DR environment, zBuRST provides the ability to perform load or stress test of new IBM Z hardware features & upgraded functions.
  • For IBM Z users whose DR sites do not match their Production environment, the zBuRST objective is to enable critical workload (E.g. use all available resource to test the mission critical workloads) testing.

From an eligibility viewpoint, if your organization is currently testing with constrained IBM Z resources, prohibiting adequate Production workload sized testing, zBuRST improves workload resiliency:

  • Can your business scale reliably & conform to SLA & KPI Metrics during seasonal or ad-hoc peak processing demands (E.g. Year End, Black Friday, Cyber Monday, et al)?
  • Is your business mission critical application impacted by change aversion, with fear of disrupting Production stability?
  • Are your agile DevOps aspirations hampered by the legacy waterfall application development approach, taking too long to adequately test changes, or introduce new features, functions, for Production workloads?
  • Do elongated Production outages (I.E. Downtime) come at an excessive or prohibitive business cost?
  • Is it too complex to provision adequate local or 3rd party IBM Z resources for large scale volume or integration tests?

The zBuRST solution has a number of prerequisites & the primary considerations are:

  • zBuRST is an extension of the IBM Z Application Development and Test Solution (DevTest Solution).
  • zBuRST Tokens are discounted at 80% from the cost of On/Off CoD capacity.
  • zBuRST can be purchased or for systems with a minimum of 10,000 installed MIPS, for up to 50-100% of Production capacity.  All MIPS capacity must reside in the same country.
  • zBuRST pre-paid tokens can be purchased up to 100% of the additional capacity needed to support Production scale stress testing.
  • zBuRST tokens allow for up to 15 days of testing; tokens can be activated for any 15 calendar days, whether consecutive or not (E.g. Preform n stress tests of n days duration).
  • zBuRST tokens expire 5-years from the IBM Z server LICC “Withdrawal from Marketing” date.
  • For DevTest Solutions, zBuRST capacity can be purchased to increase the size of the DevTest environment up to the equivalent number of Production MIPS.
  • For DR machine usage, zBuRST tokens can be purchased up to the equivalent number of Production MIPS.
  • zBuRST tokens can only be installed & exclusively used on IBM Z hardware owned by the IBM Z user (customer); zBuRST is not available to 3rd party IBM Z resource service providers.
  • zBuRST tokens are pre-paid On/Off CoD LIC records.  There can only be one On/Off CoD record active at a time.  Post-paid On/Off CoD LIC records & zBuRST tokens cannot be active at the same time on the same machine.  There cannot be mixing of pre-paid & post-paid On/Off CoD LIC records.

zBuRST can deliver greater certainty & benefit for an IBM Z organization via:

  • Change risk eradication with Production workload stress testing, increasing business resiliency, customer satisfaction & operational efficiency.
  • Faster delivery of new business features & functions at reduced risk, enabling an agile DevOps application change environment.
  • Empowering IT personnel to safely test changes, at Production workload scale, in a DevTest environment, identifying problems or anomalies that might or typically only occur at scale.
  • Higher ROI for DR resource usage (E.g. Use for stress testing, not just for DR testing).
  • Increased & comprehensive application testing capabilities for a lower cost.

When working with my customer over the last few months, the real-life lessons learned were:

  • Collaborate with the 3rd party IBM Z resource supplier, to safeguard the use of their IBM Z server based upon a days as opposed to a DR testing usage approach.  For the avoidance of doubt, contract for n days, where those n days could be used for any number of Pre-Production testing & DR usage.
  • Engage with all ISV organizations from an FYI viewpoint, informing them of this DevTest approach, where their software will be used for Pre-Production testing purposes, allowing them to safely generate temporary software license codes accordingly, as & if required.
  • Work really closely with your IBM account team, as this customer was a ~9,000 MIPS user & find a win-win situation for all.  That could be the provision of anticipated White Space CPU capacity by IBM or as a committed IBM Z Mainframe user, maybe the 10,000 MIPS watermark is just too high.
  • Educate your Operations, Applications & Business units on this zBuRST options.  Some IBM Z users might have been restricted for years if not decades, not being able to perform a 100% data & CPU resource Pre-Production workload test.  The brainstorming, collaboration & good will that manifests itself, is one of those few occasions in IT where the users of your IT services are happy to be an integral part of the change process!

My final observation is a reflection on the last few months of my day-to-day activities.  For 2-3 days per week, I have been combining IT work with being “Captain Clipboard” at a local UK COVID-19 vaccination centre, which in itself, has been so rewarding.  To see the relief on people, especially those that are of a mature age, perhaps infirmed, feeling they can be a part of the wider community again.  The parallels are obvious, zBuRST can allow those IBM Z users prohibited from performing 100% data & CPU Pre-Production testing activities, the opportunity to advance their business.  However, unlike the COVID-19 vaccination, which for the fortunate developed countries, is available to all citizens, zBuRST does have some usage restrictions.  Perhaps it’s up to the wider IBM Z user community to encourage IBM to revisit & modify their approach, perhaps reducing the MIPS capacity requirements to 5,000 MIPS.  Wherever you’re based globally, if you’re a member of SHARE (USA) or GSE (Europe), et al, maybe reach out to your Large Systems representatives & see if the global collective from the IBM Z user organizations can encourage IBM to evolve their opportunity, enabling zBuRST solution usage to a larger majority if not all IBM Z Mainframe users.

Pervasive Encryption & Compression: Why z15 Upgrade Activities Are Optimal & Strategic

A recent IBM Security sponsored Cost of a Data Breach report by the Ponemon Institute highlighted an average data breach cost of $3.86 Million.  Personally Identifiable Information (PII) accounted for ~80% of data breaches, exposing a range of between 3,400 & 99,730 compromised records.  The term Mega Breach is becoming more commonplace, classified as the exposure of 1+ Million records, where the average cost for such events increases exponentially, ~$50 Million for exposures up to 10 Million records, rising to ~$392 Million for exposures of 50+ Million records.  From an incident containment viewpoint, organizations typically require 207 days to identify & 73 days to contain a breach, totalling an average lifecycle of 280 days.  Seemingly the majority (I.E. 95%+) of data records breached were not encrypted.  I think we can all agree to agree, prevention is better than cure & the costs of these data breaches are arguably immeasurable to an organization in terms of customer trust & revenue downturn…

With the launch of IBM z14 in 2017, IBM announced its core CPU hardware included the Central Processor Assist for Cryptographic Function (CPACF) encryption feature embedded in the processor chip.  The ability to encrypt data, both at rest & in flight, for a low cost, was good news for IBM Z customers concerned about data security.  Classified as Pervasive Encryption (PE), the capability was designed to universally simplify data encryption processes, eradicating potential sources of data loss due to unwanted data breach scenarios.

It’s patently obvious that encryption inflates data & so we must consider the pros & cons of data compression accordingly.  An obvious downside of z14 data encryption is that it can render storage-level compression ineffective, because once the data is encrypted, it is not easily compressed.  A zEnterprise Data Compression (zEDC) card could be deployed to compress the data before encryption, but with added expense!  Wouldn’t it be good if data compression & encryption were performed on the CPU core?

For the IBM z15, with the Integrated Accelerator for zEnterprise Data Compression (zEDC), the industry standard compression used by zEDC is now built into the z15 core, vis-à-vis encryption via CPACF.  IBM z15 customers can now have the best of both worlds with compression, followed by encryption, delivered by the processor cores.  Therefore encryption becomes even less expensive, because after data compression, there is significantly less data to encrypt!

zEDC can perform compression for the following data classification types:

  • z/OS data (SMF logstreams, BSAM & QSAM data sets, DFSMShsm & DFSMSdss processing)
  • z/OS Distributed File Service (zFS) data
  • z/OS applications, using standard Java packages or zlib APIs
  • z/OS databases (Db2 Large Objects, Db2 Archive Logs, ContentManager OnDemand)
  • Network transmission (Sterling Connect:Direct)

Arguably the increase in remote working due to COVID-19 will increase the likelihood & therefore cost of data breaches & although encryption isn’t the silver bullet to hacking remediation, it goes a long way.  The IBM Z Mainframe might be the most securable platform, but it’s as vulnerable to security breaches as any other platform, primarily due to human beings, whether the obvious external hacker, or other factors, such as the insider threat, social engineering, software bugs, poor security processes, et al.  If it isn’t already obvious, organizations must periodically & proactively perform Security Audit, Penetration Test & Vulnerability Assessment activities, naming but a few, to combat the aforementioned costs of a security breach.

Over the decades, IBM Z Mainframe upgrade opportunities manifest themselves every several years & of course, high end organizations are likely to upgrade each & every time.  With a demonstrable TCO & ROI proposition, why not, but for many organizations, such an approach is not practicable or financially justifiable.  Occasionally, the “stars align” & an IBM Z Mainframe upgrade activity becomes significantly strategic for all users.

The IBM z15 platform is such a timeframe.  Very rarely do significant storage & security functions coincide, in this instance on-board CPU core data compression & encryption, eradicating host resource (I.E. Software, Hardware) usage concerns, safeguarding CPU (I.E. MSU, MIPS) usage optimization.  External factors such as global data privacy standards (E.g. EU GDPR, US PII) & associated data breach penalties, increase the need for strategic proactive security processes, with data encryption, high on the list of requirements.  Add in the IBM Z Tailored Fit Pricing (TFP) option, simplifying software costs, the need to compress & encrypt data without adding to the host CPU baseline, the IBM z15 platform is ideally suited for these converging requirements.  Pervasive Encryption (PE) was introduced on the IBM z14 platform, but on-board CPU core compression was not; GDPR implementation was required by 25 May 2018, with associated significant financial penalties & disclosure requirements; IBM Z Tailored Fit Pricing (TFP) was announced on 14 May 2019, typically based upon an MSU consumption baseline.

Incidentally, the IBM z15 platform can transform your application & data portfolio with enterprise class data privacy, security & cyber resiliency capabilities, delivered via a hybrid cloud.  Mainframe organisations can now get products & ideas to market faster, avoiding cloud security risks & complex migration challenges.  With the Application Discovery & Delivery Intelligence (ADDI) analytical platform, cognitive technologies are deployed to quickly analyse Mainframe applications, discovering & understanding interdependencies, minimizing change risk, for rapid application modernization. In conclusion, a year after the IBM z15 platform was announced in September 2019, field deployments are high, with the majority of promised function delivered & field tested.  With the ever-increasing cybersecurity threat of data breaches & an opportunity to simplify IBM Z software Monthly License Charges (MLC), a z15 upgrade is both strategic & savvy.  Even & maybe especially, if you’re using older IBM Z server hardware (E.g. z13, zxC12, z114/z196, z10, z9, et al), your organization can easily produce a cost justified business case, based upon reduced software costs, Tailored Fit Pricing or not, optimized compression & encryption, delivering the most securable platform for your organization & its customers.  Combined with proactive security processes to eliminate a myriad of risk register items, maybe that’s a proposition your business leaders might just willingly subscribe to…

Simplifying Db2 for z/OS CPU Optimization: Eradicating Inefficient SQL Processing

Without doubt the IBM Z Mainframe server is recognised as the de facto choice for storing mission critical System of record (SOR) data in database repositories for 92 of the top 100 global banks, 23 of the 25 top global airlines; the top 10 global insurers & ~70% of all Fortune 500 companies. ~80% of mission critical data is hosted by IBM Z Mainframe servers, processing 30+ Billion transactions per day, including ~90% of all credit card transactions. This data is accessed by ~1.3 Million CICS transactions per second, compared with a Google (mostly search) processing rate of ~70,000 transactions per second. Interestingly enough, despite processing so many mission critical transactions the IBM Z Mainframe server platform is only accountable for ~6.2% of global IT spend. One must draw one’s own conclusions as to why some IT professionals perceive the IBM Z Mainframe server as being a legacy platform, not worthy of consideration as a strategic IT server platform…

The digital transformation has delivered an exponential growth of data, typically classified as Cloud, Mobile & Social based. This current & ever-growing data source requires intelligent analytics to deliver meaningful business decisions, requiring agile application software delivery to gain competitive edge. This digital approach can sometimes deliver a myriad of micro business application changes, personalised for each & every customer, often delivering “pop-up” applications…

IBM Z Mainframe software costs are often criticized as being a major barrier to maintaining or indeed commissioning the platform. IBM have tried to minimize these costs with numerous sub-capacity pricing options over the last 30 years or so, but this is perceived by many as being overly complicated; although with a modicum of knowledge, a specialized personnel resource can easily control software costs. All that said, IBM have introduced Tailored Fit Pricing for IBM Z, in an attempt to simplify software cost management. A recent blog reviewed the Tailored Fit Pricing for IBM Z offering & whether you decide whether this IBM Z pricing mechanism is suitable for your organization, optimizing IBM Z CPU MSU/MIPS usage is mandatory. Recognizing that the IBM Z Mainframe server is the de facto database server for System of Record data, primarily via the Db2 subsystem, clearly optimizing Db2 CPU usage, whether OLTP transactions, typically via CICS, or the batch window, has been & always will be, worthwhile…

All too often, many IT disciplines can be classified with a generic 80/20 rule & typically data can be classified accordingly, where 80% of data is accessed 20% of the time & 20% of data is accessed 80% of the time. The challenge with such a blunt Rule of Thumb (ROT) is that it’s static, but it’s a good starting point. Ideally for any large data source, there would be a dynamic sampling mechanism that would identify the most active data, loading this into the highest speed memory resource to reduce I/O access times & therefore CPU usage. Dynamic management of such a data buffer would render the 80/20 rule extraneous to requirements, as each & every business has their own data access profile. However, a simple cost benefit & therefore Proof of Value (POV) analysis could ensue.

From a Db2 viewpoint, pre-defined structures such as buffer pools offer some relief in storing highly referenced data in a high-speed server memory resource, but this has a finite capacity versus performance benefit, not necessarily using the fastest memory structures available nor dynamically caching the most accessed data. The business considerations of not optimizing Db2 data access are:

  • Elongated Batch Processing: With ever increasing amounts of data to process & greater demands for 247365 availability & real-time access, data access optimization is fundamental for optimized service delivery, often measured by mission critical SLA & KPI metrics. Optimized batch processing is a fundamental requirement for acceptable customer facing business service delivery.
  • Slow Transaction Response Times: As the nature of customer requirements change, mobile device applications exponentially increasing the number of daily transactions, overall system resource capacity constraints are often stressed during peak hours. Optimized transaction response time is a fundamental requirement, being the most transparent service delivered to each & every end customer.

An easy but very expensive solution to remediate batch processing & transaction response issues is to provide more resources via a CPU server upgrade activity. A more sensible approach is to optimize the currently deployed resources, safeguarding that frequently accessed data is mostly if not always high speed cache resident, reducing the I/O processing overhead, reducing CPU usage, which in turn will optimize batch processing & transaction response times, while controlling associated IBM Z Mainframe server hardware & software costs.

The ubiquitous Db2 data access method is Structured Query Language (SQL) based, where IBM has their own implementation, SQL for Db2 for z/OS, which could be via the commonly used COBOL (EXEC SQL) programming language or a Db2 Connect API (E.g. ADO.NET, CLI, Embedded SQL, JDBC, ODBC, OLE DB, Perl, PHP, pureQuery, Python, Ruby, SQLJ). For Db2 Connect, there are 2 types of embedded SQL processing, static & dynamic SQL. Static SQL minimizes execution time by processing in advance. Though some relief is provided by Dynamic Statement Cache, dynamic SQL is processed when the SQL statement is submitted to the IBM Z Db2 server. Dynamic SQL is more flexible, but potentially slower. The decision to use static or dynamic SQL is typically made by the application programmer. There is a danger that Dynamic Statement Cache might be considered as a panacea for SQL CPU performance optimization, but as per any other performance activity, reviewing any historical changes is a good idea. The realm of possibility exists for the Db2 Subject Matter Expert (SME) to be pleasantly surprised that more often than not, there are still significant SQL CPU optimization opportunities…

From a generic Db2 viewpoint, with static SQL, you cannot change the form of SQL statements unless you make changes to the program. However, you can increase the flexibility of static statements by using host variables. Obviously, application program changes are not always desirable.

Dynamic SQL provides flexibility, if an application program needs to process many data types & structures, dictating that the program cannot define a model for each one, dynamic SQL overcomes this challenge. Dynamic SQL processing is facilitated by Query Management Facility (QMF), SQL Processing Using File Input (SPUFI) or the UNIX Systems Services (USS) Command Line Processor (CLP). Not all SQL statements are supported when using dynamic SQL. A Db2 application program that processes dynamic SQL accepts as input, or generates, an SQL statement in the form of a character string. Programming is simplified when you can structure programs not to use SELECT statements, or to use only those that return a known number of values of known types.

For Db2 data access, SQL statement processing requires an access path. The major SQL statement performance factors to consider are the amount of time that Db2 uses to determine the access path at run time & whether the access path is efficient. Db2 determines the SQL statement access path either when you bind the plan or package that contains the SQL statement or when the SQL statement executes. The repeating cost of preparing a dynamic SQL statement can make the performance worse when compared with static SQL statements. However, if you execute the same SQL statement often, using the dynamic SQL statement cache decreases the number of times dynamic statements must be prepared.

Typically, organizations have embraced static SQL over dynamic because static is more predictable, showing little or no change, while dynamic implies ever changing & unpredictable. Db2 performance optimization functions have been incorporated into base Db2 (E.g. Buffer Pools) & software products (E.g. IBM Db2 AI for z/OS, IBM Db2 for z/OS Optimizer, IBM Db2 Analytics Accelerator, IBM Z Table Accelerator, IZTA), with varying levels of benefit & cost. Ultimately IBM Z Mainframe customers need simple cost-efficient off-the-shelf solutions of a plug & play variety & without doubt, optimizing static SQL data processing is a pragmatic option for reducing Db2 subsystem CPU usage.

In Db2 Version 10, support for 64-bit run time was introduced, providing Virtual Storage Constraint Relief (VSCR), improving the vertical scalability of Db2 subsystems. With Db2 Version 11, the key z/Architecture benefit of 64-bit virtual addressing support was finally introduced, increasing capacity of central memory & virtual address spaces from 2 GB to 16 EB (Exabytes), eliminating most storage constraints. It therefore follows that any Db2 CPU performance optimization solution should also exploit the z/Architecture 64-bit feature, to support the ever-increasing data storage requirements of today’s digital workloads.

As we have identified, Db2 can consume significant amounts of z/OS CPU accessing & retrieving the same static frequently used data elements repetitively. Upon analysis, these static frequently used data elements are typically identified originating from a small percentage of Db2 tablespaces. Typically, at first glance these simple SQL programs are considered as low risk, but are repeatedly processed, often in peak processing times, consuming excessive CPU & increasing processing cost accordingly, typically z/OS Monthly Licence Charges (MLC) related. Db2 optimization tools for access path or buffer pool management provide some benefit, but this is not always significant & may require application changes. Patently there is a clear & present requirement for a simple plug & play solution, transparent to Db2 processing, maintaining an optimized high-performance in-memory cache of frequently used Db2 data, safeguarding data integrity in environments various, including SYSPLEX, Data Sharing, et al…

QuickSelect is a plug-in solution dynamically activated in a batch or OLTP environment (I.E. CICS, IMS/TM) intercepting repetitive SQL statements from Db2 application programs, storing the most active result set, not necessarily the entire tablespace, in a high-performance in-memory cache, returning to applications the same result set as per Db2, but much faster & using less CPU accordingly. QuickSelect is completely transparent to z/OS applications, eliminating any requirement to change/recompile/relink application source or rebind packages. QuickSelect processing can be switched on or off using a single keystroke, either defaulting to standard Db2 SQL processing or to benefit from the QuickSelect high-speed cache for optimized CPU resource usage.

The 64-bit QuickSelect server, implemented as a started task, intelligently caching data in self-managed memory above the bar, supporting up to 16 EB of memory, eliminating concerns of using any other commonly used storage areas (E.g. ECSA). The intelligent caching mechanism safeguards that only highly active data is retained, optimizing the associated cache memory size required.

QuickSelect caches frequently requested Db2 SQL result sets, returning these results to the application from QuickSelect cache, when a repetition of the same SQL is encountered. For data integrity purposes, QuickSelect immediately invalidates result sets upon detection of changes to underlying tables, implicitly validating each cache resident SQL result set. Changes to Db2 data by application programs are captured by a standard Db2 VALIDPROC process, attached to the typically small subset of frequently accessed tables of interest to QuickSelect. Db2 automatically activates the VALIDPROC routine whenever the table contents are changed by INSERT, DELETE, UPDATE or TRUNCATE statements, invalidating cached data from the updated tables automatically. For standard Db2 utilities such as LOAD/REPLACE, REORG/DISCARD & RECOVER, table-level changes are identified by a QuickSelect utility-trap, invalidating cached data from the updated tables automatically. QuickSelect also supports SYSPLEX & Data Sharing environments, supporting update activity via the same XCF functions & processes used by Db2.

QuickSelect delivers the following benefits:

  • CPU Savings: Meaningful reduction (E.g. 20%) in the Db2 SQL direct processing; 10%+ peak time CPU reduction is not uncommon.
  • Faster Processing: Optimized CPU usage delivers shorter batch processing & OLTP transaction response times, for related SLA & KPI objective compliance.
  • Transparent Implementation: No application changes required, source code, load module or Db2 package.
  • Survey Mode: Unobtrusive & minimal Db2 workload overhead data sampling to identify potential CPU savings from repetitive SQL & tables of interest, before implementation.
  • Staggered Deployment: Granular criteria (E.g. Job, Program, Table, Transaction, Etc.) implementation ability.
  • Reporting & Analytics: Extensive information detailing cache usage for Db2 programs & tables.

Since 1993 Db2 has evolved dramatically, in line with the evolution of the IBM Z Mainframe server. When considering today’s requirement for a digital world, processing ever increasing amounts of mission critical data, a base requirement to optimize CPU processing for Db2 SQL data access is mandatory. In a hybrid support environment where today’s IBM Z Mainframe support resource requires an even blend of technical & business skills, plug & play, easy-to-use & results driven solutions are required to optimize CPU usage, transparent to the subsystem & related application programs. QuickSelect is such a solution, fully exploiting 64-bit z/Architecture for ultimate scalability, identifying & resolving a common CPU consuming data access problem, for a mission critical resource, namely the Db2 subsystem, maintaining mission-critical System of Record data.

z/OS CPU optimization is a mandatory requirement for every organization, to reduce associated software & hardware costs & in theory, as a mandatory pre requisite for deploying the Tailored Fit Pricing for IBM Z pricing mechanism. Tailored Fit Pricing uses the previous 12 Months SCRT submissions to establish a baseline for MSU charging over a contracted period, typically 3 years. If there are any unused MSU resources, these are carried forward to the next year, but if those MSU resources remain unused at the end of the contracted period, they are lost, meaning the organization has paid too much. If the MSU resource exceeds the agreed Tailored Fit Pricing, excess MSU resources are charged at a discounted rate. Clearly achieving an optimal MSU baseline before embarking on a Tailored Fit Pricing contract is arguably mandatory & it therefore follows that optimizing CPU forever more, safeguards optimal z/OS MLC charging during the Tailored Fit Pricing contract. QuickSelect for Db2 is a seamless CPU optimization product that will perpetually deliver benefit, assisting organizations minimize their z/OS MLC costs, whether they continue to proactively manage the R4HA, submitting monthly SCRT reports or they embark on a Tailored Fit Pricing contract…

Smartphone Security Dependency: Applying Mainframe Common Sense To Real Life…

I’m by no means a security expert, for that discipline we must acknowledge RSM Partners in the IBM Mainframe space & I congratulate Mark Wilson, their Management Team & personnel on their recent acquisition by BMC.

One way or another, for 25 years since 1995 I have been a carer for my parents who both died of brain cancer & dementia, my Father in 2003 & my Mother in the last few months.  Other than to pick up mail & perform minimal house maintenance duties, I haven’t lived at my house since October 2018.  Of all my achievements in life, keeping both of my parents out of a specialized care setting is without doubt my greatest, on my own, being a widow & having outlived my only sibling when I was 9 years old.  Indeed, when I look back on things, how I have managed to balance this family activity with any type of career development seems incredulous to me.  Perhaps I can now concentrate on my alleged Mainframer day job…

It’s amazing the skills you can learn away from your day job & even in recent bereavement, dealing with the bureaucracy of probate can teach you something, especially at this current juncture, where we finally seem to be in the midst of a widespread password to Multi-Factor Authentication (MFA) security evolution!

Having to deal with a probate estate, including property, there are some recurring costs you have to pay, primarily, power, water, telecommunications, local authority, et al, while you await grant of probate & eventually sell the house.  Of course, you need a bank account to do this & for want of a better term, I decided to make lemonade out of lemons for this seemingly mundane activity.  Currently, in the UK, many of the major current account providers want your business & offer switching inducements of ~£100-£175.  I have switched current accounts 3 times in the last few months, accumulating ~£500 that I have donated to a homeless charity.  As somebody much wiser than I once noted, there’s always somebody in a worse situation than you & having to face my first Christmas without a blood relative, this year I volunteered for said homeless charity, which once again, was a real eye opener.

What became obvious while I was subscribing to & switching from these largely UK clearing bank current accounts, was the changeover from a password & memorable information account authentication system, to a password & One Time Passcode (OTP) via Mobile Phone SMS (Text Message) protocol.  Each of these clearing banks deploy the latest IBM Z Mainframe for their System Of Record (SOR) data & security management, but technology doesn’t make for a bulletproof system, because as always, there is the human user of these systems.  My experiences of dealing with my elderly & frail Mother in her last few years then became pertinent, as in her heyday, Mum had the most amazing memory, used & commissioned mini computers herself in the early 1980’s, but the degeneration of her motor & neurological abilities, rendered her largely helpless trying to use a smartphone.  Of course, this will apply to many people, of all ages with health challenges various; do technology advances exclude them from 21st century technology & services?

In theory, hopefully most organizations are now realizing that passwords are a major vulnerability, at least from a human viewpoint & I guess us IT folks all know the statistics of how long it takes to crack a password of various lengths & character composition.  Even from my own viewpoint, for many years I have been using a Password Manager, where my password to access this system exceeds 50 characters in length.  I have tens of passwords in this system, I don’t know any of them, they’re all automatically generated & encrypted.  However, if this Password Manager is compromised, I don’t expose one resource, I expose tens!  Once again, even for this system, Multi-Factor Authentication via a password & One Time Passcode (OTP) via Mobile Phone SMS (Text Message) is the access protocol.  It then occurred to me, from a generic viewpoint, most security access systems ask you to register several pieces of memorable information; what’s your favourite book; mother’s maiden name; favourite sports team; pets name, et al.  Maybe, some of this information is duplicated & although not as vulnerable as having the same password for all of your account access, there’s a lot of duplicated personal information that could compromise many accounts…

Additionally, in the last several years, the evolution towards a cashless society has become more pervasive.  I myself use a mobile wallet, a mobile payment app, with NFC (Near Field Communication) for contactless payment convenience.  The convenience factor of these systems is significant, but once again, for those people with health challenges, can they easily use these systems?  We then must consider, how much information is accessed or even stored on a smartphone, to operate these financial accounts?

To recap, knowing the major UK banking institutions, I know my financial account password is stored in a secure Mainframe Server repository (I.E. ACF2, RACF, TopSecret) & associated account data is most likely protected at rest & in-flight via Pervasive Encryption (PE) or other highly secure encryption techniques.  However, to access these highly secure Mainframe systems, the client I’m using is a smartphone, with a hopefully highly secure Operating System, Mobile Banking App & Password Manager.  If I’m a bad actor, historically I would try to hack the most pervasive Operating System of the time, Microsoft Windows via desktop & laptop PC’s.  Today, perhaps I’m going to focus on the most pervasive client, namely mobile devices, typically operating via iOS & Android.  Of course, it’s no surprise that are increasing reports & activity of security exposures in these mobile operating systems & associated web resources, servers & browsers.

Additionally, in recent times, a well know financial institution was compromised, revealing the key personal information of ~145 Million US citizens, due to the well-known “Apache Struts” vulnerability.  This financial institution does deploy an IBM Mainframe, which historically would have afforded a tightly controlled Mainframe centric environment with no public Internet links; evolving to a decentralized environment, maybe globally outsourced, with a myriad of global Internet connected devices.  If only we could all apply the lessons & due diligence measures learned over the many decades of our IBM Mainframe experience.  However, this notable data breach happened at an organization that had been deploying a Mainframe for decades, proving that it’s human beings that typically make the most costly high profile mistakes!

Being a baby boomer & a proud Mainframer, I know what can go wrong & have planned accordingly.  I have separate accounts for mobile contactless payments, credit as opposed to debit based & more than one bank current account.  Whether by account isolation or the Consumer Credit Act, I’m limiting or eliminating any financial loss risk should my smartphone or financial account information be compromised.  For belt & braces protection, I always carry a modicum of cash, because how many times, even recently, have Mainframe based banks had card processing or cash machine access outages?  I’m just applying life experience & business continuity to my own daily life requirements, but how many people in the general public apply these due diligence measures?  Once again, please consider these members of the general public might be your family member, an inexperienced child or young adult, or more likely, perhaps a vulnerable aging parent.

Once again, applying my Mainframe Disaster Recovery & Business Continuity experience, how can I safeguard 99.999%+ availability for my day-to-day life if my smartphone is lost or Password Manager is compromised?  It’s not easy, a standby phone, sure, but what is the cost of the latest premium smartphone; how easy is it to synchronize two Password Manager solutions, from different software providers?  From my viewpoint, this is somewhat analogous to the IBM Mainframe hot versus warm or cold start DR process.  If you want high availability, you have to duplicate your expensive hardware, in the unlikely event you suffer a hardware outage.  Unlike the IBM Mainframe System Of Record (SOR) data, where of course must have the same software & data on both system images, if somebody compromises your Password Manager, was that a human or software error?  I don’t have the answers, I just try to apply due diligence, but I’m not sure how many members of the general public possess the life & vocational experience a Mainframe baby boomer has.

Without doubt, eliminating passwords is a great step forward, but is Multi-Factor Authentication (MFA) the “silver bullet”; I don’t think so.  Humans beings are just that, human, born to make mistakes.  Software is just that; prone to bugs & exposures, inadvertent or otherwise.  Centralizing your whole life in a smartphone has many advantages, but is it as vulnerable as keeping your life savings under the mattress?

Finally, thank you Mum & Dad for giving me this life opportunity & showing me dignity & strength in your dying days.  Thank you to the Mainframe community for providing me with so many opportunities to learn.  Maybe you can all give something back to the wider world for the causes that mean something to you.  The local charity I discovered & supported was the Northampton Hope Centre that tackles poverty & homelessness.  There but for the grace of god certainly applies to all of us, at one time or another, so let’s try & support as many people we can, those close to home & those in need.  It only occurred to me when I lost my Mother that eventually, if we live long enough, we all become orphans & a few weeks before I became an orphan, Coldplay released a song, Orphans.  There’s a line in that song, “I want to know when I can go, back & feel home again”.  For me, hopefully after about 18 Months, the end of March 2020 might be that day!

Tailored Fit Pricing for IBM Z: A Viable R4HA Alternative?

In a previous blog entry, I discussed the pros and cons of IBM Z Solution Consumption License Charges (SCLC): A Viable R4HA Alternative.  Recently on 14 May 2019 IBM announced Tailored Fit Pricing for IBM Z, introducing two comprehensive alternatives to the Rolling 4 Hour Average (R4HA) based pricing model, for both new and existing workloads, with a General Availability (GA) date of 21 June 2019.

To digress a little, for those of us in the Northern Hemisphere, June 21 is considered as the Summer Solstice, where the date might vary, one day before or after, namely June 20-22.  You can then further complicate things with confusing Midsummer’s Day with the Summer Solstice and Astronomical versus Meteorological seasons, but whatever, it’s a significant timeframe, with many traditions throughout Europe.  Once again, Midsummer’s Day can be any date between June 19 and June 24.  Having considered my previous review of SCLC and now the Tailored Fit Pricing announcement, I was reminded of a quotation from A Midsummer Night’s Dream by William Shakespeare, “so quick bright things come to confusion”…

The primary driver for Tailored Fit Pricing for IBM Z is to help mitigate unpredictable costs whilst continuing to deliver optimal business outcomes in the world of Digital Transformation & Hybrid Cloud.  Depending on the type of workload activity in your organisation, a tailored pricing model may be far more competitive when compared to pay-as-you-go schemes that have been typical on many x86 based cloud implementations.  Combining technology with cost competitive commercial models delivered through Tailored Fit Pricing strongly challenges the mindset that IT growth must be done on a public cloud in order to make economic sense.  Put another way, this is the IBM Marketing stance to compete with the ever-growing presence of the major 3 Public Cloud providers, namely Amazon Web Services (AWS), Microsoft Azure and Google Cloud, totalling ~60% of Public Cloud customer spend.

In essence a significant portion of The Tailored Fit Pricing for IBM Z announcement is a brand renaming activity, where the Container Pricing for IBM Z name changes to Tailored Fit Pricing for IBM Z.  The IBM Application Development and Test Solution and the IBM New Application Solution that were previously introduced under the Container Pricing for IBM Z name, are now offered under the Tailored Fit Pricing for IBM Z name.  Tailored Fit Pricing for IBM Z pricing introduces two new pricing solutions for IBM Z software running on the z/OS platform.  The Enterprise Consumption and Enterprise Capacity Solutions are both tailored to your environment and offer flexible deployment options:

  • Enterprise Consumption Solution: a tailored usage-based pricing model where compute power is measured on a per MSU basis.  MSU consumption is aggregated hourly, providing a measurement system better aligned with actual system utilization, when compared with R4HA.  Software charges are based on the total annual MSU usage, assisting users with seasonal workload pattern variations.  A total MSU used charging mechanism is designed to remove MSU capping, optimizing SLA and response time metrics accordingly.
  • Enterprise Capacity Solution: a tailored full-capacity licensing model, offering the maximum level of cost predictability.  Charges are based on the overall size of the physical hardware environment.  Charges are calculated based on the estimated mix of workloads running, while providing the flexibility to vary actual usage across workloads. Charges include increased capacity for development and test environments and reduced pricing for all types of workload growth.  An overall size charging mechanism is designed to remove MSU capping, optimizing SLA and response time metrics accordingly.

The high-level benefits associated with the Enterprise Consumption and Enterprise Capacity solutions can be summarized as:

  • Licensing models that eradicate cost control capping activities, enabling clients to fully exploit the CPU capacity installed
  • Increased CPU capacity for Development and Test (DevTest) environments, enabling clients to dramatically increase DevTest activities, without cost consideration
  • Optimized and potential lower pricing for all types of workload growth, without requiring additional IBM approvals, or additional tagging and tracking

Enterprise Solution License Charges (ESLC) are a new type of Monthly License Charge (MLC) pricing methodology for Enterprise Solutions, tailored for each individual and specific client environment and related requirements.  It was forever thus, whatever the pricing mechanism, the ubiquitous z/OS, CICS, Db2, MQ, IMS, WAS software products are the major considerations for MLC pricing mechanisms.  The Key prerequisites for Tailored Fit Pricing for IBM Z are IBM z14 Models M01-M05 or z14 Model ZR1, running the z/OS 2.2 and higher Operating System.

For new Mission Critical workloads and existing or new Development and Test (DevTest) workloads, Tailored Fit Pricing for IBM Z is clearly a great fit.  The restriction of z14 hardware is a little disappointing, where Solution Consumption License Charges (SCLC) included support for the z13 and z13s server.  I’m guessing that IBM are relying upon a significant z14 field upgrade programme in the next few years, largely based upon the Pervasive Encryption (PE) functionality.  However, for those customers that have run the IBM Z platform for decades and might have invested in cost optimization activities, including but not limited to capping, the jump to these new Enterprise Solution License Charges (ESLC) might take a while…

We could review this isolated announcement to the nth degree, but I’m not sure how productive that might be.  For sure, there is always devil in the detail, but sometimes we need to consider the big picture…

As a baby boomer myself, I see my role as passing on my knowledge to the next generations, although still wanting and striving to learn each and every day.  At this time of year, where the weather is better and roads drier, I drive my classic car a lot more and I enjoy the ability to tune the engine with my ears, hands, eyes and a strobe; getting my hands dirty!  I wonder whether the future of the IBM Z platform ecosystem is somewhat analogous to that of the combustion engine.  Several decades ago, electronics and Engine Management Systems became common place for combustion engines and now the ubiquitous laptop is plugged into the engine bay, to retrieve codes to diagnose and in theory repair faults.  For the consumer, arguably a good thing from a vehicle reliability viewpoint, but from a mechanical engineer viewpoint, have these folks become deskilled?  If you truly want your modern vehicle fixed, you will probably need a baby boomer to do this, one that doesn’t rely on a laptop, but their experience.  Although a sweeping generalisation, as there are always exceptions to any rule, the same applies to the IBM Z environment, where it was forever thus, compute power (MSU/MIPS) optimization relies upon a tune, tune, tune approach.

Whether R4HA or Full Capacity based, software cost charges will only be truly optimized if the system and ultimately application code is tuned.  A possible potential downside of not paying close attention to MSU usage, especially when considering these Enterprise Solution License Charges, is a potential isolated activity to “fix” IBM Z software costs forevermore, based upon a high MSU baseline.  Just as the combustion engine management systems simplify fault or diagnostic data collection, they don’t necessarily highlight that the vehicle owner left their cargo carrier on the vehicle roof, harming fuel efficiency.  A crude analogy for sure, but experience counts for a lot.  We have all probably encountered the Old Engineer & The Hammer story before and ultimately it’s incumbent upon us all, to safeguard that we don’t enable a rapid “death of expertise”.  Once the skills are lost, they’re lost.  Whether iStrobe from Compuware, TurboTune from Critical Path Software Inc. or the myriad of other System Monitor options, engage the experienced engineer and safeguard MSU optimization.  At this point, deploy the latest IBM Z pricing mechanism, namely Tailored Fit Pricing for IBM Z, and you will have truly optimized software costs…

IBM Z Solution Consumption License Charges (SCLC): A Viable R4HA Alternative?

In the same timeframe as the recent IBM z14 and LinuxONE Enhanced Driver Maintenance (GA2) hardware announcements, there were modifications to the Container Pricing for IBM Z mechanism, namely Solution Consumption License Charges (SCLC) and the Application Development and Test Solution.  Neither of these new pricing models are dependent on the IBM z14 GA2 hardware announcement, but do require the latest IBM z13, IBM z13s, IBM z14 or IBM z14 ZR1 servers and z/OS V2.2 and upwards for collocated workloads and z/OS V2.1 and upwards for separate LPAR workloads.

For many years, IBM themselves have attempted to introduce new sub-capacity software pricing models to encourage new workloads to the IBM Z server and associated z/OS operating system.  Some iterations include z Systems New Application License Charges (zNALC), Integrated Workload Pricing (IWP) and z Systems Collocated Application Pricing (zCAP), naming but a few.  The latest iteration appears to be Container Pricing for IBM Z, announced in July 2017, with three options, namely the aforementioned Application Development and Test Solution, the New Application Solution and Payments Pricing Solution.  This recent October 2018 announcement adapts the New Application Solution option, classifying it as the Solution Consumption License Charges (SCLC) mechanism.  For the purposes of this blog, we will concentrate on the SCLC mechanism, although the potential benefits of the Application Development and Test Solution for non-Production workloads should not be under estimated…

From a big picture viewpoint, z/OS, CICS, Db2, IMS and MQ are the most expensive IBM Z software products and of course, IBM Mainframe users have designed their environments to reduce software costs accordingly, initially with sub-capacity and then Workload Licence Charging (WLC) and the associated Rolling 4 Hour Average (R4HA).  Arguably CPU MSU management is a specialized capacity and performance management discipline in itself, with several 3rd party ISV options for optimized soft-capping (I.E. AutoSoftCapping, iCap, zDynaCap/Dynamic Capacity Intelligence).  IBM thinks that this MSU management discipline has thwarted new workloads being added to the IBM Z ecosystem, unless there was a mandatory requirement for CICS, Db2, IMS or MQ.  Hence this recent approach of adding new and qualified workloads, outside of the traditional R4HA mechanism.  These things take time and with a few tweaks and repairs, maybe the realm of possibility exists and perhaps the Solution Consumption License Charges (SCLC) is a viable and eminently usable option?

SCLC offers a new pricing metric when calculating MLC software costs for qualified Container Pricing workloads.  SCLC is based on actual MSU consumption, as opposed to the traditional R4HA WLC metric.  SCLC delivers a pure and consistent metered usage model, where the MSU resource used is charged at the same flat rate, regardless of hourly workload peaks, delivering pricing predictability.  Therefore, SCLC directly reflects the total workload cost, regardless of consumption, on a predictable “pay for what you use” basis.  This is particularly beneficial for volatile workloads, which can significantly impact WLC costs associated with the R4HA.  There are two variations of SCLC for qualified and IBM verified New Applications (NewApp):

  • The SCLC pay-as-you-go option offers a low priced, per-MSU model for software programs within the NewApp Solution, with no minimum financial commitment.
  • The SCLC-committed MSU option offers a saving of 20% over the pay-as-you-go price points, with a monthly minimum MSU commitment of just 25,000 MSUs.

SCLC costs are calculated and charged per MSU on an hourly basis, aggregated over an entire (SCRT) month.  For example, if a NewApp solution utilized 50 MSU in hour #1, 100 MSU in hour #2 and 50 MSU in hour #3, the total chargeable MSU for the 3-hour period would be 200 MSU.  Hourly periods continue to be calculated this way over the entire month, providing a true, usage-based cost model.  We previously reviewed Container Pricing in a previous blog entry from August 2017.  At first glance, the opportunity for a predictable workload cost seems evident, but what about the monthly MSU commitment of 25,000 MSU?

Let’s try and break this down at the simplest level, using the SCLC hourly MSU base metric.  In a fixed 24-hour day and an arbitrary 30-day month, there would be 720 single MSU hours.  To qualify for the 25,000 MSU commitment, the hourly workload would need to average ~35 MSU (~300 MIPS) in size.  For the medium and large sized business, generating a 35 MSU workload isn’t a consideration, but probably is for the smaller IBM Mainframe user.  The monthly commitment also becomes somewhat of a challenge, as a calendar month is 28/29 days, once per year, 30 days, four times per year and 31 days, seven times per year.  This doesn’t really impact the R4HA, but for a pay per MSU usage model, the number of MSU hours per month does matter.  One must draw one’s own conclusions, but it’s clearly easier to exceed the 25,000 MSU threshold in a 31-day month, when compared with a 30, 29 or 28 day month!  From a dispassionate viewpoint, I can’t see any reason why the 20% discount can’t be applied when the 25,000 MSU threshold is exceeded, without a financial commitment form the customer.  This would be a truly win-win situation for the customer and IBM, as the customer doesn’t have to concern themselves about exceeding the arbitrary 25,000 MSU threshold and IBM have delivered a usable and attractive pricing mechanism for the desired New Application workload.

The definition of a New Application workload is forever thus, based upon a qualified and verified workload by IBM, assigned a Solution ID for SCRT classification purposes, integrating CICS, Db2, MQ, IMS or z/OS software.  Therefore existing workloads, potentially classified as legacy will not qualify for this New Application status, but any application re-engineering activities should consider this lower price per MSU approach.  New technologies such as blockchain could easily transform a legacy application and benefit from New Application pricing, while the implementation of DevOps could easily transform non-Production workloads into benefiting from the Application Development and Test Solution Container Pricing mechanism.

In conclusion, MSU management is a very important discipline for any IBM Z user and any lower cost MSU that can be eliminated from the R4HA metric delivers improved TCO.  As always, the actual IBM Z Mainframe user themselves are ideally placed to interact and collaborate with IBM and perhaps tweak these Container Pricing models to make them eminently viable for all parties concerned, strengthening the IBM Z ecosystem and value proposition accordingly.

IBM Z Mainframe VTL End Of Support (EOS): A Problem Or Opportunity?

For ~20 years, since 1996 when IBM announced their IBM TotalStorage Virtual Tape Server Model B16 (3494-B16), typically known as the VTS, followed by the StorageTek (Oracle) Virtual Storage Manager (VSM) in 1998, there has been evident IBM Mainframe VTL innovation and product line refreshes, offering a granularity of offerings for all users, regardless of size.  The consolidation of the IBM Mainframe VTL marketplace in the ~2017-2019 period is notable.  IBM have consolidated their options to the high-end TS7760, retiring their TS7720 and TS7740 models. Similarly, Oracle have also delivered significant performance and enhancements to their VSM offering, where the latest VSM 7 delivers significant resource when compared with the VSM 6 and older predecessors (NB. The VSM 6 platform replaced the proprietary VSM 5 platform with Sun servers & Sun JBOD disk storage).  Similarly, EMC have consolidated their DLm offerings to the DLm8500, retiring their DLm1000, DLm1020, DLm2000, DLm2100, DLm6000 and DLm8000 models.

A high-level review of the mainstream market place offerings, namely EMC DLm8500, IBM TS7760 and Oracle VSM 7 demonstrates Enterprise Class VTL solutions, delivering significant availability, capacity and performance capabilities, mandatory for the higher echelons of IBM Z Mainframe user.  Conversely, it follows that such attributes and associated cost become somewhat of a concern for the small to medium sized IBM Mainframe user.  When any product becomes End Of Support (EOS), End Of Life (EOL) or even End Of Marketing (EOM), the viability and associated TCO becomes a consideration.  Typically, there are several options to address such an issue:

  • Do nothing (because we’re decommissioning the IBM Mainframe sometime soon)
  • Secure a long-term support contract (E-g. 3-5 years) ASAP, to reduce increasing support costs
  • Perform a technology refresh to the latest supported supplier offering
  • Review the marketplace and migrate to a more suitable supported solution

Only the incumbent IBM Mainframe VTL user can decide the best course action for their organization, but from a dispassionate viewpoint, reviewing these respective options generates the following observations:

  • Do nothing: The cost of doing nothing is always expensive. The perpetual “we’re moving away from the IBM Mainframe in the next 3-5 years” might have been on many “to-do” lists, for decades”!  The IBM Mainframe platform is strategic!
  • Long-term support contract: This delays the inevitable and potentially generates data availability challenges, as the equipment ages and potentially becomes more unreliable, with limited or expensive OEM support.
  • Technology refresh: In theory, the best option, upgrading the incumbent technology to the latest offering. In this instance, the cost might be significant for the small to medium sized user, as EMC, IBM and Oracle no longer offer “entry to medium-sized” solutions.
  • Migrate: By definition migration is perceived as introducing risk, migrating from a tried and tested to a new solution. However, generally the best products come from suppliers with a focus on their flagship solution, as opposed to a large company, with many offerings…

The IBM Mainframe VTL marketplace does include other suppliers, including FUJITSU, LUMINEX, Visara, naming but a few, and one must draw one’s own conclusions as to their respective merits.  What is always good is a new marketplace entrant, with a credible offering, a different approach or demonstrable expertise.

Optica Technologies is a privately held technology company headquartered in Louisville, Colorado, USA. Optica have been providing high-quality data centre infrastructure solutions since 1967. Optica has been an IBM strategic partner since 2002 and has received the most extensive IBM qualification available for third party solutions. Optica products have been successfully deployed in many major enterprise data centres worldwide.

The Optica Prizm FICON to ESCON Protocol Converter designed to enable IBM mainframe customers to invest in the latest System Z platforms (I.E. zEC12/zBC12 upwards), while preserving the ability to connect to critical ESCON and Bus/Tag device types that remain.

The next generation zVT Virtual Tape Node (VTN) exploits the latest Intel server technology, delivering outstanding performance, resiliency and scalability to serve a broad range of IBM Z customers. Each zVT VTN is modular and packaged efficiently with (2) FICON channels in an industry standard 2U rack format. The zVT VTN supports up to 512 3490/3590 Virtual Tape Drive (VTD) resources, delivering ~500 MB/S performance for the typical IBM Mainframe tape workload. As per some of the architectural design characteristics of the IBM Z Mainframe server (I.E. z13, z14), the zVT VTN server is enabled for operation in warmer environments than traditional data centres and engineered for extreme conditions such as high humidity, earthquakes and dust. To support the diversity of IBM Z Mainframe customer environments, from the smallest to largest, the flexible zVT solution is available in three different formats:

  • zVT 3000i: for IBM Mainframe users with more limited requirements, the fully integrated zVT 3000i model leverages the same Enterprise Class zVT VTN, incorporating 16 Virtual Tape Drive (VTD) resources and 8 TB of RAID-6 disk capacity, delivering 20 TB of effective capacity via the onboard hardware compression card (2.5:1 compression). The fundamental cost attributes of the zVT 3000i make a very compelling argument for those customers on a strict budget, requiring an Enterprise Class IBM Mainframe storage solution.
  • zVT 5000-iNAS: the flagship zVT 5000-iNAS solution is available in a fully redundant, high availability (HA) base configuration that combines (2) VTNs and (2) Intelligent Storage Nodes (ISNs). The entry-level zVT 5000-iNAS HA offering incorporates 512 (256 per VTN) Virtual Tape Drive (VTD) resources, delivering ~1 GB/Sec performance, 144 TB RAW and ~288 TB of effective capacity using a conservative 4:1 data reduction metric. zVT 5000-iNAS can scale to a performance rating of ~4 GB/Sec and capacity in excess of 11 PB RAW.
  • zVT 5000-FLEX: For IBM Mainframe users wishing to leverage their investments in IP (NFS) or FC (SAN) disk arrays, the zVT 5000-FLEX offering can be configured with (2) 10 GbE (1 GbE option) or (2) 8 Gbps Fibre Channel ports. Virtual Tape Drive (VTD) flexibility is provided with VTD options of 16, 64 or 256, while onboard hardware compression safeguards optimized data reduction.  Enterprise wide DR is simplified, as incumbent Time Zero (E.g. Flashcopy, Snapshot, et al) functions can be utilized for IBM Mainframe tape data.

In summary, Optica zVT reduces the IBM Mainframe VTL technology migration risk, when considering the following observations:

  • Technical Support: With 50+ years IBM Mainframe I/O connectivity experience, Optica have refined their diagnostics collection and processing activities, safeguarding rapid problem escalation and rectification, with Level 1-3 experts, located in the same geographical location.
  • Total Cost of Acquisition (TCA): zVT is a granular, modular and scalable solution, with a predictable, optimized and granular cost metric, for the smallest to largest of IBM Mainframe user, regardless of IBM Z Operating System.
  • Total Cost of Ownership (TCO): Leveraging from the latest software and hardware technologies and their own streamlined support processes, Optica deliver world class cradle-to-grave support for an optimized on-going cost.
  • Flexibility: Choose from an all-in-one solution for the smallest of users (I.E. zVT 3000i), a turnkey high-availability solution for simplified optimized usage (I.E. zVT 5000-iNAS) and the ability to leverage from in-house disk storage resources (I.E. zVT 5000-FLEX).
  • Simplified Migration: A structured approach to data migration, simplifying the transition from the incumbent VTL solution to zVT. zVT also utilizes the standard AWSTAPE file format, meaning data migration from zVT is simple, unlike the proprietary AWS file formats used by other VTL offerings.

In conclusion sometimes End Of Support (EOS) presents an opportunity to review the incumbent solution and consider a viable alternative and in the case of an IBM Mainframe VTL, for the small to medium sized user especially, having a viable target option, might just allow an organization to maintain, if not improve their current IBM Mainframe VTL expenditure profile…

Enabling IBM Z Security For The Cloud: Meltdown & Spectre Observations

The New Year period of 2018 delivered unpleasant news for the majority of IT users deploying Intel chips for their Mission Critical workloads.  Intel chips manufactured since 1995 have been identified as having a security flaw or bug.  This kernel level bug has been identified as leaking memory, allowing hackers access to read sensitive data, including passwords, login keys, et al, from the chip itself.  It therefore follows, this vulnerability allows malware inserts.  Let’s not overlook that x86 chips don’t just reside in PCs, their use is ubiquitous, including servers, the cloud and even mobile devices and the bug impacts all associated operating systems, Windows, Linux, macOS, et al.  Obviously, kernel access just bypasses everything security related…

From a classification viewpoint, Meltdown is a hardware vulnerability affecting a plethora of Intel x86 microprocessors, ten or so IBM POWER processors, and some ARM and Apple based microprocessors, allowing a rogue process to read all memory, even when not authorized.  Spectre breaks the isolation between different applications, allowing attackers to trick error free programs, which actually follow best practices, into leaking sensitive data and is more pervasive encompassing nearly all chip manufacturers.

There have been a number of software patches issued, firstly in late January 2018, which inevitably caused other usability issues and the patch reliability has become more stable during the last three-month period.  Intel now claim to have redesigned their upcoming 8th Generation Xeon and Core processors to further reduce the risks of attacks via the Spectre and Meltdown vulnerabilities.  Of course, these patches, whether at the software or firmware level are impacting chip performance, and as always, the figures vary greatly, but anything from 10-25% seems in the ball-park, with obvious consequences!

From a big picture viewpoint, if a technology is pervasive, it’s a prime target for the hacker community.  Windows being the traditional easy target, but an even better target is the CPU chip itself, encompassing all associated Operating Systems.  If you never had any security concerns from a public cloud viewpoint, arguably that was a questionable attitude, but now these rapidly growing public cloud providers really need to up their game from an infrastructure (IaaS) provision viewpoint.  What other chip technologies exist that haven’t been impacted (to date), by these Meltdown and Spectre vulnerabilities; IBM Z, perhaps not?

On 20 March 2018 at Think 2018 IBM announced the first cloud services with Mainframe class data protection:

  • IBM Cloud Hyper Protect Crypto Services: deliver FIPS 140-2 Level 4 security, the highest security level attainable for cryptographic hardware. This level of security is required by the most demanding of industries, for example Financial Services, for data protection.  Physical security mechanisms provide a complete envelope of protection around the cryptographic module with the intent of detecting and responding to all unauthorized attempts at physical access.  Hyper Protect Crypto Services deliver these highest levels of data protection from IBM Z to IBM Cloud.  Hyper Protect Crypto Services secures your data in a Secure Service Container (SSC), providing the enterprise-level of security and impregnability that enterprise customers have come to expect from IBM Z technology.  Hardware virtualisation protects data in an isolated environment.  SSC safeguards no external data access, including privileged users, for example, cloud administrators.  Data is encrypted at rest, in process and in flight.  The available support for Hardware Security Modules (zHSM) allows for digital keys to be protected in accordance with industry regulations.  The zHSM provides safe and secure PKCS#11 APIs, which makes Hyper Protect Crypto Services accessible by popular programming languages (E.g. Java, JavaScript, Swift, et al).
  • IBM Cloud Hyper Protect Containers: enable enterprises to deploy container-based applications and microservices, supported through the IBM Cloud Container service, managing sensitive data with a security-rich Service Container Systems environment via the IBM Z LinuxONE platform. This environment is built with IBM LinuxONE Systems, designed for EAL5+ isolation and Secure Services Containers technology designed to prevent privileged access from malicious users and Cloud Admins.

From an IBM and indeed industry viewpoint, security concerns should not be a barrier for enterprises looking to leverage from cloud native architecture to transform their business and drive new revenue from data using higher-value services including Artificial Intelligence (AI), Internet of Things (IoT) and blockchain.  Hyper Protect Crypto Services is the cryptography function used by the that IBM blockchain platform.  The Hyper Protect Crypto Services – Lite Plan offers free experimental usage of up to 10 crypto slots and is only deleted after 30 days of inactivity.

In a rapidly changing landscape, where AI, Blockchain and IoT are driving rapid cloud adoption, the ever-increasing cybersecurity threat is a clear and present danger.  The manifestation of security vulnerabilities in the processor chip, whether Apple, AMD, Arm, IBM, Intel, Qualcomm, et al, has been yet another wake-up alert and call for action for all.  Even from an IBM Z ecosystem viewpoint, there were Meltdown and Spectre patches required, and one must draw one’s own conclusions as to the pervasive nature of these exposures.

By enabling FIPS 140-2 Level 4 security via Cloud Hyper Protect Crypto Services and EAL5+ isolation via Cloud Hyper Protect Containers IBM Z LinuxONE, if only on the IBM Cloud platform, IBM are offering the highest levels of security accreditation to the wider IT community.  Noting that it was the Google Project Zero team that identified the Meltdown and Spectre vulnerability threats, hopefully Google might consider integrating these IBM Z Enterprise Class security features in their Public Cloud offering?  It therefore follows that all major Public Cloud providers including Amazon, Microsoft, Alibaba, Rackspace, et al, might follow suit?

In conclusion, perhaps the greatest lesson learned from the Meltdown and Spectre issue is that all major CPU chips were impacted and in a rapidly moving landscape of ever increasing public cloud adoption, the need for Enterprise Class security has never been more evident.  A dispassionate viewpoint might agree that IBM Z delivers Enterprise Class security and for the benefit of all evolving businesses, considering wider and arguably ground-breaking collaboration with technologies such as blockchain, wouldn’t it be beneficial if the generic Public Cloud offerings incorporated IBM Z security technology…

Maximizing IBM Z System Of Record (SOR) Data Value: Is ETL Still Relevant?

A generic consensus for the IBM Z Mainframe platform is that it’s the best transaction and database server available, and more recently with the advent of Pervasive Encryption, the best enterprise class security server.  It therefore follows that the majority of mission critical and valuable data resides in IBM Z Mainframe System Of Record (SOR) database repositories, receiving and passing data via real-time transaction services.  Traditionally, maximizing data value generally involved moving data from the IBM Mainframe to another platform, for subsequent analysis, typically for Business Intelligence (BI) and Data Warehouse (DW) purposes.

ETL (Extract, Transform, Load) is an automated and bulk data movement process, transitioning data from source systems via a transformation engine for use by target business decision driven applications, via an installation defined policy, loading the transformed data into target systems, typically data warehouses or specialized data repositories.  Quite simply, ETL enables an organization to make informed and hopefully intelligent data driven business decisions.  This ubiquitous IT industry TLA (Three Letter Acronym) generated a massive industry of ETL solutions, involving specialized software solutions, involving various Distributed Systems hardware platforms, both commodity and specialized.  However, some ~30 years since the first evolution of ETL processes, is ETL still relevant in the 21st Century?

The 21st Century has witnessed a massive and arguably exponential data explosion, from cloud, mobile and social media sources.  These dynamic and open data sources demand intelligent analytics to process the data in near real-time and the notion of having a time delay between the Extract and Load part of the ETL process is becoming increasingly unacceptable for most data driven organizations.  During the last several years, there has been increased usage of Cloud BI, with a reported increase from ~25-80% of public cloud users, deploying Cloud BI solutions.

For cloud resident data warehouses, an evolution from ETL to ELT (Extract, Load, Transform) has taken place.  ELT is an evolutionary and savvy method for of moving data from source systems to centralized data repositories without transforming the data before it’s loaded into the target systems.  The major benefit of the ELT approach is the near real-time processing requirement of today’s data driven 21st Century business.  With ELT, all extracted raw data resides in the data warehouse, where powerful and modern analytical architectures can transform the data, as per the associated business decision making policies.  Put simply, the data transformation occurs when the associated analytical query activities are processed.  For those modern organizations leveraging from public cloud resources, ELT and Cloud BI processes make sense and the growth of Cloud BI speaks for itself.  However, what about the traditional business, which has leveraged from the IBM Z Mainframe platform for 30-50+ years?

Each and every leading Public Cloud supplier, including IBM (Watson) has their own proprietary analytical engine, integrating that technology into their mainstream offerings.  As always, the IBM Z Mainframe platform has evolved to deliver the near real-time requirements of an ELT framework, but are there any other generic solutions that might assist any Mainframe organization in their ETL to ELT evolution process?

B.O.S. Software Service und Vertrieb GmbH offer their tcVISION solution, which approaches this subject matter from a data synchronization viewpoint.  tcVISION is a powerful Change Data Capture (CDC) platform for users of IBM Mainframes and Distributed Systems servers.  tcVISION automatically identifies the changes applied to Mainframe and Distributed Systems databases and files.  No programming effort is necessary to obtain the changed data.  tcVISION continuously propagates the changed data to the target systems in real-time or on a policy driven time interval period, as and when required.  tcVISION offers a rich set of processing and controlling mechanisms to guarantee a data exchange implementation that is fully audit proof.  tcVISION contains powerful bulk processors that perform the initial load of mass data or the cyclic exchange of larger data volumes in an efficient, fast and reliable way.

tcVISION supports several data capture methods that can be individually used as the application and associated data processing flow requires.  These methods are based upon a Real-Time or near Real-Time basis, including IBM Mainframe DBMS, Logstream, Log and Snapshot (compare) data sources.  A myriad of generic database repositories are supported:

  • Adabas: Realtime/Near Realtime, Log Processing, Compare Processing
  • Adabas LUW: Real-time/Near Real-time, log processing, compare processing
  • CA-Datacom: Log processing, compare processing
  • CA-IDMS: Real-time/Near real-time, log processing, compare processing
  • DB2: Real-time/Near real-time, log processing, compare processing
  • DB2/LUW: Real-time/Near real-time, log processing, compare processing
  • Exasol: Compare processing
  • IMS: Real-time/Near real-time, log processing, compare processing
  • Informix: Real-time/Near real-time, log processing, compare processing
  • Microsoft SQL Server: Real-time/Near real-time, log processing, compare processing
  • Oracle: Real-time/Near real-time, log processing, compare processing
  • PostgreSQL: Real-time/Near real-time, log processing, compare processing
  • Sequential file: Compare processing
  • Teradata: Compare processing
  • VSAM: Real-time/Near real-time, log processing, compare processing
  • VSAM/CICS: Real-time/Near real-time, log processing, compare processing

tcVISION incorporates an intelligent bulk load component that can be used to unload data from a Mainframe or Distributed Systems data source, loading the data into a target database, either directly or by using a loader file.  tcVISION comes with an integrated loop-back prevention for bidirectional data exchange, where individual criteria can be specified to detect and ignore changes that have already been applied.  tcVISION incorporates comprehensive monitoring, logging and integrated alert notification.  Optional performance data may be captured and stored into any commercially available relational database.  This performance data can be analyzed and graphically displayed using the tcVISION web component.

From an ETL to ELT evolution viewpoint, tcVISION delivers the following data synchronization benefits:

  • Time Optimization: Significant reduction in data exchange implementation processes and data synchronization processing.
  • Heterogenous Support: Independent of database supplier, offering support for a myriad of source and target databases.
  • Resource Optimization: Mainframe MIPS reduction and data transfer optimization via intelligent secure compression algorithms.
  • Data Availability: Real-time data replication across application and system boundaries.
  • Implementation Simplicity: Eradication of application programming and data engineer resources.
  • Security: Full accountability and auditability all data movements.

In conclusion, the ETL process has now been superseded by the real-time data exchange requirement for 21st Century data processing via the ELT evolution.  Whether viewed as an ELT or data synchronization requirement, tcVISION delivers an independent vendor agnostic solution, which can efficiently deliver seamless data delivery for analytical purposes, while maintaining synchronized data copies between environments in real-time.