Would we want the ability to use SMP/E to install z/OS Open Tools (still unsupported)?

[MikeFultonDev]

I am just noodling on whether people would want to have these tools 'fit in' to a more traditional installation model (SMP/E).
I am envisioning it would be a blob you download to z/OS, validate the signature, then run SMP/E against to install.
I wouldn't expect SMP/E to be 'mandatory' in any sense - just an alternative.

Curious what the systems folks think - and maybe it's a non-starter because people wouldn't install unsupported stuff into SMP/E?

[lbdyck]

SMP/E - wouldn't that be overkill?

Seriously - I can see some shops that will like the SMP/E install model - so FMID/SYSMOD/JCLIN/etc.

But this is a different model and I'm happy with it the way it is. SMP/E adds, imho, a lot of unnecessary work on both sides.

[MikeFultonDev]

I just know some folks really like SMP/E... I was envisioning the value would be if we had a particular tool that required a z/OS PTF, and if it required other tools at a particular level, and people didn't want to install the 'zopen' tooling.
My goal is to make these tools accessible to as many shops as possible. If this doesn't move the needle, I'm keen to not do it 😁

[pjfarleyiii]

You may find that there are large shops (especially ones with "facilities management contracts" who do not have their own sysprogs) who will not install anything unless it is SMP/E installable, and looking to the future perhaps will not install anything that does not have a ZOSMF script to do the installation. They will refuse to learn open-source installation processes, they will only use the ones IBM provides/requires. I think you have to treat the Open Tools project like you are an ISV and do all the things an ISV would do to get a sale, however distasteful or burdensome.

[catherdersoftware]

I think it would depend, of course. I'm sure some companies would allow alternate download distribution sources, especially if it's explicitly stated somewhere that z/OS Open Tools are not an IBM product. However, SMP/E makes things easier. I could contribute to creating an SMP/E distribution…I don't usually like to let on that I'm pretty decent at SMP/E, including packaging. (File that for future reference.)

[pjfarleyiii]

@MikeFultonDev, I don't know if a shop without it's own sysprogs and bureaucracy that demands SMP/E installs would install "unsupported" software. My guess is "no they would not". For example at my own employer there is a currently significant movement towards forcefully deleting all the old un-supported software libraries lying around and never deleted and upgrading all products to "supported" levels in order to satisfy the corporate auditors. Shakespeare was almost right - first get rid of all the auditors, the lawyers are pussycats in comparison to the auditors . . .

[MikeFultonDev]

Thanks for the feedback. Is there any difference from an auditors perspective between a dev and prod machine ?

[pjfarleyiii]

I have not seen such a difference. In the very large shops in which I have worked they don't often have a totally separate dev machine. It is often just one of the LPAR's in the sysplex that isn't as heavily targeted for the production workload (Prod CICS's/other OLTP's/DB2/etc.), and often that same LPAR serves as the main QA platform. IOW, dev has to share and is not separate (though it may have more-or-less-dedicated DASD pools). In such environments, the auditor rules about software are definitely applied.

[pjfarleyiii]

What I am thinking is that z/OS Open Tools is going to need a commercial entity willing to make and honor "support contracts" for some "stable" level of the toolset. That will require level 1 / level 2 support personnel, a variety of support environments at multiple levels of z/OS, etc., etc., etc. Like what Rocket Tools does for their open-source ports. It stinks that such a beast is necessary, but I don't see a way around it, especially for FinTech and other such companies that have legally enforceable obligations to their clients (and/or to government entities) to run only supported software to process their clients' proprietary data.

[lbdyck]

Our product is SMP/E installable from us and IBM resells it and a customer can get it from ShopZ.

You will have to decide is it one FMID, or one per package. And then how are updates released - new FMID or as a PTF.

And will you have an installation workflow with zOSMF

[lbdyck]

one more thing - some companies do NOT allow access to github or the installation of non-vendor tools - having a SMP/E installation process gives it the impression of a vendor and if it comes via ShopZ then no github access requried.

[MikeFultonDev]

I don't think we could use ShopZ unless IBM lets non-IBMers put stuff into ShopZ - I don't want to have this become an 'IBM product-y' type of thing...

[v1gnesh]

Would this be a good intermediate step - install via zOSMF, but not SMP/E?
Doing it with zOSMF aligns with future direction.
Not using SMP/E to start with, we can test the waters.
Answer to SMP/E fans will be - we're delivering this via the future s/w interface, zOSMF.

Unsupported can't go in SMP/E... comes much later than the above, but it's still something to tackle :)

[MikeFultonDev]

Here is an important discussion in IBM-main on the value of SMP/E for sysprogs: https://www.mail-archive.com/ibm-main@listserv.ua.edu/msg132033.html

[MikeFultonDev]

I am re-reading the notes above and now and have a few more questions.

One of the things SMP/E does well is keep track of dependencies, if you set things up right.
One of the things that can cause grief with SMP/E is that it's very flexible.
Different customers will want different tools that they want to install.
Everyone will want the ability to get updates easily.

Would this be of value:

• sysprog goes to a web site (either interactively or via RESTful API) and says what software packages they want (e.g. make, git, bash)
• the website builds the set of software packages that are required to run that set of tools (e.g. make, git, bash also need perl, less, ncurses)
• the website creates a custom SMP/E installable that can be downloaded and installed using SMP/E
• the SMP/E installable would also verify on install that all the runtime dependencies are right
• the choice for when the sysprog chooses install location is interesting - typically with SMP/E this would be at install but in it would be much easier (and likely less error prone) if the location was chosen as part of building the software package in the first place

This SMP/E installable would NOT be supported - that's not part of this community's charter - but it would eliminate sysprogs having to learn a new tool (zopen) for software management and would fit into their existing schemes.

Does this have value? Is this a useful path?

[DevonianTeuchter]

Have to sit and think through how to properly meld a package manager (zopen's or OpenShift's zpm or something else/in between) and SMP/E.
What if... the generated [and non-ShopZ hosted] SMPE installable images were downloaded by suitable "z/OS Unix" tooling (eg 'zopen smpe') which was to then generate and submit suitable JCL and trigger SMPE under-the-covers. Would allow "normal" management of the SMPE environment, while still enabling the "package manager" style interface and without requiring zOSMF etc. Consumers could still download and apply the SMPE maintenance themselves in a traditional way, or the tooling would automate the process similar to the relationship between dnf&rpm or apt&dpkg, front-ending SMPE.

[v1gnesh]

Yup yup yup. This is the right way to rely on the strenghts of zOS.
Not building a wrapper but depending on rock solid parts that already exist (SMPE in this case).
dnf:rpm == apt:dpkg == zpm:smpe

It may not be easy, but when complete, it will be a work of art.
Complex under the covers, allowing manual recovery for those nightmare scenarios.
Easy interface so that it doesn't get in the way.

[v1gnesh]

One example of SMPE embracing USS stuff is Java, but ignoring the MVS datasets part of it.
As a reference, the outcome should look something like Java's... where Java via SMPE delivers /usr/lpp/java/whatever.

[MikeFultonDev]

i had that thought experiment but then thought it wouldn't be popular with the 'traditionalist' because they have to install a new package manager (zopen) for these 'crazy open source tools' just to do stuff... For me personally, once i've installed zopen, doing 'zopen install -y ...' is pretty easy, so it seems to me the main reason people would object to that would be having to download the 'open source tools specific package manager'?

[v1gnesh]

The zopen interface will be the preferred one, but if SMPE packages are supplied, something from the SMPE world can be used to install without needing zopen.
The components underlying the 'zopen smpe' subcommand that @DevonianTeuchter mentions can be supplied as skeleton JCLs etc., so that traditionalists can use them directly. Supplying usable samples upfront is better than pointing to a guide/manual, I think.

It's definitely going to be easier with 'zopen install', so the value prop for it will be obvious & enticing 😛

[couckearthur]

@MikeFultonDev has a conclusion already been made about this discussion?
For us an SMP/E package and delivery through receive order would be a significant added value.

[lbdyck]

My $0.01 (not sure they are worth more)

Just as there needs to be an air gap solution, there also needs to be an SMP/E solution for the zot sites. As for how to package the tools in SMP/E format that I've no idea. The latest IBM Packaging pub is from 2003 and discusses HFS and JAR and not in an easy to follow text. One would hope that doc, at least internally in IBM, would have been updated by now and perhaps that will help with this discussion.

[couckearthur]

@lbdyck It is indeed like that, I think what is needed is quite similar like the Rocket ported tools. If you have IDz Enterprise installed this is included in the SMP/E setup.

The FMID is HAL2111 (FUNCTION Rocket Git for z/OS). There are MVS target datasets with SAMP JCLs and USS target datasets.
What happens during apply is copying one script to /usr/lpp/**/IBM. And afterwards the script deletes everything that exists on the path and unpax the new files again. The function is called Git for z/OS but it includes several other ported tools.

If interested I can give more detail, but I have to be careful what to disclose in public

[lbdyck]

Info from an IBM-Main query:

From Kurt Quackenbush:

Sadly, the -10 version is the most recent version of that book, but it does contain information about z/OS UNIX stuff. Do you have specific questions or concerns?

Kurt Quackenbush
IBM | z/OS SMP/E and z/OSMF Software Management | kurtq@us.ibm.com

Chuck Norris never uses CHECK when he applies PTFs.

============================

From David Jousma:

I haven’t seen the book, but Ive written usermods to replace SMPE controlled data. Basically you have the tar’d payload, a script that is aware of SMPE phase/actions, etc, and based on an apply or restore does the work. Whether you are writing a usermod or installing a FMID I think the contruction would be very similar. This is the job stream to build usermod to apply updates to java.security

This script will either create or delete the backup copies of the
security policy jar files.
Policy jar files are installed/removed from the J11.0_64/conf/security
directory. The script uses the following environment variables
for input:

SMP_Directory - directory in which the file resides
SMP_File - name of the HFS file
SMP_Phase - indicates whether the shell script is being called
before or after SMP/E has processed the file
SMP_Action - the action that SMP/E is performing: COPY or DELETE

Here is a jobstream that builds the usermod. I cannot take total credit for it, another member here on IBM-MAIN (whom I cannot remember) got me started.

//E008058E JOB (DP,8710),'MAJV116BUILD',CLASS=X,MSGCLASS=T,
// NOTIFY=E008058
//HEADER EXEC PGM=IEBGENER
//SYSUT1 DD DATA,DLM=$$
//E008058E JOB (DP,8710),'MAJV116',CLASS=X,MSGCLASS=T,
// NOTIFY=E008058
//PROCLIB JCLLIB ORDER=(E008058.SMPE.ZOS25.CLONE.JCL.PROCLIB)
//SMPAPPLY EXEC SMPE
//SMPHOLD DD DUMMY
//SYSIN DD *
SET BDY(GLOBAL) .
RECEIVE S(MAJV116) SYSMODS BYPASS(APPCHK).
SET BDY(MVSTZN).
APPLY S(MAJV116) REDO.
/*
//* RESTORE S(MAJV116) .
//* REJECT S(MAJV116) BYPASS(APPCHK).
//SMPPTFIN DD DATA,DLM=ZZ
++ USERMOD (MAJV116) /*
IBM JAVA 11.0 (64-Bit) platform /.
++ VER (Z038)
FMID(HJVBB00) PRE(UI94773)
/
USERMOD DESCRIPTION(S):
MAJV116 -
****************************************************************
* WARNING: DO NOT modify this usermod. It is created and *
* built in SMPE.ZOSnnn.USERMODS.BUILD *
****************************************************************
* USERMOD DESCRIPTION: *
* - Enabling CSF key support by adding entry to *
* java.security for IBMJCECCA *
* inserted as the security.provider.2= entry. *
* *
* The JMVPRC16 proc is here to insure that these changes *
* to Java 11.0 can not be accidently lost due to a new *
* level of maintence being installed via SMP/E without *
* some notification of the regression. *
* *
* D.jousma 11/14/22 - updated Build of MAJV116(USERMOD), *
* MAJV116J(HFS), MAJV116S(Script) *
* in SMPE.ZOS25.USERMODS.BUILD *
* to support JAVA 11.0 *
* *
****************************************************************
/.
++ HFS (MAJV116J) / $java_home/lib/security/java.security.sklm /
DISTLIB (AAJVHFS)
SYSLIB (SAJVB0P)
PARM (PATHMODE(0,6,4,4))
LINK ('../conf/security/java.security.sklm')
SHSCRIPT (MAJV116S,POST)
TEXT .
$$
//SYSUT2 DD DSN=&&TEMP,DISP=(NEW,PASS),RECFM=FB,LRECL=80,
// BLKSIZE=0,SPACE=(CYL,16)
//SYSPRINT DD SYSOUT=
//SYSIN DD DUMMY
//JAVA EXEC PGM=GIMDTS
//SYSPRINT DD SYSOUT=*
//SYSUT1 DD DSN=SMPE.ZOS25.USERMODS.BUILD(MAJV116J),DISP=SHR
//SYSUT2 DD DSN=&&TEMP,DISP=(MOD,PASS)
//MIDDLE EXEC PGM=IEBGENER
//SYSUT1 DD DATA,DLM=$$
++ SHELLSCR (MAJV116S) /* $java_home/MAJV116s /
DISTLIB (AAJVHFS)
SYSLIB (SAJVB0P)
PARM (PATHMODE(0,7,5,5))
TEXT .
$$
//SYSUT2 DD DSN=&&TEMP,DISP=(MOD,PASS)
//SYSPRINT DD SYSOUT=
//SYSIN DD DUMMY
//SCRIPT EXEC PGM=GIMDTS
//SYSPRINT DD SYSOUT=*
//SYSUT1 DD DSN=SMPE.ZOS25.USERMODS.BUILD(MAJV116S),DISP=SHR
//SYSUT2 DD DSN=&&TEMP,DISP=(MOD,PASS)
//END EXEC PGM=IEBGENER
//SYSUT1 DD DATA,DLM=$$
++ PROC (JVMPRC16) DISTLIB(APROCLIB) SYSLIB(PROCLIB) TXLIB(APROCLIB).
$$
//SYSUT2 DD DSN=&&TEMP,DISP=(MOD,PASS)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//END EXEC PGM=IEBGENER
//SYSUT1 DD DATA,DLM=$$
ZZ
//
$$
//SYSUT2 DD DSN=&&TEMP,DISP=(MOD,PASS)
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SAVEIT EXEC PGM=IEBGENER
//SYSUT1 DD DSN=&&TEMP,DISP=(MOD,PASS)
//SYSUT2 DD DSN=SMPE.ZOS25.USERMODS(MAJV116),DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY

Dave Jousma
Vice President | Director, Technology Engineering

[lbdyck]

SMP Packaging Rules.pdf

[lbdyck]

Isn't zoau packaged in SMP/E format if ordered from IBM? And Java and other z/OS products/features are packaged in SMP/E - thus there should be some resources with IBM that could potentially help ????