Tape support

[Pacjunk]

Info

• Which version of Pi are you using: ZeroW
• Which github revision of software: develop
• Which board version: FULLSPEC
• Which computer is the RaSCSI connected to: OpenVMS

Describe the issue

Has anyone thought about SCSI tape support? I know this is probably not a thing in the Mac world, but it is a big thing in the workstation space, as they were often used for software distribution. There are websites where people have archived software distribution tapes (in the .tap format) and these can be loaded in to simulators.

I was thinking a good place to look for inspiration would be the SIMH project, as they have a defined tape file format and the source code for a SCSI tape driver available. Running the VAX simulator in SIMH, I can read and create "tapes" without a problem. It might be nice to do this on real hardware.

Thanks,

[sidick]

I'd be quite interested in that to try out Amiga UNIX on my A3000 too actually

[ppuskari]

I would like to register my interest in tape emulation as well. I am currently looking into what it would take on the scsi2sd as well to emulate the 3M 40m tape drive which has the neat ability to also be a full blockdevice with read/write any block on the formatted tape. So the hard part is essentially done there. The easier but still alluding me part is handling EOT emulation and tape direction changes. Would love to use this in the Apple Tape 40SC and rig up proper lights and simulate the tape action.

Other tape drive types would be interesting as well through the SCSI QIC types. I have a few SUN tapes in the archive as well. Then there is the whole early DAT/ADAT and the DDS variants. Some of the older software requires the vendor check etc to validate before use so setting that is the first part.

[uweseimet]

@Pacjunk @ppuskari Any chance that you start working on this ticket soon?

[Pacjunk]

I'm not that good at coding. It was just a suggestion/wishlist type of thing. It is probably not worth the effort for the small audience. It looks like @ppuskari may have a better chance than me.

[uweseimet]

@Pacjunk This is a ticket (like some others we have), which make sense and is jiustified, but is out of scope because of the required resources. Only somebody who has enough time to spare, a tape drive hardware, a vintage computer with the required drivers and good C++ and SCSI knowledge can resolve it.
If it turns out that also @ppuskari cannot work on it I suggest to close it because it is one of several wishlist tickets we have which require a big effort without resources for them being available. It can be re-opened any time if somebody wants to work on it. This project IMHO needs better grooming of tickets, and in order to make grooming less time-consuming we should be more stringent on which tickets are accepted and which are not.
I hope you agree and do not feel offended.

[bog-dan-ro]

I'd like to give it a try.

According to http://www.bitsavers.org/components/ncr_symbios/scsi/SCSI-2_Standard_1994.pdf Table 174 • Commands for sequential-access devices (see page 226) there are only a few (mandatory) commands that are needed to be handled, which most of them are already handled by Disk class.
The mandatory missing commands are:

• 10.2.1 ERASE
• 10.2.5 READ BLOCK LIMITS
• 10.2.11 REWIND
• 10.2.12 SPACE
• 10.2.15 WRITE FlLEMARKS

Because Disk class is way too heavy for streamers, I'm going to move some stuff to it's parent class StorageDevice and use StorageDevice as base class for Streamer.

Also ZuluSCSI implemented it https://github.com/ZuluSCSI/ZuluSCSI-firmware/blob/main/src/ZuluSCSI_tape.cpp ;-)

[uweseimet]

@bog-dan-ro In the meantime I have added a branch issue_93 with initial tape drive support to SCSI2Pi. This implementation is completely new, i.e. it is not based on any existing code.

@akuker @rdmark I would like to avoid incompatibilities between a potential future PiSCSI streamer implementation and the SCSI2Pi implementation, which would at least affect users of the SCSI Control app. It would be great if we could agree on these conventions:

• Default extensions for streamer image files are at least .tar and .tap (and potentially others). Please see below for details on the difference as far as SCSI2Pi is concerned.
• The streamer device type shall have enum value 10 in the .proto file and shall either be called SCST or SCTP. I vote for SCTP because BlueSCSI already uses TP, and SCST IMHO has some minor disadvantages I won't elaborate on.

Please provide feedback on these conventions.

In order to not only support files that tar has written to a filesystem, but also more elaborate tools that use filemarks, SCSI2Pi switches to a tar file compatibility mode when the file extension is .tar. Any other extension results in the full-featured mode with filemarks support.
Filemarks and other meta data cannot be supported with files tar has written to the filesystem, because these files only contain the actual data bytes. For filemarks support you have to add proprietary meta data to these files. Note that when writing to a tape device, tar writes a filemark at the end, which it does not do when writing directly to a filesystem.

[rdmark]

@uweseimet Supporting tar and tap file endings sounds reasonable to me. Right now the PR against piscsi uses SCST so I'd like to get @bog-dan-ro's thoughts on this.

I'm in the middle of a major life event so my own ability to work on the piscsi project is severely limited (access to hardware, living situation, priorities etc.) This is one reason activities in this project has stalled.

[uweseimet]

@rdmark Thank you for your feedback.

@bog-dan-ro Can you please comment on the device type? I'd prefer SCTP, because BlueSCSI uses TP (and not ST), and the .tap extension is used for files, and not .st or something like that. This is why I think SCTP is more consistent.
To be honest, in the meantime I also prefer SCTP for an additional reason: It is already being used in many places in the SCSI2Pi develop branch, and I am not that keen on changing it everywhere ;-).

@bog-dan-ro I just noticed that you have not added any tape device to the .proto file? Without protobuf support clients like scsictl and the SCSI Control app cannot support tape devices. I guess that the web UI would also suffer from your device not being supported in the protobuf interface.
As already suggested, in order to ensure compatibility PiSCSI and SCSI2Pi should use the same enum value. The next available (unused) value is 10.

[bog-dan-ro]

@rdmark @uweseimet it doesn't really matter what name are we using I choose SCST instead of SCTP as STreamer naming was using the the scsi standard (it seems there are streamers not only tapes).

Regarding the .tap extension, I don't know if it's a good decision as, AFAIK, some old unix systems was using it as an alternative(/alias?) to tar or cpio.

@bog-dan-ro I just noticed that you have not added any tape device to the .proto file?

Where is that file?

As already suggested, in order to ensure compatibility PiSCSI and SCSI2Pi should use the same enum value. The next available (unused) value is 10.

IMHO the best option is to have only one implementation.
My computer which has unix and SCSI is stil missing the RAM modules, once I have them I'll test your implementation and if it works then I'll backport it to PiSCSI.

[uweseimet]

@bog-dan-ro You need more than one extension, because you cannot use regular .tar files as image files when non-block objects like filemarks and end-of-data are supported. .tar files do not contain/support these, so you need an additional file type. You have chosen .tap, and I am fine with this. I don't think that .cpio should be used, because the non-tar format is proprietary. This is because of the non-block objects, where the representation in the image file may differ depending on the device implementation, just like the phyiscal formats used by different streamers differ.

>find -name *.proto
./cpp/piscsi_interface.proto

Let's decide on SCTP with enum value 10, and extensions .tar and .tap (the latter for files with non-block objects). This way PiSCSI and SCSI2Pi will be compatible (at least for tar files) even in case the implementations should differ. The extensions are not really that important, but the protobuf definitions are, because they describe the external interface.
Are you fine with this?

[bog-dan-ro]

@uweseimet I'm okay with SCTP and enum value 10.

[uweseimet]

@bog-dan-ro Great, thank you!

[uweseimet]

@bog-dan-ro By the way, when backporting I recommend using grep to find all occurrences of sctp, otherwise you might miss something else besides the protobuf-related changes.
I guess backporting will take at least a manweek, because you may also have to change the controller and the mode page handling.

[uweseimet]

@bog-dan-ro Please see my latest comment in uweseimet/scsi2pi#93, regarding the fact that the .tap extension is already a well-defined file format, which is not compatible with what SCSI2Pi and (maybe) PiSCSI is using.
This means that we cannot use .tap for raw tape files, at least not for files that are not compatible with the .tap format. In order to avoid issues I suggest to follow your initial suggestion for a different extension, i.e. ".st". Would you agree? Support for the .tap extension should only be added in case PiSCSI or SCSI2Pi support the simh compatible format, which uses .tap files.

[bog-dan-ro]

@uweseimet wait a minute, I thought that we just see .tap files as a raw images and the software will deal with the format...
Yup, I'm okay with something like .st extension for our own format, unless we can use .tap for that job i.e. it supports all the features what we need, this way we don't need to invent yet another file format.

[uweseimet]

@bog-dan-ro Yes, initially I also thought that we would use .tap for raw images with our own (potentially proprietary) format. But .tap appears to be used for a well-defined format already, i.e. if we support .tap the files with this extension would have to be compatible with this format.
Looks as if we agree: We are going to use .st for any kind of raw file, and we are going to use .tap only for files that are compatible with the simh .tap format. This way we avoid any confusion. Otherwise it appears as if we were supporting the simh format, even when we do not.

[Pacjunk]

I've seen lots of emulators use ".tap" files (e.g. early home computer cassettes), not just simh. I'm sure they are in no way compatible. Since the data structure is up to the operating system, I assume the only unique thing about simh is the tape mark used?

[uweseimet]

@Pacjunk Without knowing details on the simh format it is impossible to tell which features it supports. But using .tap as an extension for non-simh files is not such a good idea. You tried to use simh .tap files with SCSI2Pi, because you thought .tap support in SCSI2Pi means support for simh files, didn't you? At least that is what can easily happen if you use well-defined extensions in the wrong context: You download a .tap file (simh) and expect it to work.

[Pacjunk]

No, of course I didn't expect that! I didn't know anything about tape formats and assumed a raw stream of data might be readable. I certainly didn't think or expect that SCSI2Pi supported simh. I'm not really expecting virtual tapes to work on my system anyway - I was just trying to help you out with testing. On a positive note, I managed to label a tape with your code.

Out of interest, the spec is here: http://simh.trailing-edge.com/docs/simh_magtape.pdf

[uweseimet]

@Pacjunk Sorry for the misunderstanding: my impression was that you expected simh .tap to be the same as SCSI2Pi .tap. Other users might think so, and using a different default extension will prevent that. In /etc/s2p.conf you can configure additional extensions any time and make .tap an additional default extension.

Thank you for the link. I will have a look at it later, but for the present I would like to concentrate on the native tape drive support. This would also be usable with platforms for which there is no simh/tap support.