When your HP3000 gives you One Liners

I was recently posed with a question from a customer about FTP. The issue was that when they downloaded from a remote server the resulting data in the file on the 3000 only had one long line. Now when some people encounter this, one of the things they try to do is add an LF or change a line ending in LF to have a CRLF to the end of each line.

If you're considering doing this, don't, stop. Save yourself a lot of trouble and read on, because in my experience this never works.

So the specific situation was that a customer has a file from a third party that they downloaded to an NT server from a Unix server. When they downloaded from the NT server in this manner as they were told the file was fixed ascii 532 bytes:

get remotefile test;rec=-532,1,f,ascii;disc=100000
When they tried to process the file they found to have only one record and this was regardles of using ASCII or BINARY transfer methods.

ACCOUNT=  GREEN       GROUP=  NEIL    
	
FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX
TEST              532B  FA           1     100000   1       16  1  8
So the solution in this case is to do the following:
open servername
bytestream
get remotefile testbyte;REC=,,B;DISC=16384000
This results in a file that would like similar to this:
ACCOUNT=  GREEN       GROUP=  NEIL    
	
FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX
	
TESTBYTE            1B  BA      280079   16384000   1     1104  4 32
From there you can make the file fixed length with a specific size with either Qedit or the FROMBYTE utility.
t TESTBYTE
set keep var off
set len 532
k newfile,unn
The file will now look like this:
ACCOUNT=  GREEN       GROUP=  NEIL    
	
FILENAME  CODE  ------------LOGICAL RECORD-----------  ----SPACE----
                  SIZE  TYP        EOF      LIMIT R/B  SECTORS #X MX
	
NEWFILE           532B  FA         751      10000   1     1568  4  8

FTP on MPE

Earlier we reported on how to configure and install FTP on your HP e3000. FTP is without a doubt the universal method for transporting files from computer to computer. (See also our article on password problems with Security/3000.)

There are other methods, such as Reflection File transfer for PC to 3000, which is popular in the 3000 world, but almost non-existent in the HP 9000 arena. There is Kermit; however, this is more prevalent in the 9000 world and frankly I've never really gotten it to work very successfully.

Besides, our development environment includes MPE, HP-UX, and Windows and FTP come for free on all boxes and can be used in batch.

FTP on the MPE platform is very similar to FTP on HP-UX, Linux or Windows but with some interesting MPE'isms. The first is the exitonerror command, which I use in all FTP transfers. It simply means that the FTP script will immediately stop upon any error. Since we do most transfers in batch we know if/when a transfer has failed, as the ftp program aborts with the error number in a variable called ftplasterr.

Below is a typical ftp batch transfer that we use in production. In this case it gets the self-extracting version of the Suprtool Winhelp Installation from our NTDEV server. Note that we echo the commands out to a temporary file and then just run the ftp program with the stdin equated to that file.

We also refer to our machines by variables, this way if we ever need to move a machine we just change the value of the global variable and the files will be picked up from that new machine.

!comment
!comment  Get self-extracting Zip file from server and
!comment  put onto dev.
!comment
!
!file ftptemp;temp
!purge suprhelp.whelpnew
!
!echo exitonerror                        >> ftptemp
!echo user username password             >> ftptemp
!echo binary                             >> ftptemp
!echo cd /d/robdev/winhelp/              >> ftptemp
!echo get suprhelp.exe suprhelp.whelpnew >> ftptemp
!echo quit                               >> ftptemp
!
!setvar ftplasterr 0
!run ftp.arpa.sys;info="!NTDEV";stdin=*ftptemp
!if ftplasterr <> 0 then
!   showvar ftp@
!   setjcw jcw fatal
!endif
!purge ftptemp,temp
!

You may also notice that we check the ftplasterr variable and show all of the values for the other variables.

Finally you may have noticed the cd /d/ command in the ftp portion of the script, the /d/ stands for the drive letter on the NT server which has multiple drives.

MPE File Attributes

When transferring files from one HP 3000 to another there is no longer a need to specify the attributes of the file, such as:

  ;rec=-80,1,f,ascii

MPE now magically keeps track of that for you and has done since a MPE 6.0.

When transferring a file to an MPE system from a non-MPE system or transferring through a non-MPE system you will need to specify the file attributes on the target MPE system as in:

  put mympefile myfile;rec=-80,1,f,ascii;disc=3000

The default file attributes can be specified for a file transferred to your MPE system by changing the corresponding line in the file BLDPARMS.ARPA.SYS which is shown below:

;REC=-80,,F,ASCII;DISC=204800
;REC=-256,,V,BINARY;DISC=204800
;REC=,,B;DISC=16384000
Note: only the first three lines are read; everything after them is ignored.

You may modify the first three lines as long as you keep the same syntax. I.e., you may change the numbers, or F to V, but don't add anything bizarre. Anything after a space is ignored, so don't insert any spaces. If the file is missing (or any line in it), the old hard-coded defaults will be used as a back up. These are:

   ;REC=-80,,F,ASCII;DISC=204800   for ASCII mode,
   ;REC=-256,,V,BINARY;DISC=204800 for binary mode.
   ;REC=,,B;DISC=16384000          for byte stream mode.

Also, if either the REC= part or the DISC= part of either line has bad syntax, the default for that part will be reverted to.

Users may make local copies of this file and set their own defaults via a file equation:

   :file bldparms.arpa.sys=myfile

Host Commands

You can execute commands on your local 3000 by putting a colon in front of your command such as:
ftp> :listf ,2
You can find out what commands you can do remotely with the remotehelp command:
ftp> remotehelp
214-HP ARPA FTP SERVER recognizes the following commands (* - unimplemented):
   USER    PORT    STOR    MSAM*   RNTO    NLST    MKD     CDUP
   PASS    PASV*   APPE    MRSQ*   ABOR*   SITE    XMKD    XCUP
   ACCT*   TYPE    MLFL*   MRCP*   DELE    SYST    RMD     STOU
   SMNT*   STRU    MAIL*   ALLO*   CWD     STAT    XRMD    SIZE
   REIN*   MODE    MSND*   REST*   XCWD    HELP    PWD     MDTM
   QUIT    RETR    MSOM*   RNFR    LIST    NOOP    XPWD
214 Refer to FTPDOC.ARPA.SYS for more FTP Server information.

Typically I just stream jobs on the remote system with FTP's site command by doing the following in the FTP client:

ftp> site stream robelle.job.robelle
200 STREAM command ok.

Site is a standard FTP command, but what host commands the FTP server at the other end supports varies from server to server.

In fact the Qedit for Window Server installation has it's own FTP client which ftp's the server and streams the "robelle" job to set the attributes of the Robelle account.

Filenames

On MPE the default namespace for a given file is typically the MPE namespace. For example if you put a file to your MPE system with the following FTP command:
put myfile mympefile
The file will go to the group you are currently logged into.

If you want to put files into the HFS namespace then you can just specify using the typical POSIX notation:

put myfile /MYACCOUNT/MYGROUP/mydirectory/myhfsfile

Documentation

For more information on the ins and outs of FTP you can read the latest ftp documentation at:

http://docs.hp.com/mpeix/pdf/36957-90159.pdf


What About Secure FTP?

The new buzzword on the Internet is "secure FTP". This can mean several things, as explained in this introduction:
http://www.intranetjournal.com/articles/200208/se_08_14_02a.html
"FTP was originally defined in the early 1970s to transfer files to and from various ARPANET nodes. However, there are a few problems with ye ol' standard FTP that we all grew up with in the early days of the Internet. First of all, it doesn't use strong authentication. It is based on password logins which can be guessed, or discovered by cybercriminals using a sniffer. Even if the password is not guessed or sniffed, with standard FTP none of the files being transferred to and from their destinations are encrypted. FTP sends files in clear plain-text exposing them to the plethora of bad guys out there who have nothing better to do than violate the privacy of others, pilfer confidential information such as credit card information, and attempt to obtain classified information that could compromise national security."

When considering FTP, there are 3 areas that can be secured: login (user authentication), the command channel (where you transmit the PUT and GET commands, etc.) and the data channel (where you transfer the files).

For example, here is a free client for secure ftp that uses SSL (it requires Java 2 version 1.3 on your machine):

http://www.glub.com/products/secureftp/
HP-UX has secure FTP login via the optional "Secure Internet Services", but I could not find clear documentation about secure command and data channels.

MPE/iX does not have secure FTP. Perhaps someone could configure a wrapper, such as the GLUB Secure FTP Wrapper (again requires Java 2, v 1.3):

http://www.glub.com/products/ftpswrap/


First-Time FTP Installation on an e3000

FTP stands for File Transfer Protocol and is the Internet standard way to move data and programs between systems, even systems of radically different OS architecture.

Starting with MPE/iX 6.0, it has been very easy to enable the ftp server on your HP e3000. Once enabled, the ftp server makes it easy for you to deliver output to your PCs, NT servers, UNIX boxes, even to servers across the world.

Here are the installation steps on MPE/iX:

  1. You need to stream a one-time configuration job, which configures the 3 files SERVICES.NET.SYS, PROTOCOL.NET.SYS & INETDCNF.NET.SYS.
       :STREAM  FTPCNFJ.ARPA.SYS
    
  2. Once that is done you need to stream the background job
       :STREAM JINETD.NET.SYS
    
    This second job needs to be always running for the ftp server to be enabled. So if you want to stop access to your ftp server then simply abort this job.
If you have any problems, there is a short manaul titled "Installing and Managing HP ARPA File Transfer Protocol Network Manager's Guide HP 3000 MPE/iX Computer Systems" (HP Part Number: 36957-90157) that has useful information on installing ftp. This manual also shows you how to easily setup Anonymous ftp access.

Security concerns: If your HP e3000 is on a network that is connected to the internet, then you've just opened it up to whole world. Make sure your accounts are password protected. If you rely on 3rd party products like VeSoft's Security/3000 then you may need current versions that support the MPE/iX 6.0 INETD approach. You may also need the BACKG job running with 'HELLO' executing and you may need to add a "$CONFIG FTP ..." line to your HELLO.DATA.VESOFT file. Check with your 3rd party vendor.


FTP Passwords with Vesoft's Security/3000

By François Desrochers, Robelle

I ran into a problem accessing our HP3000 with ftp. It kept on rejecting the connection request. More specifically, it was rejecting the login information.

   c:\>ftp my3000
   Connected to my3000.robelle.com.
   220 HP ARPA FTP Server [A0011001] (C) Hewlett-Packard Co. 1990
   User (my3000.robelle.com:(none)): francois/longpassword,user.account
   530 User log on unsuccessful
   Login failed.

I was able to connect using other protocols like telnet and NS/VT. After a number of tries monitoring the logon process, I realized that something was wrong with my password.

We use the session password feature of VESoft's Security/3000. One of the clues was in Security/3000's logfile. Whenever I entered an invalid password, I could see:

   FTP :HELLO FRANCOIS,USER.ACCOUNT : BAD PASSWORD
   13:44/142/AIF:PE CHLOGON: Refused chlogon FRANCOIS,USER.ACCOUNT: BAD PASSWORD

Whenever I entered my password, the login request was rejected, but nothing was logged. The request was apparently not getting to Security/3000. I changed my password to a temporary 3-character password and was able to use ftp. The logfile then showed:

   FTP :HELLO FRANCOIS,USER.ACCOUNT : OK
   13:46/142/AIF:PE CHLOGON: Allowed chlogon FRANCOIS,USER.ACCOUNT: CHECK ok

As I tried to set my password back to its original value, I realized that it had 9 characters. I set it to the first 8 characters and everything worked from that point.

Login requests have to go through the ftp server and MPE's AIF:PE before getting to Security/3000. I suspect that either ftp server or AIF:PE truncates the password to 8 characters before passing it to Security/3000.

We didn't have any password edit mask in Security/3000. I was not able to find the default maximum length but, just for fun, I have been able to enter an 80-character password. So, how do you define a password maximum length?

Enter a $VEPASS-FORBID directive similar to this one in the securcon file:

   $VEPASS-FORBID "len(p)>8" "Password must have 1-8 characters" francois,@.@

The first string is the condition that you want to check. "LEN" is a Security/3000 function which returns the length of the variable enclosed in parentheses. "p" is the internal variable name for the password itself. The second string is a message to be displayed if the condition is true. The last piece of information is optional. If not specified, the condition is checked for all users. In the example above, it is only checked when the session name is "francois". As with all VESoft software, $VEPASS-FORBID directives are flexible and can do almost anything you need.