Bringing Robelle KB To The Web

by Ken Robertson


To aid customer support and provide call and problem tracking, Robelle tech-support has been using an in-house knowledge-base (KB) system for over a decade.

When a call is logged into the TurboIMAGE database, it is filter-broadcast to Robelle technical staff (aka "techies") by email. Comments and suggestions can then be added to the database by anyone who receives the broadcast, which are then re-broadcast. An e3000 application searches the database via a combination of keywords and free-form text to aid in customer support - whether to help jog the memory for solutions, or to track outstanding trouble-calls and enhancement requests.

Although KB works very well internally, there were no hooks in the system to allow external access by our customers.

In our emerging 21st century web-centric society, our computer-savvy customers fully expect to have information available to them at a mouse-click's notice. From this perspective, Robelle KB has a problem: its current interfaces are terminal-based.

The Original Dumb Client: Designed to be used over a 1200-baud modem!

Our first thought was to write a web-based thin-client that accessed the HP e3000 database directly. However, the issue of our customers' privacy came up. A frequent occurrence within the call text was "Another contact is so-and-so at 555-1212". Our database is huge! Sanitizing the data manually would be a tremendous job, and costly.

For our final solution we wanted reliability, compatibility, easy maintenance, and of course customer accessibility. We decided to export each individual entry as an automatically sanitized web page, allowing the customer to easily search and view the entries. The tools that we used were Suprtool to extract the data, Qedit to format it nicely, and FTP to transfer the files from the HP 3000 to our HP-UX web server. (Suprtool does provide some basic HTML output through STEXPORT, but I wanted finer control over the pages.) This is all tied together by MPE Command Interpreter programming.

Each KB entry web page would look like the following example:

Robelle Technical Database Reference KB14712

Product: Qedit for Windows 4.7
Subject: want host commands
Status: Closed
Date Created: 1999.02.01  08:32
Date Modified: 2001.05.01  09:29
Short Description:    want host commands

Comments from Richard's HP-UX users regarding Qedit for Windows (trial): What I really miss about remote host editing is the ability to pop up a window and send some quick commands to the host system. Yea I can toggle between Qedit and Reflection, but Windows doesn’t have a quick convenient way to switch programs, and I still have to login twice.

Append: (E:E) NICKY GUNTHER This one is important to them 22 Mar99 11:39 AM
Host commands from within the client is one of three KBs that are the most important to Say It With Hares. Rich understands that we are working on this for 1999.

Append: (E:C) HANS HENDRIKS Implemented 01 May01 9:29 AM
Host commands have been available for some time now...

To see the actual web page for this KB entry, visit the Robelle KB archive.

Data Base Structure

The native database on an HPe3000 is TurboIMAGE. It is a network database, and the simplified structure of the KB database is shown below. Records are stored in datasets, analogous to tables in SQL.

The M-CALLS set contains information unique to each entry, and is called a Master set.

The Detail Set, D-CALL-TEXT, contains doubly-linked chains of compressed text in 512-byte blocks. Although each block is a fixed length, there may be as few as two characters in the block!

The nature of IMAGE databases is that the sets contain fixed-length records. In order to save space, I came up with the scheme of ending lines with a carriage return and packing them together contiguously. This is great for storage, but not that easy to unpack and export. Because a lot of the text is formatted by the techie in a particular way on purpose, I could not just blindly wrap the text. I would have to trust that the line breaks were in the correct spot.

Creating The Output Data...One Step At A Time

Everytime we tackle a seemingly difficult task using Suprtool, we tend to surprise the programmers who wrote it. This task was no different.

The algorithm I chose to create the web pages is

  1. Using Suprtool, extract the call numbers of the entries to be formatted for the web from the KB database into a file.
    Robelle staff flag the entries to be published.
  2. Format the entries into a header for each web page. Each header will be made up of formatted html and occupy a single record in the output file.
    We now have two files in the same sequence: call-numbers and formatted-headers.
  3. We will process the first record in the files.
  4. Based on the call-number in the first record, we extract all of the text from the database. We will also set a variable to the call-number that we wll later use to allow us to name the web-page-file.
  5. Using Qedit, we format the first header and the extracted text into a file. At this point, we have the opportunity to remove e-mail addresses and translate any special characters. Characters such as '<' and '>' will cause unpredictable results in the HTML scripting. We need to change the '<' and '>' to &lt; and &gt; to display correctly.
  6. Using the variable we set above, we name and save the file as KB<call-number>.html. We save this file as a variable-length, or bytestream file, so as to occupy the least amount of space. If we were to save the file as fixed-length, a 100-line text-web pages could become 100K in size!
  7. Using a simple FTP script that we build on the fly, we will copy the file to our web server.
  8. Delete the first entry from the call numbers file, and from the headers file.
  9. Repeat steps 3 through 8 until the call numbers file is empty.
Here's the abbreviated main loop of the jobstream...
!while finfo("hdr","EOF") > 0
! run;parm=4;info="use kbs31450"
! stsetvar
! run;parm=4;info="use setva2.cmd"
! stsetva2
! echo !<html!> > qsect1
! echo !<title!>KB !callnbr: !KBTitle !</title!> 
  >> qsect1
! echo !<h2!>Robelle Technical Database Reference 
  KB!callnbr!</h2!> >> qsect1 
! run;parm=4;info="use qwebuse"
! ftpit.cmd.mis kbtogo, kb!callnbr.html, /users/WWW/kbs,

A few things went wrong with this algorithm. For example, we needed to create 'Use' files for Suprtool and Qedit before the execution of the loop. If we do not, MPE will cause an EOF on the STDIN during the second pass of the loop; as the input stream will have been exhausted. We also have to convert special characters that are emebedded in the text.

Algorithm Implementation

Two of the off-the-shelf tools used in the processes are Qedit and Suprtool, both from Robelle. These are versatile "swiss-army" tools that allow you to extract and manipulate your data in virtually unlimited ways. The key is, of course, knowing how to use these tools effectively.

If you're unfamiliar with Qedit or Suprtool, you can we go.

Step 1: Extract the call numbers. This is the easiest part. :-)

We only select KB entries with a special "WebPost" keyword and which have been modified within the last 2 days. The philosophy is that technical support will mark those calls which are suitable for publication. Some calls are too boring for publication ("Please send a pre-release tape to customer X."). And some contain customer private information from dumps and log files.

We deliberately do not extract the customer-information from the call dataset into the file, since we want the web listings to be anonymous. This step is accomplished with Suprtool.

base   kb,5,secretpassword
get    m-calls
define MoveToWeb,keyword-block,7
define short-desc1,short-desc(1)
define short-desc2,short-desc(2)
item   date-modified,date,yyyymmdd
if     $upper(MoveToWeb) = "WEBPOST" and date-modified > $today(-2)
ext    call-nbr, status-FLAG, call-taker, date-created, time-created
ext    date-modified, time-modified, product-code,subject
ext    short-desc1, short-desc2, keyword-block
out    refs2mov,link,temp

Step 2: Get the records from refs2mov, and format them into headers. Each header contains unique information about the entry. We'll format it into html table rows for later processing. Recall that each header line only takes up one line in the file, which allows us to keep a synchronization with the call numbers file.

in   refs2mov
define  dc-yyyy,date-created,4
define  dc-mm,date-created[5],2
define  dc-dd,date-created[7],2
define  tc-hh,time-created,2
define  tc-mm,time-created[3],2
define  dm-yyyy,date-modified,4
define  dm-mm,date-modified[5],2
define  dm-dd,date-modified[7],2
define  tm-hh,time-modified,2
define  tm-mm,time-modified[3],2
comment  Build a table for the main details of the header for each entry. Tables look nice.
comment  We extract vertical bars instead of &, as this is our *own* internal representation.  
ext  "|lt;tr|gt;|lt;td|gt;Product:  |lt;/td|gt;|lt;td|gt;", product-code,"|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;td|gt;Subject:|lt;/td|gt;|lt;td|gt;", subject,"|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;td|gt;Status:|lt;/td|gt;|lt;td|gt;%%", status-flag, "|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;td|gt;Date Created:|lt;/td|gt;|lt;td|gt;", dc-yyyy,".",dc-mm,".", dc-dd,"&nbsp;&nbsp;",tc-hh,":"
ext  tc-mm,"|lt;/td|gt;|lt;/tr|gt;"-mm,"|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;td|gt;Date Modified:|lt;/td|gt;|lt;td|gt;",dm-yyyy,".",dm-mm,".", dm-dd,"&nbsp;&nbsp;",tm-hh,":"
ext  tm-mm,"|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;td valign='top'|gt;Short Description:&nbsp;&nbsp;&nbsp;|lt;/td|gt;|lt;td|gt;",short-desc1
ext  short-desc2,"|lt;/td|gt;|lt;/tr|gt;"
ext  "|lt;/table|gt;|lt;pre|gt;|lt;hr|gt;"
ext  "|lt;font color=blue|gt;Originator: ",call-taker,"|lt;/font|gt;|lt;br|gt;"
out  hdr,temp,ascii

Now you might be thinking....why go to all that trouble to insert |lt; and |gt; when I could be outputting < and > ?
The reason is that the data extracted from the database could very well contain < and >, which would create html errors when viewing the page. In the sanitizing step, we will change all occurrences of < and > to &lt; and &gt; then change |lt; to < and |gt; to > to complete the html formatting.

Step 3: Process the first record of the file.
We're going to name each web page as KB<referencenumber>.html. This will provide a simple method of maintaining files on our web site.
While we're at it, we'll set the title to a variable so that we can insert it into the web page easily in our formatting step.

Step 4: Get the Text of the Entry
This step is going to be repeated over and over again until the call numbers file is empty. An interesting quirk of MPE and re-directing STDIN from a job is that you can't use it in a loop. Although the code below on the left looks sane, it will NOT work!

Bad! Good!
setvar count 1
While count < 10
    in foo
    list standard
    setvar count count + 1
setvar count 1
echo in foo > usem
echo list standard >> usem
echo exit >> usem
While count < 10
    run;info="use usem";parm=4
    setvar count count + 1

In the "Bad" example, the second time through the loop we will get an EOF error on the STDIN of the job. To get around this, we need to put our Suprtool commands inside a "use" file. Since all of the STDIN is inside a file, Suprtool re-opens it every time and the problem goes away.

Here is the "Use" file that we will need.

!comment **** get the first rec from the call numbers file
!echo purge ref2mov,temp > kbs31450
!echo setjcw cierror 0 >> kbs31450
!echo in refs2mov >> kbs31450
!echo numrecs 1 >> kbs31450
!echo out ref2mov,link,temp >> kbs31450
!echo xeq >> kbs31450
By using the numercs 1 command, we are only allowing
one record in the output file. We'll use this as a key to retreive our text
and set some variables. It will also give us a 'current entry' to work with
inside the loop.
!comment **** setup the two Use files for setting the variables
!echo purge stsetvar,temp >> kbs31450
!echo purge stsetva2,temp >> kbs31450
!echo setjcw cierror 0 >> kbs31450
!echo in ref2mov >> kbs31450
!echo ext "setvar callnbr ", call-nbr >> kbs31450
!echo out stsetvar,temp,ascii >> kbs31450
!echo xeq >> kbs31450
This creates the first of two small command files used for setting variables.
We will use the Callnbr variable to name the web page.
The Stsetva2 variable will become part of the <title> construct.

setvar callnbr <callnbr>
!echo >> kbs31450
!echo in ref2mov >> kbs31450
!echo ext short-desc1, short-desc2 >> kbs31450
!echo out stsetva2,temp,ascii >> kbs31450
!echo xeq >> kbs31450
Here is the contents of the second command file:

setvar stsetva2 <short description>

!comment **** Get all of the text records for that entry
!echo base kb,5,
secretpassword>> kbs31450
!echo table refs,call-nbr,file ref2mov >> kbs31450
!echo chain d-call-text,call-nbr=refs >> kbs31450
!echo ext text-block >> kbs31450
!echo out textdata >> kbs31450
!echo purge textdata >> kbs31450
!echo xeq >> kbs31450
!echo exit >> kbs31450
Using the refs file created above, we extract all of the text for the
current entry.
Recall that all of the individual lines are concatenated with
carriage-returns as separators.

Step 5: Use Qedit to get the appropriate header and glue everything together.
This step is truly ugly. Similar to the problem with Suprtool, we have to stuff all of our commands into a "Use" file.

We format the header and the extracted text into a single file. At this point, we have the opportunity to cleanup and sanitize the text:

Here is the code that does all the steps above:

!comment main qedit use file.
set lang data > qwebuse
!echo set len 1000 >> qwebuse
aq]=emptyln >> qwebuse
!echo dq all >> qwebuse
!echo aq ]=hdr [ >> qwebuse
cq "<"&lt;"@ >> qwebuse
cq ">"&gt;"@ >> qwebuse
cq "|lt;"!<" >> qwebuse
cq "|gt;"!>" >> qwebuse
!echo lq all >> qwebuse
!echo keep hdrchg,temp,yes >> qwebuse
!echo dq all >> qwebuse
!echo aq]=textdata >> qwebuse
!echo cq ">"!>"@ >> qwebuse
!echo cq "<"!<"@ >> qwebuse
!echo cq "Email: [ -~]*" (regexp) "" @ >> qwebuse
!echo l $c $o >> qfowebuse
!echo keep textdata,yes >> qwebuse
!echo dq all >> qwebuse
!echo set len 1000 >> qwebuse
!echo aq]= qsect1 >> qwebuse
!echo a]=hdrchg >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo cq "             "" * >> qwebuse
!echo c
q "%%S"!<font color=blue!>Signed-Off!</font!>"@ >> qwebuse
!echo cq "%%B"!<font color=red!>Bug!</font!>"@ >> qwebuse
!echo cq "%%E"!<font color=blue!>Enhancement Request!</font!>"@ >> qwebuse
!echo cq "%%C"!<font color=blue!>Closed!</font!>"@ >> qwebuse
!echo cq "%%M"!<font color='#FF8C00'!>Mystery!</font!>"@ >> qwebuse
!echo cq "%%T"!<font color=green!>Tape sent to customer!</font!>"@ >> qwebuse
!echo cq "%%P"!<font color=green!>Pre-Release Available!</font!>"@ >> qwebuse
!echo cq "%%O"!<font color=blue!>Open!</font!>"@ >> qwebuse
!echo aq]=textdata >> qwebuse
!echo set decimal on >> qwebuse
cq '13'13 '13 @ >> qwebuse
cq '13 '13'10 @ >> qwebuse
!echo cq '27 "~"@ >> qwebuse
!echo cq "~&dH"!<hr!>"@ >> qwebuse
!echo aq]=qsectf >> qwebuse
!echo cq "LQ "Qedit " @ >> qwebuse
!echo cq "ST "Suprtool " @ >> qwebuse
!echo cq "QW "Qedit for Windows "@ >> qwebuse
!echo cq "* Note: "!<font color=blue!>Append: "@ >> qwebuse
cq " AM *" AM !</font!>"@ >> qwebuse
!echo cq " PM *" PM !</font!>"@ >> qwebuse
!echo cq "* "'13'10 ""@ >> qwebuse
set keep var on >> qwebuse
!echo keep kbtogo,yes >> qwebuse
!echo t hdr,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo t refs2mov,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo exit >> qwebuse

There are a few tricks in the above code...

cq '13'13 '13@
cq '13 '13'10@
'13 is the decimal value for Carriage Return (cr). Changes two carriage returns into one. Eliminates some blank lines.
Since our data contains cr's as separators, and windows needs crlf (carriage-return line-feed), change all cr's to crlf. '10 is the decimal value for line-feed.
cq " AM *" AM !</font!>"@ We put exclamations in front of the > and < so that the HP CI doesn't mistake them for I/O re-direction commands.
cq "%%S"!<font color=blue!>... Although there are many status code changes, only one of them will actually take effect. It's sort of like falling through an if...end constuct.
cq "<"&lt;"@
cq ">"&gt;"@
Change all occurrences of < and > to &lt; and &gt; This ensures that the data in the entry won't cause any html tag problems.
cq "|lt;"!<" @
cq "|gt;"!>" @
Change our coding of |lt; and |gt; to be < and >. We prefix the < and > with an ! so that MPE won't be confusing our data with I/O redirection.
set lang=data We do this so that we can have at least 1000 characters per line. Set lang=text only allows 256 characters.
set keep var on This ensures that we don't keep all 1000 characters per line! The recordsize is variable.
aq]=emptyln This forces the switch from any defaults that may have been set regarding file type.

Step 6: Save the HTML as a variable-length file.

Most importantly, the file is kept as a bytestream, or variable-length file. This helps minimize the size of the webpages. If the files were fixed length, each line of the file would require 1000 bytes. So...a 40 line page would be 40k...just in text! Some KB entries are hundreds of lines long. With little effort on our part, using variable-length files makes the web-surfer's load-time much less.

Step 7: Copy the file to the web using FTP.

ftpit.cmd kbtogo, kb!callnbr.html, /users/WWW/kbs,

The contents of the ftpit command file is

parm fromfile="?",tofile="",todir="",tocpu=""
if "!fromfile"="?" then
echo fromfile="?",tofile="",todir="",tocpu=""
echo user prodsend > ftpin
echo !PRODPASS >> ftpin
echo cd !todir >> ftpin
echo put !fromfile !tofile >> ftpin
echo site chmod 666 !tofile >> ftpin
ftp !tocpu <ftpin >ftpout

One important note to remember is to do a chmod 666 of the file. This allows non-owners the right to view the file. Depending upon how your FTP logon and web server directory are set up, you may need different chmod permissions.

Step 8 and 9. Delete the first entry from the headers and call-numbers files, then loop until done.

!echo t hdr,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse
!echo t refs2mov,labels >> qwebuse
!echo d [ >> qwebuse
!echo k,yes >> qwebuse

Much of the code written in the above steps was to set up the loop for execution. The loop itself is fairly simple and straightforward.

!while finfo("hdr","EOF") > 0
! run;parm=4;info="use kbs31450"
! comment
! comment First html section to add in QW script
! comment
! stsetvar
! run;parm=4;info="use setva2.cmd"
! stsetva2
! echo !<html!> > qsect1
! echo !<title!>KB !callnbr: !KBTitle !</title!> >> qsect1
! echo !<link rel="stylesheet" href="/style.css" type="text/css"!> >> qsect1
! echo !<body bgcolor="#ffffff" background="/gif/background.gif"!> >> qsect1
! echo >> qsect1
! echo !<!-- std header for root web directory --!> >> qsect1
! echo !<!--#include virtual="/header.txt" --!> >> qsect1
! echo >> qsect1
! echo !<h2!>Robelle Technical Database Reference KB!callnbr!</h2!> >> qsect1
! echo !<table!> >> qsect1
! run;parm=4;info="use qwebuse"
! ftpit.cmd.mis kbtogo, kb!callnbr.html, /users/WWW/kbs,
! tell ken,mgr.mis Moved!callnbr.html !hptimef
! print ftpout


That's it. The resulting KB-to-Web task runs every night on our e3000 server and generates new and replacement web pages for our web server. To complete the project we wrote server Perl scripts which allow you to search the KB web pages, append comments to an existing KB entry or submit a new one (all submissions go to Robelle support who then forward them to KB). You can try the finished application at

As you can see, the method in this project was to use scripts to combine a few dependable tools that manipulate data and text. The tools that we used were Suprtool to extract the data, Qedit to format it nicely, and FTP to transfer the files from the HP 3000 to our HP-UX web server. This is all tied together by MPE Command Interpreter programming. No actual COBOL or C programming required. The conversion isn't very fast, but it runs late at night and doesn't have that great a volume to process each day. For more implementation details, read the expanded version of this article at

Qedit Primer

Qedit has been one of the mainstay editors on the HPe3000 for decades. It understands how to handle the different types of files on the HP, as well as some from other platforms. You can use it as a text-entry tool, as source-code creation and maintenance, or in batch as a text/data-processing tool.

Qedit has many text-manipulation 'line-mode' style commands. You can also use almost every MPE CI command within the Qedit shell.

Here are the ones that I've used in the KBsToWeb process.

Command as Used Long Form Description
set lang data set language data Sets the current filetype to allow up to 1000 character-wide records.
set len 1000 set length 1000 Sets the length of each fixed record to 1000
aq]=emptyln addq last=<filename> Append the contents of a named file to the end of the current file.
The 'q' stands for quiet, and does not list the contents of the file being copied to stdout.
dq all deleteq all Delete all of the records in the file. It's in Quiet mode, too.
cq "<"&lt;"@ Changeq "string1"string2" all Change all occurrences of string1 to string2 within the entire file. Quiet mode.
lq all listq all List all the lines of the file without line numbers
keep hdrchg,temp,yes Keep <filename>,temp,yes Save the contents of the file to the temporary file <filename>.
The "yes" indicates to write over the file if it exists..
l $c $o list $char $octal I see that I've left a debugging statement in the code.
Lists the current line with half of the display in readable characters, the other half in its octal equivalent.
HP computers initially used Octal to display binary data in readable form.
You can use the $hex option if you are more comfortable with that!
cq "             "" * Changeq "string1"string2" all This removes bunches of blank spaces. Essentially, change " <spaces> " to null.
set decimal on set decimal on This allows you to use a single quote to define characters for use in change and search.
For example, change '27"[esc]"@ will change all occurrences of the escape character to [esc].
set keep var on set keep variable on This instructs qedit to set the keep options for the current file to variable records.
t refs2mov,labels text <filename>,labels Copy the contents of filename into the current working file. If the file has filelabels, then remember them
when it comes time to save the file.
This is useful for when you are editing Self-Describing files.

Suprtool Primer

Suprtool has been around almost as long as Qedit has. It is one of the foremost data extraction tools available for the HPe3000 and HP-UX computers. Using low-level tightly-controlled code, it's main use is to extract data as quickly as possible. Usually some criteria is used in extracting the data, such as customer numbers, date ranges, etc. You can automate the update of data in databases using Suprtool, or use it to create and format reports for the web.

Suprtool also has some text manipulation tools, but they are not as strong as Qedit's. Together these two products create an impressive tool for data extraction and maipulation.

Suprtool has its own command set. Here are the ones that I've used in the KBsToWeb process.

Command as Used Long Form Description
base kb,5,secretpassword base dbname, openmode ,secretpassword Opens the named database.
get m-calls get <setname> Specify the input source to be a dataset within the currently opened database.
define MoveToWeb,keyword-block,7 define NewName, Fieldname, length Define a new fieldname, starting at the beginning of Fieldname for a certain number of bytes.
item date-modified,date,yyyymmdd item Fieldname,date,yyyymmdd The item Fieldname is of type Date, and its internal format is yyyymmdd.
if $upper(MoveToWeb) = "WEBPOST" and date-modified > $today(-2) IF Command This is not a boolean operation, but a filter on the data extraction. A simple example would be IF CALL-NBR=5000. This would set up a filter to retrieve all records whose call-nbr was equal to the value 5000. The IF command is extremely complex and requires much study.
$upper $UPPER(fieldname) Convert the fieldname to uppercase. This is used in conjunction with the IF command.
$today(-2) $TODAY(num) Convert $today(num) to some date value relative to today. This will become an internal constant during the IF operation.
ext call-nbr, status-FLAG, call-taker, date-created, time-created EXTRACT Fieldname1, Fieldname2... Extracts the fieldnames and contenates them into an output file. You can also extract constants and mathematical results of fields. For example, you can
Extract fielda=fieldb + fieldc
You can also use extract to do datatype conversion
out refs2mov,link,temp OUTPUT filename, link, temp Specifies the name of the output file to be created, its disposition, and the type. The type of LINK will cause the file to become self-describing. That is, the file will contain a mini-dictionary of the record layout.
xeq XEQ Execute all of the steps in the last specification. The entire specification comprises either from the beginning of program execution, or the last XEQ command.
in refs2mov INPUT filename The source of input will be a flat file. It may be self-describing, but it doesn't have to be. Suprtool currently only handles fixed-length files as input.

May 25, 2001