APLawrence.com -  Resources for Unix and Linux Systems, Bloggers and the self-employed

A Y2K Problem solved with Expect

© May 1999 Tony Lawrence

Small SCO systems running RealWorld Accounting are very common. Like every other software vendor's products, older RealWorld systems are not Y2K ready. In most cases, a simple upgrade to their most current release is the best answer, but some RealWorld systems were also heavily customized, and that can require expensive and time consuming rewrites of the customized portions.

I recently was approached by a local RealWorld reseller who had a customer with just that problem: the Order Entry module had been extensively modified to meet the needs of the customer's business.

Typically, Order Entry and Inventory Control are the places where businesses differ. Most people can get by with standardized General Ledger and Payables, but it's the sales side of the business where customization is needed. Modern versions of these packages offer much more flexibility in those areas, so some of the "customization" can be done without access to source code. That was not true for the older versions.

Because of the extent of the modifications, and the expense of rewriting, the RealWorld reseller came up with the idea of upgrading to the latest versions of Accounts Receivable and General Ledger, but retaining the old Order Entry version. The two digit dates are not really significant within Order Entry, at least not during the actual data entry, so this would allow the customer to continue to use his specialized entry fields without the high expense of rewriting.

The necessary condition for making this work is that the data from the old Order Entry could be transferred into the new, Y2K compliant modules. RealWorld makes no provision for transferring data from older versions beyond the initial upgrade procedure, but all RealWorld versions have always had the capability of exporting and importing their data files. That capability can be used to rebuild damaged files, or to export or import data from foreign systems. So what we planned was to set up a procedure to regularly export data from the old module, massage it slightly, and import it into the new modules.

Unfortunately, the import/export procedures, and of course the necessary adjustments to the data along the way, are more than the users of the system would want to do. They want a simple menu choice to transfer orders between modules, and that's what I was asked to provide.

The tool I chose for this is Expect (see Book Review: Expect). You can install Expect from Skunkware , or download the source from https://www.nist.gov. If you install from Skunkware, the script will also install TCL (Expect is a TCL program). You should download the source anyway, because the Skunkware install doesn't include the example scripts.

If you are a Perl programmer and have never used TCL, you probably won't like it. It lacks many of the really nice features of Perl, and its syntax is just similar enough to screw you up badly. Fortunately, you don't need a lot of TCL to find Expect useful.

What Expect does is control other programs. It's called "Expect" because at its core, what it does is "expect" some data to be produced by the program, and then sends appropriate responses. It handles all the nasty input/output details for you, and it is amazingly easy to write quite complex interactions quickly and without a great deal of effort. What I needed to control for the first step of this project was the export from the old programs.

This version exports from a program called rwc-arfu. After execution, it first brings up a screen that looks like this:

File recovery utilities                                                         
Version 4.0A1

        Please enter company-ID: __

Company-ID must be two characters (A-Z, 0-9)

Therefore, my first attempt at automating this started out like this:

#!/usr/local/bin/expect -f
spawn ./rwc-arfu
expect "9)"
send "AE\r"
The expect distribution also includes "autoexpect", which lets you build expect scripts just by running your program and interacting with it. This does have some limitations and the generated scripts will usually require hand-editing anyway, so you might as well learn how to do it manually first.

A couple of things to note here: The "9)" is down at the bottom of the screen, but the typing takes place up at "ID:". Obviously the RealWorld program had to have sent the lines at the bottom of the screen before moving back up to accept input, so it's the "9)" that I "expect". I don't need to concern myself with what came before that; Expect just ignores generated output until it gets to what you told it to look for.

The "AE" is the "Company ID". I followed that with a "/r", which is a carriage return rather than the "/n" (newline) that we're all used to typing in C and Perl programming. That's because the RealWorld program is in "raw" mode. If I had used "/n", it wouldn't have recognized it and would not have worked.

Finally note the "interact". That handy little statement turns control back to me at the keyboard. When I run this script, it enters the "AE" company id, and then everything else will need to be typed by me. That's a handy feature; it lets you start up interactive programs that ask a bunch of annoying questions on the way in, and in this case, it lets me test my script as I go along, with me always in control, noting the next required step.

In this case, the next step is a prompt for "Right Company?" followed by a "Y" already filled at the bottom of the screen:

File recovery utilities                                                         
Version 4.0A1

        Please enter company-ID: AE ATHLETIC ESSENTIALS, INC.

Right company ? Y

I thought that the proper expect string would have been "any ? Y", or even just "Y", but neither of those worked. It isn't Expect's fault; programmers sometimes output data in a different order than where it ends up on the screen. So the next part of the script looked like this:

#!/usr/local/bin/expect -f
spawn ./rwc-arfu
expect "9)"
send "AE\r"
expect "any ?"
send "\r"

I continued in this manner, incrementing the script step by step, but ran into a problem a little further on. It was actually simply that I got confused and missed a step in the procedure, so the script wasn't working right. Expect can help you when that happens with two variables it keeps for you: $expect_out(buffer), which holds everything that the program sent between the last match and the current match, and $expect_out(0,string), which holds the string you just matched. To find what I was missing, I used:

expect -re "(.*)"
send_user "I got-$expect_out(buffer)-"

The "expect -re" tells Expect that you want to use regular expressions (wildcard patterns). The "send_user" sends its output to the standard output, not to the spawned application. In this simple case, there's no necessity for putting the ".*" in parentheses, but it can become useful when you want to break the string into multiple parts. Consider this small program:

#!/usr/local/bin/expect -f
expect -re "(.*)=(.*)"
send_user "Results\n0: $expect_out(0,string)\n1: $expect_out(1,string)\n2: $expect_out(2,string)"

If you run this and type "TERM=ansi", you'll see that the parentheses cause the $expect_out string array to be filled:

0: TERM=ansi

2: ansi

Note that (0,string) also includes the linefeed from the input!

The rest of the program just continues along with expect/send pairs until we get to the point where something can go wrong. If RealWorld doesn't have any records to export, or if the file requested is in use by another user, the export fails. We handle those conditions like this:

expect {
"Press F1" { send "\033\[M" }
default exit
sleep 1
expect "Press F1"
send "\033\[M"
expect "Field"
send "\r"
expect "Conversion"
send "\t\t\t\t"
#end of script

The braces following "expect" allow multiple choices. In this case, if the program has responded with "Press F1", we can continue by sending the F1 string (obviously this program expects to be run on an Ansi terminal; if we need it to run on other terminals a more sophisticated program would be necessary). We could test for each possible string that could be sent and react appropriately, but for this situation, if we don't get "Press F1", we just want to get out. If we do get it, we continue with the rest of the send/expect sequences, and finally exit out with a series of Tabs (\t's).

Did you notice the "sleep 1" before the expect for the second "Press F1"? That shouldn't be necessary, but in some testing runs, I found I got hung up without it. I'm really not sure why, but keep in mind that Expect isn't perfect and outside influences like the present load on the machine can affect your results. The problem I had is surely a timing issue, and is probably related to getting the second escape sequence too quickly; in other words, the program wasn't really ready to accept it even though it had issued the prompt. These sorts of problems can happen even when humans are driving the keyboard, so it isn't unreasonable that Expect sometimes goofs.

Expect has lots of cute tricks up its sleeves, though. One of the options to "send" is the "-h" option, which sends characters in a way that resembles a human typist.

Anyone who has ever tried to drive another program with a script will immediately appreciate how useful Expect is. The simplistic use shown here is only the beginning; you can do some really incredible things with simple Expect scripts.

See for an example of using Expect to capture output.

Got something to add? Send me email.

(OLDER)    <- More Stuff -> (NEWER)    (NEWEST)   

Printer Friendly Version

-> A Y2K Problem solved with Expect


Inexpensive and informative Apple related e-books:

Take Control of the Mac Command Line with Terminal, Second Edition

Are Your Bits Flipped?

Take Control of OS X Server

Photos: A Take Control Crash Course

Take Control of Automating Your Mac

More Articles by © Tony Lawrence

Wed Mar 22 10:22:48 2006: 1804   anonymous

--- snip ---
"If you are a Perl programmer and have never used TCL, you probably won't like it. It lacks many of the really nice features of Perl, and its syntax is just similar enough to screw you up badly. Fortunately, you don't need a lot of TCL to find Expect useful."
--- snip ---
Now .. now.. Mr Lawrence, thats a bit of a harsh slug on the old TCL don't you think ? ... ;-)
Bruce Baumann

Wed Mar 22 11:16:30 2006: 1806   TonyLawrence

Ayup. I think TCL is a pretty poor language. Obviously you can do god things in bad languages: Expect is a good example.


Printer Friendly Version

Have you tried Searching this site?

This is a Unix/Linux resource website. It contains technical articles about Unix, Linux and general computing related subjects, opinion, news, help files, how-to's, tutorials and more.

Contact us

Printer Friendly Version

Technology is a word that describes something that doesn’t work yet. (Douglas Adams)

Linux posts

Troubleshooting posts

This post tagged:





Unix/Linux Consultants

Skills Tests

Unix/Linux Book Reviews

My Unix/Linux Troubleshooting Book

This site runs on Linode