A Y2K Problem solved with Expect

Small SCO systems running RealWorld Accounting are very common. Like every other software vendor's products, older RealWorld systems are not Y2K ready. In most cases, a simple upgrade to their most current release is the best answer, but some RealWorld systems were also heavily customized, and that can require expensive and time consuming rewrites of the customized portions.

I recently was approached by a local RealWorld reseller who had a customer with just that problem: the Order Entry module had been extensively modified to meet the needs of the customer's business.

Typically, Order Entry and Inventory Control are the places where businesses differ. Most people can get by with standardized General Ledger and Payables, but it's the sales side of the business where customization is needed. Modern versions of these packages offer much more flexibility in those areas, so some of the "customization" can be done without access to source code. That was not true for the older versions.

Because of the extent of the modifications, and the expense of rewriting, the RealWorld reseller came up with the idea of upgrading to the latest versions of Accounts Receivable and General Ledger, but retaining the old Order Entry version. The two digit dates are not really significant within Order Entry, at least not during the actual data entry, so this would allow the customer to continue to use his specialized entry fields without the high expense of rewriting.

The necessary condition for making this work is that the data from the old Order Entry could be transferred into the new, Y2K compliant modules. RealWorld makes no provision for transferring data from older versions beyond the initial upgrade procedure, but all RealWorld versions have always had the capability of exporting and importing their data files. That capability can be used to rebuild damaged files, or to export or import data from foreign systems. So what we planned was to set up a procedure to regularly export data from the old module, massage it slightly, and import it into the new modules.

Unfortunately, the import/export procedures, and of course the necessary adjustments to the data along the way, are more than the users of the system would want to do. They want a simple menu choice to transfer orders between modules, and that's what I was asked to provide.

The tool I chose for this is Expect (see Book Review: Expect). You can install Expect from Skunkware , or download the source from http://www.nist.gov. If you install from Skunkware, the script will also install TCL (Expect is a TCL program). You should download the source anyway, because the Skunkware install doesn't include the example scripts.

If you are a Perl programmer and have never used TCL, you probably won't like it. It lacks many of the really nice features of Perl, and its syntax is just similar enough to screw you up badly. Fortunately, you don't need a lot of TCL to find Expect useful.

What Expect does is control other programs. It's called "Expect" because at its core, what it does is "expect" some data to be produced by the program, and then sends appropriate responses. It handles all the nasty input/output details for you, and it is amazingly easy to write quite complex interactions quickly and without a great deal of effort. What I needed to control for the first step of this project was the export from the old programs.

This version exports from a program called rwc-arfu. After execution, it first brings up a screen that looks like this:

File recovery utilities                                                         
Version 4.0A1






        Please enter company-ID: __






Company-ID must be two characters (A-Z, 0-9)
 

Therefore, my first attempt at automating this started out like this:

#!/usr/local/bin/expect -f
spawn ./rwc-arfu
expect "9)"
send "AE\r"
interact
 

A couple of things to note here: The "9)" is down at the bottom of the screen, but the typing takes place up at "ID:". Obviously the RealWorld program had to have sent the lines at the bottom of the screen before moving back up to accept input, so it's the "9)" that I "expect". I don't need to concern myself with what came before that; Expect just ignores generated output until it gets to what you told it to look for.

The "AE" is the "Company ID". I followed that with a "/r", which is a carriage return rather than the "/n" (newline) that we're all used to typing in C and Perl programming. That's because the RealWorld program is in "raw" mode. If I had used "/n", it wouldn't have recognized it and would not have worked.

Finally note the "interact". That handy little statement turns control back to me at the keyboard. When I run this script, it enters the "AE" company id, and then everything else will need to be typed by me. That's a handy feature; it lets you start up interactive programs that ask a bunch of annoying questions on the way in, and in this case, it lets me test my script as I go along, with me always in control, noting the next required step.

In this case, the next step is a prompt for "Right Company?" followed by a "Y" already filled at the bottom of the screen:

File recovery utilities                                                         
Version 4.0A1






        Please enter company-ID: AE ATHLETIC ESSENTIALS, INC.






Right company ? Y
 

I thought that the proper expect string would have been "any ? Y", or even just "Y", but neither of those worked. It isn't Expect's fault; programmers sometimes output data in a different order than where it ends up on the screen. So the next part of the script looked like this:

#!/usr/local/bin/expect -f
spawn ./rwc-arfu
expect "9)"
send "AE\r"
expect "any ?"
send "\r"
interact
 

I continued in this manner, incrementing the script step by step, but ran into a problem a little further on. It was actually simply that I got confused and missed a step in the procedure, so the script wasn't working right. Expect can help you when that happens with two variables it keeps for you: $expect_out(buffer), which holds everything that the program sent between the last match and the current match, and $expect_out(0,string), which holds the string you just matched. To find what I was missing, I used:

expect -re "(.*)"
send_user "I got-$expect_out(buffer)-"
 

The "expect -re" tells Expect that you want to use regular expressions (wildcard patterns). The "send_user" sends its output to the standard output, not to the spawned application. In this simple case, there's no necessity for putting the ".*" in parentheses, but it can become useful when you want to break the string into multiple parts. Consider this small program:

#!/usr/local/bin/expect -f
expect -re "(.*)=(.*)"
send_user "Results\n0: $expect_out(0,string)\n1: $expect_out(1,string)\n2: $expect_out(2,string)"
 

If you run this and type "TERM=ansi", you'll see that the parentheses cause the $expect_out string array to be filled:

Results
0: TERM=ansi

1: TERM
2: ansi
 

Note that (0,string) also includes the linefeed from the input!

The rest of the program just continues along with expect/send pairs until we get to the point where something can go wrong. If RealWorld doesn't have any records to export, or if the file requested is in use by another user, the export fails. We handle those conditions like this:

expect {
"Press F1" { send "\033\[M" }
default exit
}
sleep 1
expect "Press F1"
send "\033\[M"
expect "Field"
send "\r"
expect "Conversion"
send "\t\t\t\t"
#end of script
 

The braces following "expect" allow multiple choices. In this case, if the program has responded with "Press F1", we can continue by sending the F1 string (obviously this program expects to be run on an Ansi terminal; if we need it to run on other terminals a more sophisticated program would be necessary). We could test for each possible string that could be sent and react appropriately, but for this situation, if we don't get "Press F1", we just want to get out. If we do get it, we continue with the rest of the send/expect sequences, and finally exit out with a series of Tabs (\t's).

Did you notice the "sleep 1" before the expect for the second "Press F1"? That shouldn't be necessary, but in some testing runs, I found I got hung up without it. I'm really not sure why, but keep in mind that Expect isn't perfect and outside influences like the present load on the machine can affect your results. The problem I had is surely a timing issue, and is probably related to getting the second escape sequence too quickly; in other words, the program wasn't really ready to accept it even though it had issued the prompt. These sorts of problems can happen even when humans are driving the keyboard, so it isn't unreasonable that Expect sometimes goofs.

Expect has lots of cute tricks up its sleeves, though. One of the options to "send" is the "-h" option, which sends characters in a way that resembles a human typist.

Anyone who has ever tried to drive another program with a script will immediately appreciate how useful Expect is. The simplistic use shown here is only the beginning; you can do some really incredible things with simple Expect scripts.

2 comments

More Articles by Tony Lawrence - Find me on Google+

Click here to add your comments




Wed Mar 22 10:22:48 2006: 1804   anonymous


--- snip ---
"If you are a Perl programmer and have never used TCL, you probably won't like it. It lacks many of the really nice features of Perl, and its syntax is just similar enough to screw you up badly. Fortunately, you don't need a lot of TCL to find Expect useful."
--- snip ---
Now .. now.. Mr Lawrence, thats a bit of a harsh slug on the old TCL don't you think ? ... ;-)
Regards
Bruce Baumann



Wed Mar 22 11:16:30 2006: 1806   TonyLawrence


Ayup. I think TCL is a pretty poor language. Obviously you can do god things in bad languages: Expect is a good example.

Don't miss responses! Subscribe to Comments by RSS or by Email

Click here to add your comments

If you want a picture to show with your comment, go get a Gravatar

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.