Expect: How a 20-Year-Old Tool Saved My Project

I joined zulily in August of 2010, and at the time the company as a whole consisted of only 35 people. One of my first projects involved integrating one of our systems with a system owned by a much larger, more established partner company. The details of what these systems did aren’t relevant, except that the integration mechanism was for our system to drop XML files on an SFTP server that the partner company owned and operated.

At the time we were a 100% PHP shop (this has since changed), so I implemented our side of the integration in PHP, and used an open source PHP library called phpseclib to handle the actual SFTP data transfer. The partner company didn’t have any clients who used PHP, and didn’t officially support this library, but it worked great throughout development and integration testing. The integration test phase of the project took approximately 3 months, and not once during that time did we ever have even the slightest hiccup when transferring data between systems.

However, once we went to full production, we started seeing file corruption — specifically, sometimes files we transferred to the partner’s SFTP server would be truncated. There was no discernible pattern to the truncation; it happened at different points every time, and often a file that failed once would work fine when it was retried.

Naturally, this caused some consternation, as our code hadn’t changed, and it had been working for months without fail. When I pointed the exact same code at their test server, and sent the exact same file content, everything worked flawlessly. When I pointed it back to their production server, the files were truncated.

Clearly, to my mind at least, the problem was on their end. After a couple of days of frenzied troubleshooting, we discovered that the version of the SFTP server software running on their test machines was newer than what they ran in production. Presumably, we were hitting some bug in that software that was fixed between the two versions.

Since they were a very large company with many other clients using these servers, they were unwilling to upgrade their SFTP software on our behalf. Also, given that we were already well into the go-live phase of the integration, rewriting our system in another language wasn’t an appealing option, especially since there was no guarantee that the new implementation would work any better.

One option that the parent company did officially support, though, was the sftp client included with OpenSSL. I tried manually transferring a few files this way…and the file truncation issue disappeared.

There were problems with this approach, though: for one, the SFTP server required authentication, and the partner company was unwilling to set up SSH keys for us to do non-interactive authentication. OpenSSL’s sftp client doesn’t support setting the authentication credentials via command line parameters, leaving us stuck with authenticating interactively. This obviously wasn’t an acceptable long-term solution, since these systems needed to communicate with each other without human intervention.

I don’t recall exactly when I stumbled across it, but somewhere in the midst of searching for a solution I came across Expect.

Quoting the introduction on the Expect homepage:

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial.

This sounded exactly like what I needed! I bought a copy of “Exploring Expect”, read enough of it on the train ride home to get started, and after a couple of hours I had whipped up a PHP script that did the following:

  1. Dump the XML data we wanted to transfer into a local temp file.

  2. Build an Expect script in memory by doing some variable interpolation in a PHP string.

  3. Write the resulting Expect script to another local temp file.

  4. Exec() the Expect script, capturing the process return code and anything the script wrote to stdout or stderr.

  5. Write the process output to our application log for later troubleshooting if anything went wrong.

  6. Finally, if the process return code was zero (indicating success), delete the two temp files. Otherwise, send an email to our notification alias so I could investigate.

Here’s the meat of the Expect script generation in all its glory:

$expectContents =
 '#!/usr/bin/expect' . "\n"
 . 'set timeout ' . $this->m_timeout . "\n"
 . 'spawn sftp -o Port=' . $this->m_port . ' ' . $this->m_user . '@' . $this->m_host . "\n"
 . 'expect {' . "\n"
 . ' default {exit 1}' . "\n"
 . ' "Connecting to ' . $this->m_host . '..."' . "\n"
 . '}' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 2}' . "\n"
 . ' "continue connecting (yes/no)?" {send "yes\n"; exp_continue}' . "\n"
 . ' "ssword"' . "\n"
 . '}' . "\n"
 . 'send "' . $this->m_pass . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 3}' . "\n"
 . ' "ermission denied" {exit 4}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "cd /inbound\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 5}' . "\n"
 . ' "not found" {exit 6}' . "\n"
 . ' "No such file" {exit 7}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "put ' . $filename . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 8}' . "\n"
 . ' "not found" {exit 9}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "quit\n"' . "\n"
 . 'exit 0' . "\n";

Yes, I’m aware that there are more elegant ways to interpolate strings in PHP than this, but at the time I was still fairly new to PHP and under a ton of pressure to get something — anything — working.

Coming from a background in statically typed compiled languages, this just felt wrong somehow. Even my stints in the land of Perl didn’t feel this hacky. It was an ugly, nasty, weird abomination…but it worked.

And it kept working, without fail, for the entire lifespan of this system. Over time, we encountered bugs in many other areas, as is inevitable with any complex system, but this little corner of the codebase never had a single issue.

What this taught me is that sometimes you should go slowly, be methodical, and take the time necessary to create a simple, elegant solution to your problem.

And sometimes you just need to break out the duct tape.