Lackey – GUI Automation with Image Recognition

In my line of work, I’ve found scripting to be an absolute lifesaver. I am far more efficient (and accurate) when I can have the computer do my manual grunt work for me. Some of this involves GUI automation.

I ran into an issue with one of our business applications, which is published via Citrix. The “traditional” kind of GUI automation using accessibility hooks doesn’t work for published applications (since they’re actually running on a remote Citrix server). I needed a workaround that could recognize windows and regions of the screen. My search led me to SikuliX.

I liked it right away because it used Python (well, Jython), which I had already been working with. The Sikuli IDE made it trivial to quickly write scripts and capture images, and I soon had a stockpile of scripts to update accesses, copy data from our live system into our test system, etc.

Eventually I had enough scripts that I wanted to run them via remote desktop on a different machine, so I could keep using my desktop computer for other tasks. However, because of our enterprise policies, the remote server doesn’t have Java installed. I searched for a way to run Sikuli scripts with pure Python, but found nothing at the time.

So, I decided to build it.

Lackey is the result of that endeavor.

Existing Sikuli scripts can be run with minimal modification just by importing the Sikuli shim at the top of the script:

from sikuli import *

Since Sikuli script overrides some Python native functions (like input() ), you can also just import Lackey and call the automation functions directly:

Lackey is currently in an alpha state. It’s built to be cross-platform compatible, but the only platform it currently runs on is Windows (only because that’s all I run it on). I’ll eventually expand it to run on Linux as well. Other “future features” to get up to speed with Sikuli will include OCR, event handling, and speed optimizations on the image search.

If you’d like to try it out, check out the repo for installation instructions. Lackey is published via pip, so it’s easy to set up and get started with. If you run into any bugs, file an issue on GitHub and I’ll get it fixed. If you’d like to contribute, feel free to submit a pull request.


3 thoughts on “Lackey – GUI Automation with Image Recognition”

  1. Thanks your great work, however, it seems to me that lackey can not control a window in background (e.g. with handle method). Can lackey go smoothly without interfering my activities? Or do you know some alternatives in Python can run in background?

    1. Hi Ronnie,

      Unfortunately you are correct: one of the limitations of Lackey is that it interacts with your applications the same way you do, via mouse and keyboard input, so you won’t be able to do things at the same time.

      You might try a library like pywinauto, that taps into the Windows UI Automation API. It might have more success automating background windows, but it’s limited in that not all applications support it (the reason I developed Lackey in the first place).

      If that fails, what I usually do in a production environment is set up the script on a machine I can RDP into. Note the RDP session has to stay active for the script to stay running.

      Hope this helps!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.