Legacy Application Automation | nut.js Examples

Legacy applications—those Win32 apps, Java Swing UIs, and enterprise systems built decades ago—often lack modern automation APIs. They're the backbone of many organizations but notoriously difficult to test. nut.js provides the tools to automate these applications reliably.

The Legacy Challenge

Legacy applications typically have these automation obstacles:

Challenge	Impact	nut.js Solution
No DOM or web technologies	Can't use Playwright/Selenium	Image and OCR search
Limited accessibility APIs	Element inspection unreliable	Visual automation
Custom UI frameworks	Standard locators don't work	Template matching
Non-standard controls	No consistent identifiers	Color and region-based detection

Strategy Overview

The key to legacy app automation is visual reliability:

Launch and detect - Start the app and verify it's ready
Navigate visually - Use image matching to find UI elements
Interact precisely - Click, type, and wait for responses
Verify results - Check screen content for expected outcomes

Basic Setup

typescript

import {
  screen,
  keyboard,
  mouse,
  Key,
  Button,
  Region,
  sleep,
  centerOf,
  straightTo,
  imageResource,
  getWindows,
  getActiveWindow,
} from "@nut-tree/nut-js";
import { exec } from "child_process";
import { promisify } from "util";

const execAsync = promisify(exec);

// Configure for legacy app speed
keyboard.config.autoDelayMs = 50;
mouse.config.autoDelayMs = 50;
mouse.config.mouseSpeed = 1000;

// Set resource directory for reference images
screen.config.resourceDirectory = "./reference-images";

// Lower confidence for anti-aliased legacy UIs
screen.config.confidence = 0.9;

Launching Legacy Applications

Windows Applications

typescript

async function launchWindowsApp(exePath: string, windowTitle: string): Promise<void> {
  // Launch the application
  await execAsync(`start "" "${exePath}"`);

  // Wait for window to appear
  const maxAttempts = 30;
  for (let i = 0; i < maxAttempts; i++) {
    await sleep(1000);

    const windows = await getWindows();
    for (const win of windows) {
      const title = await win.title;
      if (title.includes(windowTitle)) {
        // Focus the window
        await win.focus();
        await sleep(500);
        return;
      }
    }
  }

  throw new Error(`Application window "${windowTitle}" not found after ${maxAttempts} seconds`);
}

// Usage
await launchWindowsApp("C:\\Program Files\\LegacyApp\\app.exe", "Legacy Application");

macOS Applications

typescript

async function launchMacApp(appPath: string, windowTitle: string): Promise<void> {
  await execAsync(`open "${appPath}"`);

  const maxAttempts = 30;
  for (let i = 0; i < maxAttempts; i++) {
    await sleep(1000);

    const windows = await getWindows();
    for (const win of windows) {
      const title = await win.title;
      if (title.includes(windowTitle)) {
        await win.focus();
        await sleep(500);
        return;
      }
    }
  }

  throw new Error(`Application window "${windowTitle}" not found`);
}

// Usage
await launchMacApp("/Applications/LegacyApp.app", "Legacy Application");

Java Applications

typescript

async function launchJavaApp(jarPath: string, mainClass: string, windowTitle: string): Promise<void> {
  // Launch Java app with specific JVM options
  const javaCmd = `java -jar "${jarPath}"`;
  // For main class: java -cp "${jarPath}" ${mainClass}

  exec(javaCmd); // Don't await - it blocks

  const maxAttempts = 60; // Java apps can be slow to start
  for (let i = 0; i < maxAttempts; i++) {
    await sleep(1000);

    const windows = await getWindows();
    for (const win of windows) {
      const title = await win.title;
      if (title.includes(windowTitle)) {
        await win.focus();
        await sleep(1000); // Extra time for Java UI
        return;
      }
    }
  }

  throw new Error(`Java application window not found`);
}

Finding and Clicking UI Elements

Image-Based Button Clicks

typescript

async function clickButton(buttonImageName: string, timeout: number = 10000): Promise<void> {
  const button = await screen.waitFor(imageResource(buttonImageName), timeout);
  await mouse.move(straightTo(centerOf(button)));
  await mouse.click(Button.LEFT);
  await sleep(300); // Wait for UI response
}

// Usage
await clickButton("login-button.png");
await clickButton("submit-form.png");
await clickButton("menu-file.png");

Region-Limited Searches

For faster, more reliable searches:

typescript

async function clickButtonInRegion(
  buttonImage: string,
  region: Region
): Promise<void> {
  const button = await screen.find(imageResource(buttonImage), {
    searchRegion: region,
  });
  await mouse.move(straightTo(centerOf(button)));
  await mouse.click(Button.LEFT);
}

// Define known regions
const TOOLBAR_REGION = new Region(0, 0, 1920, 80);
const SIDEBAR_REGION = new Region(0, 80, 250, 800);
const MAIN_CONTENT_REGION = new Region(250, 80, 1670, 800);

// Search only in toolbar
await clickButtonInRegion("save-icon.png", TOOLBAR_REGION);

Handling Multiple Similar Elements

typescript

async function clickNthMatch(buttonImage: string, index: number): Promise<void> {
  const matches = await screen.findAll(imageResource(buttonImage));

  if (matches.length <= index) {
    throw new Error(`Only found ${matches.length} matches, expected at least ${index + 1}`);
  }

  // Sort by position (top-to-bottom, left-to-right)
  matches.sort((a, b) => {
    if (Math.abs(a.top - b.top) < 10) {
      return a.left - b.left;
    }
    return a.top - b.top;
  });

  await mouse.move(straightTo(centerOf(matches[index])));
  await mouse.click(Button.LEFT);
}

// Click the third checkbox
await clickNthMatch("checkbox-unchecked.png", 2);

Text Input in Legacy Forms

Simple Text Fields

typescript

async function fillTextField(fieldImage: string, value: string): Promise<void> {
  // Find and click the field
  const field = await screen.find(imageResource(fieldImage));
  await mouse.move(straightTo(centerOf(field)));
  await mouse.click(Button.LEFT);
  await sleep(100);

  // Clear existing content
  await keyboard.pressKey(Key.LeftControl, Key.A);
  await keyboard.releaseKey(Key.LeftControl, Key.A);
  await sleep(50);

  // Type new value
  await keyboard.type(value);
}

await fillTextField("username-field.png", "admin");
await fillTextField("password-field.png", "secretpassword");

Many legacy forms use Tab key navigation:

typescript

interface FormField {
  value: string;
  isPassword?: boolean;
}

async function fillFormWithTabs(startFieldImage: string, fields: FormField[]): Promise<void> {
  // Click the first field
  const firstField = await screen.find(imageResource(startFieldImage));
  await mouse.move(straightTo(centerOf(firstField)));
  await mouse.click(Button.LEFT);
  await sleep(100);

  for (const field of fields) {
    // Clear and type
    await keyboard.pressKey(Key.LeftControl, Key.A);
    await keyboard.releaseKey(Key.LeftControl, Key.A);
    await keyboard.type(field.value);

    // Tab to next field
    await keyboard.pressKey(Key.Tab);
    await keyboard.releaseKey(Key.Tab);
    await sleep(100);
  }
}

// Fill a registration form
await fillFormWithTabs("first-name-field.png", [
  { value: "John" },
  { value: "Doe" },
  { value: "john.doe@example.com" },
  { value: "555-123-4567" },
]);

Handling Special Characters

typescript

async function typeWithShift(text: string): Promise<void> {
  for (const char of text) {
    if (char === char.toUpperCase() && char !== char.toLowerCase()) {
      // Uppercase letter
      await keyboard.pressKey(Key.LeftShift);
      await keyboard.type(char.toLowerCase());
      await keyboard.releaseKey(Key.LeftShift);
    } else if ("!@#$%^&*()_+{}|:\"<>?".includes(char)) {
      // Shift symbols
      await keyboard.pressKey(Key.LeftShift);
      await keyboard.type(char);
      await keyboard.releaseKey(Key.LeftShift);
    } else {
      await keyboard.type(char);
    }
    await sleep(20);
  }
}

Handling Menus and Dropdowns

Cascading Menus

typescript

async function navigateMenu(menuPath: string[]): Promise<void> {
  for (let i = 0; i < menuPath.length; i++) {
    const menuItem = await screen.waitFor(imageResource(menuPath[i]), 5000);
    await mouse.move(straightTo(centerOf(menuItem)));

    if (i === menuPath.length - 1) {
      // Last item - click it
      await mouse.click(Button.LEFT);
    } else {
      // Intermediate item - hover to open submenu
      await sleep(500);
    }
  }
}

// Navigate: File > Export > PDF
await navigateMenu([
  "menu-file.png",
  "menu-export.png",
  "menu-export-pdf.png",
]);

typescript

async function selectDropdownOption(
  dropdownImage: string,
  optionImage: string
): Promise<void> {
  // Click dropdown to open
  const dropdown = await screen.find(imageResource(dropdownImage));
  await mouse.move(straightTo(centerOf(dropdown)));
  await mouse.click(Button.LEFT);
  await sleep(500);

  // Find and click option
  const option = await screen.waitFor(imageResource(optionImage), 5000);
  await mouse.move(straightTo(centerOf(option)));
  await mouse.click(Button.LEFT);
  await sleep(300);
}

await selectDropdownOption("country-dropdown.png", "country-usa.png");

For dropdowns that respond to keyboard:

typescript

async function selectDropdownByTyping(
  dropdownImage: string,
  searchText: string
): Promise<void> {
  const dropdown = await screen.find(imageResource(dropdownImage));
  await mouse.move(straightTo(centerOf(dropdown)));
  await mouse.click(Button.LEFT);
  await sleep(300);

  // Type to filter/select
  await keyboard.type(searchText);
  await sleep(200);
  await keyboard.pressKey(Key.Enter);
  await keyboard.releaseKey(Key.Enter);
}

// Type "Cal" to select "California" from state dropdown
await selectDropdownByTyping("state-dropdown.png", "Cal");

Handling Dialogs

typescript

async function waitForDialog(dialogImage: string, timeout: number = 10000): Promise<Region> {
  return await screen.waitFor(imageResource(dialogImage), timeout);
}

async function dismissDialog(okButtonImage: string): Promise<void> {
  const okButton = await screen.find(imageResource(okButtonImage));
  await mouse.move(straightTo(centerOf(okButton)));
  await mouse.click(Button.LEFT);
  await sleep(500);
}

// Wait for confirmation dialog and click OK
await waitForDialog("confirmation-dialog.png");
await dismissDialog("dialog-ok-button.png");

System Dialogs (Open/Save)

typescript

async function handleSaveDialog(filename: string, folder?: string): Promise<void> {
  await sleep(1000); // Wait for dialog

  if (folder) {
    // Navigate to folder using keyboard shortcut
    if (process.platform === 'darwin') {
      await keyboard.pressKey(Key.LeftSuper, Key.LeftShift, Key.G);
      await keyboard.releaseKey(Key.LeftSuper, Key.LeftShift, Key.G);
    } else {
      // Windows: Click in address bar
      await keyboard.pressKey(Key.LeftAlt, Key.D);
      await keyboard.releaseKey(Key.LeftAlt, Key.D);
    }
    await sleep(300);
    await keyboard.type(folder);
    await keyboard.pressKey(Key.Enter);
    await keyboard.releaseKey(Key.Enter);
    await sleep(500);
  }

  // Type filename
  await keyboard.type(filename);
  await sleep(200);

  // Save
  await keyboard.pressKey(Key.Enter);
  await keyboard.releaseKey(Key.Enter);
}

async function handleOpenDialog(filePath: string): Promise<void> {
  await sleep(1000);

  // Use Go to folder on macOS or address bar on Windows
  if (process.platform === 'darwin') {
    await keyboard.pressKey(Key.LeftSuper, Key.LeftShift, Key.G);
    await keyboard.releaseKey(Key.LeftSuper, Key.LeftShift, Key.G);
    await sleep(300);
  } else {
    await keyboard.pressKey(Key.LeftAlt, Key.D);
    await keyboard.releaseKey(Key.LeftAlt, Key.D);
    await sleep(300);
  }

  await keyboard.type(filePath);
  await keyboard.pressKey(Key.Enter);
  await keyboard.releaseKey(Key.Enter);
}

OCR for Dynamic Content

When text changes but you need to find it:

typescript

import { screen, mouse, straightTo, centerOf, Button, Region, textLine } from "@nut-tree/nut-js";
import { configure, Language, LanguageModelType, preloadLanguages } from "@nut-tree/plugin-ocr";

configure({
  dataPath: "./ocr-data",
  languageModelType: LanguageModelType.BEST
});

await preloadLanguages([Language.English]);

async function clickTextButton(buttonText: string): Promise<void> {
  const button = await screen.find(textLine(buttonText), {
    providerData: {
      lang: [Language.English],
      partialMatch: true,
      caseSensitive: false
    }
  });
  await mouse.move(straightTo(centerOf(button)));
  await mouse.click(Button.LEFT);
}

async function verifyScreenContains(expectedText: string): Promise<boolean> {
  try {
    await screen.find(textLine(expectedText), {
      providerData: {
        lang: [Language.English],
        partialMatch: true,
        caseSensitive: false
      }
    });
    return true;
  } catch {
    return false;
  }
}

async function readFieldValue(fieldRegion: Region): Promise<string> {
  return await screen.read(fieldRegion);
}

// Usage
await clickTextButton("Submit Order");
const hasConfirmation = await verifyScreenContains("Order Confirmed");
const orderNumber = await readFieldValue(new Region(400, 200, 150, 30));

Verification and Assertions

Visual Verification

typescript

async function assertImageVisible(
  imageName: string,
  message: string = "Image not found"
): Promise<void> {
  try {
    await screen.waitFor(imageResource(imageName), 5000);
  } catch {
    // Capture screenshot for debugging
    await screen.capture(`./failures/assertion-${Date.now()}.png`);
    throw new Error(message);
  }
}

async function assertImageNotVisible(
  imageName: string,
  message: string = "Image should not be visible"
): Promise<void> {
  try {
    await screen.find(imageResource(imageName));
    await screen.capture(`./failures/assertion-${Date.now()}.png`);
    throw new Error(message);
  } catch (error) {
    if (error.message === message) throw error;
    // Expected: image not found
  }
}

// Assertions
await assertImageVisible("success-icon.png", "Success icon should appear after save");
await assertImageNotVisible("error-dialog.png", "No error should appear");

Text Verification

typescript

async function assertTextInRegion(
  region: Region,
  expectedText: string
): Promise<void> {
  const actualText = await screen.read(region);

  if (!actualText.includes(expectedText)) {
    await screen.capture(`./failures/text-assertion-${Date.now()}.png`);
    throw new Error(`Expected "${expectedText}" but found "${actualText}"`);
  }
}

// Verify status bar shows "Ready"
const STATUS_BAR = new Region(0, 780, 400, 20);
await assertTextInRegion(STATUS_BAR, "Ready");

Complete Example: Legacy CRM Automation

typescript

import {
  screen,
  keyboard,
  mouse,
  Key,
  Button,
  Region,
  sleep,
  centerOf,
  straightTo,
  imageResource,
  getWindows,
} from "@nut-tree/nut-js";
import { exec } from "child_process";

// Configuration
screen.config.resourceDirectory = "./crm-images";
screen.config.confidence = 0.9;
keyboard.config.autoDelayMs = 50;

class LegacyCRMAutomation {
  private appTitle = "Acme CRM v3.2";

  async launch(): Promise<void> {
    exec('start "" "C:\\Program Files\\AcmeCRM\\crm.exe"');

    for (let i = 0; i < 60; i++) {
      await sleep(1000);
      const windows = await getWindows();
      for (const win of windows) {
        if ((await win.title).includes(this.appTitle)) {
          await win.focus();
          await sleep(1000);
          return;
        }
      }
    }
    throw new Error("CRM failed to launch");
  }

  async login(username: string, password: string): Promise<void> {
    // Wait for login dialog
    await screen.waitFor(imageResource("login-dialog.png"), 10000);

    // Fill credentials
    const usernameField = await screen.find(imageResource("username-field.png"));
    await mouse.move(straightTo(centerOf(usernameField)));
    await mouse.click(Button.LEFT);
    await keyboard.type(username);

    await keyboard.pressKey(Key.Tab);
    await keyboard.releaseKey(Key.Tab);
    await keyboard.type(password);

    // Click login
    await this.clickButton("login-button.png");

    // Wait for main window
    await screen.waitFor(imageResource("main-dashboard.png"), 30000);
  }

  async searchCustomer(searchTerm: string): Promise<void> {
    // Navigate to customers
    await this.navigateMenu(["menu-customers.png", "menu-search-customer.png"]);

    // Fill search
    await screen.waitFor(imageResource("search-dialog.png"), 5000);
    await keyboard.type(searchTerm);
    await this.clickButton("search-button.png");

    // Wait for results
    await sleep(2000);
  }

  async createNewCustomer(customer: {
    firstName: string;
    lastName: string;
    email: string;
    phone: string;
  }): Promise<string> {
    // Open new customer form
    await this.navigateMenu(["menu-customers.png", "menu-new-customer.png"]);
    await screen.waitFor(imageResource("customer-form.png"), 5000);

    // Fill form fields using Tab navigation
    const fields = [
      customer.firstName,
      customer.lastName,
      customer.email,
      customer.phone,
    ];

    const firstField = await screen.find(imageResource("first-name-field.png"));
    await mouse.move(straightTo(centerOf(firstField)));
    await mouse.click(Button.LEFT);

    for (const value of fields) {
      await keyboard.type(value);
      await keyboard.pressKey(Key.Tab);
      await keyboard.releaseKey(Key.Tab);
      await sleep(100);
    }

    // Save
    await this.clickButton("save-button.png");

    // Wait for confirmation and extract customer ID
    await screen.waitFor(imageResource("customer-created.png"), 10000);

    // Read the customer ID from a known region
    const idRegion = new Region(300, 200, 100, 25);
    const customerId = await screen.read(idRegion);

    // Close dialog
    await this.clickButton("close-dialog.png");

    return customerId.trim();
  }

  private async clickButton(buttonImage: string): Promise<void> {
    const button = await screen.find(imageResource(buttonImage));
    await mouse.move(straightTo(centerOf(button)));
    await mouse.click(Button.LEFT);
    await sleep(300);
  }

  private async navigateMenu(menuPath: string[]): Promise<void> {
    for (let i = 0; i < menuPath.length; i++) {
      const item = await screen.waitFor(imageResource(menuPath[i]), 5000);
      await mouse.move(straightTo(centerOf(item)));

      if (i === menuPath.length - 1) {
        await mouse.click(Button.LEFT);
      } else {
        await sleep(400);
      }
    }
  }

  async close(): Promise<void> {
    await keyboard.pressKey(Key.LeftAlt, Key.F4);
    await keyboard.releaseKey(Key.LeftAlt, Key.F4);
    await sleep(500);

    // Handle "Save changes?" dialog if it appears
    try {
      await screen.waitFor(imageResource("save-changes-dialog.png"), 2000);
      await this.clickButton("dont-save-button.png");
    } catch {
      // No dialog, app closed
    }
  }
}

// Test usage
async function runTest() {
  const crm = new LegacyCRMAutomation();

  try {
    await crm.launch();
    await crm.login("testuser", "testpass123");

    const customerId = await crm.createNewCustomer({
      firstName: "Jane",
      lastName: "Smith",
      email: "jane.smith@example.com",
      phone: "555-987-6543",
    });

    console.log(`Created customer with ID: ${customerId}`);

    await crm.searchCustomer("jane.smith@example.com");

    // Verify customer appears in search results
    await screen.waitFor(imageResource("customer-row.png"), 5000);

  } finally {
    await crm.close();
  }
}

Best Practices

Image Reference Management

text

reference-images/
├── dialogs/
│   ├── login-dialog.png
│   ├── save-dialog.png
│   └── error-dialog.png
├── buttons/
│   ├── submit-button.png
│   ├── cancel-button.png
│   └── ok-button.png
├── fields/
│   ├── username-field.png
│   └── password-field.png
└── states/
    ├── loading-spinner.png
    └── success-checkmark.png

Capture Images at Test Resolution

Always capture reference images at the same resolution and scale you'll run tests at.

Handle Timing Variability

Legacy apps can be slow. Build in appropriate waits:

typescript

// Good: Wait for element to appear
await screen.waitFor(imageResource("result.png"), 30000);

// Avoid: Fixed sleep without verification
await sleep(5000);
await clickButton("result.png"); // May fail if app is slow

Isolate Test State

Reset application state between tests:

typescript

afterEach(async () => {
  // Close any open dialogs
  await keyboard.pressKey(Key.Escape);
  await keyboard.releaseKey(Key.Escape);
  await sleep(300);

  // Return to home screen
  await keyboard.pressKey(Key.LeftControl, Key.Home);
  await keyboard.releaseKey(Key.LeftControl, Key.Home);
});

The Legacy Challenge

Strategy Overview

Basic Setup

Launching Legacy Applications

Windows Applications

macOS Applications

Java Applications

Finding and Clicking UI Elements

Image-Based Button Clicks

Region-Limited Searches

Handling Multiple Similar Elements

Text Input in Legacy Forms

Simple Text Fields

Tab-Based Form Navigation

Handling Special Characters

Handling Menus and Dropdowns

Cascading Menus

Dropdown Selection

Keyboard-Based Dropdown

Handling Dialogs

Waiting for Modal Dialogs

System Dialogs (Open/Save)

OCR for Dynamic Content

Verification and Assertions

Visual Verification

Text Verification

Complete Example: Legacy CRM Automation

Best Practices

Image Reference Management

Capture Images at Test Resolution

Handle Timing Variability

Isolate Test State