Browser Integration

Playwright Bridge

Use nut.js image search and OCR capabilities inside Playwright-controlled browser viewports.

Overview

The @nut-tree/playwright-bridge plugin integrates nut.js with Playwright, redirecting image search and OCR operations to work within browser viewports instead of the entire desktop. This enables visual automation inside web pages.

Browser-Scoped Search

Image search within the viewport only

screen.find(imageResource("btn.png"))

Position Locator

Find elements by screen position

locateByPosition

Test Matchers

Custom assertions for @playwright/test

expect(screen).toShow(img)

Installation

typescript

npm install @nut-tree/playwright-bridge

Subscription Required

This package is included in Solo, Team, OCR, and nl-matcher subscription plans.

Quick Reference

useContext

useContext({ context: BrowserContext })

void

Redirect nut.js operations to a Playwright browser context

usePage

usePage({ page: Page, trackPageChanges?: boolean })

void

Redirect nut.js operations to a specific page. Enable trackPageChanges for automatic context updates on navigation.

locateByPosition

locateByPosition

SelectorEngine

Selector engine for finding elements by screen position. Must be registered with Playwright's selectors.register().

Basic Usage

Using a Browser Context

Pass a Playwright BrowserContext to redirect all nut.js operations to the browser:

typescript

import { chromium } from "playwright";
import { screen, mouse, Button, imageResource, centerOf, straightTo } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";
import { useContext } from "@nut-tree/playwright-bridge";

useNlMatcher();

const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

// Redirect nut.js to the browser context
useContext({ context });

await page.goto("https://example.com");

// screen.find() now searches within the browser viewport
const button = await screen.find(imageResource("login-button.png"));
await mouse.move(straightTo(centerOf(button)));
await mouse.click(Button.LEFT);

Using a Specific Page

For multi-page scenarios, target a specific page:

typescript

import { chromium } from "playwright";
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";
import { usePage } from "@nut-tree/playwright-bridge";

useNlMatcher();

const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page1 = await context.newPage();
const page2 = await context.newPage();

// Direct nut.js operations to page1
usePage({ page: page1 });
await page1.goto("https://example.com/dashboard");

// Image search is scoped to page1's viewport
const chart = await screen.find(imageResource("sales-chart.png"));

// Switch to page2
usePage({ page: page2 });
await page2.goto("https://example.com/reports");

// Now searches happen in page2
const report = await screen.find(imageResource("report-header.png"));

Automatic Page Tracking

Enable trackPageChanges to automatically update the nut.js context when navigation occurs, eliminating manual context updates:

typescript

import { chromium } from "playwright";
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";
import { usePage } from "@nut-tree/playwright-bridge";

useNlMatcher();

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

// Enable automatic context updates
usePage({ page, trackPageChanges: true });

await page.goto("https://example.com");
// Context is automatically updated

await page.click('a[href="/dashboard"]');
// Context updates automatically after navigation

// No need to call usePage() again
const dashboard = await screen.find(imageResource("dashboard-header.png"));

Position-Based Locators

The bridge provides a position-based locator for finding Playwright elements by their screen coordinates. This is useful when you need to convert a visual match back to a Playwright element.

Registration

typescript

import { chromium, selectors } from "playwright";
import { locateByPosition } from "@nut-tree/playwright-bridge";

// Register before any page creation
await selectors.register("atPosition", locateByPosition);

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();

Registration Order

Custom selectors like atPosition must be registered with selectors.register() before creating any Page instances. Playwright locks the selector registry once a page is created, so registering after that point will fail.

Finding Elements by Position

Combine screen.find() with the position selector:

typescript

import { screen, imageResource, centerOf } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";
import { useContext, locateByPosition } from "@nut-tree/playwright-bridge";

useNlMatcher();

// Find element visually
const buttonRegion = await screen.find(imageResource("submit-button.png"));

// Get the center position
const buttonPosition = await centerOf(buttonRegion);

// Get the Playwright element at that position
// Point's toString() outputs the coordinates in the expected format
const buttonElement = await page.locator(`atPosition=${buttonPosition}`);

// Use standard Playwright methods
await buttonElement.click();
const text = await buttonElement.textContent();

Playwright Test Matchers

The bridge provides custom matchers for @playwright/test that integrate nut.js assertions into your test suite:

Setup

typescript

import { test, expect } from "@playwright/test";
import { playwrightMatchers } from "@nut-tree/playwright-bridge";

// Extend expect with nut.js matchers
expect.extend(playwrightMatchers);

Matcher Examples

toBeAt — Verify the mouse is at an exact position:

typescript

import { mouse, Point } from "@nut-tree/nut-js";

await expect(mouse).toBeAt(new Point(100, 100));
await expect(mouse).not.toBeAt(new Point(10, 10));

toBeIn — Verify the mouse is within a region:

typescript

import { mouse, Region } from "@nut-tree/nut-js";

await expect(mouse).toBeIn(new Region(10, 25, 70, 30));
await expect(mouse).not.toBeIn(new Region(0, 0, 10, 10));

toShow — Verify content is visible on the page:

typescript

import { screen, imageResource, textLine, pixelWithColor, windowWithTitle, RGBA } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Verify image is shown
await expect(screen).toShow(imageResource("login-button.png"));

// Verify text is shown (requires OCR plugin)
await expect(screen).toShow(textLine("Welcome"));

// Verify color is shown
await expect(screen).toShow(pixelWithColor(new RGBA(255, 0, 0, 255)));

// Verify window title (useful for hybrid testing)
await expect(screen).not.toShow(windowWithTitle(/Error/));

// With options
await expect(screen).toShow(imageResource("button.png"), {
    confidence: 0.9,
    providerData: { validateMatches: true }
});

toWaitFor — Wait for content to appear (default timeout: 5000ms, interval: 500ms):

typescript

import { screen, imageResource, textLine, pixelWithColor, RGBA } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Wait for image with custom timeout and interval
await expect(screen).toWaitFor(imageResource("loading-complete.png"), 6000, 300);

// Wait for text with OCR options
await expect(screen).toWaitFor(textLine("Ready"), 7000, 500, {
    providerData: { preprocessConfig: { binarize: false } }
});

// Wait for color
await expect(screen).toWaitFor(pixelWithColor(new RGBA(0, 255, 0, 255)));

// Wait for text matching regex pattern
await expect(screen).toWaitFor(textLine(/^Status: OK$/), 10000, 500, {
    confidence: 0.8
});

toHaveColor — Verify the color at a specific point:

typescript

import { Point, pixelWithColor, RGBA } from "@nut-tree/nut-js";

const point = new Point(100, 100);
await expect(point).toHaveColor(pixelWithColor(new RGBA(255, 0, 0, 255)));
await expect(point).not.toHaveColor(pixelWithColor(new RGBA(0, 0, 255, 255)));

Matcher Reference

toBeAt

expect(mouse).toBeAt(point: Point)

Promise<void>

Verify the mouse is at an exact position

toBeIn

expect(mouse).toBeIn(region: Region)

Promise<void>

Verify the mouse is within a region

toShow

expect(screen).toShow(needle, options?)

Promise<void>

Verify content is visible. Needle can be imageResource(), textLine(), pixelWithColor(), or windowWithTitle()

toWaitFor

expect(screen).toWaitFor(needle, timeout?, interval?, options?)

Promise<void>

Wait for content to appear. Default timeout: 5000ms, default interval: 500ms

toHaveColor

expect(point: Point).toHaveColor(pixelWithColor(rgba))

Promise<void>

Verify the color at a specific point

Full Test Example

typescript

import { useNlMatcher } from "@nut-tree/nl-matcher";
import {
    Button,
    Key,
    centerOf,
    imageResource,
    keyboard,
    mouse,
    screen,
    sleep,
    textLine,
} from "@nut-tree/nut-js";

useNlMatcher();
import { expect, test } from "@playwright/test";
import { playwrightMatchers, usePage } from "@nut-tree/playwright-bridge";

expect.extend(playwrightMatchers);

test("create new program", async ({ page }) => {
    usePage({ page });

    // Find and click "New" button
    const newButton = await screen.waitFor(
        imageResource("new_button.png"),
        7000,
        500
    );
    await mouse.setPosition(await centerOf(newButton));
    await mouse.click(Button.LEFT);

    // Find and click action button with validation
    const newActionButton = await screen.waitFor(
        imageResource("new_action.png"),
        7000,
        500,
        { providerData: { validateMatches: true } }
    );
    await mouse.setPosition(await centerOf(newActionButton));
    await mouse.click(Button.LEFT);

    await sleep(300);
    await keyboard.type("my_new_test");
    await keyboard.type(Key.Enter);

    // Wait for success text with regex pattern
    await expect(screen).toWaitFor(
        textLine(/^my_new_test . Ready$/),
        10000,
        500,
        { confidence: 0.8 }
    );

    // Verify no error is shown
    await expect(screen).not.toShow(imageResource("error.png"), {
        providerData: { validateMatches: true },
    });
});

Using with OCR

Combine the bridge with the OCR plugin to find and read text on the page:

typescript

import { chromium } from "playwright";
import { screen, mouse, centerOf, straightTo, Button, singleWord, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";
import { usePage } from "@nut-tree/playwright-bridge";
import { useOcrPlugin, configure, LanguageModelType } from "@nut-tree/plugin-ocr";

useNlMatcher();
useOcrPlugin();

// Configure OCR
configure({
  dataPath: "./ocr-data",
  languageModelType: LanguageModelType.BEST
});

const browser = await chromium.launch({ headless: false });
const page = await browser.newPage();
usePage({ page });

await page.goto("https://example.com/products");

// Find element by its text content
const addToCart = await screen.find(singleWord("Add"));
await mouse.move(straightTo(centerOf(addToCart)));
await mouse.click(Button.LEFT);

// Read text from a specific region
const searchRegion = await screen.find(imageResource("price-area.png"));
const text = await screen.read({ searchRegion });
console.log("Text:", text);

Advanced Configuration

Fine-tune nut.js behavior for your Playwright tests:

typescript

import { chromium, selectors } from "playwright";
import { locateByPosition, useContext } from "@nut-tree/playwright-bridge";
import {
    centerOf,
    ConsoleLogLevel,
    imageResource,
    keyboard,
    mouse,
    screen,
    useConsoleLogger
} from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Register position selector before creating pages
await selectors.register("atPosition", locateByPosition);

// Enable debug logging
useConsoleLogger({ logLevel: ConsoleLogLevel.DEBUG });

const runHeadless = false;
const browser = await chromium.launch({ headless: runHeadless });
const context = await browser.newContext();

// Use different reference images for headless vs headed mode
if (runHeadless) {
    screen.config.resourceDirectory = "./images/headless";
} else {
    screen.config.resourceDirectory = "./images/headed";
}

// Configure input speeds
keyboard.config.autoDelayMs = 10;     // Delay between keystrokes
mouse.config.mouseSpeed = 3000;       // Mouse movement speed (pixels/sec)

// Enable visual highlighting during development
screen.config.autoHighlight = true;

// Initialize the bridge
useContext({ context });

const page = await context.newPage();
await page.goto("https://example.com");

// Search with confidence and match validation
const button = await screen.find(imageResource("button.png"), {
    confidence: 0.9,
    providerData: { validateMatches: true }
});

const buttonPosition = await centerOf(button);
await page.locator(`atPosition=${buttonPosition}`).click();

await context.close();
await browser.close();

Configuration Options

screen.config.resourceDirectory — Base path for image files
screen.config.autoHighlight — Visually highlight found regions
screen.config.confidence — Default match confidence (0-1)
mouse.config.mouseSpeed — Mouse movement speed in pixels/sec
keyboard.config.autoDelayMs — Delay between keystrokes in ms

Best Practices

Image Preparation

Capture reference images from the actual browser viewport
Use consistent browser window sizes for reliable matching
Avoid images with dynamic content (timestamps, random IDs)

Headless Mode

Visual automation typically requires headless: false. For CI environments, use Xvfb on Linux or similar virtual display solutions.

When to Use the Bridge

Use the Playwright Bridge when you need to find elements by their visual appearance inside the browser—useful for canvas, WebGL, complex CSS, or when selectors aren't practical. For native dialogs and system UI, use nut.js directly alongside Playwright (see the Electron Testing example).

Playwright Bridge Example

Practical example of visual form testing with the Playwright Bridge

Bridge vs Hybrid Approach

Choose the right approach based on what you need to automate:

Aspect	Playwright Bridge	Hybrid (Side-by-Side)
Search scope	Browser viewport only	Entire desktop
Use case	In-browser visual testing	Native dialogs, system UI
Coordinates	Relative to page	Absolute screen position
Best for	Canvas, WebGL, complex CSS	File pickers, notifications