Playwright Bridge Example | nut.js Examples

This example demonstrates using @nut-tree/playwright-bridge to automate a browser visually. The bridge redirects nut.js operations into the browser viewport, enabling image-based automation within web pages.

Setup

typescript

import { chromium, selectors } from "playwright";
import { locateByPosition, useContext } from "@nut-tree/playwright-bridge";
import {
    centerOf,
    ConsoleLogLevel,
    imageResource,
    keyboard,
    mouse,
    screen,
    sleep,
    useConsoleLogger,
    useDefaultMouseProvider,
    useDefaultWindowProvider,
    windowWithTitle,
} from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();

// Register position selector before creating any pages
await selectors.register("atPosition", locateByPosition);

Important: Custom selectors like atPosition must be registered with selectors.register() before creating any Page instances. Playwright locks the selector registry once a page is created, so registering after that point will fail.

typescript

useConsoleLogger({ logLevel: ConsoleLogLevel.DEBUG });

const browser = await chromium.launch({ headless: false });
const ctx = await browser.newContext();

screen.config.resourceDirectory = "./images";
keyboard.config.autoDelayMs = 10;
mouse.config.mouseSpeed = 3000;
screen.config.autoHighlight = true;

// Redirect nut.js operations to the browser context
useContext({ context: ctx });

Finding and Clicking Elements

Use screen.find() and screen.waitFor() to locate elements by their appearance within the viewport:

typescript

const page = await ctx.newPage();
await page.goto("https://example.com");

// Find an element visually with match validation
const target = await centerOf(
    screen.find(imageResource("target.png"), {
        confidence: 0.9,
        providerData: { validateMatches: true },
    })
);

// Use the position to click via Playwright's locator
await page.locator(`atPosition=${target}`).click();

The atPosition selector converts a nut.js Point into a Playwright locator. Point.toString() outputs coordinates in the expected format, so you can interpolate it directly.

Interacting with Located Elements

Once you have a Playwright locator, you can use standard Playwright methods:

typescript

const searchBox = await screen.find(imageResource("search.png"), {
    confidence: 0.79,
    providerData: { validateMatches: true },
});
const searchBoxPosition = await centerOf(searchBox);

// Get a Playwright locator at the found position
const box = await page.locator(`atPosition=${searchBoxPosition}`);
await box.click();
await box.fill("nut.js");
await box.press("Enter");

Waiting for Visual Changes

Wait for elements to appear after an action:

typescript

const result = await screen.waitFor(
    imageResource("result.png"),
    7000,
    1000,
    {
        confidence: 0.9,
        providerData: { validateMatches: true },
    }
);

Capturing Screenshots of Matched Regions

Combine a nut.js match region with Playwright's screenshot API:

typescript

const result = await screen.waitFor(imageResource("result.png"), 7000, 1000);

// Use the nut.js Region to clip a Playwright screenshot
await page.screenshot({
    path: "result.png",
    clip: {
        x: result.left,
        y: result.top,
        width: result.width,
        height: result.height,
    },
});

Switching Between Browser and Desktop

A powerful pattern is switching between the browser context and the desktop. This lets you manipulate the actual browser window (move, resize) and then continue with in-browser automation:

typescript

// Start in browser context
useContext({ context: ctx });
const page = await ctx.newPage();
await page.goto("https://example.com");

const pageTitle = await page.title();

// Switch to desktop providers to manipulate the browser window
useDefaultWindowProvider();
useDefaultMouseProvider();

const browserWindow = await screen.find(windowWithTitle(pageTitle));
await browserWindow.move({ x: 100, y: 500 });
await browserWindow.resize({ width: 1000, height: 800 });

// Switch back to browser context for in-viewport operations
useContext({ context: ctx });

const button = await screen.find(imageResource("button.png"));
// ...continue with in-browser automation

Using with @playwright/test

For test suites, use playwrightMatchers with expect:

typescript

import { test, expect } from "@playwright/test";
import { screen, imageResource } from "@nut-tree/nut-js";
import { useNlMatcher } from "@nut-tree/nl-matcher";

useNlMatcher();
import { playwrightMatchers, usePage } from "@nut-tree/playwright-bridge";

expect.extend(playwrightMatchers);

test("should display the welcome screen", async ({ page }) => {
    usePage({ page });
    await page.goto("https://example.com");

    await expect(screen).toShow(imageResource("welcome-dialog.png"));
    await expect(screen).toWaitFor(imageResource("loaded.png"), 5000, 500);
});

Cleanup

Always close the context and browser when done:

typescript

await ctx.close();
await browser.close();

Key Points

useContext() redirects all nut.js operations to a Playwright browser context
usePage() targets a specific page (useful in @playwright/test with trackPageChanges)
locateByPosition must be registered with selectors.register() before creating pages
useDefaultWindowProvider() / useDefaultMouseProvider() switch back to desktop for window manipulation
useNlMatcher() must be called to activate the image matching provider
Use providerData: { validateMatches: true } for higher accuracy at the cost of speed

For complete API documentation, see the Playwright Bridge docs.