How would you design a scalable Selenium Grid architecture using Docker and Java to support high‑concurrency UI tests while ensuring test isolation and optimal resource utilization?

Question

How would you design a scalable Selenium Grid architecture using Docker and Java to support high‑concurrency UI tests while ensuring test isolation and optimal resource utilization?

Answer

Design a production‑grade Selenium Grid that can elastically handle hundreds of concurrent UI tests, guarantees test isolation, and integrates cleanly with CI/CD pipelines.

1. Architecture Overview

+-------------------+          +-------------------+
|   CI Runner (Jenkins/GitHub Actions)   |
+----------+--------+          +----------+--------+
           |                           |
           |  docker‑compose up        |
           v                           v
+-------------------+          +-------------------+
|   Selenium Hub    | <--->    |  Selenium Nodes   |
| (grid4-hub:4444)  |          | (chrome/ff:5555)  |
+-------------------+          +-------------------+
           ^                           ^
           |  Selenium RemoteWebDriver |
           +---------------------------+

• Hub: Central router (selenium/hub:4.x). Exposes the Grid API and the /status endpoint.
• Nodes: Stateless containers (selenium/node-chrome / selenium/node-firefox) that register themselves with the hub on start‑up.
• Orchestrator: Docker Compose for local/dev, Docker Swarm or Kubernetes for production auto‑scaling.

2. Docker‑Compose Definition (local/dev)

version: "3.8"

services:
  hub:
    image: selenium/hub:4.15.0
    container_name: selenium-hub
    ports:
      - "4444:4444"
    environment:
      - SE_EVENT_BUS_HOST=event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    networks:
      - selenium

  event-bus:
    image: selenium/event-bus:4.15.0
    container_name: selenium-event-bus
    ports:
      - "4442:4442"
      - "4443:4443"
    networks:
      - selenium

  chrome:
    image: selenium/node-chrome:4.15.0
    depends_on:
      - hub
      - event-bus
    environment:
      - SE_EVENT_BUS_HOST=event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=5          # limit per container
      - SE_NODE_SESSION_TIMEOUT=300
      - SE_NODE_MAX_INSTANCES=5
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: "4g"
    networks:
      - selenium

  firefox:
    image: selenium/node-firefox:4.15.0
    depends_on:
      - hub
      - event-bus
    environment:
      - SE_EVENT_BUS_HOST=event-bus
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=5
      - SE_NODE_SESSION_TIMEOUT=300
      - SE_NODE_MAX_INSTANCES=5
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: "4g"
    networks:
      - selenium

networks:
  selenium:
    driver: bridge

• Event Bus decouples hub ↔ nodes, enabling horizontal scaling.
• SE_NODE_OVERRIDE_MAX_SESSIONS=true lets each container run up to MAX_SESSIONS concurrent sessions.
• Resource limits prevent a single node from starving the host.

3. Auto‑Scaling in Production (Docker Swarm)

# Create an overlay network shared across the swarm
docker network create --driver overlay selenium

# Deploy the stack with a replica count that can be changed dynamically
docker stack deploy -c docker-compose.yml selenium-grid

• Increase replicas for chrome/firefox services based on CI queue length.
• Use Docker Swarm autoscaling tools (e.g., docker service scale selenium_grid_chrome=20).

4. Java Test Configuration (TestNG + Parallel Execution)

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.testng.annotations.*;

import java.net.URL;
import java.util.concurrent.TimeUnit;

public class ParallelSeleniumTest {

    private ThreadLocal<WebDriver> driver = new ThreadLocal<>();

    @BeforeMethod(alwaysRun = true)
    @Parameters({"browser"})
    public void setUp(@Optional("chrome") String browser) throws Exception {
        DesiredCapabilities caps = new DesiredCapabilities();
        caps.setBrowserName(browser);
        // Each test gets its own isolated profile directory
        caps.setCapability("goog:chromeOptions", Map.of(
                "args", List.of("--no-sandbox", "--disable-dev-shm-usage"),
                "prefs", Map.of("profile.default_content_settings.popups", 0)
        ));

        URL hubUrl = new URL("http://localhost:4444/wd/hub");
        driver.set(new RemoteWebDriver(hubUrl, caps));
        driver.get().manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
    }

    @Test(threadPoolSize = 10, invocationCount = 10, timeOut = 300000)
    public void sampleTest() {
        WebDriver wd = driver.get();
        wd.get("https://example.com");
        // ... test steps ...
    }

    @AfterMethod(alwaysRun = true)
    public void tearDown() {
        if (driver.get() != null) {
            driver.get().quit();
        }
        driver.remove();
    }
}

• ThreadLocal guarantees each TestNG thread works with its own WebDriver instance.
• threadPoolSize + invocationCount drives parallelism; TestNG will spin up as many sessions as the Grid can allocate.
• Profiles are isolated via Chrome options; Firefox can use moz:firefoxOptions similarly.

5. CI Integration (GitHub Actions example)

name: UI Tests

on:
  push:
    branches: [ main ]

jobs:
  selenium-tests:
    runs-on: ubuntu-latest
    services:
      selenium-hub:
        image: selenium/hub:4.15.0
        ports: [ "4444:4444" ]
      selenium-node-chrome:
        image: selenium/node-chrome:4.15.0
        env:
          SE_EVENT_BUS_HOST: selenium-hub
          SE_EVENT_BUS_PUBLISH_PORT: 4442
          SE_EVENT_BUS_SUBSCRIBE_PORT: 4443
          SE_NODE_MAX_SESSIONS: 5
          SE_NODE_MAX_INSTANCES: 5
        options: --network=host
    steps:
      - uses: actions/checkout@v3
      - name: Set up JDK 17
        uses: actions/setup-java@v3
        with:
          java-version: '17'
          distribution: 'temurin'
      - name: Build & Test
        run: |
          ./mvnw clean verify -DskipUnitTests -Dtestng.parallel=methods -Dtestng.threadCount=20

• The services block spins up a hub and a node in the same job, ensuring isolation.
• Adjust threadCount to match the node capacity.

6. Key Considerations for Production‑Ready Grid

7. TL;DR Checklist

1. Docker Compose for local dev, Swarm/K8s for production auto‑scaling.
2. Event Bus decouples hub ↔ nodes, enabling horizontal scaling.
3. ThreadLocal WebDriver + TestNG parallel config for high concurrency.
4. Resource limits (cpus, memory) + SE_NODE_MAX_SESSIONS to avoid contention.
5. CI service containers or external Grid endpoint; expose only needed ports.
6. Observability: Grid metrics, container health checks, per‑test logs.

By following this blueprint, you can run thousands of UI tests in parallel, keep each test sandboxed, and maintain a stable, cost‑effective CI pipeline.

📲 Practice Offline on Mobile: Download the free QA Automation & SDET Prep app on Google Play & App Store.