Dynamic Port Registry for a Go Device Farm

Build a crash-resilient port allocator with MySQL leases for device farms.

Introduction

30 Second Summary

Every phone in a device farm needs its own set of network ports. When your fixed port pool runs dry or a crashed process leaves a port stuck, test sessions silently fail with cryptic errors.

In this project, you will build a portregistry Go package that replaces a hardcoded port pool with dynamic, OS-assigned ephemeral ports tracked in MySQL. A background health checker automatically reclaims ports from dead processes.

What You'll Build

A Go package that uses the OS kernel to allocate ports on demand, maintains a blocklist of known developer tool ports, records all allocations as expiring leases in MySQL, and detects crashed Appium sessions to recover their ports automatically.

By the end of this project, you'll have:

  • A portregistry Go package with a dynamic allocator, blocklist, and health checker.
  • A MySQL migration for the port_leases table that tracks every port allocation with expiring leases.
  • Integration points for your existing Device Watcher and Gateway Session Manager.
  • Secret Mission: Add a port dashboard API and real-time conflict detection for operational visibility.

Are there any prerequisites?

This project builds on an existing Go device farm with a Gin-based gateway, MySQL, Selenium Grid 4, Appium, and physical Android devices connected via USB. You should have a working gateway and at least one device connected before starting.

Audit Your Port Landscape and Create the Lease Table

Your device farm's fixed port pool covers ports 4760 through 4820. That gives you about 30 usable port pairs, which works for a small fleet of devices but falls apart once you push past 30 connections.

The fix is a dynamic allocator that pulls from the entire macOS ephemeral range. Before building it, you need to map your current port landscape, create a MySQL table for tracking leases, and scaffold the new package.

In this step, get ready to:
  • Audit your Mac's port usage and identify conflict ports.
  • Create the port_leases table in MySQL.
  • Scaffold the portregistry package directory.
Audit current port usage

Your new allocator will pull ports from the macOS ephemeral range instead of a hardcoded pool. Before you build it, you need to confirm that range and see which processes already occupy ports your device farm cares about.

  • Check your Mac's ephemeral port range by running this command:
sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last

You should see two lines of output. The first value is the start of your ephemeral range (typically 49152) and the second is the end (65535). This is the pool your dynamic allocator will draw from.

What is an ephemeral port?

Ephemeral ports are temporary port numbers the operating system assigns on demand. The range 49152 through 65535 gives you over 16,000 available ports, compared to the ~30 in your fixed pool.

  • Scan for processes currently listening on ports in your device farm's range and known conflict ports by running:
lsof -iTCP -sTCP:LISTEN -nP | grep -E ':(4[4-9][0-9]{2}|[5-9][0-9]{3}|8888|9090|5037)'

What does this command do?

This filters all TCP listeners on your Mac for ports in the 4400-9999 range, plus specific known conflict ports. You will likely see entries from:

  • Appium processes bound to ports in your current 4760-4820 pool.
  • ADB server on port 5037.
  • Selenium Grid on ports 4442, 4443, or 4444.
  • Charles Proxy on port 8888 or Proxyman on port 9090.

No output at all?

If the command returns nothing, that means no processes are currently listening in those port ranges. That is fine. It means your device farm services are not running right now. The important thing is that you confirmed the ephemeral range above.

Try starting an Appium session or your device watcher, then run the command again to see what ports they grab.

Help me debug why lsof shows no output for my device farm ports.

  • Write down every port and process name you see in the output. This becomes the seed data for the blocklist you will build in the next step.
Create the port_leases MySQL table

Every dynamically allocated port needs a record that tracks which device owns it, when it was assigned, and when it expires. The port_leases table stores these records so allocations survive process crashes and gateway restarts.

  • Create a new file called 003_port_leases.sql inside your migrations directory, then add this SQL:
CREATE TABLE port_leases (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    device_udid VARCHAR(255) NOT NULL,
    port INT NOT NULL,
    port_type VARCHAR(50) NOT NULL,
    pid INT NOT NULL DEFAULT 0,
    allocated_at DATETIME NOT NULL,
    expires_at DATETIME NOT NULL,
    UNIQUE KEY uk_port (port),
    INDEX idx_device_udid (device_udid),
    INDEX idx_expires_at (expires_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

What does this table do?

Each row represents a single port lease. The key columns are:

  • device_udid ties the port back to the physical device that owns it.
  • port has a UNIQUE constraint (uk_port) so two devices can never hold the same port simultaneously.
  • port_type distinguishes appium, grid_node, and system_port allocations.
  • expires_at sets a TTL on every lease. If your gateway crashes, expired leases get cleaned up automatically.
  • Save 003_port_leases.sql.

✔️ Awesome, I've got everything!

Great. Make sure you have saved the file before running the migration.

ⓧ I'd like to double check the full code

CREATE TABLE port_leases (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    device_udid VARCHAR(255) NOT NULL,
    port INT NOT NULL,
    port_type VARCHAR(50) NOT NULL,
    pid INT NOT NULL DEFAULT 0,
    allocated_at DATETIME NOT NULL,
    expires_at DATETIME NOT NULL,
    UNIQUE KEY uk_port (port),
    INDEX idx_device_udid (device_udid),
    INDEX idx_expires_at (expires_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
  • Run the migration against your device farm database:
mysql -u root -p [[DB_NAME="your device farm database name"]] < migrations/003_port_leases.sql

What happens when you run this?

The mysql client will prompt you for your MySQL password, then execute the SQL file against your database. Adjust -u root if your database uses a different user.

  • Verify the table was created by running:
mysql -u root -p [[DB_NAME="your device farm database name"]] -e "DESCRIBE port_leases;"

You should see a table with seven rows listing id, device_udid, port, port_type, pid, allocated_at, and expires_at. The id column should show PRI in the Key column and auto_increment in the Extra column.

Seeing "Table already exists"?

If you ran the migration before, MySQL will refuse to create the table again. Drop it first with DROP TABLE port_leases; in a MySQL session, then re-run the migration file.

If you see an access denied error, double-check your MySQL username and password. Make sure you are using the same credentials you use for your existing device farm tables.

Help me troubleshoot my port_leases migration error.

Create the portregistry package structure

The registry code will live in its own Go package. You need three files: one for the core allocator, one for the blocklist, and one for the health checker.

  • Create the package directory inside your Go project by running:
mkdir -p portregistry
  • Create the three package files by running:
touch portregistry/registry.go portregistry/blocklist.go portregistry/health.go
  • Confirm the files exist by running:
ls portregistry/

You should see blocklist.go, health.go, and registry.go listed in the output.

Directory not found?

Make sure you are running these commands from the root of your Go project (the directory that contains your go.mod file). Run ls go.mod to confirm you are in the right place.

Help me find the right directory for my Go device farm project.

Your port landscape is mapped, your lease table is ready, and your package structure is in place. Next up, you will build the core PortRegistry with OS-assigned port allocation and a blocklist to dodge every conflict you just documented.

Build the PortRegistry with OS-Assigned Allocation

In the last step, you audited your Mac's port landscape and created the port_leases table to track allocations. Now you have a clear picture of which ports your dev tools claim and a database ready to record new leases.

In this step, you will build the core portregistry package. Instead of picking ports from a fixed range, the Registry asks the OS for a guaranteed-available port. Combined with a blocklist of known-conflict ports and MySQL-backed leases, this eliminates port collisions entirely.

In this step, get ready to:
  • Implement a blocklist of known-conflict ports for developer tools and infrastructure.
  • Build the core Registry with OS-assigned port allocation and MySQL-backed leases.
  • Verify the Registry allocates and releases ports correctly.
Implement the blocklist

Your Mac runs dozens of developer tools that grab well-known ports. If the Registry accidentally hands one of those ports to Appium, you get a silent bind failure. The blocklist prevents this by maintaining a map of ports the allocator must never return.

  • Open portregistry/blocklist.go in your editor and paste the following code:
package portregistry

func defaultBlocklist() map[int]string {
	bl := map[int]string{
		5037: "ADB server",
		4442: "Selenium Grid Event Bus",
		4443: "Selenium Grid Event Bus",
		4444: "Selenium Grid Hub",
		8888: "Charles Proxy",
		9090: "Proxyman",
		8080: "common HTTP dev server",
		8081: "Gateway / React Native packager",
		3000: "common dev server",
		3306: "MySQL",
	}

	for p := 5554; p <= 5585; p += 2 {
		bl[p] = "Android emulator console"
		bl[p+1] = "Android emulator ADB"
	}

	return bl
}

What does this code do?

The defaultBlocklist() function returns a map where each key is a port number and each value is a human-readable reason why that port is off-limits.

  • Ports 5037 (ADB), 4442-4444 (Selenium Grid), and 8888 (Charles Proxy) are hardcoded because these tools bind on startup and rarely change.
  • The for loop generates 16 Android emulator port pairs (5554-5585) instead of listing each one individually. Each emulator claims a console port and an ADB port.
  • Save blocklist.go.

Why these specific ports?

Each port in the blocklist came from your audit in Step 1. Charles Proxy defaults to 8888, ADB always listens on 5037, and Selenium Grid's Event Bus claims 4442 and 4443. If the OS assigned one of these ports to Appium, Appium would fail to bind and the session would silently die.

✔️ Awesome, I've got everything!

Great. Double check you have saved blocklist.go.

ⓧ I'd like to double check the full code

package portregistry

func defaultBlocklist() map[int]string {
	bl := map[int]string{
		5037: "ADB server",
		4442: "Selenium Grid Event Bus",
		4443: "Selenium Grid Event Bus",
		4444: "Selenium Grid Hub",
		8888: "Charles Proxy",
		9090: "Proxyman",
		8080: "common HTTP dev server",
		8081: "Gateway / React Native packager",
		3000: "common dev server",
		3306: "MySQL",
	}

	for p := 5554; p <= 5585; p += 2 {
		bl[p] = "Android emulator console"
		bl[p+1] = "Android emulator ADB"
	}

	return bl
}
Implement the core Registry

The Registry is the heart of this package. It asks the OS kernel for an available port, checks that port against the blocklist, records a lease in MySQL, and returns the port number to the caller.

You will build registry.go in four chunks. Each chunk adds a group of related methods.

  • Open portregistry/registry.go in your editor and paste the following foundation code:
package portregistry

import (
	"database/sql"
	"fmt"
	"log"
	"net"
	"sync"
	"time"
)

// PortType labels the purpose of a leased port
type PortType string

const (
	PortTypeAppium     PortType = "appium"
	PortTypeGridNode   PortType = "grid_node"
	PortTypeSystemPort PortType = "system_port"
)

// PortLease represents a single port allocation stored in MySQL
type PortLease struct {
	ID          int64
	DeviceUDID  string
	Port        int
	PortType    PortType
	PID         int
	AllocatedAt time.Time
	ExpiresAt   time.Time
}

// Registry manages dynamic port allocation with blocklist and lease tracking
type Registry struct {
	db        *sql.DB
	mu        sync.Mutex
	blocklist map[int]string
	leaseTTL  time.Duration
}

// New creates a Registry with the default blocklist and given lease TTL
func New(db *sql.DB, leaseTTL time.Duration) *Registry {
	return &Registry{
		db:        db,
		blocklist: defaultBlocklist(),
		leaseTTL:  leaseTTL,
	}
}

// AddBlockedPort adds a port to the blocklist at runtime
func (r *Registry) AddBlockedPort(port int, reason string) {
	r.mu.Lock()
	defer r.mu.Unlock()
	r.blocklist[port] = reason
}

What does this code do?

  • PortType constants label each allocation as appium, grid_node, or system_port. The HealthChecker (built in the next step) uses these labels to decide which ports to health-check.
  • The Registry struct holds a database connection, a mutex-protected blocklist, and a configurable lease TTL. The mutex serializes allocations so two goroutines cannot grab the same port in the same instant.
  • New() initializes the Registry with the blocklist from blocklist.go. AddBlockedPort() lets you extend the blocklist at runtime if you discover new conflicts.
  • Add the core Allocate method below AddBlockedPort by pasting the following code:
// Allocate asks the OS for an ephemeral port, checks it against the blocklist
// and existing leases, then records a lease row in MySQL.
func (r *Registry) Allocate(deviceUDID string, portType PortType, pid int) (int, error) {
	r.mu.Lock()
	defer r.mu.Unlock()

	const maxRetries = 10
	for i := 0; i < maxRetries; i++ {
		port, err := acquireEphemeralPort()
		if err != nil {
			log.Printf("ephemeral port acquisition failed (attempt %d): %v", i+1, err)
			continue
		}

		if reason, blocked := r.blocklist[port]; blocked {
			log.Printf("port %d blocked (%s), retrying", port, reason)
			continue
		}

		taken, err := r.isPortLeased(port)
		if err != nil {
			return 0, fmt.Errorf("checking lease for port %d: %w", port, err)
		}
		if taken {
			continue
		}

		now := time.Now()
		expiresAt := now.Add(r.leaseTTL)

		_, err = r.db.Exec(
			`INSERT INTO port_leases (device_udid, port, port_type, pid, allocated_at, expires_at)
			 VALUES (?, ?, ?, ?, ?, ?)`,
			deviceUDID, port, string(portType), pid, now, expiresAt,
		)
		if err != nil {
			log.Printf("lease insert failed for port %d (attempt %d): %v", port, i+1, err)
			continue
		}

		return port, nil
	}

	return 0, fmt.Errorf("failed to allocate port after %d retries", maxRetries)
}

What does the Allocate method do?

Each iteration of the retry loop follows three steps: acquire a port from the OS, screen it against the blocklist and existing leases, then write a lease row to MySQL.

  • acquireEphemeralPort() (you will add this helper shortly) calls net.Listen("tcp", "127.0.0.1:0") to get a guaranteed-available port from the kernel. The listener is immediately closed, and the port number is returned.
  • If the port is in the blocklist or already leased in MySQL, the loop retries with a fresh OS-assigned port. After 10 failures, it returns an error.
  • The MySQL UNIQUE KEY uk_port (port) constraint acts as a second safety net. Even if two processes race past the Go mutex, only one insert can succeed.

What is TOCTOU and why does the retry loop handle it?

There is a small time-of-check-to-time-of-use window between closing the listener and Appium binding to the port. Another process could theoretically grab it in between. The retry loop with blocklist checking and the MySQL uniqueness constraint make real collisions extremely rare. Apple DTS actually recommends this port-0 binding pattern over manually scanning for free ports, which is even more racy.

  • Add the group allocation and release methods below Allocate by pasting the following code:
// AllocateGroup atomically allocates three ports for a device.
// If any allocation fails, previously allocated ports are released.
func (r *Registry) AllocateGroup(deviceUDID string, pid int) (appiumPort, gridNodePort, systemPort int, err error) {
	appiumPort, err = r.Allocate(deviceUDID, PortTypeAppium, pid)
	if err != nil {
		return 0, 0, 0, fmt.Errorf("allocating appium port: %w", err)
	}

	gridNodePort, err = r.Allocate(deviceUDID, PortTypeGridNode, pid)
	if err != nil {
		r.releaseUnlocked(appiumPort)
		return 0, 0, 0, fmt.Errorf("allocating grid node port: %w", err)
	}

	systemPort, err = r.Allocate(deviceUDID, PortTypeSystemPort, pid)
	if err != nil {
		r.releaseUnlocked(appiumPort)
		r.releaseUnlocked(gridNodePort)
		return 0, 0, 0, fmt.Errorf("allocating system port: %w", err)
	}

	return appiumPort, gridNodePort, systemPort, nil
}

// Release deletes the lease for a single port
func (r *Registry) Release(port int) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	return r.releaseUnlocked(port)
}

// releaseUnlocked deletes a lease without acquiring the mutex
func (r *Registry) releaseUnlocked(port int) error {
	_, err := r.db.Exec(`DELETE FROM port_leases WHERE port = ?`, port)
	return err
}

// ReleaseByDevice frees all ports allocated to a device at once
func (r *Registry) ReleaseByDevice(deviceUDID string) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	_, err := r.db.Exec(`DELETE FROM port_leases WHERE device_udid = ?`, deviceUDID)
	return err
}

Why does AllocateGroup roll back on failure?

Each device needs three ports: Appium, Grid Node, and system. If the second or third allocation fails, the ports that already succeeded must be released immediately. Without this rollback, a partial failure would leak port leases that never get used.

The releaseUnlocked() helper does a direct database delete without acquiring the mutex. This avoids a deadlock since Allocate() already handles its own locking internally.

  • Add the lease management methods and helper functions below ReleaseByDevice by pasting the following code:
// RenewLease extends a port's expiry by the registry's TTL
func (r *Registry) RenewLease(port int) error {
	expiresAt := time.Now().Add(r.leaseTTL)
	_, err := r.db.Exec(
		`UPDATE port_leases SET expires_at = ? WHERE port = ?`,
		expiresAt, port,
	)
	return err
}

// CleanExpired deletes all leases past their expiry time
func (r *Registry) CleanExpired() (int64, error) {
	result, err := r.db.Exec(
		`DELETE FROM port_leases WHERE expires_at < ?`,
		time.Now(),
	)
	if err != nil {
		return 0, err
	}
	return result.RowsAffected()
}

// ListLeases returns all active leases for monitoring
func (r *Registry) ListLeases() ([]PortLease, error) {
	rows, err := r.db.Query(
		`SELECT id, device_udid, port, port_type, pid, allocated_at, expires_at
		 FROM port_leases ORDER BY allocated_at`,
	)
	if err != nil {
		return nil, err
	}
	defer rows.Close()

	var leases []PortLease
	for rows.Next() {
		var l PortLease
		var pt string
		if err := rows.Scan(&l.ID, &l.DeviceUDID, &l.Port, &pt, &l.PID, &l.AllocatedAt, &l.ExpiresAt); err != nil {
			return nil, err
		}
		l.PortType = PortType(pt)
		leases = append(leases, l)
	}
	return leases, rows.Err()
}

// isPortLeased checks if a port has an active (non-expired) lease
func (r *Registry) isPortLeased(port int) (bool, error) {
	var count int
	err := r.db.QueryRow(
		`SELECT COUNT(*) FROM port_leases WHERE port = ? AND expires_at > ?`,
		port, time.Now(),
	).Scan(&count)
	if err != nil {
		return false, err
	}
	return count > 0, nil
}

// acquireEphemeralPort binds to port 0 so the OS assigns an available port
func acquireEphemeralPort() (int, error) {
	listener, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		return 0, err
	}
	port := listener.Addr().(*net.TCPAddr).Port
	listener.Close()
	return port, nil
}

// IsPortListening checks if a process is actively listening on a port
func IsPortListening(port int) bool {
	conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", port), 2*time.Second)
	if err != nil {
		return false
	}
	conn.Close()
	return true
}

What do these methods do?

  • RenewLease() extends a port's expiry time. The HealthChecker (built in the next step) calls this for every healthy Appium port, so active sessions never expire.
  • CleanExpired() deletes all lease rows past their TTL. This is the crash-recovery mechanism. If the gateway dies without cleanup, leases expire on their own.
  • acquireEphemeralPort() is the key helper. It calls net.Listen("tcp", "127.0.0.1:0") to get a port from the macOS ephemeral range (typically 49152-65535), then immediately closes the listener so Appium can bind to that port.
  • IsPortListening() does a TCP dial with a 2-second timeout. The HealthChecker uses this to detect dead Appium processes.
  • Save registry.go.
  • Verify the package compiles by running:
go build ./portregistry/

If the command returns with no output, your package compiles cleanly.

Seeing a compile error?

  • Check that your package portregistry declaration appears at the top of both registry.go and blocklist.go. Both files must belong to the same package.
  • Make sure all the import paths in the import block are spelled correctly. Go will refuse to compile with unused or misspelled imports.
  • Verify you pasted the code chunks in order. The methods reference each other (e.g. AllocateGroup calls Allocate and releaseUnlocked), so all chunks must be present.

Help me debug my portregistry compile error

✔️ Awesome, I've got everything!

Great. Double check you have saved registry.go and that go build ./portregistry/ returns no errors.

ⓧ I'd like to double check the full code

package portregistry

import (
	"database/sql"
	"fmt"
	"log"
	"net"
	"sync"
	"time"
)

type PortType string

const (
	PortTypeAppium     PortType = "appium"
	PortTypeGridNode   PortType = "grid_node"
	PortTypeSystemPort PortType = "system_port"
)

type PortLease struct {
	ID          int64
	DeviceUDID  string
	Port        int
	PortType    PortType
	PID         int
	AllocatedAt time.Time
	ExpiresAt   time.Time
}

type Registry struct {
	db        *sql.DB
	mu        sync.Mutex
	blocklist map[int]string
	leaseTTL  time.Duration
}

func New(db *sql.DB, leaseTTL time.Duration) *Registry {
	return &Registry{
		db:        db,
		blocklist: defaultBlocklist(),
		leaseTTL:  leaseTTL,
	}
}

func (r *Registry) AddBlockedPort(port int, reason string) {
	r.mu.Lock()
	defer r.mu.Unlock()
	r.blocklist[port] = reason
}

func (r *Registry) Allocate(deviceUDID string, portType PortType, pid int) (int, error) {
	r.mu.Lock()
	defer r.mu.Unlock()

	const maxRetries = 10
	for i := 0; i < maxRetries; i++ {
		port, err := acquireEphemeralPort()
		if err != nil {
			log.Printf("ephemeral port acquisition failed (attempt %d): %v", i+1, err)
			continue
		}

		if reason, blocked := r.blocklist[port]; blocked {
			log.Printf("port %d blocked (%s), retrying", port, reason)
			continue
		}

		taken, err := r.isPortLeased(port)
		if err != nil {
			return 0, fmt.Errorf("checking lease for port %d: %w", port, err)
		}
		if taken {
			continue
		}

		now := time.Now()
		expiresAt := now.Add(r.leaseTTL)

		_, err = r.db.Exec(
			`INSERT INTO port_leases (device_udid, port, port_type, pid, allocated_at, expires_at)
			 VALUES (?, ?, ?, ?, ?, ?)`,
			deviceUDID, port, string(portType), pid, now, expiresAt,
		)
		if err != nil {
			log.Printf("lease insert failed for port %d (attempt %d): %v", port, i+1, err)
			continue
		}

		return port, nil
	}

	return 0, fmt.Errorf("failed to allocate port after %d retries", maxRetries)
}

func (r *Registry) AllocateGroup(deviceUDID string, pid int) (appiumPort, gridNodePort, systemPort int, err error) {
	appiumPort, err = r.Allocate(deviceUDID, PortTypeAppium, pid)
	if err != nil {
		return 0, 0, 0, fmt.Errorf("allocating appium port: %w", err)
	}

	gridNodePort, err = r.Allocate(deviceUDID, PortTypeGridNode, pid)
	if err != nil {
		r.releaseUnlocked(appiumPort)
		return 0, 0, 0, fmt.Errorf("allocating grid node port: %w", err)
	}

	systemPort, err = r.Allocate(deviceUDID, PortTypeSystemPort, pid)
	if err != nil {
		r.releaseUnlocked(appiumPort)
		r.releaseUnlocked(gridNodePort)
		return 0, 0, 0, fmt.Errorf("allocating system port: %w", err)
	}

	return appiumPort, gridNodePort, systemPort, nil
}

func (r *Registry) Release(port int) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	return r.releaseUnlocked(port)
}

func (r *Registry) releaseUnlocked(port int) error {
	_, err := r.db.Exec(`DELETE FROM port_leases WHERE port = ?`, port)
	return err
}

func (r *Registry) ReleaseByDevice(deviceUDID string) error {
	r.mu.Lock()
	defer r.mu.Unlock()
	_, err := r.db.Exec(`DELETE FROM port_leases WHERE device_udid = ?`, deviceUDID)
	return err
}

func (r *Registry) RenewLease(port int) error {
	expiresAt := time.Now().Add(r.leaseTTL)
	_, err := r.db.Exec(
		`UPDATE port_leases SET expires_at = ? WHERE port = ?`,
		expiresAt, port,
	)
	return err
}

func (r *Registry) CleanExpired() (int64, error) {
	result, err := r.db.Exec(
		`DELETE FROM port_leases WHERE expires_at < ?`,
		time.Now(),
	)
	if err != nil {
		return 0, err
	}
	return result.RowsAffected()
}

func (r *Registry) ListLeases() ([]PortLease, error) {
	rows, err := r.db.Query(
		`SELECT id, device_udid, port, port_type, pid, allocated_at, expires_at
		 FROM port_leases ORDER BY allocated_at`,
	)
	if err != nil {
		return nil, err
	}
	defer rows.Close()

	var leases []PortLease
	for rows.Next() {
		var l PortLease
		var pt string
		if err := rows.Scan(&l.ID, &l.DeviceUDID, &l.Port, &pt, &l.PID, &l.AllocatedAt, &l.ExpiresAt); err != nil {
			return nil, err
		}
		l.PortType = PortType(pt)
		leases = append(leases, l)
	}
	return leases, rows.Err()
}

func (r *Registry) isPortLeased(port int) (bool, error) {
	var count int
	err := r.db.QueryRow(
		`SELECT COUNT(*) FROM port_leases WHERE port = ? AND expires_at > ?`,
		port, time.Now(),
	).Scan(&count)
	if err != nil {
		return false, err
	}
	return count > 0, nil
}

func acquireEphemeralPort() (int, error) {
	listener, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		return 0, err
	}
	port := listener.Addr().(*net.TCPAddr).Port
	listener.Close()
	return port, nil
}

func IsPortListening(port int) bool {
	conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", port), 2*time.Second)
	if err != nil {
		return false
	}
	conn.Close()
	return true
}
Verify the Registry works

The package compiles, but you need to confirm that AllocateGroup actually returns three distinct ports in the ephemeral range and records them in MySQL. A quick allocation test proves the full lifecycle works.

  • Create a test file at portregistry/registry_test.go and paste the following verification code:
package portregistry

import (
	"database/sql"
	"fmt"
	"testing"
	"time"

	_ "github.com/go-sql-driver/mysql"
)

func TestAllocateGroup(t *testing.T) {
	// Use your existing database connection string
	db, err := sql.Open("mysql", "root:your-password-here@tcp(127.0.0.1:3306)/your-database-name?parseTime=true")
	if err != nil {
		t.Fatal(err)
	}
	defer db.Close()

	// Create a registry with a 5-minute lease TTL
	r := New(db, 5*time.Minute)

	// Allocate three ports for a test device
	appium, grid, sys, err := r.AllocateGroup("test-device-verify", 0)
	if err != nil {
		t.Fatal(err)
	}
	fmt.Printf("Allocated ports - Appium: %d, Grid: %d, System: %d\n", appium, grid, sys)

	// Verify all ports are distinct
	if appium == grid || appium == sys || grid == sys {
		t.Errorf("ports are not distinct: %d, %d, %d", appium, grid, sys)
	}

	// Verify all ports are in the ephemeral range
	for _, p := range []int{appium, grid, sys} {
		if p < 49152 || p > 65535 {
			t.Errorf("port %d is outside ephemeral range 49152-65535", p)
		}
	}

	// Clean up the test leases
	if err := r.ReleaseByDevice("test-device-verify"); err != nil {
		t.Fatal(err)
	}
	fmt.Println("All leases released successfully")
}

What does this test do?

  • It connects to your existing MySQL database, creates a Registry, and calls AllocateGroup to request three ports for a fake device.
  • It checks that all three ports are distinct and fall within the macOS ephemeral range (49152-65535).
  • It cleans up by calling ReleaseByDevice to delete the test lease rows.
  • Update the connection string in the test to match your existing database credentials and database name.
  • Save registry_test.go and run the test:
go test -v ./portregistry/ -run TestAllocateGroup

You should see output showing three port numbers in the 49152-65535 range and a PASS result.

  • While the test leases exist (before the cleanup runs), verify the rows in MySQL by running:
mysql -u root -p -e "SELECT port, port_type, device_udid FROM port_leases;" your-database-name

If the test already completed, the table will be empty because ReleaseByDevice cleaned up the rows. That is the expected behavior. An empty table after the test confirms that both allocation and release work correctly.

Test failing with a connection error?

  • Verify your MySQL server is running. Check with mysql -u root -p -e "SELECT 1;" to confirm connectivity.
  • Make sure the connection string in the test matches the credentials you use for your device farm database. The database name, username, and password must all match.
  • Check that the port_leases table exists. Run DESCRIBE port_leases; in your MySQL client to confirm the migration from Step 1 was applied.

Help me debug my portregistry test failure

Your PortRegistry is built and verified. It can allocate OS-assigned ports, screen them against 40+ known conflicts, record leases in MySQL, and clean up after itself. Next up, you will wire the Registry into your Device Watcher to replace the fixed port pool with dynamic allocation.

Integrate with Your Device Watcher

Your portregistry package is built, tested, and allocating OS-assigned ephemeral ports from a pool of over 16,000 candidates.

Now it's time to rip out your fixed PortPool and wire the Registry into your running device farm.

Your Device Watcher currently grabs a single port from a static range every time a device connects. Your Gateway Session Manager returns that port to the pool during cleanup. You'll replace both of these with Registry calls that lease-track every allocation in MySQL.

In this step, get ready to:
  • Replace PortPool.Next() with registry.AllocateGroup in your Device Watcher.
  • Update device disconnect handling to release leased ports via ReleaseByDevice.
  • Add port release to your Gateway Session Manager's cleanup flow.
Replace the fixed PortPool in your Watcher Daemon

Your Device Watcher loop currently allocates a single port from a hardcoded pool each time a new device appears. The AllocateGroup method replaces this with a single call that returns three distinct, OS-assigned ports for Appium, the Selenium Grid Node, and the UiAutomator2 system port.

  • Open your Device Watcher file and add an import for your portregistry package.
  • Make sure your watcher has access to a *portregistry.Registry instance, either by passing one in from your main function or by creating one with your existing *sql.DB.

Where should I initialize the Registry?

Your gateway's main() function is the best place to create the Registry, since it already holds your database connection. Pass the Registry into your watcher as a dependency, the same way you pass *sql.DB today.

Initialize it with a 5-minute lease TTL. This means if your gateway crashes without running cleanup, orphaned leases expire within 5 minutes and can be reclaimed.

  • In your watcher's device-connected handler, find the line where you allocate a port from your PortPool. It looks something like this:
port := portPool.Next()
  • Replace that allocation with a call to registry.AllocateGroup, so the line now looks like this:
appiumPort, gridNodePort, sysPort, err := registry.AllocateGroup(udid, 0)

What changed here?

Instead of pulling one port from a fixed pool, AllocateGroup asks the OS for three separate ephemeral ports. Each one is checked against the blocklist, verified as unleased in MySQL, and recorded with a TTL.

The second argument is 0 because Appium hasn't started yet, so you don't have a PID. You can update the lease with the real PID after launch if you want tighter tracking.

With three ports returned, you now need to use each one in the right place.

  • Update your Appium startup command to use appiumPort instead of the old fixed port. This is the -p argument you pass to the Appium server.
  • Update your Grid Node registration to use gridNodePort. This is the port the node listens on when it registers with the Selenium Grid hub.
  • Update the UiAutomator2 system port argument to use sysPort. This is the systemPort capability that Appium passes to the UiAutomator2 driver.
  • Remove any references to PortPool from your watcher. It's no longer needed for port allocation.

Handle the error from AllocateGroup

If AllocateGroup returns an error, log it and skip provisioning for that device. The built-in rollback means any partially allocated ports are already released.

Update device disconnect handling

When your watcher detects a device disconnect via ADB polling, it currently returns the port back to the fixed pool. You need to replace that with a Registry call that deletes all three lease rows at once.

  • In your watcher's device-disconnected handler, find the code that returns the port to your PortPool.
  • Replace it with a call to ReleaseByDevice:
// Release all three leased ports for this device
if err := registry.ReleaseByDevice(udid); err != nil {
    log.Printf("failed to release ports for device %s: %v", udid, err)
}

Why release by device instead of by port?

Each device has three leased ports (Appium, Grid Node, system port). ReleaseByDevice deletes all rows matching the UDID in a single query. This avoids the risk of releasing only one or two ports if something fails partway through.

Update the Gateway Session Manager

Your CleanupWorker handles session teardown after a test finishes. It stops the recording, pulls the file, uploads to S3, and deletes the downstream session.

Adding a port release here creates a second safety net. Even if the Device Watcher missed the disconnect, the session cleanup path still frees the ports.

  • Open your CleanupWorker code in your Gateway Session Manager.
  • Find the end of your cleanup sequence, after the downstream session is deleted.
  • Add a ReleaseByDevice call after the existing cleanup steps:
// Free all leased ports even if the watcher missed the disconnect
if err := registry.ReleaseByDevice(udid); err != nil {
    log.Printf("cleanup: failed to release ports for device %s: %v", udid, err)
}

Why release ports in two places?

The Device Watcher releases ports when it detects a USB disconnect. The CleanupWorker releases ports when a test session ends. These are independent code paths that can trigger in different order depending on how the session ended.

Calling ReleaseByDevice on already-released ports is a no-op (the DELETE query finds no matching rows). There is no harm in calling it twice.

  • Make sure your CleanupWorker has access to the same *portregistry.Registry instance you passed to the watcher.
  • Save all modified files.
Verify the integration with a physical device

All three code paths are updated. Time to rebuild your gateway and confirm that ports flow through the Registry end to end.

  • Rebuild and restart your Go gateway.
  • Connect a physical Android device via USB.
  • Wait for your Device Watcher to detect the device and start Appium.
  • Verify the port allocations by querying the port_leases table:
SELECT device_udid, port, port_type, allocated_at, expires_at
FROM port_leases
ORDER BY allocated_at;

What should I see?

You should see three rows for your connected device. Each row has a different port_type value: appium, grid_node, and system_port.

All three port numbers should be above 49152 (the start of the macOS ephemeral range). None of them should match any port in your blocklist.

  • Confirm that Appium started on the port shown in the appium row by checking your gateway logs.
  • Confirm that the Grid Node registered on the port shown in the grid_node row.
  • Disconnect the device from USB.
  • Run the same query again to verify that all three lease rows are deleted:
SELECT COUNT(*) FROM port_leases;

The count should be 0 if no other devices are connected. If you have multiple devices, verify the specific device's UDID is no longer in the table.

Leases not appearing after device connects?

Make sure your watcher is using the updated code path. Rebuild with go build and restart the gateway process.

Check that the *portregistry.Registry instance is not nil. If you forgot to pass it to the watcher, the AllocateGroup call will panic on a nil pointer.

Verify your MySQL connection is working by running SELECT 1 from your application. A connection timeout would cause all allocations to fail silently.

Help me debug why port leases aren't being created when my device connects.

Leases not deleted after device disconnects?

Confirm your disconnect handler is calling registry.ReleaseByDevice(udid) and not the old port-return-to-pool function.

Check that the udid value in your disconnect handler matches the one stored in the device_udid column. Case and whitespace matter.

Help me figure out why port leases aren't being cleaned up on disconnect.

Your device farm is now allocating and releasing ports dynamically. Next up, you'll add a background HealthChecker that catches the cases cleanup misses entirely, like a crashed Appium process that never triggered a disconnect.

Add Session Health Monitoring and Auto-Recovery

Your device watcher and session manager now allocate and release ports dynamically through MySQL-backed leases. But what happens when an Appium process crashes without triggering your watcher's disconnect handler?

The lease rows stay in port_leases until they expire, and those ports are locked out of allocation. In this step, you will build a background health checker that periodically dials each leased Appium port, reclaims ports from dead processes, and renews leases for healthy ones.

In this step, get ready to:
  • Implement the HealthChecker in health.go with TCP health probes and automatic lease cleanup.
  • Wire the HealthChecker into your gateway startup so it runs continuously in the background.
  • Test crash recovery by killing an Appium process and verifying automatic port reclamation.
Implement the HealthChecker

The HealthChecker is a background goroutine that wakes up on a timer, checks whether leased Appium ports are still responding, and takes action. Dead ports get released. Healthy ports get their lease renewed.

  • Open portregistry/health.go in your editor.
  • Add the package declaration, imports, struct definition, constructor, and lifecycle methods by pasting this code:
package portregistry

import (
	"log"
	"time"
)

// HealthChecker periodically probes leased ports and reclaims dead ones
type HealthChecker struct {
	registry *Registry
	interval time.Duration
	stopCh   chan struct{}
}

// NewHealthChecker creates a checker that ticks at the given interval
func NewHealthChecker(registry *Registry, interval time.Duration) *HealthChecker {
	return &HealthChecker{
		registry: registry,
		interval: interval,
		stopCh:   make(chan struct{}),
	}
}

// Start launches the health check loop in a background goroutine
func (h *HealthChecker) Start() {
	go h.run()
	log.Printf("port health checker started (interval: %s)", h.interval)
}

// Stop signals the background goroutine to exit
func (h *HealthChecker) Stop() {
	close(h.stopCh)
	log.Println("port health checker stopped")
}

What does this code do?

The HealthChecker struct holds a reference to the Registry, a configurable tick interval, and a stopCh channel for clean shutdown.

  • Start() kicks off the background loop in its own goroutine so it does not block your gateway.
  • Stop() closes the channel, which signals the goroutine to return. This ensures a clean exit when your gateway shuts down.

Next, you need the ticker loop that drives the health checks on a schedule.

  • Add the run() method below the Stop() function by pasting this code:
// run is the main loop that ticks at the configured interval
func (h *HealthChecker) run() {
	ticker := time.NewTicker(h.interval)
	defer ticker.Stop()

	for {
		select {
		case <-ticker.C:
			// Each tick triggers a full health check pass
			h.check()
		case <-h.stopCh:
			return
		}
	}
}

What does the select statement do?

The select block waits on two channels simultaneously. When the ticker fires, it calls check(). When stopCh is closed, the goroutine returns and exits cleanly.

Now for the core logic. The check() method is where the health checker actually probes each port and decides whether to renew or release.

  • Add the check() method below run() by pasting this code:
// check runs one health check pass across all active leases
func (h *HealthChecker) check() {
	// First, purge any leases that have outlived their TTL
	cleaned, err := h.registry.CleanExpired()
	if err != nil {
		log.Printf("error cleaning expired leases: %v", err)
	} else if cleaned > 0 {
		log.Printf("cleaned %d expired port leases", cleaned)
	}

	// Fetch all active leases for probing
	leases, err := h.registry.ListLeases()
	if err != nil {
		log.Printf("error listing leases for health check: %v", err)
		return
	}

	for _, lease := range leases {
		// Only check Appium ports - Grid Node and system ports depend on Appium
		if lease.PortType != PortTypeAppium {
			continue
		}

		if !IsPortListening(lease.Port) {
			// Appium is dead - release all ports for this device
			log.Printf("UNHEALTHY: appium port %d for device %s is not listening, releasing all ports",
				lease.Port, lease.DeviceUDID)
			if err := h.registry.ReleaseByDevice(lease.DeviceUDID); err != nil {
				log.Printf("error releasing ports for device %s: %v", lease.DeviceUDID, err)
			}
		} else {
			// Appium is healthy - extend the lease so it doesn't expire
			if err := h.registry.RenewLease(lease.Port); err != nil {
				log.Printf("error renewing lease for port %d: %v", lease.Port, err)
			}
		}
	}
}

Why only check Appium ports?

Grid Node and system ports are lifecycle-dependent on Appium. If the Appium process is dead, all three ports for that device should be released. If Appium is alive, the other two ports are implicitly healthy.

IsPortListening() (already in registry.go) does a TCP dial to 127.0.0.1:{port} with a 2-second timeout. This avoids unnecessary network probes while still catching crashed processes quickly.

  • Save portregistry/health.go.
  • Verify your code compiles by running:
go build ./portregistry/...

You should see no output, which means the package compiled successfully.

Seeing a compilation error?

  • Make sure your package declaration is package portregistry at the top of health.go. It must match the other files in the package.
  • Check that you imported both "log" and "time". Missing either import causes an undefined reference error.
  • Verify that IsPortListening and PortTypeAppium are defined in your registry.go. The health checker depends on both of these.

Help me debug a compilation error in my portregistry health checker

✔️ Awesome, I've got everything!

Great. Make sure you have saved health.go before moving on.

ⓧ I'd like to double check the full code

Here is the complete portregistry/health.go file. Compare it against yours line by line.

package portregistry

import (
	"log"
	"time"
)

type HealthChecker struct {
	registry *Registry
	interval time.Duration
	stopCh   chan struct{}
}

func NewHealthChecker(registry *Registry, interval time.Duration) *HealthChecker {
	return &HealthChecker{
		registry: registry,
		interval: interval,
		stopCh:   make(chan struct{}),
	}
}

func (h *HealthChecker) Start() {
	go h.run()
	log.Printf("port health checker started (interval: %s)", h.interval)
}

func (h *HealthChecker) Stop() {
	close(h.stopCh)
	log.Println("port health checker stopped")
}

func (h *HealthChecker) run() {
	ticker := time.NewTicker(h.interval)
	defer ticker.Stop()

	for {
		select {
		case <-ticker.C:
			h.check()
		case <-h.stopCh:
			return
		}
	}
}

func (h *HealthChecker) check() {
	cleaned, err := h.registry.CleanExpired()
	if err != nil {
		log.Printf("error cleaning expired leases: %v", err)
	} else if cleaned > 0 {
		log.Printf("cleaned %d expired port leases", cleaned)
	}

	leases, err := h.registry.ListLeases()
	if err != nil {
		log.Printf("error listing leases for health check: %v", err)
		return
	}

	for _, lease := range leases {
		if lease.PortType != PortTypeAppium {
			continue
		}

		if !IsPortListening(lease.Port) {
			log.Printf("UNHEALTHY: appium port %d for device %s is not listening, releasing all ports",
				lease.Port, lease.DeviceUDID)
			if err := h.registry.ReleaseByDevice(lease.DeviceUDID); err != nil {
				log.Printf("error releasing ports for device %s: %v", lease.DeviceUDID, err)
			}
		} else {
			if err := h.registry.RenewLease(lease.Port); err != nil {
				log.Printf("error renewing lease for port %d: %v", lease.Port, err)
			}
		}
	}
}
Wire the HealthChecker into your gateway startup

The HealthChecker needs to start when your gateway boots and stop when it shuts down. You will add it right after your Registry initialization in your gateway's main() function.

  • Open your gateway's main() function (or wherever you initialize the server).
  • Find where you create your Registry instance.
  • Update the Registry initialization and add the HealthChecker startup so it looks like this:
// Initialize the port registry with a 5-minute lease TTL
registry := portregistry.New(db, 5*time.Minute)

// Start background health checks every 30 seconds
checker := portregistry.NewHealthChecker(registry, 30*time.Second)
checker.Start()
defer checker.Stop()

What do these values mean?

  • 5*time.Minute is the lease TTL. If your gateway crashes without cleanup, orphaned ports are automatically reclaimed within 5 minutes.
  • 30*time.Second is the health check interval. Every 30 seconds, the checker probes all Appium ports and renews or releases their leases.
  • defer checker.Stop() ensures the background goroutine shuts down cleanly when the gateway process exits.

Why 5 minutes for the TTL?

Appium sessions typically last seconds to minutes. A 5-minute TTL means if the gateway crashes without running cleanup, ports are reclaimed within 5 minutes at most. The HealthChecker renews leases for healthy ports every 30 seconds, so active sessions never expire.

  • Save the file.
  • Rebuild your gateway by running:
go build ./...

You should see no compilation errors.

Build failing after adding the HealthChecker?

  • Make sure you have imported "time" in your gateway's main file. The 5*time.Minute and 30*time.Second literals require it.
  • Verify your import path for the portregistry package matches your module name in go.mod.

Help me fix a build error after wiring the HealthChecker into my gateway

Test crash recovery

The real test of a health checker is whether it detects and recovers from an actual process crash. You will simulate this by killing an Appium process and watching the health checker respond.

  • Start your gateway.
  • Connect a physical device. Your device watcher should allocate three dynamic ports and start Appium.
  • Verify the lease rows exist by running this query against your database:
mysql -u root -p -e "SELECT port, port_type, device_udid FROM port_leases;"

You should see three rows for the connected device: one for appium, one for grid_node, and one for system_port.

  • Find the Appium process ID for that device by running:
lsof -iTCP -sTCP:LISTEN -nP | grep appium
  • Note the PID from the output (the second column).
  • Kill the Appium process to simulate a crash by running:
kill -9 [[APPIUM_PID="the PID from the previous command"]]
  • Watch your gateway's log output. Within 30 seconds, the health checker should log a message that looks like:
UNHEALTHY: appium port <port> for device <udid> is not listening, releasing all ports
  • Verify the lease rows have been removed by querying the database again:
mysql -u root -p -e "SELECT port, port_type, device_udid FROM port_leases;"

The three lease rows for that device should be gone. The device is now available for re-provisioning by the watcher.

Lease rows still there after 30 seconds?

  • Confirm the HealthChecker is actually running. Look for the port health checker started log message in your gateway output at startup.
  • Check that your check() method filters on PortTypeAppium. If the port type stored in the database does not match "appium", the checker will skip those leases.
  • Verify the port you killed is actually dead by running lsof -iTCP -sTCP:LISTEN -nP | grep <port>. If another process grabbed it, the dial succeeds and the checker thinks it is healthy.

Help me debug why my health checker is not releasing dead Appium ports

Secret mission

Port Dashboard API and Conflict Detection

Your registry can allocate ports and recover from crashes. But how do you know what is happening across all your devices right now? In this mission, you will add a real-time port dashboard API and automatic conflict detection to your registry, giving you operational visibility into every port allocation on your device farm.

Clean Up Your Resources

Clean Up Your Resources

This project extends your existing device farm with new Go code and a MySQL table. There are no cloud resources or ongoing costs to worry about.

Resources you used:

  • port_leases table in your MySQL database.
  • portregistry Go package (registry.go, blocklist.go, health.go, api.go).
  • migrations/003_port_leases.sql migration file.

Keep everything running

No action needed. Choose this if your device farm is actively running and you want the PortRegistry and HealthChecker to keep managing ports.

  • The PortRegistry and HealthChecker run as part of your existing gateway process. There are no separate services to manage.
  • The port_leases table self-cleans via the HealthChecker's CleanExpired() calls on every tick.
  • If you notice table bloat over time, you can run OPTIMIZE TABLE port_leases in your MySQL client to reclaim disk space.

Pause - I'll come back to this later

Stop the gateway process to free up resources. Your code and database table stay intact for when you return.

  • Stop your Go gateway process. The HealthChecker stops automatically with it.
  • Any active port leases will remain in the port_leases table but will expire based on their TTL. They cause no harm while the gateway is stopped.
  • When you restart the gateway, the HealthChecker will automatically clean up any stale leases on its first tick.

Delete - I don't want to use this again

Remove all project resources and revert to your previous port management approach.

  • Drop the port_leases table by running this in your MySQL client:
DROP TABLE port_leases;
  • Remove the portregistry package directory by running:
rm -rf portregistry
  • Remove the migration file by running:
rm migrations/003_port_leases.sql
  • Revert your Device Watcher to use the previous PortPool allocation instead of registry.AllocateGroup().
  • Remove the registry.ReleaseByDevice(udid) call from your Device Watcher's disconnect handler.
  • Remove the registry.ReleaseByDevice(udid) call from your Gateway Session Manager's CleanupWorker.
  • Remove the Registry and HealthChecker initialization from your gateway's main() function.
  • Remove the GET /admin/ports and GET /admin/ports/conflicts routes from your Gin router.

Nice Work!

Nice Work!

Nice work! You just built a production-grade dynamic port management system for your Go device farm.

Your device farm can now scale to 40+ devices without port collisions, recover automatically from crashed sessions, and coexist peacefully with Charles Proxy, ADB, and every other tool on your Mac.

You've learned how to:

  • Build a dynamic PortRegistry that uses OS-assigned ephemeral ports via net.Listen("tcp", "127.0.0.1:0"), backed by a MySQL lease table with TTLs that survive process crashes and gateway restarts.
  • Implement atomic group allocation that assigns Appium, Grid Node, and system ports per device with automatic rollback on failure, plus a blocklist of 40+ known developer tool ports to avoid collisions.
  • Create an automated HealthChecker that detects zombie Appium processes via TCP health checks and reclaims their ports for reuse.
  • Secret Mission: Added a port dashboard API with real-time conflict detection for operational visibility.

Ready to quiz yourself?