Build a 6G-Style RL Link Controller

Train a Q-learning controller that adapts a simulated wireless link with Sionna.

Introduction

30 Second Summary

Wireless connections constantly trade speed for stability. A signal that carries more data in clear conditions can become unreliable as the channel weakens.

In this project, you will build a 6G-style adaptive wireless link controller in Google Colab with NVIDIA Sionna. You will train it with Q-learning to choose a modulation scheme as simulated channel quality changes.

What You'll Build

You will run one notebook that shows the controller changing its modulation as simulated channel quality rises and falls.

By the end of this project, you'll have:

  • A wireless link simulation that lets you compare QPSK, 16-QAM, and 64-QAM as channel quality changes.
  • A BER table plus curve set that reveals where each modulation meets or misses the project reliability target.
  • A trained Q-learning policy that selects a modulation for each Eb/N0 state. Benchmark charts compare its reliability-gated goodput with its outage rate against fixed choices.
  • Secret Mission: Retrain the controller with noisy channel-state observations to test whether its policy becomes more robust.

Are there any prerequisites?

You need a Google account plus basic Python experience with lists, loops, functions, and dictionaries. Google Colab can run the experiment on CPU when GPU access is unavailable.

Before We Start

Before the experiment begins, lock in the controller's job so every later result has a clear purpose. It observes an Eb/N0 channel-state bin. It controls the QPSK, 16-QAM, or 64-QAM modulation action. Success means maximizing reliable delivered bits while keeping BER at or below the project target of 0.05.

Set Up the Colab Environment

Your controller goal is locked in. The experiment now needs a stable place to run.

Google Colab provides a hosted runtime that avoids local dependency conflicts. It lets you prove the wireless environment works before you add controller logic.

A pinned package set gives NVIDIA Sionna, PyTorch, and Matplotlib known versions for the experiment.

In this step, get ready to:
  • Create a Colab notebook stored in Google Drive.
  • Configure the runtime for the available device.
  • Prepare the pinned wireless simulation environment.
Create the notebook and choose a runtime

A Colab notebook combines editable code cells with a remote computer. A GPU can accelerate later simulation work.

A CPU remains a working fallback when Colab cannot provide a GPU.

  • Go to Google Colab in your browser.
  • Sign in with your Google account.
  • Create a new notebook from the Colab home screen.

Good start. Your blank notebook is now open from Google Drive with its first code cell ready.

  • Open the Runtime menu.
  • Select Change runtime type.
  • Choose GPU when it appears as an accelerator option.
  • Apply the runtime choice.

The runtime settings now show GPU when Colab made one available. The default CPU runtime is also suitable for this project.

Why does the CPU fallback work?

Sionna can run on a CPU when no GPU is available. The calculations can take longer in later steps, but the experiment remains runnable.

Install the pinned packages

Package pins keep the notebook on the versions used to build this experiment. The setup cell installs 2.1.0 of Sionna PHY, 2.14.0 of PyTorch, and 3.11.2 of Matplotlib.

  • Paste this setup code into the first empty notebook cell:
import subprocess

# Install the exact package versions used by this experiment
subprocess.check_call(
    [
        "pip",
        "install",
        "sionna-no-rt==2.1.0",
        "torch==2.14.0",
        "matplotlib==3.11.2",
    ]
)

What does this setup cell do?

  • The subprocess.check_call function runs the package installation from Python. It stops the cell if the installer reports a failure.
  • Each version pin keeps the experiment on a known package release.

This installation can take a few minutes while Colab downloads the packages. A long stream of installer output is expected during this part.

  • Run the setup cell using the play button at its left edge.
  • Confirm that the cell finishes without a package-install failure.

Package installation failed?

Reconnect the Colab runtime before running the setup cell again. A temporary network interruption can stop a package download.

Help me diagnose the failed Colab package installation.

Verify package versions and the active device

A version check proves that the runtime loaded the intended packages. The device check reveals whether PyTorch can use CUDA in this session.

  • Add a new code cell below the setup cell.
  • Paste this version-check code into the new cell:
from importlib.metadata import version

import matplotlib.pyplot as plt
import torch
import sionna.phy


def print_environment():
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")


print_environment()

What does this version check do?

  • The package imports confirm that the wireless simulation dependencies can load.
  • The version() calls read the installed package metadata.
  • The torch.cuda.is_available() check selects the active device label.

Before you run this check, consider whether Colab reports cuda or cpu for your session.

  • Run the version-check cell using the play button at its left edge.

You should see sionna-no-rt 2.1.0 plus torch 2.14.0 in the output.

You should also see matplotlib 3.11.2. The device line ends in cuda or cpu.

✔️ I see the pinned versions

That is the environment locked down. Your notebook can now import every package needed for the wireless experiment.

  • Continue to the checkpoint below.

ⓧ I see an older version

  • Return to the setup cell.
  • Run the setup cell again.
  • Restart the current Colab runtime from the Runtime menu.
  • Run the version-check cell again.

The repeated check should now report the three pinned versions.

Still seeing another version?

The earlier package may still be loaded in memory. Restarting the runtime clears that loaded copy before the next import.

Help me fix a Colab runtime that keeps loading the wrong package versions.

ⓧ A package is missing

  • Return to the setup cell.
  • Run the setup cell again.
  • Confirm that its installer output finishes without a failure.
  • Run the version-check cell again.

The check should now import all three packages before printing their versions.

Import still failing?

Check the setup cell for a mistyped package name. Confirm that each version pin matches the reference below.

Help me diagnose a missing package in my Colab environment.

✔️ Awesome, I've got everything!

Great. Your notebook contains the setup cell and the version-check cell.

ⓧ I'd like to double check the full code

Your first notebook cell should contain this setup code.

import subprocess

# Install the exact package versions used by this experiment
subprocess.check_call(
    [
        "pip",
        "install",
        "sionna-no-rt==2.1.0",
        "torch==2.14.0",
        "matplotlib==3.11.2",
    ]
)

Your second notebook cell should contain this version-check code.

from importlib.metadata import version

import matplotlib.pyplot as plt
import torch
import sionna.phy


def print_environment():
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")


print_environment()

Your Colab environment is ready for wireless simulation. Next up, you will measure where fixed modulation starts to fail.

Measure Fixed-Modulation BER

Your pinned Google Colab runtime is ready. The next question is how a fixed wireless link behaves as channel quality changes.

Choosing 64-QAM for every transmission promises more bits per symbol. In this step, you will measure bit error rate to test whether that choice stays reliable across weak and strong channel states.

In this step, get ready to:
  • Build fixed QPSK, 16-QAM, and 64-QAM links.
  • Measure BER across eight Eb/N0 states.
  • Plot three reliability curves against the project BER target.
Build the link simulator

The simulator uses NVIDIA Sionna to map random bits onto modulation symbols. An AWGN channel adds noise before the demapper estimates the transmitted bits.

What Does This Channel Model Include?

AWGN adds complex additive noise at each chosen Eb/N0. The later SNR random walk changes Eb/N0 slowly to approximate path loss or shadowing variation.

This experiment leaves out fast Rayleigh fading and Rician fading. It also leaves out Doppler from motion.

  • Add a new code cell below the version-check cell in your notebook.
  • Create wireless_rl.py with the simulator foundation by copying this cell:
%%writefile wireless_rl.py
from importlib.metadata import version
import random

import matplotlib.pyplot as plt
import torch
import sionna.phy
from sionna.phy.channel import AWGN
from sionna.phy.mapping import BinarySource, Constellation, Demapper, Mapper
from sionna.phy.utils import ebnodb2no

SNR_STATES_DB = [-2.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0]
MODULATION_BITS = [2, 4, 6]
MODULATION_NAMES = ["QPSK", "16-QAM", "64-QAM"]
BLOCK_LENGTH = 1200
BATCH_SIZE = 128
TARGET_BER = 0.05
SEED = 42

random.seed(SEED)
torch.manual_seed(SEED)
sionna.phy.config.seed = SEED


def print_environment():
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")

What Does This Foundation Set Up?

  • The imports provide the Sionna channel blocks, modulation blocks, plotting tools, and Eb/N0 conversion utility.
  • The constants define eight channel states, three modulation actions, and the project BER target.
  • The three seeds make repeated experiment runs more consistent.
  • The print_environment() function reports the pinned packages and active compute device.
  • Run the cell.
  • Select the Files panel in Colab's left sidebar.
  • Confirm that wireless_rl.py is listed in the temporary runtime.

Good progress. Your runtime now contains the Python file that holds the wireless experiment.

File Missing From the Runtime?

Run the file-writing cell again. Refresh the Files panel if the new file is not listed.

Make sure %%writefile wireless_rl.py is the first line of the cell.

Help me diagnose why wireless_rl.py was not created in Colab.

Each modulation needs its own constellation, mapper, and demapper. The links share one random bit source and one noise channel so the comparison uses the same simulation structure.

  • Open wireless_rl.py from the Files panel.
  • Place your cursor below print_environment().
  • Add the link-building function by copying this code:
def build_links():
    source = BinarySource()
    channel = AWGN()
    links = []

    for bits_per_symbol in MODULATION_BITS:
        constellation = Constellation("qam", bits_per_symbol)
        mapper = Mapper(constellation=constellation)
        demapper = Demapper("app", constellation=constellation)
        links.append((bits_per_symbol, mapper, demapper))

    return source, channel, links

How Are the Links Built?

  • BinarySource generates the uncoded bits sent through each link.
  • Constellation creates QAM constellations with two, four, or six bits per symbol.
  • Mapper converts each bit group into a complex modulation symbol.
  • Demapper produces bit likelihoods from the noisy received symbols.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Switch back to the notebook from earlier.
  • Add a new code cell below the file-writing cell.
  • Confirm that the simulator foundation runs by copying this code:
exec(open("wireless_rl.py", encoding="utf-8").read())
print_environment()

What Does This Check Prove?

The first line loads the current contents of wireless_rl.py into the notebook session. The second line calls the environment-reporting function from that file.

  • Run the new notebook cell.

You should see sionna-no-rt 2.1.0, torch 2.14.0, and matplotlib 3.11.2 in the output. The final line should report cuda or cpu.

Simulator Foundation Not Loading?

Save wireless_rl.py before running the notebook cell. Check the copied imports for missing lines if Python reports an import or name error.

Help me fix the simulator foundation error.

Measure the BER table

A BER estimate counts how often detected bits differ from transmitted bits. Eb/N0 supplies the channel-quality state for each measurement.

  • Switch back to wireless_rl.py in the file editor.
  • Place your cursor below build_links().
  • Add the BER measurement function by copying this code:
@torch.no_grad()
def measure_ber_table():
    source, channel, links = build_links()
    ber_table = torch.zeros(
        (len(SNR_STATES_DB), len(MODULATION_BITS)),
        dtype=torch.float32,
    )

    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        for action_index, (bits_per_symbol, mapper, demapper) in enumerate(links):
            bits = source([BATCH_SIZE, BLOCK_LENGTH])
            symbols = mapper(bits)
            no = ebnodb2no(
                ebno_db=ebno_db,
                num_bits_per_symbol=bits_per_symbol,
                coderate=1.0,
            )
            received = channel(symbols, no)
            llr = demapper(received, no)
            detected_bits = (llr > 0).to(bits.dtype)
            ber = (bits != detected_bits).float().mean()
            ber_table[state_index, action_index] = ber.cpu()

    return ber_table

How Is BER Measured?

  • torch.no_grad() prevents gradient tracking because this stage only simulates and measures the links.
  • ebnodb2no() converts each Eb/N0 value into the noise level used by the channel.
  • llr > 0 converts positive bit likelihoods into detected ones.
  • ber_table stores one measured error rate for every channel-state and modulation pair.

💡 How Does This Compare With Practical Link Adaptation?

Practical link adaptation usually chooses a modulation and coding scheme against a block error rate target. Outer-loop designs use HARQ ACK or NACK feedback to correct imperfect channel estimates.

Ten percent BLER is a common operating target because it balances throughput and reliability. 3GPP specifications use different targets for different procedures, services, and CQI tables.

BER remains the right metric for this lab because the simulator sends uncoded bits. A meaningful BLER experiment would need coded transport blocks and HARQ behavior.

Grouping uncoded bits into packets would only rename the measurement. It would not create the decoding and retransmission behavior that BLER represents.

  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Confirm that measure_ber_table() appears between build_links() and the end of the file.

Measurement Function Misaligned?

Keep every line inside both loops indented consistently. Make sure return ber_table aligns with the first line inside the function.

Help me check the indentation in measure_ber_table().

A numeric table makes the reliability gap visible at every state. The column labels keep the three modulation results aligned with the same Eb/N0 row.

  • Place your cursor below measure_ber_table().
  • Add the table-printing function by copying this code:
def print_ber_table(ber_table):
    header = "Eb/N0" + "".join(f"{name:>14}" for name in MODULATION_NAMES)
    print("\nMeasured BER")
    print(header)
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        values = "".join(
            f"{ber_table[state_index, action_index].item():14.5f}"
            for action_index in range(len(MODULATION_BITS))
        )
        print(f"{ebno_db:>5.1f}{values}")

How Is the Table Formatted?

  • The header creates one right-aligned column for each modulation name.
  • Each row begins with its Eb/N0 state.
  • Each BER estimate is displayed with five decimal places so small reliability differences remain visible.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Switch back to the notebook from earlier.

Before you run the sweep, which modulation do you expect to lose reliability first as channel quality drops?

  • Measure every link and print the BER table by running this code in a new notebook cell:
exec(open("wireless_rl.py", encoding="utf-8").read())
ber_table = measure_ber_table()
print_ber_table(ber_table)

What Does This Run Do?

The cell reloads the saved simulator before measuring all 24 channel-state and modulation combinations. It then prints the resulting tensor as a labeled table.

You should see a Measured BER table with eight rows. In weaker channel states, 64-QAM should show more errors than the more robust options.

That shortfall is intentional. The highest-rate fixed choice cannot maintain the same reliability across every channel state.

BER Table Missing or Incomplete?

Save the latest version of wireless_rl.py before reloading it. Check that SNR_STATES_DB contains eight values if rows are missing.

Help me diagnose my missing BER table.

Plot the fixed-modulation curves

A logarithmic BER chart makes both large and small error rates readable. The horizontal target line reveals where each fixed modulation meets the experiment's reliability requirement.

  • Switch back to wireless_rl.py in the file editor.
  • Place your cursor below print_ber_table().
  • Add the plotting function by copying this code:
def plot_ber_curves(ber_table):
    plt.figure(figsize=(8, 5))
    for action_index, name in enumerate(MODULATION_NAMES):
        curve = torch.clamp(ber_table[:, action_index], min=1e-6)
        plt.semilogy(SNR_STATES_DB, curve.tolist(), marker="o", label=name)

    plt.axhline(TARGET_BER, color="black", linestyle="--", label="Project BER target")
    plt.xlabel("Eb/N0 (dB)")
    plt.ylabel("Measured BER")
    plt.title("Fixed Modulation Reliability")
    plt.grid(True, which="both", alpha=0.3)
    plt.legend()
    plt.tight_layout()
    plt.show()

What Does the Chart Reveal?

  • torch.clamp() keeps extremely small estimates visible on the logarithmic axis.
  • plt.semilogy() draws one BER curve for each modulation.
  • TARGET_BER supplies the dashed reliability threshold used throughout the experiment.
  • plt.show() displays the finished chart inside the notebook.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Switch back to the notebook from earlier.

Before you run the completed sweep, how do you expect the three curves to change as Eb/N0 rises?

  • Reload the simulator and display the fixed-modulation results by running this code in a new notebook cell:
exec(open("wireless_rl.py", encoding="utf-8").read())
ber_table = measure_ber_table()
print_ber_table(ber_table)
plot_ber_curves(ber_table)

What Should You See?

The notebook prints the BER table before displaying a chart with separate QPSK, 16-QAM, and 64-QAM curves. Each curve should improve as Eb/N0 rises.

The dashed project target makes unreliable channel-state and modulation combinations easy to spot. Small differences between repeated BER estimates are expected because the values come from simulated samples.

🙋‍♀️ Chart Not Appearing?

Confirm that the final line calls plot_ber_curves(ber_table). Save the file again if the function is missing after reloading it.

Check that each plotting line inside plot_ber_curves() uses consistent indentation.

Help me diagnose why my BER chart is not appearing.

You have exposed the fixed-link trade-off. Your simulator now shows how channel quality changes the reliability of all three modulation choices.

✔️ Awesome, I've got everything!

Your BER table and three fixed-modulation curves confirm that the simulator is working. Make sure wireless_rl.py is saved before moving on.

ⓧ I'd like to double check the full code

from importlib.metadata import version
import random

import matplotlib.pyplot as plt
import torch
import sionna.phy
from sionna.phy.channel import AWGN
from sionna.phy.mapping import BinarySource, Constellation, Demapper, Mapper
from sionna.phy.utils import ebnodb2no

SNR_STATES_DB = [-2.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0]
MODULATION_BITS = [2, 4, 6]
MODULATION_NAMES = ["QPSK", "16-QAM", "64-QAM"]
BLOCK_LENGTH = 1200
BATCH_SIZE = 128
TARGET_BER = 0.05
SEED = 42

random.seed(SEED)
torch.manual_seed(SEED)
sionna.phy.config.seed = SEED


def print_environment():
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")


def build_links():
    source = BinarySource()
    channel = AWGN()
    links = []

    for bits_per_symbol in MODULATION_BITS:
        constellation = Constellation("qam", bits_per_symbol)
        mapper = Mapper(constellation=constellation)
        demapper = Demapper("app", constellation=constellation)
        links.append((bits_per_symbol, mapper, demapper))

    return source, channel, links


@torch.no_grad()
def measure_ber_table():
    source, channel, links = build_links()
    ber_table = torch.zeros(
        (len(SNR_STATES_DB), len(MODULATION_BITS)),
        dtype=torch.float32,
    )

    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        for action_index, (bits_per_symbol, mapper, demapper) in enumerate(links):
            bits = source([BATCH_SIZE, BLOCK_LENGTH])
            symbols = mapper(bits)
            no = ebnodb2no(
                ebno_db=ebno_db,
                num_bits_per_symbol=bits_per_symbol,
                coderate=1.0,
            )
            received = channel(symbols, no)
            llr = demapper(received, no)
            detected_bits = (llr > 0).to(bits.dtype)
            ber = (bits != detected_bits).float().mean()
            ber_table[state_index, action_index] = ber.cpu()

    return ber_table


def print_ber_table(ber_table):
    header = "Eb/N0" + "".join(f"{name:>14}" for name in MODULATION_NAMES)
    print("\nMeasured BER")
    print(header)
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        values = "".join(
            f"{ber_table[state_index, action_index].item():14.5f}"
            for action_index in range(len(MODULATION_BITS))
        )
        print(f"{ebno_db:>5.1f}{values}")


def plot_ber_curves(ber_table):
    plt.figure(figsize=(8, 5))
    for action_index, name in enumerate(MODULATION_NAMES):
        curve = torch.clamp(ber_table[:, action_index], min=1e-6)
        plt.semilogy(SNR_STATES_DB, curve.tolist(), marker="o", label=name)

    plt.axhline(TARGET_BER, color="black", linestyle="--", label="Project BER target")
    plt.xlabel("Eb/N0 (dB)")
    plt.ylabel("Measured BER")
    plt.title("Fixed Modulation Reliability")
    plt.grid(True, which="both", alpha=0.3)
    plt.legend()
    plt.tight_layout()
    plt.show()

How to Use This Reference

This reference combines every simulator function added in this step. Compare it with your saved wireless_rl.py file to find missing lines or indentation differences.

Your fixed-modulation benchmark is ready. Next, you will turn these BER measurements into rewards that teach a controller when each modulation is worth using.

Train the Q-Learning Controller

Your Sionna simulator in Google Colab now exposes how each modulation behaves across changing channel conditions.

A fixed modulation cannot balance reliability against rate across every state. In this step, Q-learning turns the measured BER values into a policy that selects one modulation for each observed channel state.

In this step, get ready to:
  • Convert measured BER values into rewards for each state-action pair.
  • Train a controller with epsilon-greedy exploration and Q-table updates.
  • Plot the training reward and learned modulation policy.
Shape the reward signal

A reward tells the controller which actions help its goal. This experiment rewards delivered bits when BER stays within the project target of 0.05.

Actions above that target receive a negative score. This pushes the controller toward faster modulation only when the measured link remains reliable.

  • Select wireless_rl.py in Colab's Files panel.
  • Place your cursor below plot_ber_curves().
  • Add make_reward_table() by pasting this code into the editor:
def make_reward_table(ber_table):
    # Map each modulation action to its raw bits-per-symbol rate
    rates = torch.tensor(MODULATION_BITS, dtype=torch.float32).unsqueeze(0)
    reliable_goodput = rates * (1.0 - ber_table)

    # Credit reliable actions and penalize actions above the BER target
    failure_penalty = -10.0 * ber_table
    return torch.where(
        ber_table <= TARGET_BER,
        reliable_goodput,
        failure_penalty,
    )

What Does This Code Do?

  • The rates tensor assigns each modulation its bits per symbol.
  • The reliable_goodput values credit faster actions according to the bits they deliver.
  • The failure_penalty values make unreliable actions unattractive.
  • The torch.where() call chooses the score according to TARGET_BER.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Press Cmd+F on macOS or Ctrl+F on Windows to open the editor search.
  • Enter def make_reward_table in the search field.
  • Confirm that the editor reports exactly one match.

Seeing More Than One Reward Function?

Delete any repeated copy of make_reward_table() so the file contains one definition. Save the file before returning to the notebook.

Help me remove a duplicate reward function.

  • Load the updated file and display its reward table by running this cell:
exec(open("wireless_rl.py", encoding="utf-8").read())

ber_table = measure_ber_table()
make_reward_table(ber_table)

How Is the Table Organized?

Each row represents one value from SNR_STATES_DB. Each column represents QPSK, 16-QAM, or 64-QAM.

Positive values identify actions that satisfy the project BER target. Negative values expose the actions that fail it.

You should see a tensor with eight rows and three columns. The mix of positive and negative scores gives the controller a useful learning signal.

Reward Table Missing?

Save wireless_rl.py before loading it in the notebook. Confirm that the editor search finds one complete make_reward_table() definition.

If the BER sweep fails, confirm that the pinned Colab runtime from the previous step is still active. Help me debug the reward-table check.

Train the Q-table

A Q-table stores an estimated immediate reward for every channel-state and modulation pair. The controller refines these values across 6000 training episodes.

How Does Exploration Help?

Epsilon-greedy exploration sometimes chooses a random modulation. The remaining decisions use the action with the highest current Q-value.

Epsilon decreases during training. This lets the controller explore early before relying more heavily on what it has learned.

💡 Why Use an Immediate Reward Target?

The selected modulation changes the reward for the current SNR state. It does not change which SNR state comes next.

The random walk evolves independently of the action. The learning problem therefore behaves like a contextual bandit with a tabular value estimate for each context-action pair.

The update uses only the immediate reward, which is equivalent to setting gamma = 0. Bootstrapping from the next state would add value from a transition the controller cannot influence.

  • Return to wireless_rl.py in Colab's file editor.
  • Place your cursor below make_reward_table().
  • Add train_q_learning() by pasting this code into the editor:
def train_q_learning(reward_table, episodes=6000, seed=SEED):
    rng = random.Random(seed)
    q_table = torch.zeros_like(reward_table)
    alpha = 0.2
    state = len(SNR_STATES_DB) // 2
    episode_rewards = []

    for episode in range(episodes):
        epsilon = max(0.05, 1.0 - episode / (0.8 * episodes))

        # Explore early, then favor the highest estimated immediate reward
        if rng.random() < epsilon:
            action = rng.randrange(len(MODULATION_BITS))
        else:
            action = int(torch.argmax(q_table[state]).item())

        # Update this state-action pair from its observed one-step reward
        reward = float(reward_table[state, action].item())
        target = reward
        q_table[state, action] += alpha * (target - q_table[state, action])
        episode_rewards.append(reward)

        # Move the channel independently of the selected modulation
        next_state = state + rng.choice([-1, 0, 1])
        state = max(0, min(len(SNR_STATES_DB) - 1, next_state))

    return q_table, episode_rewards

What Does This Code Do?

  • The q_table starts with one zero-valued estimate for every state-action pair.
  • The epsilon value balances random exploration with the highest current immediate-reward estimate.
  • The target equals the current reward because the action cannot influence the next SNR state.
  • The random walk moves the channel one state lower, keeps it steady, or moves it one state higher after the update.
  • Press Cmd+S on macOS or Ctrl+S on Windows to save wireless_rl.py.
  • Open the editor search with Cmd+F on macOS or Ctrl+F on Windows.
  • Search for def train_q_learning.
  • Confirm that the editor reports exactly one match.

Training Function Missing or Repeated?

Keep one complete train_q_learning() definition below make_reward_table(). Delete any repeated copy before saving the file.

Check that target = reward stays inside the training loop. Help me check the immediate-reward training function.

  • Train the controller and display its eight action indexes by running this cell:
exec(open("wireless_rl.py", encoding="utf-8").read())

reward_table = make_reward_table(ber_table)
q_table, episode_rewards = train_q_learning(reward_table)
torch.argmax(q_table, dim=1)

What Does This Check Prove?

The final expression selects the highest-valued action in each row of q_table. The resulting tensor contains one action index for each of the eight channel states.

An index of 0 represents QPSK. An index of 1 represents 16-QAM. An index of 2 represents 64-QAM.

That is the controller's first complete policy. Every channel state now maps to a learned modulation choice.

Training Cell Failing?

Confirm that reward_table has eight rows and three columns. A different shape cannot align with the controller's state and action indexes.

Rerun the reward-table check if the notebook no longer has ber_table in memory. Help me debug the Q-learning training cell.

Visualize the learned policy

Raw Q-values hide the controller's behavior. A moving reward curve shows the training trend while a state-by-state policy plot reveals which modulation the controller selects.

  • Return to wireless_rl.py in Colab's file editor.
  • Place your cursor below train_q_learning().
  • Add moving_average() by pasting this code into the editor:
def moving_average(values, window=200):
    averages = []
    running_sum = 0.0

    # Maintain one rolling reward sum across the selected window
    for index, value in enumerate(values):
        running_sum += value
        if index >= window:
            running_sum -= values[index - window]

        # Use the available sample count while the window fills
        count = min(index + 1, window)
        averages.append(running_sum / count)

    return averages

Why Smooth the Rewards?

Exploration makes individual episode rewards jump around. moving_average() smooths those jumps across a window of 200 episodes.

The smoothed curve makes the overall training direction easier to inspect.

  • Save wireless_rl.py with Cmd+S on macOS or Ctrl+S on Windows.
  • Open the editor search with Cmd+F on macOS or Ctrl+F on Windows.
  • Search for def moving_average.
  • Confirm that the editor reports exactly one match.

Moving-Average Helper Missing or Repeated?

Keep one moving_average() definition. Check that running_sum begins before the loop and return averages remains outside it.

Help me check the moving-average helper.

  • Place your cursor below moving_average() in wireless_rl.py.
  • Add plot_training_and_policy() by pasting this code into the editor:
def plot_training_and_policy(q_table, episode_rewards):
    # Convert each state row into its highest-valued modulation action
    policy = torch.argmax(q_table, dim=1)
    selected_bits = [MODULATION_BITS[int(action)] for action in policy]

    # Show the reward trend beside the state-by-state policy
    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    axes[0].plot(moving_average(episode_rewards))
    axes[0].set_title("Immediate-Reward Learning")
    axes[0].set_xlabel("Episode")
    axes[0].set_ylabel("Moving-average reward")
    axes[0].grid(True, alpha=0.3)

    axes[1].step(SNR_STATES_DB, selected_bits, where="mid", marker="o")
    axes[1].set_title("Learned Modulation Policy")
    axes[1].set_xlabel("Eb/N0 (dB)")
    axes[1].set_ylabel("Selected modulation")
    axes[1].set_yticks(MODULATION_BITS, MODULATION_NAMES)
    axes[1].grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()
    return policy

What Do the Plots Reveal?

The first chart uses Matplotlib to show the smoothed reward across training episodes. Its overall movement reveals whether the controller is finding more valuable actions.

The second chart converts each action index into bits per symbol. This makes every learned modulation choice visible across the eight Eb/N0 states.

  • Save wireless_rl.py.
  • Open the editor search with Cmd+F on macOS or Ctrl+F on Windows.
  • Search for def plot_training_and_policy.
  • Confirm that the editor reports exactly one match.

Plotting Function Missing or Repeated?

Keep one plot_training_and_policy() definition after moving_average(). Delete any repeated copy before saving.

Confirm that the function title reads Immediate-Reward Learning and that it returns policy. Help me check the plotting function.

  • Place your cursor below plot_training_and_policy() in wireless_rl.py.
  • Add print_policy() by pasting this code into the editor:
def print_policy(policy):
    print("\nLearned policy")

    # Translate each stored action index into its modulation label
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        action = int(policy[state_index].item())
        print(f"Eb/N0 {ebno_db:>5.1f} dB -> {MODULATION_NAMES[action]}")

What Does This Code Do?

The function walks through the eight Eb/N0 states. It translates each stored action index into its modulation name.

The printed table gives you a readable version of the same policy shown in the chart.

  • Save wireless_rl.py.
  • Open the editor search with Cmd+F on macOS or Ctrl+F on Windows.
  • Search for def print_policy.
  • Confirm that the editor reports exactly one match.

Policy Printer Missing or Repeated?

Keep one print_policy() definition. Confirm that its for loop and final print line remain inside the function.

Help me check the policy printer.

The next step extends the script's benchmark. A complete entry point now gives that extension a stable place to connect.

  • Place your cursor below print_policy() in wireless_rl.py.
  • Add the baseline entry point by pasting this code into the editor:
def main():
    # Rebuild the measured link outcomes for this run
    print_environment()
    ber_table = measure_ber_table()
    print_ber_table(ber_table)
    plot_ber_curves(ber_table)

    # Train and display the immediate-reward policy
    reward_table = make_reward_table(ber_table)
    q_table, episode_rewards = train_q_learning(reward_table)
    learned_policy = plot_training_and_policy(q_table, episode_rewards)
    print_policy(learned_policy)


if __name__ == "__main__":
    main()

What Does the Baseline Harness Run?

  • The first section rebuilds the BER table and fixed-modulation curves.
  • The second section trains the immediate-reward controller and displays its policy.
  • The __main__ guard runs the experiment when the file executes directly.
  • Save wireless_rl.py.
  • Search for def main in the editor.
  • Confirm that the editor reports exactly one match.
  • Search for if __name__ == "__main__":.
  • Confirm that the guard appears once at the end of the file.

Entry Point Missing or Repeated?

Keep one main() definition and one __main__ guard. Place the guard at the far-left indentation level after the function.

Help me check the baseline entry point.

✔️ Awesome, I've got everything!

Your controller functions and baseline entry point are now saved in wireless_rl.py. Each definition appears once, so notebook cells only run the saved file.

ⓧ I'd like to double check the full code

Part 1: Simulator and BER measurement

from importlib.metadata import version
import random

import matplotlib.pyplot as plt
import torch
import sionna.phy
from sionna.phy.channel import AWGN
from sionna.phy.mapping import BinarySource, Constellation, Demapper, Mapper
from sionna.phy.utils import ebnodb2no

# Define the experiment's channel states, actions, and sampling settings
SNR_STATES_DB = [-2.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0]
MODULATION_BITS = [2, 4, 6]
MODULATION_NAMES = ["QPSK", "16-QAM", "64-QAM"]
BLOCK_LENGTH = 1200
BATCH_SIZE = 128
TARGET_BER = 0.05
SEED = 42

# Seed each random source for more reproducible measurements
random.seed(SEED)
torch.manual_seed(SEED)
sionna.phy.config.seed = SEED


def print_environment():
    # Report the packages and compute device used for this run
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")


def build_links():
    # Share one bit source and AWGN channel across all modulations
    source = BinarySource()
    channel = AWGN()
    links = []

    # Build one mapper and demapper for each modulation order
    for bits_per_symbol in MODULATION_BITS:
        constellation = Constellation("qam", bits_per_symbol)
        mapper = Mapper(constellation=constellation)
        demapper = Demapper("app", constellation=constellation)
        links.append((bits_per_symbol, mapper, demapper))

    return source, channel, links


@torch.no_grad()
def measure_ber_table():
    source, channel, links = build_links()
    ber_table = torch.zeros(
        (len(SNR_STATES_DB), len(MODULATION_BITS)),
        dtype=torch.float32,
    )

    # Measure every channel-state and modulation combination
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        for action_index, (bits_per_symbol, mapper, demapper) in enumerate(links):
            bits = source([BATCH_SIZE, BLOCK_LENGTH])
            symbols = mapper(bits)
            no = ebnodb2no(
                ebno_db=ebno_db,
                num_bits_per_symbol=bits_per_symbol,
                coderate=1.0,
            )
            received = channel(symbols, no)
            llr = demapper(received, no)
            detected_bits = (llr > 0).to(bits.dtype)
            ber = (bits != detected_bits).float().mean()
            ber_table[state_index, action_index] = ber.cpu()

    return ber_table


def print_ber_table(ber_table):
    # Format one row per channel state and one column per modulation
    header = "Eb/N0" + "".join(f"{name:>14}" for name in MODULATION_NAMES)
    print("\nMeasured BER")
    print(header)
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        values = "".join(
            f"{ber_table[state_index, action_index].item():14.5f}"
            for action_index in range(len(MODULATION_BITS))
        )
        print(f"{ebno_db:>5.1f}{values}")


def plot_ber_curves(ber_table):
    # Plot each fixed modulation on a logarithmic BER scale
    plt.figure(figsize=(8, 5))
    for action_index, name in enumerate(MODULATION_NAMES):
        curve = torch.clamp(ber_table[:, action_index], min=1e-6)
        plt.semilogy(SNR_STATES_DB, curve.tolist(), marker="o", label=name)

    plt.axhline(
        TARGET_BER,
        color="black",
        linestyle="--",
        label="Project BER target",
    )
    plt.xlabel("Eb/N0 (dB)")
    plt.ylabel("Measured BER")
    plt.title("Fixed Modulation Reliability")
    plt.grid(True, which="both", alpha=0.3)
    plt.legend()
    plt.tight_layout()
    plt.show()

Part 2: Controller and entry point

def make_reward_table(ber_table):
    # Map each modulation action to its raw bits-per-symbol rate
    rates = torch.tensor(MODULATION_BITS, dtype=torch.float32).unsqueeze(0)
    reliable_goodput = rates * (1.0 - ber_table)

    # Credit reliable actions and penalize actions above the BER target
    failure_penalty = -10.0 * ber_table
    return torch.where(
        ber_table <= TARGET_BER,
        reliable_goodput,
        failure_penalty,
    )


def train_q_learning(reward_table, episodes=6000, seed=SEED):
    rng = random.Random(seed)
    q_table = torch.zeros_like(reward_table)
    alpha = 0.2
    state = len(SNR_STATES_DB) // 2
    episode_rewards = []

    for episode in range(episodes):
        epsilon = max(0.05, 1.0 - episode / (0.8 * episodes))

        # Explore early, then favor the highest estimated immediate reward
        if rng.random() < epsilon:
            action = rng.randrange(len(MODULATION_BITS))
        else:
            action = int(torch.argmax(q_table[state]).item())

        # Update this state-action pair from its observed one-step reward
        reward = float(reward_table[state, action].item())
        target = reward
        q_table[state, action] += alpha * (target - q_table[state, action])
        episode_rewards.append(reward)

        # Move the channel independently of the selected modulation
        next_state = state + rng.choice([-1, 0, 1])
        state = max(0, min(len(SNR_STATES_DB) - 1, next_state))

    return q_table, episode_rewards


def moving_average(values, window=200):
    averages = []
    running_sum = 0.0

    # Maintain one rolling reward sum across the selected window
    for index, value in enumerate(values):
        running_sum += value
        if index >= window:
            running_sum -= values[index - window]

        # Use the available sample count while the window fills
        count = min(index + 1, window)
        averages.append(running_sum / count)

    return averages


def plot_training_and_policy(q_table, episode_rewards):
    # Convert each state row into its highest-valued modulation action
    policy = torch.argmax(q_table, dim=1)
    selected_bits = [MODULATION_BITS[int(action)] for action in policy]

    # Show the reward trend beside the state-by-state policy
    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    axes[0].plot(moving_average(episode_rewards))
    axes[0].set_title("Immediate-Reward Learning")
    axes[0].set_xlabel("Episode")
    axes[0].set_ylabel("Moving-average reward")
    axes[0].grid(True, alpha=0.3)

    axes[1].step(SNR_STATES_DB, selected_bits, where="mid", marker="o")
    axes[1].set_title("Learned Modulation Policy")
    axes[1].set_xlabel("Eb/N0 (dB)")
    axes[1].set_ylabel("Selected modulation")
    axes[1].set_yticks(MODULATION_BITS, MODULATION_NAMES)
    axes[1].grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()
    return policy


def print_policy(policy):
    print("\nLearned policy")

    # Translate each stored action index into its modulation label
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        action = int(policy[state_index].item())
        print(f"Eb/N0 {ebno_db:>5.1f} dB -> {MODULATION_NAMES[action]}")


def main():
    # Rebuild the measured link outcomes for this run
    print_environment()
    ber_table = measure_ber_table()
    print_ber_table(ber_table)
    plot_ber_curves(ber_table)

    # Train and display the immediate-reward policy
    reward_table = make_reward_table(ber_table)
    q_table, episode_rewards = train_q_learning(reward_table)
    learned_policy = plot_training_and_policy(q_table, episode_rewards)
    print_policy(learned_policy)


if __name__ == "__main__":
    main()

The final run repeats the BER sweep before training the controller. Colab may take a little time to produce both plots.

Before you run this, do you expect one modulation to work best in every channel state?

  • Run the complete training check with this cell:
exec(open("wireless_rl.py", encoding="utf-8").read())

ber_table = measure_ber_table()
reward_table = make_reward_table(ber_table)
q_table, episode_rewards = train_q_learning(reward_table)
learned_policy = plot_training_and_policy(q_table, episode_rewards)
print_policy(learned_policy)

What Does the Final Check Run?

  • The BER sweep rebuilds the measured link outcomes used by the reward table.
  • The training function produces q_table and episode_rewards.
  • The plotting function derives learned_policy with the highest-valued action for each state.
  • The policy printer converts those action indexes into modulation names.

You should see a learned policy for all eight Eb/N0 states. You should also see a reward chart beside a policy chart that uses at least two modulation schemes.

Strong result. Your controller now changes its modulation choice as the simulated channel moves between weaker and stronger states.

Only Seeing One Modulation?

Confirm that you trained from the measured reward_table produced by the current BER sweep. Check that TARGET_BER remains 0.05.

Confirm that the random-walk update can reach every index from 0 through 7. Help me investigate why my learned policy uses only one modulation.

Your adaptive controller is trained and visible. Next, you will benchmark its reliability-rate trade-off against three fixed-modulation policies.

Benchmark Adaptive and Fixed Links

Your Q-learning controller already chooses a modulation for each channel state. Its policy plot shows that the agent uses multiple modulation schemes.

A learned policy only matters when you compare its reliability-rate trade-off with simple fixed choices. This step tests every policy on the same changing channel trace.

In this step, get ready to:
  • Generate one reproducible random-walk channel trace.
  • Score each policy using reliability-gated goodput plus outage rate.
  • Compare the learned controller with three fixed-modulation baselines.
Generate one shared channel trace

A fair benchmark gives every policy the same sequence of channel states. The random-walk trace models channel quality moving between neighboring Eb/N0 states.

  • Switch back to wireless_rl.py in the Colab editor.
  • Add generate_channel_trace() below print_policy() by pasting this code:
def generate_channel_trace(length=500, seed=7):
    rng = random.Random(seed)
    state = len(SNR_STATES_DB) // 2
    trace = []

    for _ in range(length):
        trace.append(state)
        state += rng.choice([-1, 0, 1])
        state = max(0, min(len(SNR_STATES_DB) - 1, state))

    return trace

What does this code do?

  • The local random generator makes the channel sequence reproducible.
  • The trace starts in the middle of the eight channel states.
  • Each update moves down one state, stays still, or moves up one state.
  • The bounds keep every state inside SNR_STATES_DB.
  • Save wireless_rl.py.
  • Confirm the editor now shows generate_channel_trace() directly below print_policy().

Trace function in the wrong place?

Check that generate_channel_trace() begins at the far-left indentation level. A nested definition would keep the benchmark from calling it.

Help me check the placement and indentation of generate_channel_trace().

Score reliability and delivered rate

Reliability-gated goodput credits delivered bits only when the measured BER meets the project target. Outage records how often a policy selects a modulation above that target.

  • Add score_policy() below generate_channel_trace() by pasting this code:
def score_policy(ber_table, actions_by_state, trace):
    delivered_rate = 0.0
    outages = 0

    for state in trace:
        action = int(actions_by_state[state])
        ber = float(ber_table[state, action].item())

        if ber <= TARGET_BER:
            delivered_rate += MODULATION_BITS[action] * (1.0 - ber)
        else:
            outages += 1

    return {
        "goodput": delivered_rate / len(trace),
        "outage": outages / len(trace),
    }

How are the policies scored?

  • The selected action chooses one BER value for each true channel state.
  • A reliable state contributes its modulation rate adjusted by measured BER.
  • An unreliable state increases the outage count.
  • The returned averages make policies comparable across the shared trace.
  • Save wireless_rl.py.

Before you run the script, do you expect these two new functions to change the existing policy output yet?

  • Check that the updated file remains runnable by executing this in a notebook code cell:
exec(open("wireless_rl.py", encoding="utf-8").read())

What does this check prove?

This executes the current contents of wireless_rl.py. A complete run confirms that both new function definitions have valid syntax.

You will see the BER table, fixed-modulation curves, learned policy, plus the training plots from earlier. The benchmark output arrives after you connect the new functions to main().

Seeing a Python error?

Check the indentation inside both new functions. Make sure each return line remains inside its function.

Help me debug the new trace or scoring function.

Benchmark learned and fixed policies

The benchmark uses one trace for all four controllers. This isolates policy behavior because every controller faces the same channel changes.

  • Add the first version of benchmark_policies() below score_policy() by pasting this code:
def benchmark_policies(ber_table, learned_policy):
    trace = generate_channel_trace()
    policies = {
        "Learned RL": learned_policy.tolist(),
        "Fixed QPSK": [0] * len(SNR_STATES_DB),
        "Fixed 16-QAM": [1] * len(SNR_STATES_DB),
        "Fixed 64-QAM": [2] * len(SNR_STATES_DB),
    }
    metrics = {
        name: score_policy(ber_table, policy, trace)
        for name, policy in policies.items()
    }

    print("\nPolicy benchmark")
    for name, result in metrics.items():
        print(
            f"{name:>13}: goodput={result['goodput']:.3f} bits/symbol, "
            f"outage={100 * result['outage']:.1f}%"
        )

    return metrics

What does this benchmark compare?

  • The learned policy uses its chosen action for each Eb/N0 state.
  • Each fixed policy repeats one modulation action across all eight states.
  • The metrics dictionary scores every policy against the same trace.
  • The printed table exposes the trade-off between delivered rate and outage.
  • Save wireless_rl.py.
  • Scroll to main() near the bottom of the file.
  • Find this line:
    print_policy(learned_policy)

Where does the benchmark start?

This line currently ends the learned-policy workflow. The benchmark call belongs immediately below it because it reuses ber_table plus learned_policy.

  • Replace that line with this expanded section:
    print_policy(learned_policy)

    benchmark_policies(ber_table, learned_policy)

How is the benchmark connected?

The first line keeps the learned-policy output from the previous step. The new call sends the measured BER table plus the learned actions into the benchmark.

  • Save wireless_rl.py.

Before you run the updated script, which fixed modulation do you think will have the highest outage?

  • Run the benchmark in a notebook code cell by executing:
exec(open("wireless_rl.py", encoding="utf-8").read())

What does this run reveal?

The script now reaches benchmark_policies() after training. It prints one goodput value plus one outage percentage for every controller.

You will see a new Policy benchmark table with rows for Learned RL, Fixed QPSK, Fixed 16-QAM, and Fixed 64-QAM.

Missing the benchmark table?

Check that the benchmark_policies(ber_table, learned_policy) call sits inside main(). It should use the same four-space indentation as print_policy(learned_policy).

Help me find why the policy benchmark is not printing.

The table makes the trade-off measurable. Charts make the same comparison easier to present.

  • Inside benchmark_policies(), find return metrics.
  • Insert this plotting section directly above that line:
    names = list(metrics)
    goodput = [metrics[name]["goodput"] for name in names]
    outage = [100 * metrics[name]["outage"] for name in names]

    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    axes[0].bar(names, goodput)
    axes[0].set_title("Reliability-Gated Goodput")
    axes[0].set_ylabel("Bits per symbol")
    axes[0].tick_params(axis="x", rotation=20)

    axes[1].bar(names, outage)
    axes[1].set_title("Reliability Outage")
    axes[1].set_ylabel("Trace states above BER target (%)")
    axes[1].tick_params(axis="x", rotation=20)

    plt.tight_layout()
    plt.show()

What do the charts show?

  • The left chart compares reliability-gated goodput in bits per symbol.
  • The right chart compares the percentage of trace states above the BER target.
  • Matching policy names keep both views aligned with the printed table.
  • Save wireless_rl.py.

Before the final run, which controller do you expect to give the strongest balance of goodput plus low outage on this trace?

  • Run the completed experiment in a notebook code cell by executing:
exec(open("wireless_rl.py", encoding="utf-8").read())

What confirms the benchmark works?

The completed script prints metrics for all four policies. It also displays separate goodput and outage bar charts.

Compare the learned controller's two bars with every fixed baseline. The strongest balance comes from combining useful goodput with fewer states above the project BER target.

You have closed the evaluation loop. Your controller now has a reproducible benchmark against three simple alternatives.

Charts missing after the table?

Check that the plotting section sits inside benchmark_policies() before return metrics. Every plotting line should keep four spaces of indentation.

Help me debug the missing goodput or outage charts.

✔️ Awesome, I've got everything!

Great work. Make sure wireless_rl.py is saved before you capture your benchmark.

ⓧ I'd like to double check the full code

Part 1: Simulator and controller

from importlib.metadata import version
import random

import matplotlib.pyplot as plt
import torch
import sionna.phy
from sionna.phy.channel import AWGN
from sionna.phy.mapping import BinarySource, Constellation, Demapper, Mapper
from sionna.phy.utils import ebnodb2no

SNR_STATES_DB = [-2.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0]
MODULATION_BITS = [2, 4, 6]
MODULATION_NAMES = ["QPSK", "16-QAM", "64-QAM"]
BLOCK_LENGTH = 1200
BATCH_SIZE = 128
TARGET_BER = 0.05
SEED = 42

random.seed(SEED)
torch.manual_seed(SEED)
sionna.phy.config.seed = SEED


def print_environment():
    active_device = "cuda" if torch.cuda.is_available() else "cpu"
    print(f"sionna-no-rt {version('sionna-no-rt')}")
    print(f"torch {torch.__version__}")
    print(f"matplotlib {version('matplotlib')}")
    print(f"device {active_device}")


def build_links():
    source = BinarySource()
    channel = AWGN()
    links = []

    for bits_per_symbol in MODULATION_BITS:
        constellation = Constellation("qam", bits_per_symbol)
        mapper = Mapper(constellation=constellation)
        demapper = Demapper("app", constellation=constellation)
        links.append((bits_per_symbol, mapper, demapper))

    return source, channel, links


@torch.no_grad()
def measure_ber_table():
    source, channel, links = build_links()
    ber_table = torch.zeros(
        (len(SNR_STATES_DB), len(MODULATION_BITS)),
        dtype=torch.float32,
    )

    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        for action_index, (bits_per_symbol, mapper, demapper) in enumerate(links):
            bits = source([BATCH_SIZE, BLOCK_LENGTH])
            symbols = mapper(bits)
            no = ebnodb2no(
                ebno_db=ebno_db,
                num_bits_per_symbol=bits_per_symbol,
                coderate=1.0,
            )
            received = channel(symbols, no)
            llr = demapper(received, no)
            detected_bits = (llr > 0).to(bits.dtype)
            ber = (bits != detected_bits).float().mean()
            ber_table[state_index, action_index] = ber.cpu()

    return ber_table


def print_ber_table(ber_table):
    header = "Eb/N0" + "".join(f"{name:>14}" for name in MODULATION_NAMES)
    print("\nMeasured BER")
    print(header)
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        values = "".join(
            f"{ber_table[state_index, action_index].item():14.5f}"
            for action_index in range(len(MODULATION_BITS))
        )
        print(f"{ebno_db:>5.1f}{values}")


def plot_ber_curves(ber_table):
    plt.figure(figsize=(8, 5))
    for action_index, name in enumerate(MODULATION_NAMES):
        curve = torch.clamp(ber_table[:, action_index], min=1e-6)
        plt.semilogy(SNR_STATES_DB, curve.tolist(), marker="o", label=name)

    plt.axhline(TARGET_BER, color="black", linestyle="--", label="Project BER target")
    plt.xlabel("Eb/N0 (dB)")
    plt.ylabel("Measured BER")
    plt.title("Fixed Modulation Reliability")
    plt.grid(True, which="both", alpha=0.3)
    plt.legend()
    plt.tight_layout()
    plt.show()


def make_reward_table(ber_table):
    rates = torch.tensor(MODULATION_BITS, dtype=torch.float32).unsqueeze(0)
    reliable_goodput = rates * (1.0 - ber_table)
    failure_penalty = -10.0 * ber_table
    return torch.where(
        ber_table <= TARGET_BER,
        reliable_goodput,
        failure_penalty,
    )


def train_q_learning(reward_table, episodes=6000, seed=SEED):
    rng = random.Random(seed)
    q_table = torch.zeros_like(reward_table)
    alpha = 0.2
    gamma = 0.9
    state = len(SNR_STATES_DB) // 2
    episode_rewards = []

    for episode in range(episodes):
        epsilon = max(0.05, 1.0 - episode / (0.8 * episodes))

        if rng.random() < epsilon:
            action = rng.randrange(len(MODULATION_BITS))
        else:
            action = int(torch.argmax(q_table[state]).item())

        next_state = state + rng.choice([-1, 0, 1])
        next_state = max(0, min(len(SNR_STATES_DB) - 1, next_state))

        reward = float(reward_table[state, action].item())
        target = reward + gamma * float(torch.max(q_table[next_state]).item())
        q_table[state, action] += alpha * (target - q_table[state, action])

        episode_rewards.append(reward)
        state = next_state

    return q_table, episode_rewards


def moving_average(values, window=200):
    averages = []
    running_sum = 0.0

    for index, value in enumerate(values):
        running_sum += value
        if index >= window:
            running_sum -= values[index - window]
        count = min(index + 1, window)
        averages.append(running_sum / count)

    return averages


def plot_training_and_policy(q_table, episode_rewards):
    policy = torch.argmax(q_table, dim=1)
    selected_bits = [MODULATION_BITS[int(action)] for action in policy]

    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    axes[0].plot(moving_average(episode_rewards))
    axes[0].set_title("Q-Learning Reward")
    axes[0].set_xlabel("Episode")
    axes[0].set_ylabel("Moving-average reward")
    axes[0].grid(True, alpha=0.3)

    axes[1].step(SNR_STATES_DB, selected_bits, where="mid", marker="o")
    axes[1].set_title("Learned Modulation Policy")
    axes[1].set_xlabel("Eb/N0 (dB)")
    axes[1].set_ylabel("Selected modulation")
    axes[1].set_yticks(MODULATION_BITS, MODULATION_NAMES)
    axes[1].grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()
    return policy


def print_policy(policy):
    print("\nLearned policy")
    for state_index, ebno_db in enumerate(SNR_STATES_DB):
        action = int(policy[state_index].item())
        print(f"Eb/N0 {ebno_db:>5.1f} dB -> {MODULATION_NAMES[action]}")

Part 2: Benchmark and entry point

def generate_channel_trace(length=500, seed=7):
    rng = random.Random(seed)
    state = len(SNR_STATES_DB) // 2
    trace = []

    for _ in range(length):
        trace.append(state)
        state += rng.choice([-1, 0, 1])
        state = max(0, min(len(SNR_STATES_DB) - 1, state))

    return trace


def score_policy(ber_table, actions_by_state, trace):
    delivered_rate = 0.0
    outages = 0

    for state in trace:
        action = int(actions_by_state[state])
        ber = float(ber_table[state, action].item())

        if ber <= TARGET_BER:
            delivered_rate += MODULATION_BITS[action] * (1.0 - ber)
        else:
            outages += 1

    return {
        "goodput": delivered_rate / len(trace),
        "outage": outages / len(trace),
    }


def benchmark_policies(ber_table, learned_policy):
    trace = generate_channel_trace()
    policies = {
        "Learned RL": learned_policy.tolist(),
        "Fixed QPSK": [0] * len(SNR_STATES_DB),
        "Fixed 16-QAM": [1] * len(SNR_STATES_DB),
        "Fixed 64-QAM": [2] * len(SNR_STATES_DB),
    }
    metrics = {
        name: score_policy(ber_table, policy, trace)
        for name, policy in policies.items()
    }

    print("\nPolicy benchmark")
    for name, result in metrics.items():
        print(
            f"{name:>13}: goodput={result['goodput']:.3f} bits/symbol, "
            f"outage={100 * result['outage']:.1f}%"
        )

    names = list(metrics)
    goodput = [metrics[name]["goodput"] for name in names]
    outage = [100 * metrics[name]["outage"] for name in names]

    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
    axes[0].bar(names, goodput)
    axes[0].set_title("Reliability-Gated Goodput")
    axes[0].set_ylabel("Bits per symbol")
    axes[0].tick_params(axis="x", rotation=20)

    axes[1].bar(names, outage)
    axes[1].set_title("Reliability Outage")
    axes[1].set_ylabel("Trace states above BER target (%)")
    axes[1].tick_params(axis="x", rotation=20)

    plt.tight_layout()
    plt.show()
    return metrics


def main():
    print_environment()
    ber_table = measure_ber_table()
    print_ber_table(ber_table)
    plot_ber_curves(ber_table)

    reward_table = make_reward_table(ber_table)
    q_table, episode_rewards = train_q_learning(reward_table)
    learned_policy = plot_training_and_policy(q_table, episode_rewards)
    print_policy(learned_policy)

    benchmark_policies(ber_table, learned_policy)


if __name__ == "__main__":
    main()

How to use this reference

This reference shows the completed simulator, controller, plus benchmark in one file. Compare function placement plus indentation with your saved copy.

Secret mission

Train Through Noisy Channel Feedback

Perfect channel reports make the controller's job easier than a real wireless link would. Train a second policy with perturbed channel observations to test whether noisy training improves its reliability-rate trade-off.

Clean Up Your Resources

Clean Up Your Resources

This experiment uses Google Colab's free resources. Choose whether to keep your work, pause the active runtime, or delete the project.

Resources you used:

  • Google Drive: One notebook containing your experiment cells and results.
  • Google Colab runtime: The active session containing wireless_rl.py and noisy_csi.py.

Keep everything running

No action needed. Choose this if you are still experimenting with the controller or comparing its results.

  • Keep the notebook saved in Google Drive so its cells and outputs remain available.
  • Download wireless_rl.py from the active Colab runtime's file browser if you want a local copy.
  • Download noisy_csi.py from the active Colab runtime's file browser if you want to preserve the Secret Mission extension.

Colab can reclaim free runtimes. Your downloaded files protect the Python code if the temporary session disappears.

Pause - I'll come back to this later

This option stops the active session while keeping the notebook in Google Drive. Temporary runtime files can disappear after disconnection.

  • Download wireless_rl.py from the active Colab runtime's file browser.
  • Download noisy_csi.py from the active Colab runtime's file browser.
  • Disconnect the active runtime using the connection controls in the notebook header.
  • Confirm the runtime indicator no longer shows an active connection.

Your notebook remains available in Google Drive. You can start a new runtime when you return.

Delete - I don't want to use this again

Deleting the notebook removes your saved experiment cells and outputs. Downloaded copies remain available if you decide to keep them.

  • Download wireless_rl.py from the active Colab runtime if you want to keep a copy.
  • Download noisy_csi.py from the active Colab runtime if you want to keep the Secret Mission extension.
  • Locate the project notebook in Google Drive.
  • Delete the project notebook from Google Drive.
  • Switch back to the active Google Colab notebook from earlier.
  • Delete wireless_rl.py using the temporary runtime's file browser.
  • Delete noisy_csi.py using the temporary runtime's file browser.
  • Disconnect the active runtime using the connection controls in the notebook header.

You should see that no runtime is connected. No cloud infrastructure from this project remains running.

Nice Work!

Nice Work!

You made it! Your Google Colab notebook now runs a complete adaptive wireless-link experiment.

The experiment turns changing channel quality into visible reliability curves, a learned modulation policy, and side-by-side performance results.

You've learned how to:

  • Built an NVIDIA Sionna PHY simulation for QPSK, 16-QAM, and 64-QAM transmission through an AWGN channel.
  • Measured how bit error rate changes across the configured Eb/N0 states. Displayed separate curves that expose where each fixed modulation meets the project reliability target.
  • Trained a tabular Q-learning controller to select a modulation for each channel state. Benchmarked its reliability-gated goodput and outage rate against fixed QPSK, 16-QAM, and 64-QAM policies.
  • Secret Mission: Extended the controller to train with noisy channel-state observations. Compared the clean-trained policy with the noise-trained policy on the same changing channel trace.

Ready to quiz yourself?