Exponential Tilt

Because KL divergence increases with $\beta$, we solve the scalar equation at exactly the point we run out of budget. To get the exact right level, we can do this by looking for $\beta > 0$ such that

That's where the unique $\beta$ will result in our final solution $\beta^*$. Incorporated into the equation

Where $\beta^*$ is selected so that $D_{KL}(p(\beta^*) \,\|\, q) = C$

What this Equation Represents

Now that we have this cool equation, what the heck does it mean? In this section, I am going to break down each variable within the equation as it relates to our original problem and explain what happens when you tweak it.

$p_i^*$ is the optimal probability for phrase $t_i$ and the collection of all $p_i^*$ results in the $p^*$ distribution of phrases. It can be thought of as the perfect balance between resembling ordinary traffic and favoring useful phrases (those $t_i$ with high payoffs, $s_i$).

$q_i$ is the frequency that a phrase $t_i$ typically appears according to our baseline $q$. It is how often the phrase typically appears in a regular stream of messages.

$s_i$ is the success probability for the use of a phrase $t_i$ in a message.

$\beta$ is the same as $\frac{1}{\lambda}$ and is the "aggression dial" for the whole distribution. When $\beta$ is 0, $p = q$ which means the distribution looks exactly like normal traffic. Great for hiding your attacks, but not the best you can do when it comes to achieving high payoffs.

We can put together two variables ($\beta$, $s_i$) to get the factor that rewards more effective phrases. This looks like $e^{\beta s_i}$.

$q_i e^{\beta s_i}$ — Adding in $q_i$ allows us to multiply our baseline probability by those payoff preferences. This is the exponential tilt where this type of equation gets its name from. Among all distributions with limited KL divergence from $q$, the exponential tilt is the most efficient way to increase expected payoff. We are effectively spending the budget on favoring high-value phrases.

$Z(\beta)$ is our normalization constant that gets all the probabilities in the distribution to sum to 1.

$C$ is the constant that sets the whole thing in motion. In other words, it is the detectability budget our attacker can use to maximize their payoff.

What This Means for Defenders Like Doppel

While attackers can manipulate $p_i$ for any of their phrases, $t_i$, there are only three levers that defenders can control, $T$, $q$, and $C$ within this toy model. In this section, we will walk through each and talk about the costs of Type I and Type II errors.

$q$ is our modeled expectations for how often we will see any of the loaded phrases from $t$ in regular, benign traffic. A naive approach to building $q$ is to sample how often each phrase in $T = \{t_1, t_2, \ldots, t_n\}$ occurs within your current dataset.

A slightly less naive approach curates the dataset by filtering out all the phishing attempts that likely already exist within it (shout out to data labelers who do this kind of thankless work day in and day out!) This results in a clean $q$ model of how often those phrases should appear.

This assumes we have a good handle on most of $T$. If we never think to add in a phrase $t_{\mathrm{unk}}$, (in other words $t_{\mathrm{unk}} \notin T$) our detector could never trigger and attackers will just hammer that phrase until they succeed in their goal.

As defenders, we can adjust the $q_i$ for any individual phrase in hopes of more quickly catching divergences that arise from the $p$ distribution we will see in production. We can also adjust all of $q$ by calibrating our dataset, smoothing, additional labeling, or selecting new/better datasets. By appropriately setting $q$, we can effectively shape the attacker’s optimal response $p^*$ to catch bad actors and estimate the $\hat{p}$ that we will see. But this comes at a cost (more on this in a moment). A good intuition for the effects of manipulating $q$ is that setting $q_i = 0$ means that any instance of $t_i$ will trigger the alert (KL divergence would be infinite and always past the threshold of $C$.) It is probably good to note here that the whole of $q$ still sums to 1, so lowering $q_i$ moves the center of the KL ball, essentially making phrase $t_i$ more anomalous. It in effect shifts sensitivity for that phrase.

We can also manipulate $C$, or the amount of divergence we allow before we trigger an alert on a given distribution. Lowering the constant makes us more sensitive to perturbations in the distributions, effectively shrinking the attacker’s maximum achievable expected payoff by limiting the attacker's KL divergence budget. However, this comes at the same cost as many forms of adjusting $q$.

Taking benign traffic as the null hypothesis, making the detector more sensitive (e.g., lowering $q_i$ or lowering $C$) will catch a lot more attacks, but it comes at the cost of increased Type I errors. All those false positives result in an annoying experience for our end users and complaints pointed at the defenders.

Making the detector less sensitive (leaving $q_i$ as is, raising $C$) would stop many of those false positives from happening, but more true positives would sneak by as false negatives. These kinds of errors can be catastrophic, so many defenders resolve to a more risk-averse position that is overly sensitive but mostly avoids Type II errors.

A real-world example of this is the heat Anthropic is catching from their strict cyber and bio safeguards. Even in something as innocuous as this essay (in which I had asked Claude Fable to check my notations), I triggered the safeguards multiple times. This tradeoff is at the heart of most threshold-based detection software no matter the sophistication. It appears in everything from lightweight detectors like this problem to ML and DL-based classifiers with millions to billions or trillions of parameters.