IEC studies the behavior and degradation of a language model under controlled load. Language models follow instructions less well as the number of instructions grows and as the text and the answer get longer, and the quality also depends on the place of an instruction. The phenomenon is well studied, but existing tests do not allow it to be measured as a quantity: their instructions are heterogeneous, share one coherent text and are often judged by another model, so the load is confounded with the difficulty of the task itself.

Paper: H. Kutomanov. Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load. arXiv:2610.36095, 2026 — https://arxiv.org/abs/2610.36095. Code and data — https://github.com/kutomanovhennadii/instruction-capacity.

The load: a word replacement rule

The load is set by the simplest independent instructions: rules that replace one word with another in a text without meaning. A rule is written as lake -> stone: the model must replace every occurrence of the word lake with the word stone. Such a rule is an atomic instruction: it applies to one word and at every place of application admits exactly one right outcome. The rules are not linked to each other by the construction of the task, and executing a rule requires neither reasoning, nor knowledge of the world, nor understanding of the text. One rule serves as a conventional unit of load.

The text of a task is a stream of words from a fixed vocabulary of single-token nouns, without meaning and punctuation, broken into lines of ten words. The model must apply all the rules to the text, keeping it otherwise unchanged. This is one of the published tasks of the GPT-4o mini measurement — a text of 20 words and two rules:

Rules:
james -> hang
reign -> steal

Text:
reign cab reign luck cab luck reign reign cab reign
james luck james james luck james cab james cab luck

The answer of the model coincided with the right answer: all ten replacements were made.

steal cab steal luck cab luck steal steal cab steal
hang luck hang hang luck hang cab hang cab luck

Kinds of load

The instrument distinguishes five kinds of load:

• the text length — the number of words of the text;
• the word position in the answer — the number of the word at which a replacement is made;
• the replacement density — the share of words in the text that must be replaced;
• the number of rules in the list;
• the position of a rule in the list — the number of the rule being checked.

The text length, the replacement density and the number of rules are set by the point — a triple of numbers: the number of rules, the number of distinct words of the text and the number of repetitions of each word. The word position and the rule number are known for every replacement, so the contribution of each kind of load is estimated separately. The specific words of a task are determined by three seeds of the generator — the rule seed, the text seed and the rule order seed. A task at a point, together with the answer of the model to it, is called a request; the prompt of every request is reproduced byte for byte, and the same task can be presented to different models.

Measurement

The answer of the model is compared with a reference answer computed by a program; neither an evaluator — a person or a model — nor measures of text similarity enter the measurement. Every occurrence of a word to be replaced becomes a Bernoulli trial: the outcome is 1 if the right replacement stands at its place in the answer, and 0 in all other cases.

The answer is checked line by line, and three kinds of failure are distinguished: a wrong replacement with the structure of the answer preserved, a local disruption of a line after which the structure of the answer recovers, and a loss of the structure of the answer after which matching with the lines of the reference answer becomes impossible. A damaged answer is not considered a malfunction of the instrument: it gives a full measurement, and unmatchable positions are counted as unsuccessful, so the score of the model cannot be overestimated.

Laws and the passport

From the outcomes of the replacements a statistical model recovers how the probability of a right replacement depends on each kind of load. The form of each dependence is first measured over intervals of the load — bins — without assumptions about its shape, and then described by a smooth law.

On GPT-4o mini each of these dependences proved regular and is described by a simple law: linear in the logarithm for the number of rules, a saturation law for the replacement density, and a power law for the position of a rule in the list. The text length and the word position act not as a sum of losses but as two interacting constraints. Besides wrong replacements, in some of the requests the model loses the structure of the whole answer; this collapse obeys its own laws and is determined by the load of the text and the progress through the answer, not by the number of instructions. The laws of the collapse describe the probability that a request loses the text and the place in the answer at which this happens.

The laws yield parameters of the model with uncertainties. The instruction capacity (Instruction Execution Capacity, IEC) is a measure of load by instructions: the number of rules the model holds at a given share of right replacements. The guaranteed instruction capacity is a measure of load by replacement density: the same quantity at the full density, when every word of the text is replaced. The fatigue capacity is a measure of load by the number of words in the answer: the number of the word up to which a given share of right replacements is kept. A capacity is only one of the characteristics derived from the laws, while the laws themselves give a fuller description of the behavior of the model under load.

Together, the laws and the parameters form the passport of the model. Models measured under one protocol — with the same prompt template, structure of the text, rules of answer checking and vocabulary — can be compared with each other. The laws have so far been obtained on one model, and their universality has not been established; comparing the laws of different models is the nearest task of the work. The instrument measures the form of the dependences, not their causes.

What is open

The code of the instrument, the measurement data and the passport are open, and every number of the passport is recomputed from the published points by the code of the instrument: https://github.com/kutomanovhennadii/instruction-capacity. Only the technology for choosing the measurement points is closed, and the passport does not depend on it. The code is distributed under the MIT license, the data and documents under CC BY 4.0.

Passports

Citation

@misc{kutomanov2026instruction,
  title         = {Instruction Capacity: Behavior and Degradation
                   of Language Models under Controlled Load},
  author        = {Hennadii Kutomanov},
  year          = {2026},
  eprint        = {2610.36095},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG}
}