Juan JosÊ de Haro ¡ bilateria.org ¡ bilateria.org
Version 2.0
Mathematical foundation of the adaptive assessment protocol. Sections 1â6 develop the full theory; section 9 provides a step-by-step numerical example.
A traditional assessment system assigns the same questions to all students in the same order. This generates two inefficiencies:
Adaptive assessment solves this by selecting the most informative question at each moment given what is already known about the student. To do this you need three ingredients:
This document describes the mathematics behind each of those three ingredients.
Instead of assigning the student a fixed value, such as a grade or label, the system maintains a probability distribution over a set of hypotheses. In the classical case those hypotheses are mutually exclusive and exhaustive; when several errors can coexist, the state is better represented by several parallel dimensions or by a distribution over full profiles.
Let \(\mathcal{H} = \{H_1, H_2, \ldots, H_n\}\) be the set of possible hypotheses. For example:
At each moment, the system maintains a vector of probabilities:
with the restriction:
This vector expresses the system's degree of belief about the student's actual state, not a certainty.
If there is no prior information about the student, the system starts from a uniform distribution:
This choice reflects maximum ignorance: all hypotheses are equally plausible before any response is observed. If reliable prior information existed (results from previous courses, previous diagnoses), it could be used as a justified initial distribution.
In order to use the logistic function that generates the likelihoods (see §4), each hypothesis \(H_i\) needs a numerical value \(\theta_i\) that represents its position on the domain scale. The recommended convention is to center the values at zero with equal intervals:
| \(n\) hypotheses | \(\theta_i\) values |
|---|---|
| 2 | \(-1,\; +1\) |
| 3 | \(-2,\; 0,\; +2\) |
| 4 | \(-3,\; -1,\; +1,\; +3\) |
| 5 | \(-4,\; -2,\; 0,\; +2,\; +4\) |
These values are fixed and depend only on the number of hypotheses; the difficulties \(b_q\) are placed within that scale, not the other way round. The exact relationship between θ and b is explained in §8.
The student's state does not have to be a single distribution. When you want to know not only how much has been mastered but which components are failing, it is advisable to maintain several Bayesian distributions in parallel:
Each distribution \(\mathbf{p}^{(d)}\) over the hypotheses of dimension \(d\) is updated independently using the machinery of §3â§4, but it must be fed with the evidence that belongs to it: the overall result may update the belief about level and each subcriterion updates only the belief about its own dimension. If a single global correct or incorrect outcome depends on several skills at once, using it to update several independent dimensions would duplicate evidence and misattribute the cause of the error. This makes it possible to distinguish what to practise (level by category) from what to reinforce (diagnosis by dimension).
Each dimension can have its own random floor \(c^{(d)}\) (see §4.6â4.7), so the same latent value \(\theta\) produces different visible percentages depending on the dimension. The percentages between dimensions are not directly comparable; among ordinal dimensions, the common reference is \(\theta\). In nominal error factors there is no meaningful \(\theta\): their summary is the marginal probabilities and the status (present, absent, undetermined), and no expected \(\theta\) should be computed for them (§10.3). Dimensions that can coexist should not be forced within a single distribution: they are modeled separately (see §9 of the protocol).
It is worth situating this against the classic model. IRT and Rasch models assume unidimensionality (all items measure a single latent attribute) and local independence (at a given ability level, the response to one item does not depend on the responses to the others). Under those assumptions, a single distribution over \(\theta\) fully describes the student. The multidimensional state described here extends that framework: instead of forcing several attributes within a single dimension âwhich would violate local independenceâ it maintains a separate distribution for each dimension and preserves local independence within each one. It is a deliberate generalization, not a breach of the model.
When the student answers a question, that answer is evidence that should modify our estimate of the student's state. The update mechanism is Bayes' theorem:
where:
In practice, the response \(R\) is binary: correct (A) or incorrect (F). The update takes the form:
Note that the denominator is simply a normalization constant. In the implementation, it is enough to calculate the numerators for all \(i\) and divide by their sum.
If the student answers several questions, the process is applied sequentially: the posterior of one question becomes the prior of the next. This is mathematically equivalent to updating with all responses at once, as long as the responses are conditionally independent given the true hypothesis.
Under the usual assumptions of probabilistic consistency and updating from observed evidence, Bayes' rule is the natural way to update probabilities. It satisfies all of the following at once:
Alternatives such as rule networks or classical expert systems do not have these properties and can be blocked in incorrect diagnoses when the student responds unexpectedly.
The sequential updating of §3.3 gives the same weight to every response, the first and the last. That is correct if the student's state does not change during the session, but in prolonged practice resources the student learns while practising (see §11.3): the posterior drags along the old evidence and can converge towards a state that no longer exists.
The minimal correction is prior-anchored exponential forgetting: before incorporating each new response, the accumulated posterior is attenuated toward the prior \(\pi_i\) with a power \(\lambda \in (0, 1]\) and renormalised:
With \(\lambda = 1\) the standard Bayes of §3.3 is recovered, and with a uniform prior the factor \(\pi_i^{1-\lambda}\) is constant and the rule coincides with the simple form \(p_i \leftarrow p_i^{\lambda}\) renormalised. With \(\lambda < 1\), evidence from \(k\) steps ago enters with effective weight \(\lambda^k\): it decays geometrically, and the system's effective memory is approximately \(1/(1-\lambda)\) responses. Unrolling the recursion shows that the prior keeps weight exactly 1 at all times: forgetting discards old evidence, never the prior. A practical rule: \(\lambda = 1 - 1/W\), where \(W\) is the number of recent responses that should dominate the estimate. A variant with a similar effect is mixing the posterior with the prior, \(p_i \leftarrow (1-\gamma)\,p_i + \gamma\,\pi_i\); both return some mass to the discarded hypotheses and keep the estimate open to change.
Why anchoring to the prior is not optional. The simple form \(p_i \leftarrow p_i^{\lambda}\) (or mixing with the uniform) has the uniform distribution as its fixed point: with no new evidence, any distribution drifts toward it. With level hypotheses and a uniform prior this is harmless, but it destroys any informative prior. The most damaging case is that of error factors (§10): with \(P(\text{error}) = 0.25\) and \(\lambda = 0.95\), a factor receiving no evidence rises on its own to \(\approx 0.34\) in 10 steps and \(\approx 0.40\) in 20, reappearing as "undetermined" or "probable" without the learner having done anything: the very false positive the informative prior of the protocol's §5.1 avoids. Anchored to the prior, a distribution with no new evidence stays at its prior. The attenuation is applied to every distribution at every step of the session â including those receiving no evidence from that response â so all of them track the passage of time without degrading toward the uniform.
Calibrating \(\lambda\) with several parallel distributions. Attenuating every distribution on each response has a consequence worth making explicit: each distribution ages once per response, but only receives evidence when the response belongs to it. If distribution \(d\) is updated on one of every \(K_d\) responses, its memory measured in its own attempts is \(M_d \approx 1/\bigl(K_d\,(1-\lambda)\bigr)\), that is, \(K_d\) times shorter than the memory in responses. Setting a common \(\lambda\) for all of them does not treat them alike: it penalises those updated less often. Hence the target memory \(M\) should be set in attempts of that distribution and a \(\lambda\) derived per distribution:
With \(K_d = 1\) (a dimension scored on every response) this recovers \(\lambda = 1 - 1/M\). With \(M = 20\) and \(6\) categories, a dimension always scored uses \(\lambda = 0.95\) and each category uses \(\lambda = 0.95^{1/6} \approx 0.9915\); both retain a memory of about \(20\) of their own attempts, even though the category's window spans \(1/(1-\lambda) \approx 117\) responses. Applying \(0.95\) to the categories as well would leave them \(20/6 \approx 3.3\) attempts of memory, enough to erase an initial diagnosis of \(2\) attempts per category.
The same correction affects the minimum-sample window (§7.6): attempts are counted within that distribution's last \(1/(1-\lambda_d)\) responses, and they are counted unweighted. A count weighted by \(\lambda^k\) would make two consecutive attempts sum to \(1 + \lambda = 1.95\), failing an integer threshold of \(2\) and silently tightening the gate.
Example. With a uniform prior, the posterior \((0.809,\; 0.180,\; 0.011)\) and \(\lambda = 0.9\), the attenuation yields \((0.782,\; 0.202,\; 0.016)\): the dominant belief is preserved, but the alternatives regain room to react if the student's behaviour changes.
When to use it. In short-session diagnostic resources, where the state is stable, \(\lambda = 1\) should be used: forgetting would only add noise and delay convergence. In continuous practice or reinforcement (§7.6), values of \(\lambda \approx 0.9\)â\(0.98\) make the estimate track the student's current state. With forgetting active, the confidence shown should be interpreted as confidence about the recent state; entropy stays somewhat higher and the session does not close by itself, which is consistent with the no-stopping mode.
Forgetting is symmetric: it does not presuppose that the student improves, it only prevents old evidence from blocking the tracking of a changing state. If the direction of learning is to be modelled explicitly, the natural extension is a transition model (Bayesian filtering): before each update, a matrix \(T\) is applied with a small probability of moving up a level âlarger, for example, right after showing an explanationâ, so that \(\mathbf{p} \leftarrow \text{normalise}\!\left(L \circ (T^{\top}\mathbf{p})\right)\). Its two-state special case is Bayesian Knowledge Tracing (Corbett & Anderson, 1995), the classic model of intelligent tutoring systems. In exchange it introduces one more parameter to set without data âthe probability of learning per stepâ, with the cautions of §11.1.
To apply Bayes we need \(P(A \mid H_i, q)\): the probability that a student in state \(H_i\) answers question \(q\) correctly. This probability is the likelihood.
The system must generate them automatically from the parameters of each question, without requiring the teacher to fill in probability tables.
The IRT (Item Response Theory) model provides a family of functions to model \(P(A \mid \theta, q)\). The three-parameter model (3PL) is:
The three parameters are:
| Parameter | Name | Meaning |
|---|---|---|
| \(a\) | Discrimination | Slope of the curve; controls how strongly the question separates different levels |
| \(b_q\) | Difficulty | Value of \(\theta\) at which the probability of success, excluding chance, reaches 50% |
| \(c_q\) | Pseudo-chance | Minimum probability of success; in the absence of empirical data, it can be approximated as \(c_q \approx 1/m\) |
The logistic form is not merely a computational convenience. Its root lies in the family of Rasch models, where taking the logarithm of the odds \(P/(1-P)\) yields an additive scale (logit) on which the probability of success depends only on the difference \(\theta_i - b_q\). That structure is what makes it meaningful to place the student's level and the item's difficulty on the same scale and compare them. In the strict Rasch case (common discrimination \(a\) and \(c = 0\)), this property is known as specific objectivity: the comparison between two students does not depend on which items are used, nor the comparison between two items on which students answer them. The three-parameter model generalizes the curve (allowing variable \(a\) and \(c > 0\)) and relaxes that strict objectivity, but keeps the same underlying justification for using the logistic function.
When \(\theta_i \gg b_q\) (the student is well above the difficulty), the exponent \(e^{-a(\theta_i - b_q)} \to 0\) and:
In the implementation, this upper limit is not allowed to reach 1: it is capped by the mastery ceiling \(P(A) \leq 0.95\) (§8.3), which models slips and prevents a single failure on an easy item from producing near-deterministic posterior jumps.
When \(\theta_i \ll b_q\) (the student is well below the difficulty), the exponent \(\to +\infty\) and:
When \(\theta_i = b_q\), the exponent argument is zero and:
This is the inflection point of the curve: where the slope is maximum and, therefore, where the question is most discriminating.
The \(a\) parameter controls the slope of the logistic curve. Its effect can be seen by deriving the ICC with respect to \(\theta\):
The maximum slope (in \(\theta = b_q\)) is:
A high value of \(a\) produces a steeper curve: the question discriminates better between students close to its difficulty, but provides little information to students clearly above or below it. A lower value produces a smoother curve: the question is useful over a wider range of levels, but is less discriminating.
The usual values in psychometrics range between 0.5 and 2.5. For general purpose educational systems, values around \(1.0\)â\(1.5\) are reasonable starting points; the operational specification recommends fixing \(a_{\text{ef}} = 1.25\) and deriving \(a\) from each item's guessing floor.
Effective discrimination and mixed formats. The maximum slope depends on the product \(a\,(1 - c_q)\), not on \(a\) alone. It is therefore useful to distinguish the effective discrimination \(a_{\text{ef}} = a\,(1 - c_q)\) âthe maximum slope expressed without the factor \(1/4\)â from the nominal discrimination \(a\). When all questions share the same number of options, and therefore the same \(c_q\), both are proportional and it is enough to fix \(a\). But if the bank mixes formats with different \(c_q\) (for example, true/false with \(c_q = 0.5\) alongside 4-option items with \(c_q = 0.25\)), keeping \(a\) constant makes the slope vary by format alone: a well-constructed true/false question would have a lower slope than an equivalent open question, a purely mechanical bias with no pedagogical meaning.
To avoid this, fix the target effective discrimination \(a_{\text{ef}}\) and solve for the nominal one:
This way each question keeps the same slope at its inflection point regardless of the number of options. Taking \(a_{\text{ef}} = 1.25\) as the common target:
| Format | \(c_q\) | \(a = a_{\text{ef}} / (1 - c_q)\) | Effective slope \(a_{\text{ef}}\) |
|---|---|---|---|
| Open (no guessing) | 0 | 1.25 | 1.25 |
| 5 options | 0.20 | 1.5625 | 1.25 |
| 4 options | 0.25 | â1.667 | 1.25 |
| 3 options | 1/3 | 1.875 | 1.25 |
| True/false | 0.50 | 2.5 | 1.25 |
In this way \(a\) is not set arbitrarily: it is computed from a target effective discrimination and the pseudo-guessing of each question. This rule should be applied always, not only when a single test mixes formats. Two questions âor two different resourcesâ with the same nominal \(a\) but a different number of options have different maximum slopes and are not comparable in that sense: with \(a = 1.5\), a 3-option question (\(c_q = 1/3\)) yields \(a_{\text{ef}} = 1.0\), while a 4-option one (\(c_q = 1/4\)) yields \(a_{\text{ef}} = 1.125\). Fixing \(a_{\text{ef}}\) and deriving \(a\) in each case equalizes those slopes, but it does not equalize the item's full expected information: that information depends on the whole curve, the guessing floor, and the current posterior. The recommended target is \(a_{\text{ef}} = 1.25\): since \(c_q \leq 0.5\) for any item with two or more options, the derived nominal \(a\) never exceeds \(2.5\) âthe extreme case, true/false, gives exactly \(a = 2.5\)â and thus stays within the usual psychometric range (0.5â2.5). A larger target would overshoot it: \(a_{\text{ef}} = 1.5\) would give \(a = 3.0\) for true/false. The numerical examples in this document use \(a = 1.5\) directly for illustration and, for readability, do not apply the mastery ceiling of §8.3 (see the note in §9.1).
This is the likelihood that enters Bayesian updating when the learner answers incorrectly.
In multiple choice questions with \(m_q\) options, \(c_q = 1/m_q\) can be used as an initial approximation in the absence of empirical data:
In a calibrated IRT model, \(c_q\) should be estimated from real data, because pseudo-chance does not always coincide with pure chance: distractors are not equally attractive and some learners eliminate options before responding. This probability belongs to each question, not to the test as a whole. In numerical or exact-text answers where chance is irrelevant, \(c_q = 0\) is used.
A question is not always scored as a single correct or incorrect response. If the task is evaluated through several components \(j\) (steps, decisions, or subresponses), each with its own number of options \(m_j\) and weight \(w_j\), the weighted average of the chance levels of each component is not the probability of getting the whole item right, but the expected partial score by chance:
For example, an item with three components of weights \((0.5,\,0.3,\,0.2)\) and options \((2,\,4,\,6)\) has an expected score by chance:
This aggregated \(c_q\) enters the ICC (§4.2) only if the curve is interpreted as the expected score of the composite item. If the item is scored all-or-nothing, the probability of full success by chance is not the mean but the product of the chance levels of the independent components: \(\tfrac12 \cdot \tfrac14 \cdot \tfrac16 \approx 0.021\). The weights \(w_j\) must be the same as those used to score partial credit (§4.8).
When the answer allows degrees, not just correct or incorrect, but a score \(s \in [0, 1]\) obtained from weighted components, the likelihood should favor the hypotheses whose prediction \(p_i = P(A \mid H_i, q)\) is compatible with that score. If only an aggregate score is available, a coherent approximation is the geometric likelihood:
Since \(P(F \mid H_i, q) = 1 - P(A \mid H_i, q)\), if we write \(p_i = P(A \mid H_i, q)\), then:
This \(L(H_i)\) replaces the likelihood in the Bayesian update of §3.2; the rest of the process, including normalization, does not change. Boundary cases:
The score \(s\) must be calculated explicitly and in a way that can be scored automatically, typically as a weighted sum of subcriteria with \(\sum_j w_j = 1\). If the item has \(J\) approximately independent components and the strength of the evidence should be preserved, one can use \(L(H_i) \propto p_i^{sJ}(1-p_i)^{(1-s)J}\). This form is equivalent to treating the response as \(J\) independent Bernoulli trials with the same probability \(p_i\) of the whole item (the binomial coefficient does not depend on \(H_i\) and cancels in the normalisation), so it is only reasonable if the components are of similar difficulty: if they differ clearly âthe usual case in step-by-step tasks, where saturated components contribute little discriminationâ it overcounts the evidence relative to the per-component model, and it is advisable to use a \(J\) smaller than the real number of components (more conservative). If the result of each component is known, it is preferable to multiply the likelihoods component by component. Psychometrics has canonical polytomous models for graded responses âthe Graded Response Model (Samejima, 1969) and the Partial Credit Model (Masters, 1982)â; the geometric likelihood is a simpler approximation adopted here because those models require estimating per-category parameters from response data that this methodology does not have (§11.1).
Three hypotheses \(\theta \in \{-2, 0, 2\}\), item of medium difficulty (\(b_q = 0\), \(a = 1.5\), \(c_q \approx 0.36\)), uniform prior \(\mathbf{p} = (\tfrac13, \tfrac13, \tfrac13)\). The probabilities of complete success are approximately \(p = (0.40,\, 0.68,\, 0.97)\). With a partial answer \(s = 0.75\):
Applying Bayes and normalising (\(\sum = \tfrac13(0.443 + 0.563 + 0.407) \approx 0.471\)) gives:
The posterior shifts towards the middle level, because a score \(s = 0.75\) is more compatible with a prediction \(p_i = 0.68\) than with an almost perfect prediction \(p_i = 0.97\). Thus partial credit provides evidence for the level that best predicts the observed score, not only a softer push towards high mastery.
Selection with partial credit. Expected information gain (§6) is defined over the outcomes that the item models. The exact version with graded responses would require averaging over the distribution of \(s\), which is usually not known a priori. In practice, selection can keep using the binary scenarios (full success and full failure) as an approximation, while the geometric or component-wise likelihood is reserved for the update once the response has been observed.
Shannon entropy measures the uncertainty of a probability distribution:
It is measured in bits. By agreement, \(0 \cdot \log_2 0 = 0\).
Minimum entropy. If a hypothesis concentrates all the probability (\(p_k = 1\), \(p_{i \neq k} = 0\)), the entropy is zero: there is no uncertainty.
Maximum entropy. If all hypotheses are equiprobable (\(p_i = 1/n\)), the entropy is maximum:
This corresponds to total ignorance about the student's state.
Non-increasing expected entropy. The subsequent entropy can increase or decrease after a specific response, depending on the evidence observed (as illustrated by the example in §9, where the correct answer in Q2 increases the entropy with respect to the previous step). For any correctly modelled question, the subsequent entropy expected âbefore knowing the answerâdoes not exceed the current entropy, because expected gain is mutual information and cannot be negative. Maximum-information selection does not create that property: it chooses the question that reduces that expected entropy the most.
With \(n = 3\) hypothesis:
| \(\mathbf{p}\) Distribution | \(H\) Entropy (bits) | Interpretation |
|---|---|---|
| \((0.33,\; 0.33,\; 0.33)\) | \(1.58\) | Total ignorance |
| \((0.60,\; 0.30,\; 0.10)\) | \(1.30\) | High uncertainty |
| \((0.80,\; 0.15,\; 0.05)\) | \(0.88\) | Probable diagnosis |
| \((0.95,\; 0.04,\; 0.01)\) | \(0.32\) | Almost certain diagnosis |
| \((1.00,\; 0.00,\; 0.00)\) | \(0.00\) | Absolute certainty |
The probability of the most probable hypothesis (\(\max_i p_i\)) is an intuitive indicator, but entropy captures more information: it distinguishes between \((0.80, 0.15, 0.05)\) and \((0.80, 0.19, 0.01)\), which have the same maximum but different distribution from the rest. Entropy is also the natural quantity that appears in the information gain (§6), which makes the system mathematically coherent.
The system chooses the next question by seeking to maximize the expected reduction in entropy. For each candidate question \(q\), the expected information gain is calculated:
where \(\mathbf{p}'\) is the posterior distribution after answering \(q\), and the expectation is over the two possible outcomes (hit or miss).
We first define the marginal probability of success in the question \(q\), using the law of total probability:
And the marginal probability of failure:
Next we calculate the posterior conditionals, applying Bayes before knowing the real answer:
With these posteriors we calculate the entropy in each scenario:
The expected entropy after asking the question \(q\) is:
And the information gain is:
The system selects the question \(q^*\) with the highest \(IG\):
where \(\mathcal{Q}\) is the set of available questions.
The expected information gain from question \(q\) is exactly the mutual information between the response \(R_q\) and the student state \(H\):
where \(KL\) is the Kullback-Leibler divergence. Maximizing \(IG\) is equivalent to selecting the question whose answer, on average, most separates the posterior from the prior.
In practice, several questions can have identical or very close information gains, especially if they share the same parameters \(a\), \(b_q\) and \(c_q\). A deterministic selection between ties produces systematically repetitive tests between different sessions.
The recommended solution is a weighted random selection:
This combines maximum informative usefulness with thematic diversity.
The dominant criterion in classic adaptive tests (CAT) is not entropy reduction but the Item Information Function (IIF), based on Fisher information:
where \(P(\theta)\) is the item's ICC (§4.2) and \(P'(\theta)\) its derivative. The IIF measures how much information the item provides at a specific point \(\theta\) on the continuum, and the classic system selects the item that maximizes \(I_q(\hat\theta)\) at the current point estimate of ability.
This system's criterion âmaximum expected entropy reduction (§6.1)â fits especially well when the state is represented as a full distribution rather than as a single value \(\hat\theta\), for two reasons:
Both criteria converge in the limiting case of a highly concentrated distribution: when the belief is almost point-like, maximizing information gain and maximizing the IIF at \(\hat\theta\) select practically the same item. The difference matters above all in the first questions, when uncertainty is high.
The use of mutual information and KL divergence (§6.3) to select items is well established in Bayesian psychometric research, but its application in real educational tools is uncommon: what is usual in the classroom is the IIF or simpler rules (next item of the estimated level, selection by difficulty). The combination this system uses âa distribution of hypotheses, Bayesian updating and selection by Shannon entropy reduction, integrated into a tool for teachersâ is, in that sense, infrequent in the educational field.
The system should stop when it has enough confidence in the student's status. The natural criterion is: stop when the most probable hypothesis exceeds a confidence level \(p_{\min}\) (for example, 0.80).
However, checking \(\max_i P(H_i) \geq p_{\min}\) directly may not be enough, because it does not take into account how the remaining probability is distributed. Entropy is a more complete indicator.
We look for the entropy of a distribution in which the most probable hypothesis has probability \(p_{\min}\) and the remaining probability \(1 - p_{\min}\) is distributed evenly among the other \(n - 1\) hypotheses:
The entropy of this distribution is:
With \(p_{\min} = 0.80\):
| \(n\) hypotheses | \(H_{\max} = \log_2 n\) (bits) | \(H_{\text{stop}}\) (bits) | Remaining uncertainty fraction |
|---|---|---|---|
| 2 | 1.00 | 0.72 | 72% |
| 3 | 1.58 | 0.92 | 58% |
| 4 | 2.00 | 1.04 | 52% |
| 5 | 2.32 | 1.12 | 48% |
The \(H_{\text{stop}}\) formula assumes that the remaining probability is uniformly distributed, which is an approximation. Two distributions with the same maximum can have different entropies:
The second has lower entropy even though the maximum is the same, because the probability is more concentrated. A prudent implementation may check both conditions at once:
In fact, when \(H_{\text{stop}}\) is derived from the same \(p_{\min}\), the maximum-probability condition implies the entropy condition: the distribution that defines \(H_{\text{stop}}\) is the maximum-entropy distribution among those whose maximum equals \(p_{\min}\), and that threshold decreases as the maximum grows, so \(\max_i P(H_i) \geq p_{\min}\) alone guarantees \(H \leq H_{\text{stop}}\). Checking both conditions is harmless, but redundant. If an additional check that does add a requirement is desired, a separation criterion can be used:
which requires the winning hypothesis to be sufficiently ahead of the second candidate. It is worth being precise about when this adds anything: since \(P(H_{\text{second}}) \leq 1 - P(H_{\text{winner}})\), the condition \(\max_i P(H_i) \geq p_{\min}\) already guarantees a separation \(\geq 2p_{\min} - 1\). Therefore the separation criterion only adds a requirement if \(\Delta_{\min} > 2p_{\min} - 1\); with \(p_{\min} = 0.80\) that requires \(\Delta_{\min} > 0.60\), and a \(\Delta_{\min}\) of 0.3â0.4 would be as redundant as the entropy. Its real usefulness is as an alternative to a high \(p_{\min}\): when there are many hypotheses and requiring \(\max_i P(H_i) \geq 0.80\) is impractical, a moderate maximum together with \(\Delta_{\min} \geq 0.3\) captures the relevant confidence âthat the winner dominates the secondâ even if the remaining mass is spread out. With \(n = 2\), separation and \(p_{\min}\) are equivalent (\(\text{sep} = 2\max - 1\)).
In addition to the entropy criterion, the system must consider:
All of the above assumes an objective of diagnosis: estimate the student's state and stop. In open practice or reinforcement resources, that objective changes: the goal is not to converge and stop, but to sustain practice. In that mode:
Caution due to sample size. With just a few attempts, the posterior can move a long way for a single answer. In order not to communicate an unfounded domain, it is advisable to require a minimum sample by category or dimension (e.g., 2â4 attempts) before showing high levels of mastery, and marking the estimate as tentative while evidence is scarce. This caution is a presentational decision: it does not alter the Bayesian posterior, only how it is translated to the interface. With forgetting active, the minimum sample must be counted over a recent window (attempts within that distribution's last \(1/(1-\lambda_d)\) responses, unweighted; see §3.5), not over the whole session: evidence expires with forgetting, but a cumulative counter does not, and a category sampled only at the start would still count as diagnosed while its posterior has already been attenuated. That expiring counter governs the mastery gates, not what is shown to the learner: the number of exercises they see as solved is the real total. And if a category is left with no evidence inside its window, its belief has already returned to the prior: it should be presented as no recent data, not as a weakness.
The IRT 3PL logistics curve has its maximum slope at \(\theta_i = b_q\). This means that the question \(q\) is more discriminating for students whose level is close to the \(b_q\) difficulty.
If the extreme level \(\theta_{\max}\) matches the extreme difficulty \(b_{\max}\), the most difficult question places the advanced level right at the tipping point. At that point the probability of success is still not high: with 4 options and \(c_q = 0.25\), \((1+c_q)/2 = 0.625\) is valid. Therefore, getting that question right weakly confirms the advanced level.
This may cause the system to underuse extreme questions: not because they are useless, but because other questions may produce a larger expected reduction in uncertainty.
To avoid this problem, the range of \(\theta\) should be strictly greater than the range of \(b\). In this discrete system, the following practical convention is adopted:
Factor 2 is a useful heuristic, not a standard universal IRT rule. It can be adjusted according to the value of \(a\), \(c_q\) and the target probability of success for extreme levels. If you want an extreme level student to have an objective probability \(P^*\) of getting an extreme question right, you can solve it:
Thus the separation between scales is justified by a desired probability, not by a fixed factor.
Example. With 3 levels of difficulty \(b \in \{-1,\; 0,\; +1\}\):
where \(\sigma(x) = 1/(1 + e^{-x})\) is the standard sigmoid function.
The robust way to guarantee the separation is not to recompute \(\theta\) from the bank, but to fix the scale by convention and place the difficulties within it. The \(\theta_i\) values depend only on the number of hypotheses:
so that \(\theta\) runs over \(\{-(n-1), \ldots, +(n-1)\}\) with spacing 2 (this formula extends the table in §2.3 to any \(n\)). The difficulties are placed in the central half of the scale, \(b_q \in [-\theta_{\max}/2,\; +\theta_{\max}/2]\), which maintains the factor-2 proportion of §8.2 for the extreme items with any \(n\) (with the caveat about its dependence on \(n\) explained below). Since \(\theta_{\max}\) depends on \(n\), the conversion of qualitative difficulties must also depend on \(n\): with \(k\) categories, they are spread evenly over the interval,
(with \(k = 1\), \(b = 0\)). Thus, with \(n = 3\) and \(k = 3\), \(b \in \{-1, 0, +1\}\); with \(n = 3\) and \(k = 5\), \(b \in \{-1, -0.5, 0, +0.5, +1\}\); with \(n = 4\) and \(k = 3\), \(b \in \{-1.5, 0, +1.5\}\). A difficulty table independent of \(n\) would be inconsistent: with \(n = 2\) (\(\theta = \pm 1\)) and five fixed categories up to \(b = \pm 2\), the extreme difficulty would double the extreme level and the separation between scales would be inverted. Numeric \(b_q\) values outside the interval are clamped to it.
Caveat: the factor 2 maintains the shape, not the target probability. Fixing \(b_q \in [-\theta_{\max}/2,\; \theta_{\max}/2]\) preserves the proportion \(\theta_{\max} = 2\,b_{\max}\) for any \(n\), but the absolute margin between the top level and the hardest item is \(\theta_{\max} - b_{\max} = (n-1)/2\), which depends on \(n\). That is why the probability that the top level answers the hardest item correctly, \(P = c_q + (1-c_q)\,\sigma\!\big(a\,(n-1)/2\big)\), is not constant (with \(a_{\text{ef}} = 1.25\)):
| \(n\) | Margin \((n-1)/2\) | \(P\) (open, \(c=0\)) | \(P\) (4 options, \(c=0.25\)) |
|---|---|---|---|
| 2 | 0.5 | 0.651 | 0.773 |
| 3 | 1.0 | 0.777 | 0.881 |
| 4 | 1.5 | 0.867 | 0.943 |
| 5 | 2.0 | 0.924 | 0.974 |
With \(n \geq 3\) the margin is enough for the extreme items to confirm comfortably, but with \(n = 2\) (mastered / not mastered) confirmation is weak â0.65 for open, 0.77 for four optionsâ, the same mediocre value the factor 2 is meant to avoid. Two consequences follow: (i) the comparability of confidences and convergence speeds across resources promised by the invariant \(a_{\text{ef}}\cdot\Delta\theta\) is strict only between resources with the same \(n\); (ii) with \(n = 2\) it is advisable to compensate with more questions âor to define the scale from a target probability \(P^*\) using the formula in §8.2â rather than relying on the fixed spacing of 2.
The inverse rule âstretching \(\theta\) up to \(2 \cdot \max_q |b_q|\)â has two flaws. First, it makes the meaning of the hypotheses depend on the bank: two resources with the same pedagogy but different difficulty ranges produce non-comparable confidences and convergence speeds, because the strength of each update is governed by the product \(a \cdot \Delta\theta\) (discrimination times separation between adjacent hypotheses), which would be left uncontrolled. Second, it is fragile to outliers: a single question with \(b = 3\) in a bank where the rest is at \(\pm 1\) would stretch \(\theta\) to \(\pm 6\) and saturate the likelihoods of all the other questions (probabilities pinned to \(c_q\) or to 1), so that a single answer would produce almost deterministic posterior jumps. With the fixed scale, that outlier item is simply clamped to the edge of the interval and the rest of the bank keeps its behaviour. With \(a_{\text{ef}} = 1.25\) and intervals of 2, the invariant \(a_{\text{ef}} \cdot \Delta\theta = 2.5\) stays stable across resources. If most of the bank fell outside the interval, the problem is one of design (poorly defined levels and difficulties) and must be resolved with the teacher, not by stretching the scale.
What the invariant equalizes and what it does not. The product \(a_{\text{ef}} \cdot \Delta\theta = 2.5\) equalizes the maximum slope of the ICC across formats, but not the maximum strength of the evidence from a failure. That is set by the likelihood ratio between adjacent hypotheses: on the failure side, \(P(F\mid\theta) = (1-c_q)\,[1-\sigma(a(\theta-b_q))]\), the factor \(1-c_q\) cancels in the ratio and the maximum ratio tends to \(e^{a\,\Delta\theta}\) with the nominal \(a = a_{\text{ef}}/(1-c_q)\), not with \(a_{\text{ef}}\). Hence, with \(\Delta\theta = 2\), the same failure on an easy item carries very different likelihood ratios depending on the format: â 12 for open (\(c_q=0\)), â 28 for 4 options (\(c_q=0.25\)), and â 148 for true/false (\(c_q=0.5\)). So that a single failure does not produce almost deterministic posterior jumps âand to model the slip that the pure 3PL ignoresâ, the ordinal case also applies the same mastery ceiling as the nominal one (§10.5): \(P(\text{success}) \le 0.90\text{â}0.95\), never 1. With that ceiling, the failure likelihood ratio on the easy true/false item drops from â 148 to â 1 between the high levels, and the ordinal case recovers symmetry with the nominal one.
| ID | Difficulty \(b_q\) | \(m_q\) Options | \(c_q\) |
|---|---|---|---|
| Q1 | \(-1\) (easy) | 4 | 0.25 |
| Q2 | \(0\) (medium) | 4 | 0.25 |
| Q3 | \(+1\) (difficult) | 4 | 0.25 |
For each pair \((\theta_i, b_q)\) we calculate \(P(A \mid \theta_i, q)\) with the formula IRT 3PL, \(a = 1.5\), \(c = 0.25\):
For Q1 (\(b_1 = -1\)):
| Hypothesis | \(\theta_i - b_q\) | \(e^{-1.5 \cdot x}\) | \(\sigma\) | \(P(A)\) | \(P(F)\) |
|---|---|---|---|---|---|
| \(H_1\) | \(-2 - (-1) = -1\) | \(e^{1.5} = 4.48\) | \(0.182\) | \(0.387\) | \(0.613\) |
| \(H_2\) | \(0 - (-1) = +1\) | \(e^{-1.5} = 0.223\) | \(0.818\) | \(0.863\) | \(0.137\) |
| \(H_3\) | \(2 - (-1) = +3\) | \(e^{-4.5} = 0.011\) | \(0.989\) | \(0.992\) | \(0.008\) |
For Q2 (\(b_2 = 0\)):
| Hypothesis | \(\theta_i - b_q\) | \(P(A)\) | \(P(F)\) |
|---|---|---|---|
| \(H_1\) | \(-2\) | \(0.286\) | \(0.714\) |
| \(H_2\) | \(0\) | \(0.625\) | \(0.375\) |
| \(H_3\) | \(+2\) | \(0.964\) | \(0.036\) |
For Q3 (\(b_3 = +1\)):
| Hypothesis | \(\theta_i - b_q\) | \(P(A)\) | \(P(F)\) |
|---|---|---|---|
| \(H_1\) | \(-3\) | \(0.258\) | \(0.742\) |
| \(H_2\) | \(-1\) | \(0.387\) | \(0.613\) |
| \(H_3\) | \(+1\) | \(0.863\) | \(0.137\) |
With uniform prior \(\mathbf{p} = (0.333, 0.333, 0.333)\), we calculate the information gain for each question.
Initial entropy:
For Q1 (\(b = -1\)):
Posteriors after a correct answer:
Posterior entropy if correct:
Posteriors after an incorrect answer:
Posterior entropy if it fails:
Information gain:
Summary of information gains (following the same procedure for Q2 and Q3):
| Question | \(IG\) (bits) |
|---|---|
| Q1 (\(b = -1\)) | 0.280 |
| Q2 (\(b = 0\)) | \(0.275\) |
| Q3 (\(b = +1\)) | \(0.212\) |
With \(c_q = 0.25\), the probability of success by chance breaks the symmetry between easy and difficult questions: a failure on an easy question is highly diagnostic, while a success on a difficult question can still be partially explained by chance. In this configuration, the system selects Q1, although Q2 is practically tied.
The student fails Q1. We update:
Entropy after failure:
Entropy has dropped from 1.585 to 0.764 bits. The system suspects quite strongly that the student is of basic level.
With \(\mathbf{p}^{(1)} = (0.809, 0.180, 0.011)\), the system recalculates the information gains for Q2 and Q3 (Q1 has already been used). In this asymmetric state, Q2 is more informative because it helps to check whether the previous failure could have been accidental.
The student answers Q2 correctly (intermediate). We update with \(P(A \mid H_i, Q2) = (0.286, 0.625, 0.964)\):
The success on the medium question shifts part of the probability towards \(H_2\), but the initial failure on an easy question still weighs heavily. The diagnosis is between basic and medium-low.
| Step | Action | \(P(H_1)\) | \(P(H_2)\) | \(P(H_3)\) | \(H\) (bits) |
|---|---|---|---|---|---|
| 0 | Initial prior | 0.333 | 0.333 | 0.333 | 1.585 |
| 1 | Q1 failure (easy) | 0.809 | 0.180 | 0.011 | 0.764 |
| 2 | Answers Q2 correctly (medium) | 0.652 | 0.318 | 0.030 | 1.080 |
Note that the correct answer to the middle question has increased the entropy compared to step 1: the evidence has distributed the probability more between \(H_1\) and \(H_2\), increasing the uncertainty. This is correct: the system continues to almost completely rule out the advanced level, but now hesitates more between basic and intermediate.
With \(p_{\min} = 0.80\) and \(n = 3\), the threshold is \(H_{\text{stop}} = 0.92\) bits. The current entropy (1.080 bits) is above the threshold, so the test continues.
The IRT 3PL logistic function assumes that the hypotheses have an order: more θ means more level. This is appropriate for assessing mastery, but not when the hypotheses are alternative categories with no relation of order between them. They may be misconceptions, resolution strategies, different possible causes of the same error, or thematic areas with no hierarchy between them. If those categories are genuinely alternative, they can be modeled as mutually exclusive hypotheses. If, however, several errors can coexist, they should not be forced into a single nominal list: one should move to a multifactorial model or to a distribution over full profiles.
Example. Let's assume three hypotheses:
In this case there is no single scale of "more or less mastery." The logistic function is not the appropriate model.
Alternative. Define the likelihoods directly according to the expected diagnosis of each question:
| Question | \(P(A \mid H_A)\) | \(P(A \mid H_B)\) | \(P(A \mid H_C)\) |
|---|---|---|---|
| Does the mass of an object change on the Moon? | 0.20 | 0.80 | 0.95 |
| Does a car braking have acceleration? | 0.75 | 0.15 | 0.95 |
| Is force proportional to mass? | 0.50 | 0.50 | 0.90 |
These likelihoods are defined by the teacher or the AI based on knowledge of which errors each confusion produces. The Bayesian update is identical; only the source of the likelihoods changes.
To operate with exclusive non-hierarchical hypotheses, the parametric structure (\(a\), \(b_q\), \(c_q\) per question, §4) is replaced by an explicit likelihood matrix. Given a set of hypotheses \(\{H_1, \ldots, H_n\}\) and a bank of questions \(\{q_1, \ldots, q_m\}\), the model is fully specified by a matrix \(\mathbf{M}\) of size \(m \times n\) whose entries are the probability of a correct answer to each question under each hypothesis:
Each question thus contributes a row of the table in the previous example. This matrix is the equivalent of the ICC (§4.2) of the ordered case, but read from a table instead of computed with a logistic function. If one wants to exploit which specific distractor the student chose âand not just correct/incorrectâ, each question additionally defines a distribution \(P(R = r \mid H_i, q)\), with the responses summing to 1 for each pair \((q, i)\).
If errors can coexist, the same principle is applied over profiles rather than simple hypotheses. With \(k\) binary factors there are up to \(2^k\) possible profiles, and the data structure becomes a matrix \(P(R = r \mid \pi_j, q)\), where \(\pi_j\) is a full profile. In practice this may also be factorized into several parallel dimensions if interactions between errors are weak. The decision does not require the teacher to know this distinction: the AI uses independent factors when each error is evidenced and interpreted separately, and full profiles when the combination of errors changes the expected response, one error masks another, or the pedagogical intervention depends on the combination. If \(2^k\) profiles becomes unmanageable, related factors are grouped or diagnosis is done in phases.
The rest of the methodology works unchanged, replacing every occurrence of the parametric likelihood \(P(A \mid \theta_i, q)\) from the ICC (§4.2) with the corresponding entry of the matrix \(M_{qi} = P(A \mid H_i, q)\):
The only thing that does not transfer is summarizing the posterior as a point on a scale. With ordered hypotheses one reports the expected value \(\mathbb{E}[\theta] = \sum_i p_i\, \theta_i\) (and a level or color derived from it); with nominal hypotheses that weighted average is meaningless.
If the model is exclusive nominal, one may report the maximum a posteriori hypothesis (MAP), \(\hat H = \arg\max_i p_i\), and its probability \(p_{\hat H}\) as confidence. When two hypotheses compete, showing the full posterior distribution over \(\{H_i\}\) is more informative than a single label.
If the model is multifactorial, the final report should be factor-wise: for each error one computes its marginal probability and classifies it as present, absent, or undetermined according to the chosen confidence threshold. The global summary then becomes an error profile (for example, â2 of the 3 analysed errors are presentâ) rather than a single winning label. Caution with the informative prior: with \(P(\text{error}) \approx 0.2\)â\(0.3\), the "absent" classification already starts at 0.7â0.8 with no evidence at all, so the confidence threshold is not enough on its own. A minimum sample of evidence about that factor must also be required, and the report must distinguish "confirmed absent" (with evidence) from "insufficient evidence" (the prior's default value).
The entries \(M_{qi}\) are defined a priori by the teacher or the AI from didactic knowledge about which errors each confusion produces; they are not calibrated with real response data. This carries the same caveats as §11.1 and adds one: unlike the parametric case, there are no "contrasted default values" (\(a_{\text{ef}} = 1{.}25\) with each item's \(a\) derived, \(c_q \approx 1/m_q\)) to fall back on âeach cell of the matrix is a specific judgmentâ. Consequently:
Since the values \(M_{qi}\) do not come from a formula (§10.4), it is advisable to set them with an explicit criterion. Each cell answers a single question: "if the student had \(H_i\), with what probability would they answer \(q\) correctly?".
1. Pedagogical questionâhypothesis relationship. The value depends on whether the question activates the misconception: if \(q\) attacks precisely the concept that \(H_i\) distorts, the student tends to fail and \(M_{qi}\) is low; if \(q\) is unrelated to that error, it does not interfere and \(M_{qi}\) is high (similar to the mastery row); in intermediate cases, a central value. Each row \(H_i\) should thus show the signature of that error âlow where it corrupts, high where it does not reachâ; a flat row indicates that the question does not distinguish that hypothesis.
2. Anchors. Each cell is bounded between two limits: a guessing floor \(M_{qi} \geq 1/m\) when failure comes from random guessing (with \(m\) options, a student answering at random does not fall below that probability; this is the role of \(c_q\) in §4.2) âwith the single exception of point 3â and a slip ceiling for the mastery hypothesis, \(\approx 0.90\text{â}0.95\), never exactly \(1\).
3. Distractor structure. If the question is multiple-choice and one of the distractors is exactly the answer that \(H_i\) produces, the student does not succeed by chance but is drawn toward that wrong option: then \(M_{qi}\) falls below the guessing floor. The criterion is not just "will they fail?" but "which wrong answer does this error generate, and is it among the options?".
4. Coarse buckets. For Bayes what matters is not the exact decimal but the separation: in a discriminating question, the correct hypothesis must clearly exceed the one the error makes fail. It is enough to work in buckets:
| Situation | \(P(A \mid H_i, q)\) |
|---|---|
| Mastery, or the error does not affect the question | \(\approx 0.90\) |
| Partial effect, no distractor capturing the error | \(\approx 0.50\) |
| The error pushes toward a specific distractor | \(\approx 0.15\text{â}0.25\) |
| Guessing floor (minimum, except for the point-3 case) | \(\approx 1/m\) |
In the ordered case this same judgment is packaged inside the difficulty \(b_q\) and the logistic; the nominal case merely makes it explicit cell by cell. Once the matrix is filled, Monte Carlo validation (§11.8) checks whether the set of judgments separates the hypotheses, without needing real data.
The parameters \(a\), \(b_q\) and \(c_q\) are generated automatically from default values and the structure of each question (§4). They are a priori estimates, not empirically calibrated measures: a real student may respond differently from what the model predicts. If response data from a large sample of students were available, the parameters could be refined using IRT estimation methods, but that is not part of this methodology. Nevertheless, the use of well-established default values is supported by the IRT literature as a reasonable starting point in the absence of empirical data: in this methodology the discrimination is not fixed directly as \(a = 1.5\), but rather the effective discrimination \(a_{\text{ef}} = 1.25\) is fixed and each item's \(a\) is derived as \(a = a_{\text{ef}}/(1 - c_q)\) (§4.4), with \(c_q \approx 1/m_q\) (see §4.2 and §12).
Sequential Bayesian updating assumes that the responses are conditionally independent given the true hypothesis: knowing \(H_i\) makes the correlation between answers irrelevant. This assumption is violated when:
The model assumes that the student's state does not change during the session. In short diagnostic evaluation sessions this is reasonable. In long adaptive learning sessions, the student can improve during the interaction itself, which would cause the posterior to converge towards a hypothesis that no longer reflects the current state. This limitation is mitigated by updating with exponential forgetting, or by an explicit transition model, described in §3.5: both prevent old evidence from blocking the tracking of a changing state.
If the student responds systematically at random, the \(c_q\) parameter only partially protects: it reduces the upward bias in the probability of success, but does not eliminate the noise. With enough random responses, the posterior can converge toward incorrect hypotheses.
With few questions, the posterior can be biased by coincidences (a non-representative streak of successes or failures). impose a minimum number of questions before activating the stop criterion reduces this risk, at the cost of lengthening the session.
The system produces a probabilistic estimation, not an absolute truth. The results should be interpreted as an aid to educational decision, especially when:
Limits §11.2â§11.5 describe situations in which a particular student may not conform to the model. It is useful to detect them quantitatively, without external empirical data, from the student's own response pattern in the session: this is what a person-fit statistic measures.
Consider a student who has answered \(N\) questions, with outcome \(x_q \in \{0, 1\}\) (correct/incorrect) on question \(q\), and let \(p_q = P(A \mid \hat\theta, q)\) be the probability of a correct answer that the model assigns to that question under the estimated level \(\hat\theta\) (the most probable hypothesis of the final posterior). The observed log-likelihood of the pattern is:
If the student were really at level \(\hat\theta\), this quantity would have expected value and variance:
The standardized index \(l_z\) (Drasgow, Levine and Williams, 1985) is:
which, under the model, is approximately distributed as \(N(0, 1)\). Strongly negative values (as a rough guide, \(l_z < -2\)) signal a pattern that is improbable under the estimated level âtypically, getting hard questions right and easy ones wrong, slips, or random guessingâ: the diagnosis, even if the posterior reports it as confident, may not be reliable for that student. Values close to \(0\) indicate coherence.
This quantifies the qualitative signal of §11.6: entropy measures how concentrated the model's belief is, but not whether the observed pattern is compatible with that belief; person-fit covers that gap. Limitation: \(l_z\) is an asymptotic approximation; with few questions its distribution departs from the normal and the threshold is indicative, a caution signal, not a formal test. In adaptive tests the calibration is even more delicate: selection tends to present items near the maximum-uncertainty region, and the index is computed using the estimated level, not the true one. Corrections and procedures for this situation exist, such as \(l_z^*\) and person-fit work in CAT, but here \(l_z\) is kept only as a practical low-reliability warning.
Person-fit validates the diagnosis of a particular student; a different question is whether the test as a whole âbank and parametersâ separates the levels well. Since the parameters are a priori and not calibrated (§11.1), this cannot be answered with real data, but the model's own behaviour can be examined through Monte Carlo simulation. In diagnostic resources or resources that decide stage promotion, this check must be available to the resource author: if the AI can execute code, it runs it when generating the resource; if it works in a chat without execution, it leaves a separate utility or teacher/author view to run it in the browser and marks the design as pending validation.
The procedure generates synthetic respondents located at the \(\theta_i\) of each hypothesis. For a respondent of level \(\theta_i\), each response is simulated as a Bernoulli trial with probability \(P(A \mid \theta_i, q)\) given by the ICC (§4.2), applying the same adaptive selection (§6) and the same stopping criterion (§7) as for a real student. In nominal or multifactorial models, responses are simulated from the response distribution of each hypothesis or profile. Repeating this many times per level, hypothesis, or profile estimates the confusion matrix:
The diagonal \(C_{ii}\) is the hit rate per level; the off-diagonal elements, the confusions. From it follow global accuracy, balanced accuracy, the undetermined-result rate, and the average test length. As an operational default, use at least 500 simulations per hypothesis or profile (1000 if the browser handles it smoothly). If any relevant hypothesis falls below 0.70 correct classification under the model itself, or if two hypotheses are systematically confused, the bank must not be presented as well separated: add items, review difficulties or likelihoods, or state the limitation.
Essential limit: the respondents are generated with the same model that then classifies them, so this is not an empirical validation. It measures the internal coherence and separability of the design âif not even ideal respondents located exactly at the \(\theta\) of each level can be told apart, the bank does not discriminate those levelsâ, but it does not guarantee that the \(\theta_i\) and the difficulties \(b_q\) correspond to reality. It is, in Bayesian terms, a pre-posterior check of the decision procedure. Real validity still requires student data (§11.1). Even so, it is a valuable and cheap diagnostic of the design, computable before applying the test to anyone.
How to cite this document (APA 7): de Haro, J. J. (2026). Mathematical foundations of Bayesian adaptive educational systems (Version 2.0). https://jjdeharo.github.io/recursos-adaptativos/matematicas_en.html
License: CC BY-SA 4.0 ¡ Juan JosÊ de Haro ¡bilateria.org