Calibration

Alexandre Bouchard-Côté

Bayesian calibration

Thought experiment: what if a certain Bayesian inference method is used many times?

Calibration of credible intervals: a calibrated 90% credible interval should contain the true parameter 90% of the time.

What do we mean by “true parameter”?

What do we mean by “90% credible interval” a function \(C(x) = [L(x), R(x)]\) which (1) computes the posterior \(\pi(\cdot | x)\), and (2), selects left and right end points, \(l\), \(r\) such that \(\int_l^r \pi(z | x) {\text{d}}z = 0.9\). Example: HDI from last week.

What do we mean by “90% of the time”

Well specified vs misspecified models

Short exercise

Poll: make a guess!

  1. calibrated for small data, calibrated for large data
  2. not calibrated for small data, calibrated for large data
  3. only approximately calibrated for both small and large data
  4. none of the above

Readings

After doing this week’s exercise on calibration, read the following tutorial, especially sections 4 and 6: https://arxiv.org/abs/2011.01808