Syllabus: Topics in Bayesian Analysis and Decision Theory.

Description: Bayesian statistics provides a wide range of tools to approach data analysis. This course is composed of (1) a Bayesian "bootcamp" facilitated by probabilistic programming languages; (2) more specialized topics, with an emphasis on computational Bayesian statistics; (3) a final project in teams or individually.

Prerequisite: Background in probability (e.g. Stat 302 or equivalent). Exposure to inference or machine learning (e.g. STAT 460 / 560 or MATH 419 or CS 540 or equivalent). If you are not sure, come talk to me after one or two lectures.

Evaluation

• Exercises (they will involve coding, about 3 total) 40%
• Final project (teams encouraged) 50%
• Participation: 10%
  • in class and/or labs and/or piazza and/or office hours, reporting typos in notes,
  • scribing/editing activity potentially.

Handing-in files:

• Name your file using your student number followed by a description of the file, e.g. 123423423-hw1.pdf. Use a zip archive if handing-in more than one file.
• Email to TA and I

Final project:

The course project involves independent work on the topic of your choice but with the constraint that it should incorporate one of the following (in some exceptional cases a project not doing so might in principle be possible but if you want to do that a case should be made well in advance that the proposed plan is directly relevant to this course, i.e. the bulk of it is going to use or extend techniques taught in the course)

• A careful and scientific comparison of a Bayesian estimator with another one, either Bayesian or non-Bayesian. Review the literature on both side so as to be fair and critical to both sides of the comparison. State and defend the criteria you use. Consider calibration and M-open setups. Examples:

  • Bayesian vs frequentist... regression/classification, feature selection, density estimation, survival analysis, ...
  • Is there some structure that can be exploited (e.g. informed by the data types for the covariates/features, groups of related features i.e. feature templates, hierarchical approaches, etc), to get better Bayesian methods on these generic classes of inference problems?

• A Bayesian inference method over a non-standard data type. Acquire or write an efficient posterior inference method, either using a PPL or from scratch. Develop a novel Bayes estimator and implement it. Benchmark the Bayes estimator on synthetic data, comparing the performance with a naive baseline such as MAP. Examples:

  • Types of graphs such as matchings
  • Phylogenetic trees or networks
  • Multiple sequence alignments
  • Clustering or feature matrices

• Careful construction of a model for a real dataset. Defend the prior and likelihood using best practices. Validate implementation and statistical properties. Use the model to address a scientific/engineering problem.

• Create a twist on an existing MCMC sampling algorithm, or a novel one. Show it is invariant with respect to the distribution of interest. Benchmark the performance of the method against one baseline using best practices. Examples:

  • Population based moves
  • Divide-and-Conquer SMC for pseudo-marginal phylogenetics or other setups usually approached via ABC
  • Non-reversible or PDMP based methods
  • Extensions of parallel tempering methods

Note: some projects may combine many of the above.

Final project logistics:

• Teams are encouraged, in which case you should outline the final report who did what. Expectation will grow linearly in the group size (meaning group of more than 2 is probably not a great idea in most cases!).
• Submit by email to the TA and myself, with subject line 'STAT 520 Final Project':
  • A pdf document of up to 8 pages excluding references
  • A link to a public github repo containing code and tex files. I may have a quick look at commit history to ensure all parties contributed (don't obsess about it, I know line numbers are very rough).
    
• Grading: I will base the grade on the same factors one would usually consider in a paper reviewing process (but taking into account the fact that the time frame is shorter than the typical time to write a research paper). Is the goal clearly defined? Is it well motivated? Is the approach sound? Creative? Is the paper well-written? Are there interesting connections made to the existing literature?

Project timeline:

• March 4: optional discussion during office hour on the proposed projects
• March 18: during lecture/office hours, be ready to discuss potential ideas
• March 25: prepare abstract, freeze teams, each team sends a 1 pager by email
• April 24 at night: due date. Hard deadline!

Textbook: I recommend C. Robert's "The Bayesian Choice". Other optional references will be posted at the bottom of the lecture note files.

Additional readings:

• Resources on Bayesian inference (one of the many motivations for Monte Carlo).
  • Lecture notes available for my course on Bayesian statistics (2015). Notes from a previous iteration of the course (2014) that focussed more on Bayesian non-parametric statistics.
  • Bayesian Data Analysis (2013). Gelman et al.
    • Another popular reference on Bayesian inference.
• Resources on Monte Carlo:
  • Monte Carlo theory, methods and examples (2013). Art Owen.
    • Available from the author's webpage..
    • Excellent treatment of Importance Sampling in particular.
  • Monte Carlo strategies in scientific computing (2001). Jun S. Liu.
    • Available through SpringerLink behind UBC paywalls or VPN.
    • Good reference for the basics but missing important new developments from the past 15 years.
  • Handbook of Monte Carlo Methods (2011). Dirk P. Kroese, Thomas Taimre, and Zdravko I. Botev.
    • Seems to cover some slightly more recent algorithms but still missing lots of developments that happened in this decade.
• Resources on probabilistic programming
  • Blang, which efficiently supports both discrete, combinatorial and continuous random variables; especially useful for hard non-convex problems.
  • Frameworks focussing on autodiff+HMC, e.g. for continuous random variable and target not too multimodal: Stan, Edward, pyMC3, pyro (some support discrete variable at various degrees of efficiency).
  • Languages focussing on maximum expressivity: Anglican, Birch.
  • Many more.

Office hours: See Contact tab above.