Assignments

Pros and Cons

In the "Pros & Cons" post, please show that you've read the paper critically by providing the following:

• Pros: 1-2 full sentences. What is the most exciting or interesting idea or method described here? Why is it important? Don't just copy the abstract - what do you think?
• Cons: 1-2 full sentences. What is the biggest weakness of this method, model, or algorithm? Problems may include weak empirical validation, missing theory, unacknowledged assumptions, applicability to a narrow range of problems, or more.

Open Questions

In the "Questions" post, please ask a question you'd like addressed in class. Your instructor will use these questions to plan out what gets covered.

Feel free to highlight unclear sections, steps you didn't follow, assumed background knowledge you don't have, or more.

** Reading discussion format inspired by Prof. Erik Sudderth's fall 2011 course at Brown **

Brainstorming

For inspiration, you might consider:

• Skimming the Project Ideas discussion posts on Piazza
• Browsing papers from recent Bayesian Deep Learning workshops
• • 2018 workshop at NeurIPS
• • 2017 workshop at NeurIPS
• Feel free to stop by Office Hours to brainstorm with your instructor or TA

Initial Project Pitch

Deliverables:

• Due 10/09: Sign up (every student) for your chosen team & project via "Final Project Team Signup"
• Due 10/16: In Class: Give 5 minute presentations as a team

Instructions: Prepare a ~5 minute presentation with 4 slides that:

• Introduces your team (1 slide)
• • Include project title
• • Include team name
• • List names of all team members. Highlight relevant skills.
• Defines a target task (1 slide)
• • What task? (regression, clustering, etc.)
• • What dataset? (what specific real dataset? why is it exciting?)
• • What evaluation? (how will you know you've been successful? what plot will you make to prove it?)
• Defines a BDL-relevant research question (1 slide)
• • What goal/hypothesis? (e.g. "Prior A will give better performance than prior B on task X because ___")
• • What models?
• • What algorithms?
• Defines a feasible timeline (1 slide)
• • What are major milestones? How do they depend on each other?
• • How will these milestones fit into the remaining ~7 weeks of the semester?
• • What are the biggest challenges?

Checkpoint 1: Project Proposal

Instructions:

Submit a short report (about 2-3 pages) via PDF submission to Gradescope: https://www.gradescope.com/courses/61382/assignments/283795.

For checkpoint 1, you can use any word processor format (Microsoft Word, Google docs, etc.), just be sure to follow the structure below. If you want to use LaTeX, we recommend the the official LaTeX style files for the NIPS 2018 conference. You can also try online collaborative editing of LaTeX documents: NIPS 2018 template on Overleaf.

Your report should consist of several paragraphs of complete sentences, organized in the following structure:

• Section 1: Motivation
• • What task? (regression, clustering, etc.)
• • What goal/hypothesis? (e.g. "This prior will give better performance")
• • Why? Why is this exciting? What is new or unclear that this project will address?
• Section 2: Methods
• • What is the probabilistic model? Define the key random variables, as well as the specific prior and likelihood distributions. Use math equations. Try to keep notation simple.
• • What is the algorithm you'll use to estimate the posterior? MCMC? VI? Something else?
• • Specify exactly one focus method (the approach you think will achieve your goal/hypothesis best)
• • Specify at least one baseline (simpler model or simpler algorithm) which could also partially achieve the goal.
• Section 3: Experiments
• • Identify one relevant dataset (can be real data or toy/synthetic data)
• • • Describe the data to non-expert and provide citation.
• • • Provide some relevant summary statistics (e.g. num. examples, num. dimensions), either in the main paragraph or in a table.
• • • Make sure the reader understands how this data was chosen to answer your goal/hypothesis
• • Describe your intended experimental results
• • • What specific 1 or 2 figures/tables will you make to communicate your results? You can either make "cartoon" versions of the figures or describe them clearly in text
• • • How will these plots/tables either confirm or refute your hypothesis?
• • Describe your implementation plan
• • • What code resources will you use? (autograd? pytorch? any specific existing packages?)
• • • What major parts (if any) will you need to write yourself?
• Section 4: Timeline
• • What are the next steps for Checkpoint 2?
• • What are the potential roadblocks between now and Checkpoint 2 and how will you avoid them?

Checkpoint 2: Proof of Concept

Last Updated: Monday 2019-11-11 09:00 am

Due Date: Friday 2019-11-15 11:59 pm

What to Turn In:

Submit a short report via PDF submission to Gradescope: <https://www.gradescope.com/courses/61382/assignments/294228/>

If for some reason gradescope isn't working, submit via email to both Prof. Mike Hughes and TA Ramtin.

Instructions

Building on the report from Checkpoint 1, submit a revised report in BDL research paper style. Your report should keep the structure and content from Checkpoint 1, with appropriate edits as you respond to your instructor feedback and your own thinking evolves. There are two short-and-easy new sections that are now required (Changelog and References). These are denoted below with the label NEW. Use no more than 8 pages (not including references). If you have more use an appendix.

Your report should specifically include the content summarized by the bullet points below. These points will be the focus of this stage of evaluation by your instructor.

Note: The outline below is meant to be somewhat flexible. If you think your project structure is best served by a change, please talk to course staff and we will try to accommodate.

• Section 0: Changelog (NEW)
• • Short bulleted list summary of big changes you've made to your plan since Checkpoint 1, and why you made those changes. The list should have at most 5 entries (so focus on the most important changes).
• Section 1: Motivation
• • Revise your hypothesis statement so that it includes
• • • a comparison between two specific methods (models and/or algorithms)
• • • a claim about a specific measureable performance metric
• • • a "because" statement that tries to explain why one method is better
• • FYI only: For more about why we focus on hypotheses, see Alan Bundy's "THE NEED FOR HYPOTHESES IN INFORMATICS"
• Section 2: Methods
• • Revise from Checkpoint 1 so that both methods mentioned in hypothesis are clearly described (with math as needed)
• • • For each method (focus and baseline), clearly identify the model, the algorithm, and the implementation you'll use. Include citations as appropriate.
• • • • An example good description would be: "We will use a logistic regression binary classifier with an L2 penalty on weights but none on the bias parameter. We'll trained using the L-BFGS optimization algorithm, using the 'lbfgs' implementation in sklearn (Pedregosa et al. 2011)"
• Section 3: Experiments
• • For each method (focus and baseline), clearly identify the relevant hyperparameters of the model /algorithm (learning rates, regularization penalties, etc.) and explain which you will fit and which you will leave fixed.
• • • Explain how you will do any fitting/tuning (grid search? optimizing what performance metric?)
• • • Explain briefly why any chosen fixed values are justified (simplicity is an OK reason).
• • Provide key details about your intended experimental protocol
• • • How will you split data into train/validation/test? Randomly? By subject?
• • • About how long will you run any iterative algorithm (e.g. MCMC/VI). How will you assess if you have converged?
• • • Why will your experimental results confirm/refute your hypothesis?
• • • Consider following the ML "reproducibility checklist" (by Joelle Pineau, google it)
• • Provide at least one diagnostic figure, showing why your focus method is (or is not) working as expected
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Example 1: if you are doing VI, show an ELBO objective trace plot like we did in HW3
• • • Example 2: if you are doing HMC, show a trace plot of log potential energy vs iterations (as we did in HW2)
• • FYI last year's requirement for a preliminary results figure is not necessary.
• Section 4: Timeline
• • Revise as needed (focus on remaining time)
• • What are the next steps for Checkpoint 3?
• • What are the remaining potential roadblocks and how will you plan ahead to avoid them?
• References (NEW)
• • Include appropriate academic references for your focus method, your baselines, and any code you use. Be sure to include references for both models and algorithms, which are often different.
• • Full citation style can be any style you've seen before in a BDL research paper. Be sure each reference includes a title, author, venue name, year, and (if possible) a URL.
• • In-text citation should be ideally be in (Author Year) format (e.g. bibtex's apalike style, so that readers don't need to flip pages to know who you are citing. However, any format is acceptable as long as it is consistent.

Checkpoint 3: Preliminary Results

Last Updated: Monday 2019-11-18 12:00 pm

Due Date: Monday 2019-11-25 11:59 pm

What to Turn In: Submit your PDF report to Gradescope: <https://www.gradescope.com/courses/61382/assignments/299636/>

Building on the report from Checkpoint 2, submit a revised report in BDL research paper style. Your report should keep the structure and content from Checkpoint 2, with appropriate edits as you respond to your instructor feedback and your own thinking and results evolve. Use no more than 8 pages (not including references). Be concise but complete. If you really insist on needing more space, use an appendix. The instructor will focus evaluation on the contents of the first 8 pages.

Your report should specifically include the content summarized by the bullet points below. These points will be the focus of this stage of evaluation by your instructor.

• Section 0: Changelog
• • Short bulleted list summary of big changes you've made to your plan since Checkpoint 2, and why you made those changes. The list should have at most 5 entries (focus on the most important changes).
• Section 1: Motivation
• • Revise as needed, but no strictly new content required.
• • Try to answer any major points raised in previous instructor feedback (esp. about your concrete hypothesis).
• Section 2: Methods
• • Revise as needed, but no strictly new content required.
• • Try to answer any major points raised in previous instructor feedback (esp. about hyperparameters and selection strategies).
• Section 3: Experiments
• • Provide at least one near-complete results figure showing some result for both your focus method and baseline methods, on some reasonable dataset (could be a small toy dataset, or something bigger)
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Figure should be accompanied by appropriate descriptive text in the paper body.
• • • Text should make clear how the intended experiment answers the hypothesis' central question
• • • It is OK if not everything is working yet. Try to get as far as you can. Use text to indicate what is working and what is still being debugged.
• Section 4: Timeline
• • Revise as needed (focus on remaining time)
• • What are the next steps before your final presentation? (Hint: look below at what results are expected)
• • What are the remaining potential roadblocks and how will you plan ahead to avoid them?

Final Presentation

Prepare slides for an 8 minute presentation, given as a team.

You will be evaluated according to this rubric: [GoogleDoc: BDL Final Presentation Rubric]

There will not be a rigid required structure, but here is a template that could work well for most projects. Do feel free to deviate if you feel a different organization system makes sense, but be sure your choice can still be mapped to the rubric above.

• Define Project Goal (~ 1 slide)
• • Motivate the problem you are trying to solve and why it matters
• • Formulate the task (define input and output of computation system)
• • Target an audience of machine learning graduate students who have taken this class before, but who may not know much at all about your domain (e.g. reinforcement learning or computer vision or NLP or whatever)
• Present a Research Question / Hypothesis (~1 slide)
• • Make claim about comparison between focus and baseline(s) methods
• • Identify measurable performance metric(s) appropriate to goal
• • Provide because/why reasoning that is plausible to domain experts
• Define a baseline method (~1 slide)
• • Precise description of major pieces (focus on model, some details on algo/implementation)
• • Emphasize at least one strength and at least one limitation
• • Highlight an important hyperparameter and how you set it
• Define a proposed/focus method (~1 slide)
• • Precise description of major pieces (focus on model, some details on algo/implementation)
• • Emphasize at least one strength and at least one limitation
• • Highlight an important hyperparameter and how you set it
• Provide Experimental Results on ONE dataset (pick most interesting) (~1 slide)
• • Dataset and evaluation protocol (esp. train/validation/test) clear and appropriate
• • Performance metrics are clear and appropriate
• • Focus and baseline methods compared with critical interpretation
• • Preliminary answer provided to the research question/hypothesis
• Provide a Reflection and Outlook (~1 slide)
• • Short-term future work is sensible (What will you finalize for the report?)
• • Summarize one key take-home lesson about BDL methods from your project
• • Summarize one key take-home lesson about your task/evaluation/domain (try to say something more exciting than "ML is hard")

Final Report

Submit a final report via PDF submission to Gradescope: <https://www.gradescope.com/courses/61382/assignments/318807>

Your final report should be fully in BDL research paper style, using the section structure below. Use no more than 8 pages (not including references). Be concise but complete. If you really insist on needing more space, use an appendix. The instructor will focus evaluation on the contents of the first 8 pages, so these should be self-contained.

Your report should specifically include the content summarized by the bullet points below. All these points should build on the reports you've submitted from earlier checkpoints.

• Section 0: Abstract
• • Include a brief abstract in the style of a BDL paper.
• • Abstract should summarize the major ideas, contributions and results.
• • Remove the old changelog section.
• Section 1: Motivation
• • Motivate the problem you are trying to solve and why it matters
• • Formulate the intended task (define input and output of computation system)
• • Summarize related work for an audience of machine learning researchers
• • Articulate a clear research hypothesis
• Section 2: Methods
• • For both a focus method and one or more baseline(s), please include:
• • • Precise description of the model, algorithm, and implementation with relevant equations as needed.
• • • Highlight important hyperparameters and strategies for tuning or fixing values.
• Section 3: Experiments
• • Include at least 2 results figures appropriate to answering the questions raised by your hypothesis:
• • • Figures should meet professional standards: label axes, provide a self-contained caption, readable text size, etc.
• • • Figures should be accompanied by appropriate descriptive text in the paper body that critically interprets the results (why did one method outperform another? is this expected? can we understand why?).
• • • Text should make clear how the experiment answers the hypothesis
• • Optional: As needed, include any figures needed to illustrate the task, the data you are using, the approach, or justify claims in the narrative. If you would want to put the figure in a submitted BDL research paper, please include it.
• Section 4: Reflection and Outlook
• • Reflect on short-term future work: What are the easiest things to improve about the current pipeline?
• • Reflect on long-term future work: What are the most important remaining limitations and how would you attack them?
• • Provide one non-trivial take-home message about BDL methods
• References
• • Include formal citations to all relevant related work (datasets, evaluation metrics, methods, implementations, etc.)
• • Full citation style can be any style you've seen before in a BDL research paper. Be sure each reference includes a title, author, venue name, year, and (if possible) a URL.
• • In-text citation should be ideally be in (Author Year) format (e.g. bibtex's apalike style, so that readers don't need to flip pages to know who you are citing. However, any format is acceptable as long as it is consistent.