Brainstorming

For inspiration, you might consider:

• Skimming the Project Ideas discussion posts on Canvas
• • Method Ideas
• • Applications in Systems Biology Ideas
• Browsing recent papers from Bayesian Deep Learning workshops
• Discussing ideas on the the Canvas discussion forum
• Feel free to stop by Office Hours to brainstorm with your instructor or TA

Initial Project Pitch

Deliverables:

• Due 10/11: Sign up (every student) for your chosen team & project via Final Project Team Signup "quiz" on Canvas
• In-Class 10/16: Give 5 minute presentations as a team

Instructions: Prepare a ~5 minute presentation with 4 slides that:

• Introduces your team (1 slide)
• • Include project title
• • Include team name
• • List names of all team members. Highlight relevant skills.
• Defines a target task (1 slide)
• • What task? (regression, clustering, etc.)
• • What dataset? (what specific real dataset? why is it exciting?)
• • What evaluation? (how will you know you've been successful? what plot will you make to prove it?)
• Defines a BDL-relevant research question (1 slide)
• • What goal/hypothesis? (e.g. "Prior A will give better performance on metric X than prior B because ___")
• • What models?
• • What algorithms?
• Defines a feasible timeline (1 slide)
• • What are major milestones? How do they depend on each other?
• • How will these milestones fit into the 8 weeks?
• • What are the biggest challenges?

Checkpoint 1: Project Proposal

Instructions:

Submit a short report (about 2-4 pages) via PDF submission to Gradescope: https://www.gradescope.com/courses/23483/assignments/115297/.

For checkpoint 1, you can use any word processor format (Microsoft Word, Google docs, etc.), just be sure to follow the structure below. If you want to use LaTeX, we recommend the the official LaTeX style files for the NIPS 2018 conference. You can also try online collaborative editing of LaTeX documents: NIPS 2018 template on Overleaf.

Your report should have the following structure:

• Section 1: Motivation
• • What task? (regression, clustering, etc.)
• • What goal/hypothesis? (e.g. "This prior will give better performance")
• • Why? Why is this exciting? What is new or unclear that this project will address?
• Section 2: Methods
• • What models? Define the prior and likelihood with equations.
• • What algorithms? MCMC? VI? Something else?
• • Specify exactly one focus method (the approach you think will achieve your goal best)
• • Specify at least one baseline (simpler model or simpler algorithm) which could also partially achieve the goal.
• Section 3: Experiments
• • Identify one relevant "small/toy" dataset available in an accessible tidy format.
• • • Describe the data to non-expert and provide citation.
• • • Provide a table of summary statistics (num. examples, num. dimensions)
• • Identify one relevant "real-world" dataset available in an accessible tidy format.
• • • Describe the data to non-expert and provide citation.
• • • Provide a table of summary statistics (num. examples, num. dimensions)
• • • Is there a train/test split?
• • Describe your intended experimental protocol
• • • What plots will you make to communicate your results?
• • • What code resources will you use? (autograd? pytorch? any specific existing packages?)
• • • (if relevant): What hyperparameter values? What initializations?
• Section 4: Timeline
• • What are major milestones? How do they depend on each other?
• • Can all these milestones fit into the 8 weeks? Provide a chart showing how you've allocated the rest of your project time.
• • What are the next steps for Checkpoint 2?
• • What are the potential roadblocks and how will you avoid them?

Checkpoint 2: Proof of Concept

Instructions

Submit a short report via PDF submission to Gradescope: https://www.gradescope.com/courses/23483/assignments/119599/. Due Tues. Nov. 6 at 11:59pm ET.

Building on the report from Checkpoint 1, submit a revised report in BDL research paper style. Your report should keep the structure and content from Checkpoint 1, with appropriate edits as you respond to your instructor feedback and your own thinking evolves. There are two new sections that are now required, these are denoted below with the label NEW. Use no more than 8 pages (not including references). If you have more use an appendix.

Your report should specifically include the content summarized by the bullet points below. These points will be the focus of this stage of evaluation by your instructor.

Note: The outline below is meant to be somewhat flexible. If you think your project structure is best served by a change, please talk to course staff and we will try to accommodate.

• Section 0: Changelog (NEW)
• • One sentence: Which computing resources are you using/plan to use? Your options include: your laptop, [Amazon AWS provided by Tufts EECS], [Tufts HPC], as well as other options you can find on your own. Please justify your choice.
• • Short bulleted list summary of big changes you've made to your plan since Checkpoint 1, and why you made those changes. The list should have at most 5 entries (so focus on the most important changes).
• Section 1: Motivation
• • Revise your hypothesis statement so that it includes
• • • a comparison between two specific methods (models and/or algorithms)
• • • a claim about a specific measureable performance metric
• • • a "because" statement that tries to explain why one method is better
• • FYI only: For more about why we focus on hypotheses, see Alan Bundy's "THE NEED FOR HYPOTHESES IN INFORMATICS"
• Section 2: Methods
• • Revise so that both methods mentioned in hypothesis are clearly described
• • • For each method (focus and baseline), clearly identify the model, the algorithm, and the implementation you'll use (e.g. Logistic regression with L2 penalty on weights, trained with L-BFGS optimization, using implementation in sklearn). Include citations as appropriate.
• • • For each method (focus and baseline), clearly identify the relevant hyperparameters of the model/algorithm and explain which you will fit and which you will leave fixed. Explain how you will do any fitting/tuning, and why fixed values are justified.
• Section 3: Experiments
• • Provide at least one diagnostic figure, showing why your focus method is (or is not) working as expected
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Example 1: if you are doing VI, show an ELBO objective trace plot like we did in HW3
• • • Example 2: if you are doing HMC, show a trace plot of log potential energy vs iterations (as we did in HW2)
• • Provide at least one partial results figure/table showing the key evaluation metrics for a baseline approach on your small/toy dataset
• • • Example: If you said you'd analyze accuracy on MNIST with a GP classifier baseline, you should have an accuracy number bar graph or table.
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Caption should clearly help reader interpret results in broader context (is the baseline solving the problem well? what better performance should be achievable with a better method? what else has been done in literature?)
• Section 4: Timeline
• • Revise as needed (focus on remaining time)
• • What are the next steps for Checkpoint 3?
• • What are the remaining potential roadblocks and how will you plan ahead to avoid them?
• References (NEW)
• • Include appropriate academic references for your focus method, your baselines, and any code you use. Be sure to include references for both models and algorithms, which are often different.
• • Full citation style can be any style you've seen before in a BDL research paper. Be sure each reference includes a title, author, venue name, year, and (if possible) a URL.
• • In-text citation should be ideally be in (Author Year) format (e.g. bibtex's apalike style, so that readers don't need to flip pages to know who you are citing. However, any format is acceptable as long as it is consistent.

Checkpoint 3: Preliminary Results

Submit a short report via PDF submission to Gradescope: https://www.gradescope.com/courses/23483/assignments/127388/. Due Tues. Nov. 20 at 11:59pm ET.

Building on the report from Checkpoint 2, submit a revised report in BDL research paper style. Your report should keep the structure and content from Checkpoint 2, with appropriate edits as you respond to your instructor feedback and your own thinking and results evolve. Use no more than 8 pages (not including references). Be concise but complete. If you really insist on needing more space, use an appendix. The instructor will focus evaluation on the contents of the first 8 pages.

Your report should specifically include the content summarized by the bullet points below. These points will be the focus of this stage of evaluation by your instructor.

• Section 0: Changelog
• • Short bulleted list summary of big changes you've made to your plan since Checkpoint 2, and why you made those changes. The list should have at most 5 entries (focus on the most important changes).
• Section 1: Motivation
• • Revise as needed, but no strictly new content required.
• • Try to answer any major points raised in previous instructor feedback (esp. about your concrete hypothesis).
• Section 2: Methods
• • Revise as needed, but no strictly new content required.
• • Try to answer any major points raised in previous instructor feedback (esp. about hyperparameters and selection strategies).
• Section 3: Experiments
• • Provide at least one near-complete results figure for your small/toy dataset, showing both your focus method and baseline methods
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Figure should be accompanied by appropriate descriptive text in the paper body.
• • • Text should make clear how the experiment answers the hypothesis
• • Provide at least one preliminary results figure for your large dataset, showing baseline results
• • • Figure should meet professional standards: label axes, provide a caption, etc.
• • • Figure should be accompanied by appropriate descriptive text in the paper body.
• • • Text should make clear how the planned experiment answers the hypothesis
• • Provide at least one diagnostic figure for your large dataset, showing that you have tried to run your focus method
• • • Example: could be trace plots for VI or HMC
• • • If you can, try to do more than just include one line plot (e.g. show results from an experiment you would need to do anyway, comparing learning rates, architectures, etc.).
• Section 4: Timeline
• • Revise as needed (focus on remaining time)
• • What are the next steps before your final presentation?
• • What are the remaining potential roadblocks and how will you plan ahead to avoid them?

Final Presentation

Prepare slides for an 8 minute presentation, given as a team.

You will be evaluated according to this rubric: [GoogleDoc: BDL Final Presentation Rubric]

There will not be a rigid required structure, but here is a template that could work well for most projects. Do feel free to deviate if you feel a different organization system makes sense, but be sure your choice can still be mapped to the rubric above.

• Define Project Goal (~ 1 slide)
• • Motivate the problem you are trying to solve and why it matters
• • Formulate the task (define input and output of computation system)
• • Summarize related work for an audience of machine learning researchers
• Present a Research Question / Hypothesis (~1 slide)
• • Make claim about comparison between focus and baseline(s) methods
• • Identify measurable performance metric(s) appropriate to goal
• • Provide because/why reasoning that is plausible to domain experts
• Define a baseline method (~1 slide)
• • Precise description of major pieces (model/algo/implementation)
• • Emphasize at least one strength and at least one limitation
• • Highlight important hyperparameters and how you set them
• Define a proposed/focus method (~1 slide)
• • Precise description of major pieces (model/algo/implementation)
• • Emphasize at least one strength and at least one limitation
• • Highlight important hyperparameters and how you set them
• Provide Experimental Results on ONE dataset (pick one that's most interesting) (~1 slide)
• • Dataset and evaluation protocol (esp. train/validation/test) clear and appropriate
• • Performance metrics are clear and appropriate
• • Focus and baseline methods compared with critical interpretation
• • Answer provided to the research question/hypothesis
• Provide a Reflection and Outlook (~1 slide)
• • Short-term future work is sensible (What will you finalize for the report?)
• • Summarize one key take-home lesson about BDL methods from your project
• • Summarize one key take-home lesson about teamwork/planning/evaluation

Final Report

Submit a final report via PDF submission to Gradescope: https://www.gradescope.com/courses/23483/assignments/127435/. Due Tues. Dec. 18 at 11:59pm ET.

Your final report should be fully in BDL research paper style, using the section structure below. Use no more than 8 pages (not including references). Be concise but complete. If you really insist on needing more space, use an appendix. The instructor will focus evaluation on the contents of the first 8 pages, so these should be self-contained.

Your report should specifically include the content summarized by the bullet points below. All these points should build on the reports you've submitted from earlier checkpoints.

• Section 0: Abstract
• • Include a brief abstract in the style of a BDL paper.
• • Abstract should summarize the major ideas, contributions and results.
• • Remove the old changelog section.
• Section 1: Motivation
• • Motivate the problem you are trying to solve and why it matters
• • Formulate the intended task (define input and output of computation system)
• • Summarize related work for an audience of machine learning researchers
• • Articulate a clear research hypothesis
• Section 2: Methods
• • For both a focus method and one or more baseline(s), please include:
• • • Precise description of the model, algorithm, and implementation with relevant equations as needed.
• • • Highlight important hyperparameters and strategies for tuning or fixing values.
• Section 3: Experiments
• • Include at least 2 results figures, one for the smaller and one for the larger datasets:
• • • Figures should meet professional standards: label axes, provide a self-contained caption, etc.
• • • Figures should be accompanied by appropriate descriptive text in the paper body that critically interprets the results (why did one method outperform another? is this expected? can we understand why?).
• • • Text should make clear how the experiment answers the hypothesis
• • Optional: As needed, include any diagnostic figures or other figures needed to illustrate the problem, the approach, or justify claims in the narrative
• Section 4: Reflection and Outlook
• • Reflect on short-term future work: What are the easiest things to improve about the current pipeline?
• • Reflect on long-term future work: What are the most important remaining limitations and how would you attack them?
• • Provide one non-trivial take-home message about BDL methods
• References
• • Include formal citations to all relevant related work (datasets, evaluation metrics, methods, implementations, etc.)
• • Full citation style can be any style you've seen before in a BDL research paper. Be sure each reference includes a title, author, venue name, year, and (if possible) a URL.
• • In-text citation should be ideally be in (Author Year) format (e.g. bibtex's apalike style, so that readers don't need to flip pages to know who you are citing. However, any format is acceptable as long as it is consistent.