Assignment: Operational Semantics

COMP 105

Due Monday, September 28, 2020 at 11:59PM

If you’re going to talk about languages you’ve never seen before, you need a vocabulary. This assignment introduces you to the basics of operational semantics, inference rules, and syntactic proof technique. You will use these skills heavily throughout the first two-thirds of the course, and after 105 is over, when you want to understand a new language idea, you will use them again.

Some of the essential skills are

Few of these skills can be mastered in a single assignment. When you’ve completed the assignment, I hope you will feel confident in your knowledge of exactly the way judgment forms, inference rules, and derivations are written. On the other skills, you’ll have made a start.

On this assignment, there is no pair programming.

Retrieval practice

Before tackling reading-comprehension or other homework questions, you have the option to check your understanding by looking at the “retrieval practice” questions in section 1.10.1, which starts on page 75 of the textbook. Look at questions G through N, but not M. (Example question: what does the metavariable x stand for?) If you can answer these questions quickly and easily, then you are prepared to tackle the actual reading-comprehension and other homework questions. If you can’t answer a retrieval-practice question, a TA will be happy to answer it for you.

Do not submit answers to any retrieval-practice questions. They are indicators of preparedness, nothing more.

Part A: Reading comprehension (individual work, 10 percent)

Before starting other problems, answer the numbered questions below. You can download them.

For questions 1–7, please read pages 28–37 (the book sections on environments and on operational semantics of expressions). These questions are multiple-choice.

  1. ξ is an environment that maps names to

    1. only user-defined functions.
    2. only the values of formal parameters.
    3. both primitive and user-defined functions.
    4. the values of both global variables and formal parameters.
    5. only primitive functions.
    6. only the values of global variables.
  2. ϕ is an environment that maps names to

    1. only user-defined functions.
    2. only the values of formal parameters.
    3. both primitive and user-defined functions.
    4. the values of both global variables and formal parameters.
    5. only primitive functions.
    6. only the values of global variables.
  3. ρ is an environment that maps names to

    1. only user-defined functions.
    2. only the values of formal parameters.
    3. both primitive and user-defined functions.
    4. the values of both global variables and formal parameters.
    5. only primitive functions.
    6. only the values of global variables.
  4. In the operational semantics, what kind of a thing does the metavariable e stand for?

    1. an environment
    2. an Impcore variable
    3. an elaboration
    4. an expression
    5. a value
    6. a function
  5. In the operational semantics, what kind of a thing does the metavariable v stand for?

    1. an environment
    2. an Impcore variable
    3. an elaboration
    4. an expression
    5. a value
    6. a function
  6. In the operational semantics, what kind of a thing does the phrase ρ{x ↦ 7}(x) stand for?

    1. an environment
    2. an Impcore variable
    3. an elaboration
    4. an expression
    5. a value
    6. a function
  7. In the operational semantics, what kind of a thing does the phrase ρ{x ↦ 7}{x ↦ 8} stand for?

    1. an environment
    2. an Impcore variable
    3. an elaboration
    4. an expression
    5. a value
    6. a function

Questions 8 and 9 are also based on pages 28–37. Please answer a number.

  1. How many rules have the IF syntactic form in the conclusion?

  2. How many rules have the APPLY syntactic form in the conclusion? (Look at all the rules in the section, not just the summary on page 80.)

Now let’s understand a subtle point about rules. Study the FormalVar and FormalAssign rules starting on page 32. In FormalVar, the initial and final states have the same ρ. But in the FormalAssign rule, there is an initial state with ρ, an intermediate state with ρ, and a final state with ρ′{x ↦ v}. Answer these questions:

  1. In rule FormalVar, the initial and final state have the same ρ because

    1. Evaluating x might change the value of some formal parameter.
    2. Evaluating x might not change the value of any formal parameter.
    3. Evaluating x doesn’t change the value of any formal parameter.
  2. In rule FormalAssign, there is an intermediate state with ρ (rho-prime) because

    1. Evaluating e might change the value of some formal parameter.
    2. Evaluating e might not change the value of any formal parameter.
    3. Evaluating e doesn’t change the value of any formal parameter.
  3. Consider the sample derivation on page 58. The same ρ is used throughout the derivation because

    1. Every subexpression is known, and because there is no unknown subexpression, there is no need for a ρ (rho-prime).
    2. No part of the evaluation changes the value of a formal parameter.
    3. The example derivation takes a shortcut and doesn’t actually conform to the rules.

Part B: Talking operational semantics (25%)

The rest of the assignment is divided into parts B, C, and D. This assignment is almost exclusively a theory assignment, and almost everything goes into one file theory.pdf. (The exception is some code you will write for part (d) of exercise 16, in part C, which goes into file awk-icon.imp.) In this part (part B), you translate between colloquial, informal English and the language of operational semantics.

Related reading:

From exercise 10 on page 79 of Build, Prove, and Compare, complete parts (a) and (d). Your informal English must not use any symbols that refer to environments—talk about environments using your knowledge of what names stand for. Make your explanations as simple as possible, and use only language that a COMP 11 student would understand. (In other words, it is OK to talk about “variables,” but it is not OK to talk about “environments.”)

From exercise 11 on page 79 of Build, Prove, and Compare, complete parts (a), (c), and (d). Notes:

  • The convention of the field is that new metavariables are implicitly universally quantified. For example, if you just write a global-variable environment ξ, you’re assumed to be talking about any possible ξ. If what you’re saying is true for just some environments, the conventional opening is to say “there exists a ξ such that.” (Or more likely, “There exist ξ, ϕ, ρ, v, ξ, and ρ such that.”)

  • If you need to write an implication, use the words “if” and “then.” Do not use inference-rule notation—every new inference rule adds a case to every metatheoretic proof.

  • Some of the statements in this problem most easily formalized in the form “if evaluation judgment then conclusion.”

Exercise R. The following notation suggests an alternative to the WhileEnd rule of Impcore. Either explain why this rule is effectively the same as the original rule, or explain why it’s different.

          ⟨e1,ξ,φ,ρ⟩ ⇓ ⟨0,ξ′,φ,ρ′⟩
  ---------------------------------------- (WhileEnd')
  ⟨while(e1, e2), ξ, φ, ρ⟩ ⇓ ⟨0, ξ′, φ, ρ′⟩

The following notation suggests an alternative to the FormalAssign rule of Impcore. Either explain why this rule is effectively the same as the original rule, or explain why it’s different.

    x ∈ dom ρ    ⟨e,ξ,φ,ρ⟩ ⇓ ⟨v,ξ′,φ,ρ′⟩
  ------------------------------------------------ (FormalAssign')
  ⟨set(x, e), ξ, φ, ρ⟩ ⇓ ⟨v, ξ′, φ, ρ′⟩

Note: A convincing explanation of why two rules are different is should include not only an explanation in English but also a code example on which the two rules behave differently.

Part C: Operational semantics and language design (25%)

This part contains just one exercise, in which you explore two alternative ways of dealing with unbound variables, and you write code to distinguish them.

We encourage you to discuss ideas, but no other student may see your rules or your code. If you have difficulty, find a TA, who can help you work a couple of similar problems.

Do all parts of exercise 16 on page 82 of Build, Prove, and Compare. This is an exercise in language design. The exercise will give you a feel for the kinds of choices a language designer might have made in a language you have never seen before. It will also give you a tool you can use to think about the consequences of language-design choices even without an implementation.

To complete the exercise, you must analyze three variations on a design: the Impcore standard and two alternatives, which resemble the languages Awk and Icon. For the Impcore standard, you can confirm the results of your analysis using the Impcore implementation. But for the Awk-like and Icon-like variations, you don’t have an implementation that you can use to verify the results of your analysis. To get the problem right, you have two choices: think carefully about the semantics you have designed and the program you have written—or build two more interpreters, so that you can actually test your code. (Each new interpreter requires only a two-line change to file eval.c, so if you wanted to build new interpreters, you wouldn’t be deep in the weeds.)

Part (d) involves coding from scratch, and it could involve new functions. But these functions are not trying to do anything useful with data; instead, they are trying to tease out differences in language semantics. Moreover, unless you choose to build interpreters, you cannot run unit tests of the Awk-like and Icon-like semantics. For these reasons, the only steps we expect from our recommended design process are a name and a contract for each function you choose to write (steps 3 and 4).

We will assess your code by running it in three interpreters. This assessment leaves you vulnerable to these common mistakes:

  • You might define a function and forget to call it. If you forget to call your function, then when we run your code, the last thing the interpreter does will probably not be to print 0 or 1, which is what is called for in the exercise.

  • You might forget that after evaluating an expression, the interpreter prints the result of the expression.

  • You might use print or printu where you really meant println.

  • You might include unit tests in your code. In that case, the last thing the interpreter prints will be the results of running the unit tests.

Part D: Operational-semantics derivations, proofs, and metatheory (40%)

This part takes you into proof: theory, derivations, and metatheory. We encourage you to discuss ideas, but no other student may see your rules, your derivations, or your proof. If you have difficulty, find a TA, who can help you work a couple of similar problems.

Related reading:

  • For an explanation of a valid derivation and for the algorithm used to build one, see section 1.7, which starts on page 56.
  • For an example of a derivation tree, see page 58.
  • For rules of operational semantics, see section 1.5, which starts on page 29. The most important rules are summarized on pages 80–81.
  • For metatheory, see section 1.7.2, which starts on page 60.

Do exercise 12 on page 79 of Build, Prove, and Compare. The purpose of the exercise is to develop your understanding of derivations, so be sure to make your derivation complete and formal. You can write out a derivation like the ones in the book, as a single proof tree with a horizontal line over each node. If you prefer, you can write a sequence of judgments, number each judgment, and write a proof tree containing only the numbers of the judgments, which you will find easier to fit on the page.

In this exercise, or in writing any derivation, the most common mistake made is to copy judgments blindly from the rules of the semantics. This kind of copying results in superfluous primes. In the rules, the primes in ξ and ρ are a way of saying “I don’t know.” In particular, what’s unknown is the exact nature of the subexpressions, and therefore the results of evaluating them. (Notice that the syntactic forms VAR and LITERAL don’t have any subexpressions, and their rules don’t have any primes.) In the expression (begin (set x 3) x), all of the subexpressions are known, and a correct derivation doesn’t have any primes.

Do part (a) of exercise 13 on page 79 of Build, Prove, and Compare. Now that you know how to write a derivation, in this exercise you start reasoning about derivations. This problem calls for a math-class proof about formal semantics, so any formal derivations you write need to be supplemented by a few words explaining what the formal derivation is and what role it plays in the proof.

As in the previous exercise, be wary of primes. The ξ, ρ, ξ and ρ in the problem are not necessarily different from the initial environments or from each other. The primes say only that they might be different.

Exercise F: Metatheory. This final exercise requires you to raise your game again by reasoning about the set of all valid derivations. It’s metatheory. Metatheoretic proofs are probably unfamiliar, but you will have a crack at them in lecture and in recitation.

Metatheory is a fantastic tool, because it gives you results that can apply to any program written in a language. But it’s hard to get started: there are useful metatheoretic results, and there are easy metatheoretic results, but I don’t know any useful, easy metatheoretic results. Here are a couple of results that are useful but not easy:

  • Impcore is deterministic: the same program gives the same answer every time (not true of Java!).

  • Impcore can be evaluated using a call stack (like C and Java).

To make it possible to do something relatively interesting but also relatively easy, I’ll ask you to investigate a metatheoretic conjecture that’s not true:1

Conjecture: If global variable y is defined, and if y does not appear in expression e, and if expression e evaluates to some value, then the evaluation of e does not change the value of y.

To disprove a metatheoretic conjecture, it is sufficient to find a counterexample, but to show insight into how metatheory works, you will also explain what parts of the proof fail and how. To structure your explanations, you’ll rely on a stylized structure that applies to every metatheoretic proof: as described in the book in section 1.7.3, a proof needs a case for each rule in the semantics.

Using the detailed definitions below, complete the following four tasks, each of which relates to the conjecture.

  1. Exhibit a counterexample.

  2. Since the conjecture relates to a change in the value of a variable, the cases for assignment (set) are likely to be relevant. Both those cases actually work; show that they go through.

    Note that the two cases are actually different; both need to be handled in full.

    (To prove these cases, which are inductive, you will be proving an implication. Please identify the induction hypothesis, which is the left-hand side of that implication. You may wish to review the handout “Overview of Induction” by Chris Phifer.)

  3. Find one case of the metatheoretic proof that fails, and explain why the proof doesn’t go through for that case.

  4. Briefly explain, in language a COMP 11 student would understand, why the conjecture fails. (A sentence or two is plenty.)

To reduce bureaucracy, you will show that the conjecture fails in Simplified Impcore. Simplified Impcore is a restricted subset of Impcore in which:

  • There are no while or begin expressions.
  • Every function application has exactly two arguments.
  • The only primitive function is +.

Using Simplified Impcore reduces the number of cases to something manageable.

Here’s the conjecture in detail: whenever e, ξ, ϕ, ρ⟩ ⇓ ⟨v, ξ′, ϕ, ρ′⟩ and y is a variable that is in domξ but does not appear (“free”) in e, then ξ′(y) = ξ(y). We will not formalize “appearing free” just yet, but the appearance of a free variable is defined inductively over the abstract syntax. Here is the definition for Impcore:

  • x appears free in VAR(x).
  • x appears free in SET(x, e).2
  • If x appears free in any of e’s subexpressions, then it also appears free in e. For example, if x appears free in e1, then it also appears free in IF(e1, e2, e3).

This concept is so central to the proof that it benefits from notation: for the set of variables that appear free in e, we write fv(e). We can then make things more formal:

  • fv(VAR(x)) = {x}.
  • fv(SET(x, e)) = {x} ∪ fv(e).
  • If e has subexpressions, its set of free variables is the union of the free variables of the subexpressions. For example, fv(IF(e1, e2, e3)) = fv(e1) ∪ fv(e2) ∪ fv(e3). As another example, fv(APPLY(f, e1, …, en)) = fv(e1) ∪ ⋯fv(en).

And we can also formalize the conjecture:

  • Whenever e, ξ, ϕ, ρ⟩ ⇓ ⟨v, ξ′, ϕ, ρ′⟩ and y ∈ domξ and y ∉ fv(e), then ξ′(y) = ξ(y).

This is the form of the conjecture that will be most useful for your solutions.

Rewrite instructions for ϕ as follows:

  • Environments ξ and ρ must be written out in full. But writing out ϕ in full would be tiresome, so instead, you may say “ϕ contains all the functions in the initial basis, plus the following additional function definitions,” and then show the code for the definitions. (If you solve the problem without any additional definitions, that is OK.)

Notes:

  • To “find a counterexample” means to exhibit a choice of e, v, and five environments for which the evaluation judgment holds—but in which evaluating e changes the value of a defined global variable, even though the variable does not appear free in e. Environments ξ and ρ must be written out. For your environment ϕ, you may state simply “the ϕ from the initial basis,” or if you wish, you may say “ϕ includes the initial basis plus the user-defined functions added by the following definitions,” and give the definitions.

  • If e calls a function f, then f’s variables (its formal parameters and any global variables it mentions), are not considered free in e. Only variables mentioned in the actual parameters appear free in e.

Extra credit: Eliminating begin

Simplified Impcore has neither while nor begin. You already have an idea that you can often replace while with recursion. For extra credit, show that you can replace begin with function calls: complete exercise 14 on page 81 of Build, Prove, and Compare.

How to organize and submit your work

Reading comprehension is separate from other problems.

  • To complete part A, download the questions, then edit the answers into the file cqs.opsem.txt. If your editor is not good with Greek letters, you can spell out their names: ξ is “xi,” ϕ is “phi,” and ρ is “rho.”

    You won’t submit part A until you also have the files for the other parts.

  • To complete parts B, C, and D, create files awk-icon.imp and theory.pdf. Please leave your name out of your PDF—that will enable your work to be graded anonymously.

    For these exercises you will turn in your code for exercise 16 D in file awk-icon.imp; everything else goes into theory.pdf. For the theory.pdf, you could consider using LaTeX, but unless you already have experience using LaTeX to typeset mathematics, it’s a bad idea. We recommend that you write your theory homework by hand, then scan or photograph it.

    If you do already know LaTeX and you wish to use it, you may benefit by emulating our Latex source code for a simple proof system or Sam Guyer’s LaTex source code for typesetting operational semantics. You might also like Matthew Ahrens’s video tutorial on typesetting proof trees.

    Please also create a file called README, in which you tell us anything else you think is useful for us to know. We provide a template for your README at http://www.cs.tufts.edu/comp/105/homework/opsem-README-template.

    As soon as you have the files for all parts, cd into the appropriate directory and run submit105-opsem to submit a preliminary version of your work. You’ll need files README, cqs.opsem.txt, awk-icon.imp, and theory.pdf, but preliminary versions are good enough. Keep submitting and resubmitting until your work is complete; we grade only the last submission.

When you submit theory.pdf, the provide program should email you a copy of the PDF. Check the email and be sure that the PDF opens and displays what you expect. If there is a problem with the PDF, resubmit the file or ask for help on Piazza.

To help us read your work, we need for you to organize your answers carefully:

  • The answer to each question must start on a new page.

  • The theory answers must appear in this order: exercises 10, 11, R, 16 (parts A, B, and C), 12, 13 (part A), and finally F. If you choose to complete exercise 14 (see below), put it last, after F.

How your work will be evaluated

Operational semantics

Below is an extensive list of criteria for judging semantics, rules, derivations, and metatheoretic proofs. As always, you are aiming for the left-hand column, you might be willing to settle for the middle column, and you want to avoid the right-hand column.

Talking operational semantics (part B)

Exemplary Satisfactory Must improve
Talking

• Explanations are correct and use only words a COMP 11 student would understand.

• Formalism is completely correct; all primes, subscripts, and quantifiers are used correctly.

• Claimed differences with Impcore are supported by example code (which may be just a fragment; complete, running code is not necessary).

• When a rule is claimed to be effectively the same as the original, the solution explains why the notational differences are irrelevant.

• Explanations are correct, but they use words that a COMP 11 student might not understand, like “environment.”

• Explanations are almost correct, and course staff can see what went wrong.

• Formalism would be correct except for problems with quantifiers. (Quantifiers may be missing or may be in the wrong places, or may be confused with .)

• Formalism would be correct except for issues with subscripts or primes.

• Claimed differences with Impcore are not supported by any code.

• A rule is correct claimed to be effectively the same as the original, but the solution does not explain why the notational differences are irrelevant.

• There are explanations that the course staff can’t easily relate to the formalism.

• There are explanations that the course staff can’t understand.

• Formalism wouldn’t be correct even if quantifiers, subscripts, and primes were corrected.

• Differences with Impcore are not identified correctly.

Changed rules of Impcore (part C, exercise 16, parts (a) and (b))

Exemplary Satisfactory Must improve
Rules

• Every inference rule has a single conclusion which is a judgment form of the operational semantics.

• In every inference rule, every premise is either a judgment form of the operational semantics or a simple mathematical predicate such as equality or set membership.

• In every inference rule, if two states, two environments, or two of any other thing must be the same, then they are notated using a single metavariable that appears in multiple places. (Example: ρ or σ)

• In every inference rule, if two states, two environments, or two of any other thing may be different, then they are notated using different metavariables. (Example: ρ and ρ)

• New language designs use or change just enough rules to do the job.

• Inference rules use one judgment form per syntactic category.

• In every inference rule, two states, two environments, or two of any other thing must be the same, yet they are notated using different metavariables. However, the inference rule includes a premise that these metavariables are equal. (Example: ρ1 = ρ2)

• A new language design has a few too many new or changes a few too many existing rules.

• Or, a new language design is missing a few rules that are needed, or it doesn’t change a few existing rules that need to be changed.

• Notation that is presented as an inference rule has more than one judgment form or other predicate below the line.

• Inference rules contain notation above the line that does not resemble a judgment form and is not a simple mathematical predicate.

• Inference rules contain notation, either above or below the line, that resembles a judgment form but is not actually a judgment form.

• In every inference rule, two states, two environments, or two of any other thing must be the same, yet they are notated using different metavariables, and nothing in the rule forces these metavariables to be equal. (Example: ρ and ρ are both used, yet they must be identical.)

• In some inference rule, two states, two environments, or two other things may be different, but they are notated using a single metavariable. (Example: using ρ everywhere, but in some places, ρ is needed.)

• In a new language design, the number of new or changed rules is a lot different from what is needed.

• Inference rules contain a mix of judgment forms even when describing the semantics of a single syntactic category.

Program to probe Impcore/Awk/Icon semantics (part C, exercise 16, part (d))

Exemplary Satisfactory Must improve
Semantics

• The program which is supposed to behave differently in Awk, Icon, and Impcore semantics behaves exactly as specified with each semantics.

• The program which is supposed to behave differently in Awk, Icon, and Impcore semantics behaves almost exactly as specified with each semantics.

• The program which is supposed to behave differently in Awk, Icon, and Impcore semantics gets one or more semantics wrong.

• The program which is supposed to behave differently in Awk, Icon, and Impcore semantics looks like it is probably correct, but it does not meet the specification: either running the code does not print, or the last thing printed is not the 0 or 1 called for in the problem.

Derivations (part D, exercises 12 and 13)

Exemplary Satisfactory Must improve
Derivations

• In every derivation, every utterance is either a judgment form of the operational semantics or a simple mathematical predicate such as equality or set membership.

• In every derivation, every judgement follows from instantiating a rule from the operational semantics. (Instantiating means substituting for meta variables.) The judgement appears below a horizontal line, and above that line is one derivation of each premise.

• In every derivation, equal environments are notated equally. If both ρ and ρ appear, they must not be known to be equal.

• Every derivation takes the form of a tree. The root of the tree, which is written at the bottom, is the judgment that is derived (proved).

• In every derivation, new bindings are added to an environment exactly as and when required by the semantics.

• In one or more derivations, there are a few horizontal lines that appear to be instances of inference rules, but the instantiations are not valid. (Example: rule requires two environments to be the same, but in the derivation they are different.)

• In a derivation, the semantics requires new bindings to be added to some environments, and the derivation contains environments extended with the right new bindings, but not in exactly the right places.

• In one or more derivations, there are horizontal lines that the course staff is unable to relate to any inference rule.

• In one or more derivations, there are many horizontal lines that appear to be instances of inference rules, but the instantiations are not valid.

• Environments in intermediate or final states have primes or subscripts not found in the initial environment, and there is no unknown derivation (or unknown subexpression) whose result could account for a prime or a subscript.

• A derivation is called for, but course staff cannot identify the tree structure of the judgments forming the derivation.

• In a derivation, the semantics requires new bindings to be added to some environments, and the derivation contains environments extended with new bindings, but the new bindings in the derivation are not the bindings required by the semantics. (Example: the semantics calls for a binding of answer to 42, but instead answer is bound to 0.)

• In a derivation, the semantics requires new bindings to be added to some environments, but the derivation does not contain any environments extended with new bindings.

Metatheory (part D, exercise F)

Exemplary Satisfactory Must improve
Metatheory

• The counterexample includes everything mentioned in the evaluation judgment: expression, result, and all environments.

• Proofs by induction explicitly identify the induction hypothesis.

• Metatheoretic proofs operate by structural induction on derivations, and derivations are named.

• Metatheoretic proofs classify derivations by case analysis over the final rule in each derivation. Cases with similar proofs are grouped together.

• When a problem calls for a complete metatheoretic proof, the case analysis covers every possible derivation.

• Failed cases for metatheoretic proof explain the failure even in the presence of a good induction hypothesis.

• The counterexample includes an expression, but it omits the result or one or more value environments.

• A proof by induction is not explicit about what the induction hypothesis is, but course staff can figure it out.

• Metatheoretic proofs operate by structural induction on derivations, but derivations and subderivations are not named, so course staff may not be certain of what’s being claimed.

• Metatheoretic proofs classify derivations by case analysis over the final rule in each derivation, but the grouping of the cases does not bring together cases with similar proofs.

• Failed cases for metatheoretic proof are correctly identified, but they blame the induction hypothesis for the failure.

• The counterexample doesn’t work.

• In a proof by induction, course staff cannot figure out what the induction hypothesis is.

• Metatheoretic proofs don’t use structural induction on derivations (serious fault).

• Metatheoretic proofs are missing many cases (serious fault).

• Course staff cannot figure out how metatheoretic proof is broken down by cases (serious fault).

• A problem calls for a complete metatheoretic proof, but the cases that are presented don’t cover every possible derivation.

• Failed cases for metatheoretic proof are not correctly identified.


  1. A famous scientist once said that proofs in programming languages are interesting only when they are wrong.

  2. And in addition, any variable that appears free in e also appears free in SET(x, e).