Overview

In this programming assignment, you will implement the syntax and semantic analysis phases for IC. These phases require you to write the parser, the AST and symbol table packages, and the type checker.

Details

You are required to implement the following:

• The Parser. You will write the grammar as a scup spec file: ic.scup, at the root of your repository, exactly the format you used in HW 3 and HW 4. The starter includes the library source, a stub ic.scup, and the small provided host (ic.parser.ICParser) that loads it.

  You will use the grammar from the IC Language Specification as a starting point. You must refactor that grammar to be LL(1): eliminate all left recursion (use a %left block with a %binary rule for the binary operators — one declaration for the whole precedence ladder — and %chain postfix loops for forms like e.f, e[i], e.m(...), and e.length), and left-factor alternatives that share a prefix until one token of lookahead decides. The specification’s LL(1) appendix works several of these factorings for you and deliberately leaves two — the statement rule and primary, both of which hinge on what an ID can begin — for you to work out; the conflict errors will referee your attempts (and sbt "runMain scup.check ic.scup" checks the grammar without running anything else). Expect the factored statement rule to push two requirements into your action blocks: the target of = must be a location, and an expression statement must be a call. The dangling else is the one place %greedy is warranted.

  You should use the parser to only build the AST. Separate passes over the AST will build the symbol tables and perform the semantic checks, after the program has been parsed and the AST has been constructed. Tokens carry their positions — record the source line in each AST node as you build it; your error messages in this and later assignments depend on it.

• AST Construction. Design a class hierarchy for the abstract syntax tree (AST) nodes for the IC language. When the input program is syntactically correct, your checker will produce a corresponding AST for the program. The abstract syntax tree is the interface between the syntax and semantic analysis, so designing it carefully is important for the subsequent stages in the compiler. Note that your AST classes do not necessarily correspond to the non-terminals of the IC grammar. Use the grammar from the language specification only as a guideline for designing the AST. Once you have designed the AST class hierarchy, extend your parser to construct ASTs.

  Your AST nodes should be implemented as case classes.

• Symbol Tables and Types. Then design the symbol table structures and any additional structures you wish to use for representing program types. Your design should allow each AST node to access the symbol table corresponding to its current scope (e.g. class, method, or block scope).

  Your constructed symbol tables should be available to all remaining compilation phases, and I recommend that all subsequent phases refer to program symbols (e.g., variables, methods, class names, etc.) using references to their symbol table entries, and not using their string names. In other words, each AST node containing a name should be given a second field that you fill in when you resolve that name. This second field should contain enough information to uniquely identify the declaration, type, etc. of the symbol being referenced.

  There are many ways to accomplish this. Perhaps the most straightforward is to have the symbol table structure map each string name in scope to the AST node corresponding to the declaration of that name. Given that structure, you can simply augment each AST node containing a name with a pointer to the AST node for that name’s declaration, and fill in this pointer later on. For example, if you have a FieldDecl class to represent field declarations and a FieldAccess class to represent a field access, then FieldAccess should have an instance variable decl of type FieldDecl that will will be linked to the appropriate field declaration during type checking. This is briefly described, along with alternatives, in Cooper and Torczon, Chapter 5.7.

• Semantic Checks. After you have constructed the AST and the symbol tables, your compiler will analyze the program and perform semantic checks. These semantic checks include type-checking, scoping rules not enforced while building the symbol tables, and all of the other requirements described in the language specification.

  Required semantic checks. The full list is defined by the language specification, but as a checklist your compiler must perform at least the following:

  • name resolution and scoping (every use resolves to a declaration visible in its scope);

  • the type rules, as given in the specification;

  • break and continue appear only inside loops;

  • every non-void method returns a value on all paths;

  • definite assignment: every local variable is assigned before it is used, checked with a conservative, Java-style per-path analysis;

  • exactly one main method with the required signature. This is a whole-program check: perform it when checking a complete program.

• Error Handling. You must extend your error package with SyntaxError and SemanticError exceptions, and have your compiler throw such exceptions whenever it encounters errors. These exceptions must carry information about the error, such as the line number and a message describing the violation. You are not required to report more than one error; the execution may terminate after the first lexical, syntactic, or semantic error. One should be able to fix the problem immediately after reading the error message. (Real compilers differ here: production compilers recover from errors and report as many as they can — essential for IDEs and language servers — but we stop at the first error for grading simplicity.)

Command Line Invocation. As in PA 1, your compiler will be invoked with the program file name as an argument, with an optional “-d” flag:

    sbt "run <file.ic>"

Equivalently, ./icc.sh <file.ic> (included in the starter, as in PA 1) runs the same command with sbt’s log messages hidden; it takes exactly the same arguments as the compiler.

For scripted testing, the starter includes the same specTests runner as PA 1 (src/test/scala/tests/SpecTests.scala): sbt "specTests <dir>" runs your compiler on every .ic file under <dir> and compares the last line of its output to the matching .ic.expected file. Each test runs in a fresh JVM, so global state or a stray System.exit in your compiler cannot leak between tests. The starter’s tests/ directory holds one test suite per milestone:

• tests/pa2-cp1 — what must work at the first checkpoint (parsing: lexical and syntax errors are caught; semantic errors still compile “successfully”);
• tests/pa2-cp2 — the second checkpoint (symbol tables: scope and class-resolution errors are now caught too);
• tests/pa2 — the final submission (all semantic checks).

The three suites contain the same programs; only the expected verdicts differ. Test against the suite for the milestone you are working toward, e.g. sbt "specTests tests/pa2-cp1". Later assignments add new suites to tests/ — and you should add your own tests there as well.

If you prefer working with a standalone jar (for manual runs or your own scripts), build one with sbt assembly and run it directly — this avoids sbt’s startup time on every file:

    java -jar target/icc.jar <file.ic>

The compiler will parse the input file, construct the AST and symbol tables, perform the semantic checks, and report any error it encounters. In addition, your compiler must support two command-line options to print internal information about the AST and the symbol tables:

1. The “-printAST" option: prints a textual description of the constructed AST to System.out.

2. The “-printSymTab" option: prints a textual description of the symbol tables to System.out.

These options should appear after the filename, as in:

    sbt "run <file.ic> -printAST"

or

    java -jar target/icc.jar <file.ic> -printAST

You can design your own textual description of the AST and symbol table structures, but make sure your output provides all important information and is easy to read.

Output Format.

As in the previous assignment, the last line of the output must be either

Success.

or

Failed.

depending on whether any errors were found. Some of the intermediate checkpoints will require other data to be printed, but the final submission should print no information other than error messages and that one last line. The checkpoints and the final submission are graded on Gradescope against staged versions of the course test suite, so complying with the Success./Failed. protocol matters.

Package Structure.

You should implement the new components of the compiler as the following sub-packages of the ic package:

• the error module for error exceptions;

• the lex module for the lexer and token classes;

• the parser module for ICParser and associated classes;

• the ast module for the AST class hierarchy;

• the symtab module for symbol tables; and

• the tc module for the type checker.

The dot Utility.

You may find it helpful to use the graph visualization tools in the graphviz suite of tools for printing out information about the AST and the hierarchy of symbol tables. You can find information about this tool on the course web site. The most useful tool would be the dot program, which reads a textual specification for a graph and outputs a graphical image (in PDF format, jpg, or other image formats). For instance, the dot specification for the AST of the statement x = y + 1 is:

        digraph G {
          expr [label="="];
          lhs [label="x"];
          rhs [label="+"];
          leftop [label="y"];
          rightop [label="1"];
          expr -> lhs;
          expr -> rhs;
          rhs -> leftop;
          rhs -> rightop;
        }       

Running the dot tool on a file containing this description, as described in HW 4, will produce a graphical tree. Using dot is encouraged but not required.

Getting Started

I have provided a PA2 Starter project on GitHub. It contains a complete lexer for IC (an ic.slex equivalent to the PA 1 solution), the scup library source, and a stub ic.scup for you to grow into the full grammar. I suggest starting with this project even if you’d like to use your own scanner code.

Accept the GitHub invitation email to your team’s repository, which was created from this starter. Then do one piece of one-time setup in your clone: add the course starter repository as a second remote, so you can pull updates I release during the project — bug fixes now, and the PA 3 / PA 4 test suites and starter files later:

git remote add upstream https://github.com/williams-cs/cs434-f26-compiler-starter
git pull --no-rebase --allow-unrelated-histories upstream main

(--allow-unrelated-histories is needed only on this very first pull, because your repository was created from the starter template without its git history.) From then on, whenever a handout or announcement says new files are available, each team member just runs:

git pull upstream main

Releases only ever add files (tests/, handout/, lib/), so the pull will not conflict with your own work.

If you choose to use your own scanner code, you will want to change how it matches integer literals so your parser can properly handle “-”. More specifically, “-” can be interpreted in two ways in IC programs, as the binary minus operator or the unary minus operator. It will be the parser’s job to determine which is the appropriate interpretation. Thus “-10” should be scanned as two tokens, “-” and “10”, and the parser will represent that as the unary minus operator applied to 10 in the program’s parse tree.

One consequence of scanning “-” as a separate token: the most negative int is -2147483648, but the literal 2147483648 on its own is out of range. As the specification now states, the literal 2147483648 is legal only when it appears immediately under a unary minus; anywhere else it is an error.

Working with scup.

Your grammar goes in ic.scup, one rule per nonterminal, each alternative’s block building your AST nodes. Some advice from experience:

• Grow the grammar incrementally — classes, then fields and methods, then statements, then expressions — and keep a small .ic test alongside every rule you add. A rule can start out action-free (it parses to a generic labeled tree) and gain its %type and blocks once its shape settles.

• Declare a %type for every rule as you design your AST: the blocks are then statically checked against the grammar, and a wrong $i or a wrong node type is a compile error at the spec file’s own line.

• Debugging a predictive parser is ordinary debugging: you can put a Util.debug call inside any block, and you can unit test any rule in isolation — every rule is a start symbol, so grammar.parse(grammar.ruleFor("stmt"), tokens) works on a small token sequence.

• Most grammar mistakes never reach a test: conflicts are rejected when the grammar loads, naming the rule and the productions involved. When a conflict report surprises you, run sbt "runMain scup.check -dump ic.scup" — the FOLLOW sets in the dump usually explain it — and left-factor rather than reordering alternatives (there is no order).

Documentation and Testing

Testing and Documentation are crucial moving forward. A few basic thoughts, now that PA 1 is behind us:

• Comments. Follow the basic 136 rules about documenting the role of each class, what each non-trivial method does, and how each important block of code works. You need not document every line or class. For example, you will write perhaps 20–30 AST node classes. They all are roughly the same. So, write a top-level architectural comment in the root class and note conventions followed in the rest of the classes. Then comment only the most salient details or tricky parts in the rest of the files (eg, the fields linking identifiers to declarations).

• Testing. You will not survive this project if you only write one or two test files. The best way to test is to write many (eg, dozens) of small cases to cover as many cases of the IC specification as possible and test as you go. To this end, plan on writing 20–30 specification tests as you begin each phase of the project, rather than waiting until the night before it is due. And run all of your tests often to avoid unwittingly breaking correct behavior. The script from PA 1 should serve as a good starting point for this, and I can assist in setting up additional scripts if you like.

  You will also want to write unit tests for specific components, such as your parser, symbol table, etc. You can place unit tests in the src/test/scala directory, using the same MUnit style we used for unit tests last week, and run them with sbt test.

  You should also document how you test your compiler so that I can reproduce your results.

Submission

This is a substantial programming project. You have roughly 4 weeks to complete it — use your time wisely. There will be intermediate checkpoints along the way. Develop in your GitHub repository as usual, and submit your sbt project to Gradescope for each checkpoint and for the final deadline. Please be sure that your submission (and your GitHub repository) contains up-to-date versions of the following by the submission deadlines listed below:

• All of your source code and test cases (in directories /src and /test). As in the previous assignment, make sure your code is well-documented. I will likely run your compilers and browse through your code at each checkpoint.

• A brief, clear, and concise summary of where you are at each checkpoint. Place this in README.md. This document should describe where you are, what bugs or open issues remain, and how you tested this part. The writeup need not be long. A few short sentences or a bullet list should be sufficient. (Think of this is what you’d say if your boss asked you for a 30 second update on your project.) Don’t delete older prose on each submission – just add a new section to it. Start with what you wrote for PA 1.

As in the first assignment, double check your project when you submit each checkpoint by cloning out a fresh copy and verifying that it works.

Schedule

These milestones are the minimal requirements for the checkpoints. You are of course welcome (if not strongly encouraged) to do more by each deadline.

Tutorial meetings, the week of the first checkpoint:

Your parser must successfully parse valid IC programs and report syntax errors in bad ones. Come to the meetings with an initial design of your AST package, including: 1) The list of operations present in your root node class, and 2) a brief overview of the class hierarchy of node types. This need not be very detailed, but it should at least demonstrate that you have begun to think about how to lay out your ASTs.

Your project should include at least 20–30 test cases, and ideally more.

10pm, Wednesday, Oct. 21:

Your parser must generate ASTs for programs, and you must support the -printAST command-line option. The tests/pa2-cp1 suite encodes exactly what this checkpoint’s Gradescope run expects: sbt "specTests tests/pa2-cp1".

10pm, Wednesday, Oct. 28:

All of the above, plus your compiler must generate the symbol tables and perform name resolution for variable uses. That is, each variable access should be resolved to either a local variable access or a field access. Variable name resolution should be done as a separate pass over the AST after all of the symbol table information has been constructed. You may wish to resolve class names to their declarations at the same time as variable resolution. Otherwise, you’ll likely need a separate pass to do that before type checking. (Field and method name resolution will take place during the subsequent type checking phase.)

At this point your compiler should implement the -printSymTab option. Additionally, you may wish to extend your AST printer to print out some indication of how each variable access in the program has been resolved.

This checkpoint’s suite is tests/pa2-cp2.

10pm, Wednesday, Nov. 4:

PA 2 due: sbt "specTests tests/pa2" should report no failures.