A3: Lexing and Parsing

Lexing and Parsing with ocamllex/Menhir

  def fact(x : int) : int {
      if x == 0 then 1
      else if x == 1 then 1 
           else x * fact(x - 1)
  }
  fact(4)

  def fact(x : int) : int { ... }

In general, every Grumpy program consists of a number of function definitions followed by an expression (which may call one or more of the defined functions).

  def f(...) : ... { ... }
  def g(...) : ... { ... }
  ...
  def z(...) : ... { ... }
  ... some result expression here ...

Here's a second example that illustrates a couple additional features of Grumpy: mutable references and variable scope:

  def f(x:int, y:bool) : int {   // -+ x and y go into scope
    let z = ref x in             // -+ z goes into scope
    {                            //  |
      let w = !z in              // ---+ w goes into scope
      z := w + 1                 //  | |
    };                           // ---+ w goes out of scope
    !z + 1                       //  |
  }                              // -+ x,y, and z go out of scope
  f(3, false)

The function's second line introduces a new block scope with brackets { ... }. What's the effect of this scope? Any let-bound variables we declare inside it won't be accessible outside the { ... } (for example, it would be illegal to refer to w in the expression !z + 1, by rewriting it to something like !z + w).

The overall result of the program is 5: Reference z is initialized to 3. Variable w equals 3 in the update to z := w + 1 (= 3 + 1 = 4). Finally, the result of the function is the last sequenced expression in its body, !z + 1 = 4 + 1 = 5.

Your job

• To implement the lexer, you'll be using ocamllex, a lexer generator.
• To implement the parser, you'll be using Menhir, a parser generator for LR(1) grammars.

1. Download the assignment files

  $ tar xzvf a3.tgz

  src/               -- compiler source files
    Makefile         -- the project Makefile
    _tags            -- the tags file for ocamlbuild
    AST.mli          -- language-independent abstract syntax stuff
    AST.ml           -- associated helper functions
    exp.mli          -- the definition of Grumpy's abstract syntax
    exp.ml           -- associated functions
    lexer.mll        -- ocamllex source file (Part 2)
    parser.mly       -- Menhir source file (Part 3)
    grumpy.ml        -- the toplevel compiler program    
    tests/           -- test cases

$ make

  ...
  File "parser.mly", line 13, characters 15-20:
  Warning: the token WHILE is unused.
  Finished, 22 targets (0 cached) in 00:00:00.

Now try running

$ make test

  $ make test
  ocamlbuild -use-menhir -use-ocamlfind grumpy.native
  Finished, 22 targets (22 cached) in 00:00:00.
  cd tests && ./run.sh
  test01-unary-negation.gpy:1:2: Unexpected char: -
  *** test01-unary-negation.gpy FAILED ***
  test02-boolean-negation.gpy:1:2: Unexpected char: n
  *** test02-boolean-negation.gpy FAILED ***
  ... followed by many more ...

  ...
  test50-fractal.gpy
  test50-fractal.gpy.expected
  test51-loopref.gpy
  test51-loopref.gpy.expected

2. lexer.mll

Start by opening lexer.mll; now navigate to the (mostly empty) definition of rule token. You'll see that, initially, it contains only a few rules:

  rule token = parse
    "//"                        { comment lexbuf }
  | newline                     { next_line lexbuf; token lexbuf }
  | ['0'-'9']+ as lxm           { INTCONST(Int32.of_string lxm) }
  (* FILL IN A BUNCH MORE RULES HERE *)             
  | _          { raise (Syntax_err ("Unexpected char: " ^ Lexing.lexeme lexbuf)) }

In the regexp-style pattern

  ['0'-'9']+ as lxm

The definition of the tokens themselves is given in parser.mly. Here are the first few:

  %token <int32> INTCONST
  %token <float> FLOATCONST
  %token <bool> BOOLCONST
  %token <string> ID

  %token DEF LET WHILE IF THEN ELSE REF INT FLOAT BOOL UNIT TT IN

3. parser.mly

  (** Is type [t] an arithmetic type? *)
  val is_arith_ty : ty -> bool

In the assignment code, interface files AST.mli and exp.mli define the abstract syntax of the Grumpy source language. The Grumpy lexer and parser convert concrete Grumpy programs into values of type (AST.ty, unit Exp.exp) AST.prog.

Now

In general, your job in this file is to add a number of new nonterminal rules to the Menhir grammar, corresponding to the Grumpy syntax given in the Grumpy spec.

Each such rule will look something like the following:

  unop:
  | MINUS
    { UMinus }
  | NOT
    { UNot }
  | DEREF
    { UDeref }

  | MINUS
    { UMinus }

Menhir rules can be more complicated. Here's a second example:

  exp_list:
  | l = separated_list(COMMA, exp)
    { l }

In the rule, the result of parsing this list (a list of abstract syntax expressions) is bound to variable l, which may then appear within the braces { ... }. In this case, our exp_list rule just returns l. However in general, a rule might do more interesting things with intermediate expressions. For more information on Menhir's special separated_list function (and other useful functions), see Section 5.4 of the Menhir manual, which describes Menhir's "Standard Library" (a collection of useful, pre-defined parsing functions).

Here's one more:

  arg:
  | arg_id = id COLON arg_ty = mytype
    { mk_tid arg_id arg_ty }

Precedences and Associativity

To encode precedence and associativity in Menhir, add precedence and associativity directives to the top of your parser.mly file (before the %%) as documented in Section 4 of the Menhir manual.

For example, the following pair of directives encode that TIMES and DIV bind tighter than PLUS and MINUS:

  %left PLUS MINUS
  %left TIMES DIV

4. Test

  $ make test

5. Submit

6. Piazza