The Lexer
A major component of your assignment will be to write a procedure (Function) that returns a token when it is needed. Your lexer should return a record, one field for the token and another field the actual "value" of the token (lexeme), i.e. the instance of a token.
Your main program should test the lexer i.e., your program should read a file containing the source code of Rat13SU generate tokens and print out the results to a file.
Make sure that you print both, the tokens and lexemes. Basically, your main program should work as follows
while not finished (i.e. not end of the source file) do
call the lexer for a token
print the token and lexeme
endwhile
Do at least 3 test cases and make sure that you turn in proper documentation. (See the
handout).
Example
while (fahr < upper) a = 23.00;
token lexeme
keyword while
separator (
identifier fahr
operator <
identifier upper
separator )
identifier a
operator =
real 23.00
separator ;