Exercise. (a tokenizer for a small imperative language) a program that accepts a sequence of characters and returns a sequence of tokens is called a tokenizer or lexical analyzer. a compiler is a program that translates a sequence of characters, which represents a program, into a sequence of lowlevel instructions that can be executed on a machine. in its most basic form, a compiler consists of three parts: 1. tokenizer: reads a sequence of characters and outputs a sequence of tokens.2. parser: reads a sequence of tokens and outputs an abstract syntax tree (parse tree)3. code generator: traverses the syntax tree and generates low-level instructions for a real machine or an abstract machine. the proposed imperative language which should be tokenized has the following list of tokens: keywords, operators, atoms, integers, floats, and separators. the keywords are: program, void, bool, int, float, true, false, if, then, else, local, in, end, assign, call. the operators are: ‘=’, ‘+’, ‘-‘, ‘*’, ‘/’, ‘==’, ‘! =’, ‘> ’, ‘=’. an atom starts with a lowercase character, followed by any number of alphanumeric characters, but it cannot be a keyword