Tokens are the basic lexical building blocks of source code: they are the “words” of the language: characters are combined into tokens according to the rules of the programming language. There are five classes of tokens:
These are words which have a fixed meaning in the language. They cannot be changed or redefined.
These are names of symbols that the programmer defines. They can be changed and re-used. They are subject to the scope rules of the language.
These are usually symbols for mathematical or other operations: +, -, * and so on.
This is usually white-space.
Numerical or character constants are used to denote actual values in the source code, such as 1 (integer constant) or 2.3 (float constant) or “String constant” (a string: a piece of text).
In this chapter we describe all the Pascal reserved words, as well as the various ways to denote strings, numbers, identifiers etc.