In this chapter, we shall discuss in detail the design of the build system. For information of how to use the build system, please see: FCM System User Guide > The Build System.
The build system analyses the directory tree containing a set of source code, processes the configuration, and invokes make to compile/build the source code into the project executables. The system is written in a set of Perl modules. It is designed to work with GNU make. It creates the Makefile and many other dependent files automcatically. The build system uses a similar interface to the extract system. Its configuration file can be produced through the extract system. It also shares the same command line interface and many other utilities with the code management system and the extract system.
The build system has the following input:
Output from the build system includes:
The build system uses the following commands, modules and tools:
| Name | Category | Description |
|---|---|---|
| fcm | Perl executable | Top level command line interface of the FCM system. |
| fcm_internal | Perl executable | Command wrapper for the compiler and linker. |
| Fcm::Build | Perl module | Main class that controls the running of the build system. |
| Fcm::BuildTask | Perl module | A class that performs various "tasks" (such as pre-process and generate interface) for the build system. |
| Fcm::CfgFile | Perl module | A class for reading from and writing to configuration files. |
| Fcm::Compiler | Perl module | A class for wrapping the compiler and linker commands. |
| Fcm::Config | Perl module | A class that contains the configuration settings shared by all FCM components. |
| Fcm::SrcFile | Perl module | A class that controls the actions on a source file. |
| Fcm::SrcPackage | Perl module | A class that deals with the actions on a source directory sub-package. |
| Fcm::Util | Perl module | A collection of utilities shared by all FCM components. |
| Ecmwf::Fortran90_stuff | Perl module | A utility originally developed by the ECMWF for generating interface blocks for Fortran 9X source files. Modified for adoptation by the FCM system. |
| make | Unix utility | The make build utility. FCM is designed to work with the GNU version of make. |
| ksh | Unix shell | The following shell commands are used: "cp", "rm", "mv", "cd" and "touch". |
| f90aib | Fortran utility | Formerly used by the GEN system as the generator for Fortran 9X interface blocks. It is a freeware developed by Michel Olagnon at the French Research Institute for Exploitation of the Sea. Its use is still supported by FCM, but the ECMWF interface generator is now preferred. |
There are several options that can be supplied to the build command. These options are implemented as follows:
When we invoke the FCM command, it creates a new instance of Fcm::Config, which reads, processes and stores information from the central and user configuration file. The default settings in Fcm::Config is overwritten by the information provided by the central configuration file. If a user configuration file is found, its settings will take overall precedence. These settings are stored in the Fcm::Config instance, which are parsed to all other modules used by the build system. By convention, the reference to the Fcm::Config instance can normally be fetched by the "config" method for all OO "Fcm::" modules.
When we invoke the build command, it creates a new instance of Fcm::Build, which automatically creates a new instance of Fcm::CfgFile. If an argument is specified in the build command, it is used as the build configuration file if it is a regular file. Otherwise, it is used to search for the build configuration file. If no argument is specified, the current working directory is searched. Fcm::CfgFile will attempt to locate a file called "bld.cfg" under this directory. If such a file is not found, it will attempt to locate it under "cfg/bld.cfg".
Once a file is located, Fcm::CfgFile will attempt to parse it. This is done by reading and processing each line of the configuration file into separate label, value and comment fields. Each line is then pushed into an array that can be fetched using the "lines" method of the Fcm::CfgFile instance. Internally, each line is recorded as a reference to a hash table with the following keys:
The information given by each line is "deciphered" by Fcm::Build. The information is processed in the following ways:
Unless the search source flag "SEARCH_SRC" is switched off (0) in the build configuration, Fcm::Build will attempt to search the source sub-directory "src/" of the build root recursively for source directory sub-packages. The source directories obtained in the search are treated as if they are declared using "SRC::<pcks>" in the build configuration file.
As discussed in the user guide, if you declare the Fortran compiler flags without specifying a sub-package, the declaration applies globally. Otherwise, it only applies to the Fortran source files within the sub-package. This is implemented via a simple "tool selection" mechanism. You may have noticed that all TOOL declarations (and TOOL settings in Fcm::Config) are turned into an environemnt variable declaration in the generated Makefile. For example, if we have a "FFLAGS__bar__egg__ham__foo" declaration, it will be declared as an environment variable in the generated Makefile. Suppose we have a source file "foo.f90" under the sub-package "bar::egg::ham". When we invoke the compiler wrapper (i.e. "fcm_internal" and "Fcm::Compile") to compile the source file, the system will first attempt to select from the FFLAGS environment variable that matches the sub-package of the source file, which is "FFLAGS__bar__egg__ham__foo" in this case. If the environment variable does not exist, it will attempt to go one sub-package up, i.e. "FFLAGS__bar__egg__ham", and so on until it reaches the global "FFLAGS" declaration, (which should always exists).
For changes in compiler flags declaration, the build system should trigger re-compilation of required targets only. This is implemented using a "flags" file system. These "flags" files are dummy files created in the "flags/" sub-directory of the build root. They are updated by the "touch" command. The following dependencies are followed:
The system records changes in declared tools using a cache file, (called ".bld_tool", located at the ".cache/" sub-directory of the built root). It is basically a list of "TOOL::" declarations for the latest build. When an incremental build is invoked, the list is compared against the current set. If there are changes (modification, addition and deletion) in any declarations, the timestamp of the corresponding "flags" files will be updated. Files depending on the updated "flags" file will then be considered out of date by make, triggering a re-build of those files.
The build system generates an interface block file for each Fortran 9X source file. If the original source file has been pre-processed, the system uses the pre-processed source file. Otherwise, the system uses the original source file. For each source file containing standalone subroutines and functions, the system will generate an interface file containing the interfaces for the subroutines and functions. The interface files for other Fortran 9X source files are empty.
Fcm::Build controls the creation of interface files by searching for a list of Fcm::SrcFile instances containing Fortran 9X source files, by calling the "is_type ('FORTRAN9X')" method of each Fcm::SrcFile instance. For each of Fortran 9X source file, a Fcm::BuildTask is created to "build" the interface file. The build task is dependent on the interface generator. The interface files will be re-generated if we change the interface generator. The generated interface is held in an array initially. If an old file exists, it is read into an array so that it can be compared with the current one. The current interface is written to the interface file if it is not the same as the old one, or if an old one does not already exist.
FCM supports the use of f90aib and the ECMWF interface generator. The latter is the default.
For each source directory sub-package, the build system scans its source files for dependency information. The dependency scanner uses a pre-defined set of patterns and rules in Fcm::Config to determine whether a line in a source file contains a dependency. Only source files of supported types are scanned. The dependency information of a sub-package is stored in the memory as well as a cache file. The latter can be re-used by subsequent incremental builds. In an incremntal build, only those source files newer than the cache file is re-scanned for dependency. The cache file is read/written using temporary instances of Fcm::CfgFile.
The control of the source file selection process is handled by the Fcm::SrcPackage instances, while the actual dependency scans are performed via the scan_dependency method of the Fcm::SrcFile instances.
A dependency has a type. For example, it can be a Fortran module or an include file. The type of a dependency determines how it will be used by the source file during the make stage, and so it affects how the make rule will be written for the source file. In memory, the dependency information is stored in a hash table, which can be retrieved as a property of the Fcm::SrcFile instance. The keys of the hash table are the dependency items, and the values are their types.
A dependency is not added to the hash table if it matches with an exclude dependency declaration for the current sub-package.
While the dependency scanner is scanning through each line of a Fortran source file, the system also attempt to determine its internal name. This is normally the name of the first compilable program unit defined in the Fortran source file. The internal name is converted into lowercase (bearing in mind that Fortran is case insensitive), and will be used to name the compiled object file of the source file.
The package configuration file is a system to bypass the automatic dependency scanner. It can also be used to add extra dependencies to a source file in the package. The configuration file is a special file in a source package. The lines in the file is read using a temporary instance of Fcm::CfgFile created by Fcm::SrcPackage. All declarations in a package configuration file apply to named source files. The declarations set the properties of the Fcm::SrcFile instance associated with the source file. It can be used to add dependencies to a source file, and to tell the system to bypass automatic dependency scanning of the source file. Other modifications such as the internal name (object file name) of a source file, or the target name of the executable can also be set using the package configuration file in the package containing the source file.
The dependency information is used to create the Makefile fragments for the source directory sub-packages. A Makefile fragment is updated if it is older than its corresponding dependency cache file.
The following is a list of file types and their make rule targets:
| File type | Targets | |
|---|---|---|
| SOURCE | all |
|
| FPP and C |
If the original source has not been pre-processed:
|
|
| PROGRAM |
|
|
| all except PROGRAM |
|
|
| all FORTRAN except PROGRAM and MODULE |
|
|
| INCLUDE |
|
|
| EXE and SCRIPT |
|
|
| LIB |
|
|
The resulting Makefile is made up of a top level Makefile and a list of include ".mk" files, (one for each sub-package). The toplevel Makefile consists of useful environment variables, including the search path of each sub-directory, the build tools, the verbose mode and the VPATH directives for different file types. It has two top level build targets, "all" and "clean". The "all" target is the default target, and the "clean" target is for removing the previous build from the current build root. It also has a list of targets for building the top level and the container sub-package "flags" files. At the end of the file is a list of "include" statements to include the ".mk" files.
A the top of each of ".mk" files are local variables for locating the source directories of the sub-package. Below that are the rules for building source files in the sub-package.
As discussed in the user guide, the PP switch can be used to switch on pre-processing. The PP switch can be specified globally or for individual sub-packages. (However, it does not go down to the level of individual source files.) The "inheritance" relationship is similar to that of the compiler flags.
Currently, only Fortran source files with uppercase file extensions and C source files are considered to be source files requiring pre-processing. If a sub-package source directory contains such files and has its PP switch set to ON, the system will attempt to pre-process these files.
The system allows header files to be located anywhere in the source tree. Therefore, a dependency scan is performed on all files requiring pre-processing as well as all header files to obtain a list of "#include" header file dependencies. For each header file or source file requiring pre-processing, a new instance of Fcm::BuildTask is created to represent a "target". Similar to the logic in make, a "target" is only up to date if all its dependencies are up to date. The Fcm::BuildTask instance uses this logic to pre-process its files. Dependent header files are updated by copying them to the "inc/" sub-directory of the build root. The "inc/" sub-directory is automatically placed in the search path of the pre-processor command, usually by the "-I" option. Pre-processing is performed by a method of the Fcm::SrcFile instance. The method builds the command by selecting the correct set of pre-processor definition macros and pre-processor flags, using an inheritance relationship similar to that used by the compiler flags. Unlike make, however, Fcm::BuildTask only updates the target if both the timestamp and the content are out of date. Therefore, if the target already exists, the pre-processing command is only invoked if the timestamp of the target is out of date. The output from the pre-processor will then be compared with the content in the target. The target is only updated if the content has changed.
Once a source file is pre-processed, subsequent build system operations such as Fortran 9X interface block generation and dependency scan (for creating the Makefile) will be based on the pre-processed source, and not the original source file. If a source file requires pre-processing and is not pre-processed at the pre-processing stage, it will be left to the compiler to perform the task.
The build system file type register is a simple interface for modifying the default settings in the Fcm::Config module. There are two registers, one for output file type and one for input file type.
The output file register is the simpler of the two. It is implemented as a hash table, with the keys being the names of the file types as known by the system internally, and the values being the suffices that will be added to those output files. Therefore, the output file register allows us to modify the suffix added to an output file of a particular type.
The input file register allows us to modify the type flags of a file named with a particular extension. The type flags are keywords used by the build system to determine what type of input files it is dealing with. It is implemented as a list of uppercase keywords delimited by a pair of colons. The order of the keywords in the string is insignificant. Internally, the build system determines the action on a file by looking at whether it belongs to a type that is tied with that particular action. For example, a file that has the keyword "SOURCE" in its type flag will be treated as a compilable source file, and so it will be written to the Makefile with a rule to compile it.
The following items are automatically written to the Makefile:
Compile and link are handled by the fcm_internal wrapper script. The wrapper script uses the environment variables exported by the Makefile to generate the correct compiler (or linker) command for the current source (or object) file. Depending on the diagnistic verbose level, it also prints out various amount of diagnostic output.
For compilation, the wrapper does the following:
For linking, the wrapper does the following:
A build can inherit configurations, source files and other items from a previous build. At the Perl source code level, this is implemented via hash tables and search paths. At the Makefile level, this is implemented using the "vpath" directives. The following is a summary: