source: LMDZ5/branches/IPSLCM6.0.11/tools/fcm/doc/design/extract.html @ 4068

Last change on this file since 4068 was 1578, checked in by jghattas, 13 years ago
  • Add fcm in LMDZ5/tools directory

It is no longer needed to have fcm in your environement PATH variable.
Now makelmdz_fcm takes by default this fcm. It is still possible to use
another fcm, using -fcm_path argument in makelmdz_fcm.
File size: 14.3 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2
3<html>
4<head>
5  <title>FCM Detailed Design: Extract System</title>
6  <meta name="author" content="FCM development team">
7  <meta name="descriptions" content="FCM Detailed Design: Extract System">
8  <meta name="keywords" content="FCM, design">
9  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10  <link rel="stylesheet" type="text/css" href="style.css">
11</head>
12
13<body>
14  <address>
15    <a href="index.html">FCM Detailed Design</a> &gt; Extract System
16  </address>
17
18  <h1>Extract System</h1>
19 
20  <p>In this chapter, we shall discuss in detail the design of the extract
21  system. For information of how to use the extract system, please see: <a
22  href="../user_guide/extract.html">FCM System User Guide &gt; The Extract
23  System</a>.</p>
24
25  <p>The extract system extracts source directories from different branches of
26  Subversion repositories, combining them with source directories from the
27  local file system to give a source directory tree suitable for feeding into
28  the build system. The system is written in a set of Perl modules. The extract
29  system uses a similar interface to the build system. It shares the same
30  command line interface and many other utilities with the code management
31  system and the build system.</p>
32
33  <h2><a name="io">Input and Output</a></h2>
34
35  <p>The extract system should provide the following outputs:</p>
36
37  <ul class="pad">
38    <li>a directory tree with the extracted source code suitable for feeding
39    into the build system.</li>
40
41    <li>a configuration file for the build system.</li>
42
43    <li>an expanded version of the current configuration file, so that the
44    current extraction can be re-used.</li>
45  </ul>
46
47  <p>The following inputs are required by the extract system:</p>
48
49  <ul class="pad">
50    <li>the location of the destination.</li>
51
52    <li>the location of the source, e.g a repository URL or a local file
53    system path.</li>
54
55    <li>the revision of the source, if applicable.</li>
56
57    <li>extra configuration settings to be exported to the build system.</li>
58
59    <li>the location of previous extractions, if the current extract is to be
60    based on them.</li>
61  </ul>
62
63  <h2><a name="component">Components</a></h2>
64
65  <p>The extract system uses the following commands, modules and tools:</p>
66
67  <table class="pad" summary="build system components" border="1">
68    <tr>
69      <th>Name</th>
70
71      <th>Category</th>
72
73      <th>Description</th>
74    </tr>
75
76    <tr>
77      <th>fcm</th>
78
79      <td>Perl executable</td>
80
81      <td>Top level command line interface of the FCM system.</td>
82    </tr>
83
84    <tr>
85      <th>Fcm::CfgFile</th>
86
87      <td>Perl module</td>
88
89      <td>A class for reading from and writing to configuration files.</td>
90    </tr>
91
92    <tr>
93      <th>Fcm::Config</th>
94
95      <td>Perl module</td>
96
97      <td>A class that contains the configuration settings shared by all
98      FCM components.</td>
99    </tr>
100
101    <tr>
102      <th>Fcm::Extract</th>
103
104      <td>Perl module</td>
105
106      <td>Main class that controls the running of the extract system.</td>
107    </tr>
108
109    <tr>
110      <th>Fcm::ReposBranch</th>
111
112      <td>Perl module</td>
113
114      <td>A class that stores and processes information of a repository
115      branch.</td>
116    </tr>
117
118    <tr>
119      <th>Fcm::SrcDirLayer</th>
120
121      <td>Perl module</td>
122
123      <td>A class that stores and processes information of a "layer" in the
124      extraction sequence of a source directory.</td>
125    </tr>
126
127    <tr>
128      <th>Fcm::Util</th>
129
130      <td>Perl module</td>
131
132      <td>A collection of utilities shared by all FCM components.</td>
133    </tr>
134
135    <tr>
136      <th>svn</th>
137
138      <td>Subversion client</td>
139
140      <td>The following sub-commands are used: "info", "list", "export" and
141      "cat".</td>
142    </tr>
143
144    <tr>
145      <th>ksh</th>
146
147      <td>Unix shell</td>
148
149      <td>The following shell commands are used: "cp", "rm" and "mkdir".</td>
150    </tr>
151
152    <tr>
153      <th>rdist</th>
154
155      <td>Unix utility</td>
156
157      <td>A remote distribution tool for mirror the extracted source
158      directory to a remote host.</td>
159    </tr>
160
161    <tr>
162      <th>rsync</th>
163
164      <td>Unix utility</td>
165
166      <td>A remote synchronisation tool for mirror the extracted source
167      directory to a remote host.</td>
168    </tr>
169
170    <tr>
171      <th>remsh</th>
172
173      <td>Unix command</td>
174
175      <td>A command to invoke a shell on a remote host.</td>
176    </tr>
177  </table>
178
179  <h2><a name="task">Task</a></h2>
180 
181  <p>To do its job, the extract system executes the following tasks in
182  order:</p>
183
184  <ul class="pad">
185    <li>parse/decipher the extract configuration file.</li>
186
187    <li>establish the extraction sequence when dealing with extraction of
188    multiple branches of the same package.</li>
189
190    <li>extract or copy individual files from the source to the
191    destination.</li>
192
193    <li>generate of an expanded extraction configuration file to allow another
194    extraction to be based on the current one.</li>
195
196    <li>generate of a build configuration file.</li>
197
198    <li>mirror extracted source code and configuration files to a remote build
199    machine.</li>
200  </ul>
201
202  <h3><a name="task_cfg">The extract configuration</a></h3>
203
204  <p>When we invoke the FCM command, it creates a new instance of Fcm::Config,
205  which reads, processes and stores information from the central and user
206  configuration file. Configuration settings in Fcm::Config are then
207  accessible by all other modules used by the extract system.</p> 
208
209  <p>When we invoke the extract command, it creates a new instance of
210  Fcm::Extract, which automatically creates a new instance of the
211  Fcm::CfgFile. If an argument is specified in the command line, the argument
212  is used as the "basis". Otherwise, the current working directory is taken as
213  the basis. If the basis is a directory, Fcm::CfgFile will attempt to locate
214  a file called "ext.cfg" under this directory. If such a file is not found,
215  it will attempt to locate it under "cfg/ext.cfg". If the basis is a regular
216  file, the file itself is used.</p>
217 
218  <p>Once a file is located, Fcm::CfgFile will attempt to parse it. This is
219  done by reading and processing each line of the configuration file into
220  separate label, value and comment fields. If an INC declaration is
221  encountered, a new instance of Fcm::CfgFile is created to read the included
222  file as specified. The included lines are then added to the current array.
223  Each line is then pushed into an array that can be fetched as a property of
224  Fcm::CfgFile. Internally, each line is recorded as a reference to a hash
225  table with the following keys:</p>
226
227  <ul class="pad">
228    <li>LABEL: the label of a declaration.</li>
229
230    <li>VALUE: the value of a declaration.</li>
231
232    <li>COMMENT: the comment following a declaration or the comment in a
233    comment line.</li>
234
235    <li>NUMBER: the line number of the current line in the source file.</li>
236
237    <li>SRC: the name of the source file.</li>
238  </ul>
239 
240  <p>The information given by each line is "deciphered" by Fcm::Extract. The
241  information is processed in the following ways:</p>
242
243  <ul class="pad">
244    <li>The configuration file type and version declarations are stored as
245    properties of the Fcm::CfgFile instance. Fcm::Extract uses the
246    information to ensure that it is reading an extract configuration
247    file.</li>
248
249    <li>The destination directory declarations are stored in a hash table,
250    which is a property of the Fcm::Extract instance.</li>
251
252    <li>The remote destionation machine, logname and directory declarations
253    are stored in a hash table, which is another property of the Fcm::Extract
254    instance.</li>
255
256    <li>Build configuration declarations are stored in a hash table, which is
257    yet another property of the Fcm::Extract instance.</li>
258
259    <li>For each declaration of a repository branch, a new instance of
260    Fcm::ReposBranch is created, if it does not already exist. (A
261    Fcm::ReposBranch instance is identified by a "tag" property, which is the
262    combination of its package name and its branch name.) The REPOS, VERSION,
263    SRC and EXPSRC declarations set the "repos", "version", "dir" and "expdir"
264    properties of the Fcm::ReposBranch instance.</li>
265
266    <li>The override mode flag is stored as a property of the Fcm::Extract
267    instance.</li>
268
269    <li>For each declaration to USE a previous extract, a new instance of
270    Fcm::Extract is created, with its "extracted" property set to true. The
271    instance Fcm::Extract for the previous extraction creates a new instance of
272    Fcm::CfgFile for its configuration file. The configuration of the previous
273    extract is read and processed similarly to what was described above. The
274    current instance of Fcm::Extract will then attempt to inherit the settings
275    of the previous extraction where appropriate. The instances of the
276    previous extractions are stored in an array, which can be fetched as a
277    property of the current Fcm::Extract instance.</li>
278
279    <li>The INC declaration is ignored, as it is already processed by
280    Fcm::CfgFile.</li>
281
282    <li>The MIRROR declaration changes the setting (TOOL, MIRROR) in the
283    Fcm::Config instance.</li>
284  </ul>
285
286  <p>If a full extraction is required, Fcm::Extract will attempt to remove any
287  sub-directories created by previous extractions in the same location.
288  Destination directories are (re-)created as they are required.</p>
289
290  <p>For each repository branch, if the REPOS declaration is a file system
291  path, the VERSION declaration will be set automatically to the word "USER".
292  If the REPOS declaration matches a FCM URL keyword pattern, it is expanded
293  to the full URL. If REPOS is not in the local file system and the VERSION
294  declaration is not a number, the system will attempt to convert the keyword
295  back to a number. If the keyword is "HEAD", the system will use <tt>"svn
296  info"</tt> to determine the revision number. Otherwise, it will attempt to
297  match the keyword with a pre-defined FCM revision keyword. If there are any
298  expanded source directory (EXPSRC) declarations, the system will use
299  <tt>"svn ls -R"</tt> to search recursively for all normal source directories
300  containing regular files. These directories are then added to the "dir"
301  property of the Fcm::ReposBranch instance.</p>
302
303  <h3><a name="task_seq">The extraction sequence</a></h3>
304
305  <p>In the next step, the extract system converts the information given in
306  the list of repository branches into a list of source directory sub-package.
307  Each source directory sub-package will have a destination and a "stack" of
308  extraction sequence. The sequence is basically a list for locating the
309  source directories in the repository branches. The order of the sequence is
310  based on the order in which a repository branch is declared. The logic has
311  already been discussed in the user guide.</p>
312
313  <p>The sequence is implemented by a list of Fcm::SrcDirLayer instances. For
314  each Fcm::SrcDirLayer instance in an extraction sequence of a source
315  directory, the system will attempt to find out its "last commit" revision,
316  using the <tt>"svn info"</tt> command on the particular revision on the
317  given URL. This information is normally cached in a file called ".config" in
318  the cache sub-directory of the extraction destination root. For an
319  incremental extraction, the system will consult the cache to obtain the list
320  of "last commit" revisions for the source directories, instead of having to
321  go through a large number of <tt>"svn info"</tt> commands again. The cache
322  file is read/written using a temporary instance of Fcm::CfgFile. The label
323  in each line consists of the package name of the sub-package, its URL and a
324  revision number. The corresponding value is the "last commit" revision at
325  the given revision number.</p>
326
327  <h3><a name="task_ext">The extraction</a></h3>
328
329  <p>With the extraction sequence in place for each source directory, the
330  extraction itself can now take place. There are two steps in this
331  process.</p>
332
333  <p>For each "layer" in the extraction sequence of each source directory, if
334  the "layer" contains a repository URL, the system extracts from that URL the
335  source directory and place the resulting source files in a cache. From the
336  cache sub-directory of the destination root, the cache for each source
337  directory is placed under a relative path that reflects the sub-package name
338  of the source directory. Underneath this path is a list of directories
339  with names reflecting the name of the branch and the "last commit"
340  revision, (separated by double underscore "__"). These are where the cache of
341  the source files for the "layers" of the source directory are placed.</p>
342
343  <p>It is also worth noting that source files from the local file system are
344  not cached. They will be taken directly from their locations.</p>
345
346  <p>Once we have the cached "layers" (or branches) of the source directories,
347  the system will select the source files from the correct cache before
348  updating the destinations. The logic of which has already been discussed in
349  the user guide.</p>
350
351  <p>At the end of this stage, we should have a directory tree in the
352  destination source sub-directory, with the relative paths to the extracted
353  files reflecting the sub-package names of those files.</p>
354
355  <h3><a name="task_gen">The extract/build configuration generator</a></h3>
356
357  <p>If extraction completes without any error, the system will attempt to
358  write an expanded extract configuration file, where all revision keywords
359  are expanded into numbers, and all source directory packages are declared.
360  Subsequent dependent extractions will be able to re-use this configuration
361  without having to invoke the Subversion client for repository and revision
362  information.</p>
363
364  <p>The system will also attempt to produce a build configuration file for
365  feeding to the build system. The following "conversions" are performed:</p>
366
367  <ul>
368    <li>The destination root becomes the build root.</li>
369
370    <li>The destination directories are all are declared as the source
371    directories.</li>
372
373    <li>All BLD declarations are unchanged, except that the BLD prefixes are
374    removed from the label.</li>
375
376    <li>All USE extract configuration statements are converted to USE build
377    configuration statements.</li>
378  </ul>
379
380  <h3><a name="task_mir">The mirror interface</a></h3>
381
382  <p>The system uses "rdist" or "rsync" to mirror the extracted source code
383  and the generated configuration files to a remote machine.</p>
384 
385  <script type="text/javascript" src="maintain.js">
386  </script>
387</body>
388</html>
Note: See TracBrowser for help on using the repository browser.