source: LMDZ6/trunk/tools/fcm/doc/user_guide/extract.html @ 4582

Last change on this file since 4582 was 1578, checked in by jghattas, 13 years ago
  • Add fcm in LMDZ5/tools directory

It is no longer needed to have fcm in your environement PATH variable.
Now makelmdz_fcm takes by default this fcm. It is still possible to use
another fcm, using -fcm_path argument in makelmdz_fcm.
File size: 43.6 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2
3<html>
4<head>
5  <title>FCM System User Guide: The Extract System</title>
6  <meta name="author" content="FCM development team">
7  <meta name="descriptions" content="User Guide - The Extract System">
8  <meta name="keywords" content="FCM, user guide, extract, build">
9  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10  <link rel="stylesheet" type="text/css" href="style.css">
11</head>
12
13<body>
14  <address>
15    <a href="index.html">FCM System User Guide</a> &gt; The Extract System
16  </address>
17
18  <h1>The Extract System</h1>
19
20  <p>The extract system provides an interface between the revision control
21  system (currently Subversion) and the build system. Where appropriate, it
22  extracts code from the repository and other user-defined locations to a
23  directory tree suitable for feeding into the build system. In this chapter,
24  we shall use many examples to explain how to use the extract system. At the
25  end of this chapter, you will be able to extract code from the local file system
26  as well as from different branches of different repository URLs. You will
27  also learn how to mirror code to a remote machine. Finally, you will be given
28  an introduction on how to specify configurations for the build system via the
29  extract configuration file. (For further information on the build system,
30  please see the next chapter <a href="build.html">The Build System</a>.) The
31  last section of the chapter tells you what you can do in the case when
32  Subversion is not available.</p>
33
34  <h2><a name="command">The Extract Command</a></h2>
35
36  <p>To invoke the extract system, simply issue the command:</p>
37  <pre>
38fcm extract
39</pre>
40
41  <p>By default, the extract system searches for an extract configuration file
42  "ext.cfg" in "$PWD" and then "$PWD/cfg". If an extract configuration file is
43  not found in these directories, the command fails with an error. If an
44  extract configuration file is found, the system will use the configuration
45  specified in the file to perform the current extraction.</p>
46
47  <p>If the destination of the extraction does not exist, the system performs a
48  new full extraction to the destination. If a previous extraction already
49  exists at the destination, the system performs an incremental extraction,
50  updating any modifications if necessary. If a full (fresh) extraction is
51  required for whatever reason, you can invoke the extract system using the
52  "-f" option, (i.e. the command becomes "fcm extract -f"). For further
53  information on the extract command, please see <a href=
54  "command_ref.html#fcm_ext">FCM Command Reference &gt; fcm extract</a>.</p>
55
56  <h2><a name="simple">Simple Usage</a></h2>
57
58  <p>The extract configuration file is the main user interface of the extract
59  system. It is a line based text file. For a complete set of extract
60  configuration file declarations, please refer to the <a href=
61  "annex_ext_cfg.html">Annex: Declarations in FCM extract configuration
62  file</a>.</p>
63
64  <h3><a name="simple_local">Extract from a local path</a></h3>
65
66  <p>A simple example of a basic extract configuration file is given below:</p>
67
68  <table class="pad" summary="extract_example1" border="1" width="100%">
69    <tr>
70      <th>Extract configuration example 1 - extract from a local path</th>
71    </tr>
72
73    <tr>
74      <td>
75        <pre>
76cfg::type         ext       # line 1
77cfg::version      1.0       # line 2
78                            # line 3
79dest::rootdir     $PWD      # line 4
80                            # line 5
81repos::var::user  $HOME/var # line 6
82                            # line 7
83expsrc::var::user code      # line 8
84</pre>
85      </td>
86    </tr>
87  </table>
88
89  <p>The above demonstrates how to use the extract system to extract code from
90  a local user directory. Here is an explanation of what each line does:</p>
91
92  <ul>
93    <li>line 1: the label "CFG::TYPE" declares the type of the configuration
94    file. The value "ext" tells the system that it is an extract configuration
95    file.</li>
96
97    <li>line 2: the label "CFG::VERSION" declares the version of the extract
98    configuration file. The current default is "1.0". Although it is not
99    currently used, if we have to change the format of the configuration file
100    at a later stage, we shall be able to use this number to determine whether
101    we are reading a file with an older format or one with a newer format.</li>
102
103    <li>line 3: a blank line or a line beginning with a "#" is a comment, and
104    is ignored by the interpreter.</li>
105
106    <li>line 4: the label "DEST::ROOTDIR" declares the destination root
107    directory of this extraction. The value "$PWD" expands to the current
108    working directory.</li>
109
110    <li>line 5: comment line, ignored.</li>
111
112    <li>line 6: the label "REPOS::&lt;pck&gt;::&lt;branch&gt;" declares the top
113    level URL or path of a repository. The package name of the repository is
114    given by &lt;pck&gt;. In our example, we choose "var" as the name of the
115    package. (You can choose any name you like, however, it is usually sensible
116    to use a package name that matches the name of the project or system you are
117    working with.) The branch name in the repository is given by &lt;branch&gt;.
118    (Again, you can choose any name you like, however, it is usually sensible to
119    use a name such as "trunk", "user" or something that matches your branch
120    name.) In our example, the word "user" is normally used to denote a local
121    user directory. Hence the statement declares that the repository path for
122    the "var" package in the "user" branch can be found at "$HOME/var".</li>
123
124    <li>line 7: comment line, ignored.</li>
125
126    <li>line 8: the label "EXPSRC::&lt;pck&gt;::&lt;branch&gt;" declares an
127    "expandable" source directory for the package &lt;pck&gt; in the branch
128    &lt;branch&gt;. In our example, the package name is "var", and the branch
129    name is "user". These match the package and the branch names of the
130    repository declaration in line 6. It means that the source directory
131    declaration is associated with the path "$HOME/var". The value of the
132    declaration "code" is therefore a sub-directory under "$HOME/var". By
133    declaring a source directory using an "expsrc" label, the system
134    automatically searches for all sub-directories (recursively) under the
135    declared source directory.</li>
136  </ul>
137
138  <p>Invoking the extract system using the above configuration file will
139  "extract" all sub-directories under "$HOME/var/code" to "$PWD/src/var/code".
140  Note: the extract system ignores all hidden files, (i.e. directories and
141  files beginning with a "."). It will write a build configuration file to
142  "$PWD/cfg/bld.cfg". The configuration used for this extraction will be
143  written to the configuration file at "$PWD/cfg/ext.cfg".</p>
144
145  <table class="pad" summary="note - incremental extraction" border="1" width=
146  "100%">
147    <tr>
148      <th>Note - incremental extraction</th>
149    </tr>
150
151    <tr>
152      <td>Suppose you have already performed an extraction using the above
153      configuration file. At a later time, you have made some changes to some
154      of the files in the source directory. Re-running the extract system on
155      the same configuration will trigger an incremental extraction. In an
156      incremental extraction, the system will update only those files that are
157      modified. In exact words, the system checks the modification time of each
158      file in the source and destination. If a source file is newer than its
159      corresponding destination file, it checks whether the content differs.
160      The destination is only updated if its content differs from the
161      source.</td>
162    </tr>
163  </table>
164
165  <h3><a name="simple_url">Extract from a Subversion URL</a></h3>
166
167  <p>The next example demonstrates how to extract from a Subversion repository
168  URL:</p>
169
170  <table class="pad" summary="extract_example2" border="1" width="100%">
171    <tr>
172      <th>Extract configuration example 2 - extract from a Subversion URL</th>
173    </tr>
174
175    <tr>
176      <td>
177        <pre>
178cfg::type           ext                    # line 1
179cfg::version        1.0                    # line 2
180                                           # line 3
181dest::rootdir       $PWD                   # line 4
182                                           # line 5
183repos::var::trunk   svn://server/var/trunk # line 6
184version::var::trunk 1234                   # line 7
185                                           # line 8
186expsrc::var::trunk  code                   # line 9
187</pre>
188      </td>
189    </tr>
190  </table>
191
192  <ul>
193    <li>line 1 to 5: same as example 1.</li>
194
195    <li>line 6: the line declares the repository location of the "trunk" branch
196    of the "var" package to be the Subversion URL
197    "svn://server/var/trunk".</li>
198
199    <li>line 7: the label "VERSION::&lt;pck&gt;::&lt;branch&gt;" declares the
200    revision of the repository associated with the package &lt;pck&gt; in the
201    branch &lt;branch&gt;. The current line tells the extract system to use
202    revision 1234 of "svn://server/var/trunk". If the revision is not declared,
203    the default revision "HEAD" will be assumed.</li>
204
205    <li>line 8: comment line, ignored.</li>
206
207    <li>line 9: the line declares an expandable source directory in the
208    repository "svn://server/var/trunk".</li>
209  </ul>
210
211  <p>Invoking the extract system using the above configuration file will
212  extract all sub-directories under "svn://server/var/trunk/code" to
213  "$PWD/src/var/code". It will write a build configuration file to
214  "$PWD/cfg/bld.cfg". The configuration used for this extraction will be
215  written to the configuration file at "$PWD/cfg/ext.cfg".</p>
216
217  <table class="pad" summary=
218  "note - declaration of source directories for extraction" border="1" width=
219  "100%">
220    <tr>
221      <th>Note - declaration of source directories for extraction</th>
222    </tr>
223
224    <tr>
225      <td>
226        <strong>EXPSRC or SRC?</strong>
227
228        <p>So far, we have only declared source directories using the "EXPSRC"
229        statement, which stands for "expandable source directory". A source
230        directory declared using this statement will trigger the system to
231        search recursively for any sub-directories under the declared one. Any
232        sub-directories containing regular source files will be included in the
233        extraction. Empty directories, hidden directories and directories
234        containing only hidden files are ignored.</p>
235
236        <p>If you do not want the system to search for sub-directories
237        underneath your declared source directory, you can declare your source
238        directory using the "SRC" statement. The "SRC" statement is essentially
239        the same as "EXPSRC" except that it does not trigger the automatic
240        recursive search for source directories. In fact, the system implements
241        the "EXPSRC" statement by expanding it into a list of "SRC"
242        statements.</p>
243
244        <p><strong>Package and sub-package</strong></p>
245
246        <p>The second field of a repository, revision or source directory
247        declaration label is the name of the container package. It is a name
248        selected by the user to identify the system or project he/she is
249        working on. (Therefore, it is often sensible to choose an identifier
250        that matches the name of the project or system.) The package name
251        provides a unique namespace for a file container. Source directories
252        are automatically arranged into sub-packages, using the names of the
253        sub-directories as the names of the sub-packages. For example, the
254        declaration at line 9 in example 2 will put the source directory in the
255        "var::code" sub-package automatically.</p>
256
257        <p>The double colon "::" and the double underscore "__" (internal only)
258        are delimiters for package names in the extract system. Please avoid
259        using "::" and "__" for naming your files and directories.</p>
260
261        <p>You can declare a sub-package name explicitly in your source
262        directory statement. For example, the following two lines are
263        equivalent:</p>
264        <pre>
265src::var::trunk                        code/VarMod_Surface
266src::var::code::VarMod_Surface::trunk  code/VarMod_Surface
267</pre>
268
269        <p>Explicit sub-package declaration should not be used normally, as it
270        requires a lot more typing (although there are some situations where it
271        can be useful).</p>
272      </td>
273    </tr>
274  </table>
275
276  <table class="pad" summary="note - the expanded extract configuration file"
277  border="1" width="100%">
278    <tr>
279      <th>Note - the expanded extract configuration file</th>
280    </tr>
281
282    <tr>
283      <td>
284        At the end of a successful extraction, the configuration used by the
285        current extraction is written in "cfg/ext.cfg" under the extract
286        destination root. This file is an "expanded" version of the original,
287        with changes in the following declarations:
288
289        <ul>
290          <li>All revision keywords are converted into revision numbers.</li>
291
292          <li>If a revision is not defined for a repository, it is set to the
293          corresponding revision number of the HEAD revision.</li>
294
295          <li>All URL keywords are converted into the full URLs.</li>
296
297          <li>All EXPSRC declarations are expanded into SRC declarations.</li>
298
299          <li>All other variables are expanded.</li>
300        </ul>
301
302        <p>With this file, it should be possible for later extraction to
303        re-create the current configuration even if the contents of the
304        repository have changed. (This applies only to code stored in the
305        repository.)</p>
306      </td>
307    </tr>
308  </table>
309
310  <h3><a name="simple_mirror">Mirror code to a remote machine</a></h3>
311
312  <p>The next example demonstrates how to extract from a repository and mirror
313  the code to a remote machine. It is essentially the same as example 2,
314  except that it has three new lines to describe how the system can mirror the
315  extracted code to a remote machine.</p>
316
317  <table class="pad" summary="extract_example3" border="1" width="100%">
318    <tr>
319      <th>Extract configuration example 3 - mirror code to remote machine</th>
320    </tr>
321
322    <tr>
323      <td>
324        <pre>
325cfg::type           ext
326cfg::version        1.0
327
328dest::rootdir       $PWD
329
330rdest::machine      tx01                           # line 6
331rdest::logname      frva                           # line 7
332rdest::rootdir      /scratch/frva/extract/example3 # line 8
333
334repos::var::trunk   svn://server/var/trunk
335version::var::trunk 1234
336
337expsrc::var::trunk  code
338</pre>
339      </td>
340    </tr>
341  </table>
342
343  <p>Here is an explanation of what each line does:</p>
344
345  <ul>
346    <li>line 6: "RDEST::MACHINE" declares the target machine to which the code
347    will be mirrored. The example mirrors the code to the machine named
348    "tx01".</li>
349
350    <li>line 7: "RDEST::LOGNAME" declares the remote user name of the target
351    machine, to which the user has remote login access. If this is not
352    declared, the system uses the login name of the current user on the local
353    machine.</li>
354
355    <li>line 8: "RDEST::ROOTDIR" declares the root directory of the extraction
356    on the remote target machine. This is where the mirror version of the
357    extraction will be sent.</li>
358  </ul>
359
360  <p>Invoking the extract system on the above configuration will trigger an
361  extraction similar to that given in example 2, but it will also attempt to
362  mirror the contents at "$PWD/src/var/code" to
363  "/scratch/frva/extract/example3/src" on the remote machine. It will also
364  mirror the expanded extract configuration file "$PWD/cfg/ext.cfg" to
365  "/scratch/frva/extract/example3/cfg/ext.cfg" and "$PWD/cfg/bld.cfg" to
366  "/scratch/frva/extract/example3/cfg/bld.cfg". It is also worth noting that
367  the content of the build configuration file will be slightly different, since
368  it will include directory names appropriate for the remote system.</p>
369
370  <table class="pad" summary="note - mirroring command" border="1" width=
371  "100%">
372    <tr>
373      <th>Note - mirroring command</th>
374    </tr>
375
376    <tr>
377      <td>
378        The extract system currently supports "rdist" and "rsync" as its
379        mirroring tool. The default is "rsync". To use "rdist" instead of
380        "rsync", add the following line to your "$HOME/.fcm" file:
381        <pre>
382set::tool::mirror  rdist
383</pre>
384
385        <p>N.B. If you are going to mirror code to another machine, you need to
386        ensure that your account on the remote machine is set up correctly to
387        accept commands from the local machine. In our current settings of both
388        "rdist" and "rsync", all you need to do is set up your "$HOME/.rhosts"
389        file on the remote machine. For example, if you are "fred" working on
390        the local machine "eld001", you will need to have the following entry
391        in your "$HOME/.rhosts" on the remote machine:</p>
392        <pre>
393eld001  fred
394</pre>
395      </td>
396    </tr>
397  </table>
398
399  <h2><a name="advanced">Advanced Usage</a></h2>
400
401  <h3><a name="advanced_multi">Extract from multiple repositories</a></h3>
402
403  <p>So far, we have only extracted from a single location. The extract system
404  is not much use if that is the only thing it can do. In fact, the extract
405  system supports extraction of multiple source directories from multiple
406  branches in multiple repositories. The following configuration file is an
407  example of how to extract from multiple repositories:</p>
408
409  <table class="pad" summary="extract_example4" border="1" width="100%">
410    <tr>
411      <th>Extract configuration example 4 - extract from multiple
412      repositories</th>
413    </tr>
414
415    <tr>
416      <td>
417        <pre>
418cfg::type           ext
419cfg::version        1.0
420
421dest::rootdir       $PWD
422
423repos::var::trunk   fcm:var_tr              # line 6
424repos::ops::trunk   fcm:ops_tr              # line 7
425repos::gen::trunk   fcm:gen_tr              # line 8
426
427version::gen::trunk 2468                    # line 10
428
429expsrc::var::trunk  code                    # line 12
430expsrc::var::trunk  scripts                 # line 13
431expsrc::ops::trunk  code                    # line 14
432src::gen::trunk     code/GenMod_Constants   # line 15
433src::gen::trunk     code/GenMod_Control     # line 16
434src::gen::trunk     code/GenMod_FortranIO   # line 17
435src::gen::trunk     code/GenMod_GetEnv      # line 18
436src::gen::trunk     code/GenMod_ModelIO     # line 19
437src::gen::trunk     code/GenMod_ObsInfo     # line 20
438src::gen::trunk     code/GenMod_Platform    # line 21
439src::gen::trunk     code/GenMod_Reporting   # line 22
440src::gen::trunk     code/GenMod_Trace       # line 23
441src::gen::trunk     code/GenMod_UMConstants # line 24
442src::gen::trunk     code/GenMod_Utilities   # line 25
443</pre>
444      </td>
445    </tr>
446  </table>
447
448  <p>Here is an explanation of what each line does:</p>
449
450  <ul>
451    <li>line 6 to 8: these lines declare the repositories for the "trunk"
452    branches of the "var", "ops" and "gen" packages respectively. It is worth
453    noting that the values of the declarations are no longer Subversion URLs
454    but are FCM URL keywords. These keywords are normally declared in the
455    central configuration file of the FCM system, and will be expanded into the
456    corresponding Subversion URLs by the FCM system. For further information on
457    URL keywords, please see <a href=
458    "code_management.html#svn_basic_keywords">Code Management System &gt; Using
459    Subversion &gt; Basic Command Line Usage &gt; Repository &amp; Revision
460    Keywords</a>.</li>
461
462    <li>line 10: this line declares the revision number for the "trunk" branch
463    of the "gen" package, i.e. for the "fcm:gen_tr" repository. It is worth
464    noting that the revision numbers for the "var" and "ops" packages have not
465    been declared. By default, their revision numbers will be set to
466    "HEAD".</li>
467
468    <li>line 12 to 14: these line declares the source directories for the
469    "trunk" branches of the "var" and "ops" packages. For the "var" package, we
470    are extracting everything from the "code" and the "scripts" sub-directory.
471    For the "ops" package, we are extracting everything from the "code"
472    directory.</li>
473
474    <li>line 15 to 25: these line declares the source directories for the
475    "trunk" branch of the "gen" package. The source directories declared will
476    not be searched for sub-directories underneath the declared
477    directories.</li>
478  </ul>
479
480  <p>We shall end up with a directory tree such as:</p>
481  <pre>
482$PWD
483   |
484   |--- cfg
485   |      |
486   |      |--- bld.cfg
487   |      |--- ext.cfg
488   |
489   |--- src
490          |
491          |--- gen
492          |      |
493          |      |--- code
494          |              |
495          |              |--- GenMod_Constants
496          |              |--- GenMod_Control
497          |              |--- GenMod_FortranIO
498          |              |--- GenMod_GetEnv
499          |              |--- GenMod_ModelIO
500          |              |--- GenMod_ObsInfo
501          |              |--- GenMod_Platform
502          |              |--- GenMod_Reporting
503          |              |--- GenMod_Trace
504          |              |--- GenMod_UMConstants
505          |              |--- GenMod_Utilities
506          |
507          |--- ops
508          |      |
509          |      |--- code
510          |              |
511          |              |--- ...
512          |
513          |--- var
514                 |
515                 |--- code
516                 |       |
517                 |       |--- ...
518                 |
519                 |--- scripts
520                         |
521                         |--- ...
522</pre>
523
524  <table class="pad" summary="note - revision number" border="1" width="100%">
525    <tr>
526      <th>Note - revision number</th>
527    </tr>
528
529    <tr>
530      <td>
531        As seen in the above example, if a revision number is not specified for
532        a repository URL, it defaults to the "HEAD" revision. The revision
533        number can also be declared in other ways:
534
535        <ul>
536          <li>Any revision arguments acceptable by Subversion are allowed. You
537          can use a valid revision number, a date between a pair of curly
538          brackets (e.g. {"2005-05-01 12:00"}) or the keyword "HEAD". However,
539          please do not use the keywords "BASE", "COMMITTED" or "PREV" as these
540          are reserved for working copy only.</li>
541
542          <li>FCM revision keywords are allowed. These must be defined for the
543          corresponding repository URLs in either the central or the user FCM
544          configuration file. For further information on revision keywords,
545          please see <a href="code_management.html#svn_basic_keywords">Code
546          Management &gt; Using Subversion &gt; Basic Command Line Usage &gt;
547          Repository &amp; Revision Keywords</a>.</li>
548
549          <li>Do not use the keyword "USER", as it is used internally by the
550          extract system.</li>
551        </ul>
552      </td>
553    </tr>
554  </table>
555
556  <h3><a name="advanced_branches">Extract from multiple branches</a></h3>
557
558  <p>We have so far dealt with extraction from a single branch in any package.
559  The extract system can be used to "combine" changes from different branches
560  of a package. An example is given below:</p>
561
562  <table class="pad" summary="extract_example5" border="1" width="100%">
563    <tr>
564      <th>Extract configuration example 5 - extract from multiple branches</th>
565    </tr>
566
567    <tr>
568      <td>
569        <pre>
570cfg::type               ext
571cfg::version            1.0
572 
573dest::rootdir           $PWD
574 
575repos::var::trunk       fcm:var_tr
576repos::ops::trunk       fcm:ops_tr
577repos::gen::trunk       fcm:gen_tr
578 
579version::gen::trunk     2468
580 
581expsrc::var::trunk      code
582expsrc::var::trunk      scripts
583expsrc::ops::trunk      code
584src::gen::trunk         code/GenMod_Constants
585src::gen::trunk         code/GenMod_Control
586src::gen::trunk         code/GenMod_FortranIO
587src::gen::trunk         code/GenMod_GetEnv
588src::gen::trunk         code/GenMod_ModelIO
589src::gen::trunk         code/GenMod_ObsInfo
590src::gen::trunk         code/GenMod_Platform
591src::gen::trunk         code/GenMod_Reporting
592src::gen::trunk         code/GenMod_Trace
593src::gen::trunk         code/GenMod_UMConstants
594src::gen::trunk         code/GenMod_Utilities
595
596repos::var::new_stuff   fcm:var_br/frva/r1234_new_stuff   # line 27
597repos::var::bug_fix     fcm:var_br/frva/r1516_bug_fix     # line 28
598repos::ops::good_stuff  fcm:ops_br/opsrc/r3188_good_stuff # line 29
599</pre>
600      </td>
601    </tr>
602  </table>
603
604  <p>The configuration file in example 5 is similar to that of example 4 except
605  for the last three lines. Here is an explanation of what they do:</p>
606
607  <ul>
608    <li>line 27: this line declares a repository URL for the "new_stuff" branch
609    of the "var" package. From the URL of the branch, we know that the branch
610    was created by the user "frva" based on the trunk at revision at 1234. The
611    description of the branch is "new_stuff". The following points are worth
612    noting:
613
614      <ul>
615        <li>By declaring a new branch with the same package name to a
616        previously declared branch, it is assumed that both branches reside in
617        the same Subversion repository.</li>
618
619        <li>It is only by convention that the branch name "new_stuff" declared
620        using the "REPOS::var::new_stuff" label should correspond to the branch
621        description of the URL. It is not a requirement by the extract system.
622        It is perfectly valid to use "REPOS::var::ham_egg" as the label of line
623        27. In such case, the extract system will use "ham_egg" instead of
624        "new_stuff" as the internal label of the URL.</li>
625
626        <li>No revision is declared for this URL, so the default "HEAD"
627        revision is used.</li>
628
629        <li>No source directory is declared for this URL. By default, if no
630        source directory is declared for a branch repository, it will attempt
631        to use the same set of source directories as the first declared branch
632        of the package. In this case, the source directories declared for the
633        "trunk" branch of the "var" package will be used.</li>
634      </ul>
635    </li>
636
637    <li>line 28: this line declares another branch called "bug_fix" for the
638    "var" package. No source directory is declared for this URL either, so it
639    will use the same set of source directories declared for the "trunk"
640    branch.</li>
641
642    <li>line 29: this line declares a branch called "good_stuff" for the "ops"
643    package. It will use the same set of source directories declared for the
644    "ops" package "trunk" branch.</li>
645  </ul>
646
647  <p>When we invoke the extract system, it will attempt to extract from the
648  first declared branch of a package, if the last commit revision of the source
649  directory is the same in all the branches. However, if the last commit
650  revision of the source directory differs for different branches, the system
651  will attempt to obtain an extract priority list for each source directory,
652  using the following logic:</p>
653
654  <ol>
655    <li>The system looks for source directory packages from the first declared
656    branch to the last declared branch.</li>
657
658    <li>The branch in which a source directory package is first declared is
659    the "base" branch of the source directory package.</li>
660
661    <li>The commit revision of a source directory package in a subsequently
662    declared repository branch is compared with that of the base branch. If
663    the commit revision is the same as that of the base branch, the source
664    directory of this branch is discarded. Otherwise, it is placed at the end
665    of the extract priority list.</li>
666  </ol>
667
668  <p>For the "var" package in the above example, let us assume that we have
669  three source directory packages X, Y and Z under "code", and their commit
670  revisions under "trunk" are 100. Let's say we have committed some changes to
671  X and Z in the "new_stuff" branch at revision 102, and other changes to Y
672  and Z in the "bug_fix" branch at revision 104, the extract priority lists
673  for X, Y and Z will look like:</p>
674
675  <ul>
676    <li>X: trunk (100, base), new_stuff (102), bug_fix (100, discarded)</li>
677
678    <li>Y: trunk (100, base), new_stuff (100, discarded), bug_fix (104)</li>
679
680    <li>Z: trunk (100, base), new_stuff (102), bug_fix (104)</li>
681  </ul>
682
683  <p>Once we have an extract priority list for a source directory, we can
684  begin the extraction of source files in the source directory. The source
685  directory of the base branch is extracted first, followed by that in the
686  subsequent branches. If a source file in a subsequent branch has the same
687  content as the that in the base branch, it is discarded. Otherwise, the
688  following logic determines the branch to use:</p>
689
690  <ol>
691    <li>If a source file is modified in only one subsequent branch, the source
692    file in that branch is extracted.</li>
693
694    <li>If a source file is modified in two or more subsequent branches, but
695    their modifications are the same, then the source file in the first
696    modification is used.</li>
697
698    <li>If a source file is modified in two or more subsequent branches and
699    their modifications differ, then the behaviour depends on the override
700    mode setting. If override mode is false, the extraction fails. Otherwise,
701    the file in the latest declared branch takes precedence. The default
702    override mode is false, which is the safe behaviour - conflicting changes
703    should be resolved manually. However, this behaviour may not be the most
704    efficient way for development. Therefore, an override mode is provided,
705    which should be used with care. To switch on the override mode, make the
706    following declaration in the extract configuration file:
707      <pre>
708override  1
709</pre>
710    </li>
711  </ol>
712 
713  <p>Once the system has established which source files to use, it determines
714  whether the destination is out of date or not. The destination is out of
715  date if the source file does not exist or if its content differs from the
716  version of the source file we are using. The system only updates the
717  destination if it is considered to be out of date.</p>
718
719  <p>The extract system can also combine changes from branches in the Subversion
720  repository and the local file system. The limitation is that there can only
721  be one branch from the local file system. (By convention, the branch is named
722  "user".)</p>
723
724  <p>It is also worth bearing in mind that the "user" branch always takes
725  precedence over branches residing in Subversion repositories. Hence, source
726  directories from a "user" branch are always placed at the end of the extract
727  priority list.</p>
728
729  <p>Extracting from a mixture of Subversion repository and local file system
730  is demonstrated in the next example.</p>
731
732  <table class="pad" summary="extract_example6" border="1" width="100%">
733    <tr>
734      <th>Extract configuration example 6 - extract from multiple branches +
735      user paths</th>
736    </tr>
737
738    <tr>
739      <td>
740        <pre>
741cfg::type               ext
742cfg::version            1.0
743 
744dest::rootdir           $PWD
745 
746repos::var::trunk       fcm:var_tr
747repos::ops::trunk       fcm:ops_tr
748repos::gen::trunk       fcm:gen_tr
749 
750version::gen::trunk     2468
751 
752expsrc::var::trunk      code
753expsrc::var::trunk      scripts
754expsrc::ops::trunk      code
755src::gen::trunk         code/GenMod_Constants
756src::gen::trunk         code/GenMod_Control
757src::gen::trunk         code/GenMod_FortranIO
758src::gen::trunk         code/GenMod_GetEnv
759src::gen::trunk         code/GenMod_ModelIO
760src::gen::trunk         code/GenMod_ObsInfo
761src::gen::trunk         code/GenMod_Platform
762src::gen::trunk         code/GenMod_Reporting
763src::gen::trunk         code/GenMod_Trace
764src::gen::trunk         code/GenMod_UMConstants
765src::gen::trunk         code/GenMod_Utilities
766
767repos::var::new_stuff   fcm:var_br/frva/r1234_new_stuff
768repos::var::bug_fix     fcm:var_br/frva/r1516_bug_fix
769repos::ops::good_stuff  fcm:ops_br/opsrc/r3188_good_stuff
770
771repos::var::user        $HOME/var/src                     # line 31
772repos::gen::user        $HOME/gen/src                     # line 32
773</pre>
774      </td>
775    </tr>
776  </table>
777
778  <p>Example 6 is similar to example 5 except that it is also extracting from
779  local directories. Here is an explanation of the lines:</p>
780
781  <ul>
782    <li>line 31 to 32: these line declare the "repositories" for the "user"
783    branches of the "var" and "gen" packages respectively. Both are local paths
784    at the local file system. There are no declarations for source directories
785    for the "user" branches, so they use the same set of source directories of
786    the first declared branches, the "trunk" branches in both cases.</li>
787  </ul>
788
789  <table class="pad" summary="note - the inc statement" border="1" width=
790  "100%">
791    <tr>
792      <th>Note - the INC declaration</th>
793    </tr>
794
795    <tr>
796      <td>
797        You have probably realised that the above examples have many repeated
798        lines. To avoid having repeated lines in multiple extract configuration
799        files, you can use INC declarations to "include" other extract
800        configuration files. For example, if the configuration file of example 5
801        is stored in the file "$HOME/var/example5/ext.cfg", line 1 to 29 of
802        example 6 can be replaced with an INC declaration. Example 6 can then be
803        written as:
804        <pre>
805inc                     $HOME/var/example5/ext.cfg
806
807repos::var::user        $HOME/var/src
808repos::gen::user        $HOME/gen/src
809</pre>
810
811        <p>Note: the INC declaration supports the special "environment variable"
812        $HERE. If this variable is already set in the environment, it acts as a
813        normal environment variable. However, if it is not set, it will be
814        expanded into the container directory of the current extract
815        configuration file. This feature is particularly useful if you are
816        including a hierarchy of extract configurations from files in the same
817        container directory in a repository.</p>
818      </td>
819    </tr>
820  </table>
821
822  <h3><a name="advanced_incremental">Incremental extract based on a previous
823  extraction</a></h3>
824
825  <p>All the examples above dealt with standalone extraction, that is, the
826  current extraction is independent of any other extraction. If a previous
827  extraction exists in another location, the extract system can "USE" this
828  previous extraction in your current extraction. This works like a normal
829  incremental extraction, except that your extraction will only contain the
830  changes you have specified (compared with the USEd extraction) instead of the
831  full source directory tree. This type of incremental extraction is useful in
832  several ways. For instance:</p>
833
834  <ul>
835    <li>The extraction is fast, because you only have to extract and mirror
836    files that you have changed.</li>
837
838    <li>The subsequent build will also be fast, since it will use incremental
839    build.</li>
840
841    <li>You do not need write access to the original extraction. A system
842    administrator can set up a stable version in a central account, which
843    developers can then reuse.</li>
844
845    <li>You want an incremental extraction, but you need to leave the original
846    extraction unmodified.</li>
847  </ul>
848
849  <p>The following example is based on example 4 and 6. The assumption is that
850  an extraction has already been performed at the directory "~frva/var/vn22.0"
851  based on the configuration file in example 4.</p>
852
853  <table class="pad" summary="extract_example7" border="1" width="100%">
854    <tr>
855      <th>Extract configuration example 7 - incremental extract based on a
856      previous extraction</th>
857    </tr>
858
859    <tr>
860      <td>
861        <pre>
862cfg::type               ext
863cfg::version            1.0
864 
865dest::rootdir           $PWD
866
867use                     ~frva/var/vn22.0                  # line 6
868
869repos::var::new_stuff   fcm:var_br/frva/r1234_new_stuff   # line 8
870repos::var::bug_fix     fcm:var_br/frva/r1516_bug_fix     # line 9
871repos::ops::good_stuff  fcm:ops_br/opsrc/r3188_good_stuff # line 10
872
873repos::var::user        $HOME/var/src                     # line 12
874repos::gen::user        $HOME/gen/src                     # line 13
875</pre>
876      </td>
877    </tr>
878  </table>
879
880  <ul>
881    <li>line 6: this line replaces line 1 to 25 of example 6. It declares that
882    the current extraction should be based on the previous extraction located
883    at "~frva/var/vn22.0".</li>
884  </ul>
885
886  <p>Running the extract system using the above configuration will trigger an
887  incremental extraction, as if you are running an incremental extraction
888  having modified the configuration file in example 4 to that of example 6. The
889  only difference is that the original extraction using the example 4
890  configuration will be left untouched at "~frva/var/vn22.0", and the new
891  extraction will contain only the changes in the branches declared from line 8
892  to 13.</p>
893
894  <p>If you are setting up an extraction to be reused, you do not have to
895  perform a build. If you don't you will still gain the benefit of incremental
896  file extraction, but you will be performing a full build of the code.</p>
897
898  <h3><a name="advanced_build">Extract - Build Configuration</a></h3>
899
900  <p>Configuration settings for feeding into the build system can be declared
901  through the extract configuration file using the "BLD::" prefix. Any line in
902  an extract configuration containing a label with such a prefix will be
903  considered a build system variable. At the end of a successful extraction,
904  the system strips out the "BLD::" prefix before writing these variables to
905  the build configuration file. Some example entries are given between line 17
906  and 22 in the following configuration file:</p>
907
908  <table class="pad" summary="extract_example8" border="1" width="100%">
909    <tr>
910      <th>Extract configuration example 8 - setting build configuration</th>
911    </tr>
912
913    <tr>
914      <td>
915        <pre>
916cfg::type           ext
917cfg::version        1.0
918
919dest::rootdir       $PWD
920
921repos::var::trunk   fcm:var_tr
922repos::ops::trunk   fcm:ops_tr
923repos::gen::trunk   fcm:gen_tr
924
925version::gen::trunk 2468
926
927expsrc::var::trunk  code
928expsrc::var::trunk  scripts
929expsrc::ops::trunk  code
930src::gen::trunk     code/GenMod_Constants
931src::gen::trunk     code/GenMod_Control
932src::gen::trunk     code/GenMod_FortranIO
933src::gen::trunk     code/GenMod_GetEnv
934src::gen::trunk     code/GenMod_ModelIO
935src::gen::trunk     code/GenMod_ObsInfo
936src::gen::trunk     code/GenMod_Platform
937src::gen::trunk     code/GenMod_Reporting
938src::gen::trunk     code/GenMod_Trace
939src::gen::trunk     code/GenMod_UMConstants
940src::gen::trunk     code/GenMod_Utilities
941
942bld::target         VarProg_AnalysePF.exe   # line 27
943
944bld::tool::fc       sxmpif90                # line 29
945bld::tool::cc       sxmpic++                # line 30
946bld::tool::ld       sxmpif90                # line 31
947</pre>
948      </td>
949    </tr>
950  </table>
951
952  <p>The above example is essentially the same as example 4, apart from the
953  additional build configuration. The following is a simple explanation of what
954  the lines represent: (For detail of the build system, please see the next
955  chapter on <a href="build.html">The Build System</a>.)</p>
956
957  <ul>
958    <li>Line 27: the line declares a default target of the build.</li>
959
960    <li>Line 29-31: the lines declare the Fortran compiler, the C compiler and
961    the linker respectively.</li>
962  </ul>
963
964  <table class="pad" summary="note - user variable" border="1" width=
965  "100%">
966    <tr>
967      <th>Note - user variable</th>
968    </tr>
969
970    <tr>
971      <td>When you start using the extract system to define compiler flags for
972      the build system, you may end up having to make a lot of long and
973      repetitive declarations. In such case, you may want to define variables to
974      replace the repetitive parts of the declarations. In the extract system,
975      you can define a local variable by making a declaration with a label that
976      begins with a percent sign "%". For example:
977
978        <pre>
979# Declare a variable %fred
980%fred                     -Cdebug -eC -Wf,-init heap=nan stack=nan
981
982bld::tool::fflags         %fred
983# bld::tool::fflags       -Cdebug -eC -Wf,-init heap=nan stack=nan
984
985bld::tool::fflags::foo    %fred -f0
986# bld::tool::fflags::foo  -Cdebug -eC -Wf,-init heap=nan stack=nan -f0
987
988bld::tool::fflags::bar    -w %fred
989# bld::tool::fflags::bar  -w -Cdebug -eC -Wf,-init heap=nan stack=nan
990</pre>
991      </td>
992    </tr>
993  </table>
994
995  <h2><a name="verbose">Diagnostic verbose level</a></h2>
996
997  <p>The amount of diagnostic messages generated by the extract system is
998  normally set to a level suitable for normal everyday operation. This is the
999  default diagnostic verbose level 1. If you want a minimum amount of
1000  diagnostic messages, you should set the verbose level to 0. If you want more
1001  diagnostic messages, you can set the verbose level to 2 or 3. You can modify
1002  the verbose level in two ways. The first way is to set the environment
1003  variable FCM_VERBOSE to the desired verbose level. The second way is to
1004  invoke the extract system with the "-v &lt;level&gt;" option. (If set, the
1005  command line option overrides the environment variable.)</p>
1006
1007  <p>The following is a list of diagnostic output at each verbose level:</p>
1008
1009  <table class="pad" summary="List of diagnostic verbose output" border="1">
1010    <tr>
1011      <th>Verbose level</th>
1012
1013      <th>Possible output</th>
1014    </tr>
1015
1016    <tr>
1017      <th>0</th>
1018
1019      <td>
1020        <ul>
1021          <li>Report the time taken to extract the code.</li>
1022
1023          <li>Report the time taken to mirror the code.</li>
1024
1025          <li>If "rdist" is used to mirror the code, run the command with the
1026          "-q" option.</li>
1027        </ul>
1028      </td>
1029    </tr>
1030
1031    <tr>
1032      <th>1</th>
1033
1034      <td>
1035        <ul>
1036          <li>Everything at verbose level 0.</li>
1037
1038          <li>Report the name of the extract configuration file.</li>
1039
1040          <li>Report date/time at the beginning of the extract step.</li>
1041
1042          <li>Report the number of directories created, number of ignored
1043          sub-directories, number of files updated and number of removed
1044          files.</li>
1045
1046          <li>In override mode, report any files that are modified in two or
1047          more branches.</li>
1048
1049          <li>Report date/time at the beginning of the mirror step.</li>
1050
1051          <li>Report total time.</li>
1052        </ul>
1053      </td>
1054    </tr>
1055
1056    <tr>
1057      <th>2</th>
1058
1059      <td>
1060        <ul>
1061          <li>Everything at verbose level 1.</li>
1062
1063          <li>If the revision specified for a repository branch is not current
1064          (i.e. the specified revision number is less than the revision number
1065          of the last commit revision), print an information statement to
1066          inform the user of the last commit revision of the branch.</li>
1067
1068          <li>Report details of created directories, ignored sub-directories,
1069          removed files and updated files.</li>
1070
1071          <li>Report any files that have the same modifications in two or more
1072          branches.</li>
1073
1074          <li>If "rdist" is used to mirror the code, run the command without the
1075          "-q" option.</li>
1076        </ul>
1077      </td>
1078    </tr>
1079
1080    <tr>
1081      <th>3</th>
1082
1083      <td>
1084        <ul>
1085          <li>Everything at verbose level 2.</li>
1086
1087          <li>Report all shell commands invoked by the extract system with
1088          timestamp.</li>
1089
1090          <li>If "rdist" is used to mirror the code, print the "distfile"
1091          supplied to the command.</li>
1092
1093          <li>If "rsync" is used to mirror the code, invoke the command with
1094          the "-v" option.</li>
1095        </ul>
1096      </td>
1097    </tr>
1098  </table>
1099
1100  <h2><a name="nosvn">When Subversion Is Not Available</a></h2>
1101
1102  <p>The extract system can still be used if Subversion is not available.
1103  Clearly, you can only use local repositories. However, you can still do
1104  incremental extraction, mirror an extraction to another machine, or combine
1105  code from multiple local repositories.</p>
1106
1107  <p>If you are using Subversion but your server is down then clearly there is
1108  little you can do. However, if you already have an extraction then you can
1109  re-run <tt>fcm extract</tt> as long as the extract configuration file only
1110  refers to fixed revisions. If this is not the case then you can always use the
1111  expanded extract configuration file which can be found in "cfg/ext.cfg" under
1112  the extract destination root. This means that you can continue to makes changes
1113  to local code and do incremental extractions even whilst your Subversion server
1114  is down.</p>
1115
1116  <script type="text/javascript" src="maintain.js"></script>
1117</body>
1118</html>
Note: See TracBrowser for help on using the repository browser.