source: LMDZ5/branches/IPSLCM5A2.1/tools/fcm/doc/user_guide/system_admin.html @ 3206

Last change on this file since 3206 was 1578, checked in by jghattas, 13 years ago
  • Add fcm in LMDZ5/tools directory

It is no longer needed to have fcm in your environement PATH variable.
Now makelmdz_fcm takes by default this fcm. It is still possible to use
another fcm, using -fcm_path argument in makelmdz_fcm.
File size: 24.0 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2
3<html>
4<head>
5  <title>FCM System User Guide: System Administration</title>
6  <meta name="author" content="FCM development team">
7  <meta name="descriptions" content="User Guide - System Administration">
8  <meta name="keywords" content="FCM, user guide">
9  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
10  <link rel="stylesheet" type="text/css" href="style.css">
11</head>
12
13<body>
14  <address>
15    <a href="index.html">FCM System User Guide</a> &gt; System Administration
16  </address>
17
18  <h1>System Administration</h1>
19
20  <p>This chapter provides an administration guide for managers of projects or
21  systems which are using FCM.</p>
22
23  <p>Note that, where this section refers to the "FCM team" this applies only to
24  Met Office users. External users will either need to refer to the equivalent
25  team within their organisation or will need to perfom these tasks themselves.
26  </p>
27
28  <h2><a name="svn">Subversion</a></h2>
29
30  <h3><a name="svn_design">Repository design</a></h3>
31
32  <p>The FCM system assumes that each project directory has sub-directories called
33  <em>trunk</em>, <em>branches</em> and <em>tags</em> (Subversion recommends it but
34  doesn't insist on it). We recommend that each project within a Subversion
35  repository is in a sub-directory of the repository root.</p>
36      <pre>
37&lt;root&gt;
38    |
39    |-- &lt;project 1&gt;
40    |       |
41    |       |-- trunk
42    |       |-- branches
43    |       |-- tags
44    |
45    |-- &lt;project 2&gt;
46    |       |
47    |       |-- trunk
48    |       |-- branches
49    |       |-- tags
50    |
51    |-- ...
52</pre>
53  <p>In theory you could also have the project as the root directory or several levels
54  below the root directory. However, this is not tested and could cause problems with
55  some <tt>fcm</tt> commands so is best avoided.</p>
56
57  <p>You will need to decide whether to use a single project tree for your system or
58  whether to use multiple projects.</p>
59
60  <p>Advantages of a single project tree:</p>
61
62  <ul>
63    <li>Changes to any part of the system can always be committed as a single logical
64    changeset. If you split your system into multiple projects then you may have
65    occasions when a logical change involves more than one project and hence requires
66    multiple commits (and branches).</li>
67  </ul>
68
69  <p>Disadvantages of a single project tree:</p>
70
71  <ul>
72    <li>If you have a large system then your working copies may become very large
73    and unwieldy. Basic commands such as <tt>checkout</tt> and <tt>status</tt>
74    can become frustratingly slow if your working copy is too large.</li>
75       
76    <li>Depending on how you work, you may end up doing lots more merges of files
77    that are unrelated to your work.</li>
78  </ul>
79
80  <p>One common approach is to split the admin type files (things that only the
81  system manager should need to touch) into a separate project from the core
82  system files (which all the developers need access to). If you include any
83  large data files under version control you may also want to use a separate
84  project for them to avoid making your working copies very large when editing
85  code.</p>
86 
87  <p>Note that there is often no obvious right or wrong answer so you just have
88  to make a decision and see how it works out. You can always re-arrange your
89  repository in the future (although be aware that this will break any changes
90  being prepared on branches at the time).</p>
91
92  <p>You also need to decide whether your system requires its own repository (or
93  multiple repositories) or whether it can share with another system.</p>
94 
95  <ul>
96    <li>The main disadvantage of having separate repositories for each system is
97    the maintenance overhead (although this is almost all automated by the FCM
98    team so is not a big deal).</li>
99       
100    <li>We normally configure a single Trac system per repository. If the
101    repository contains multiple systems then it makes it difficult to use the
102    Trac milestones to handle system releases. However, Trac now supports
103    restricting itself to a sub-directory within a repository so, again, this is
104    not a big deal.</li>
105       
106    <li>If you share a repository with other systems then your revision numbers
107    can increase even when there are no changes to your system. This doesn't
108    matter but some people don't like it.</li>
109  </ul>
110 
111  <p>For simplicity, in most cases you will probably want your own repository for
112  your system.</p>
113
114  <p>You will not normally want to have multiple repositories for a system. One
115  exception may be if you are storing large data files where you might not want
116  to keep all the old versions for ever. Removing old versions can't be done
117  without changing all the revision numbers which would mess up all your code
118  history and Trac tickets. Storing the large data files in a separate
119  repository reduces the impact if you do decide to remove old versions in the
120  future. One disadvantage of this approach is that, for the moment at least,
121  Trac only handles one repository so you will need a separate Trac system for
122  the data files.</p>
123
124  <p>For further details please see the section
125  <a href="http://svnbook.red-bean.com/en/1.2/svn.reposadmin.projects.html#svn.reposadmin.projects.chooselayout">
126  Choosing a Repository Layout</a> from the Subversion book.</p>
127
128  <h3><a name="svn_create">Creating a repository</a></h3>
129
130  <p>Normally the FCM team will help you to set up your initial repository.
131  However, it is quite simple if you need to do it yourself. First you need to
132  issue the command <tt>svnadmin create /path/to/repos</tt>. This creates an
133  empty repository which is now ready to accept an initial import. To do so,
134  you should create a directory tree in a suitable location, and issue the
135  <tt>svn import</tt> command. At the top level of your directory tree should be
136  the project directories. Each project should then contain three directories
137  "trunk", "branches" and "tags". The directories "branches" and "tags" should be
138  empty.  The directory "trunk" should contain your source files in a directory
139  structure you have chosen.  For example, if your directory tree is located
140  at "$HOME/foo", you will do the following to import it to a new repository:</p>
141
142  <table class="pad" summary="creating a repository and importing a project tree" border="1" width="100%">
143    <tr>
144      <th>Creating a repository and importing a project tree</th>
145    </tr>
146
147    <tr>
148      <td>
149        <pre>
150(SHELL PROMPT)$ svnadmin create FOO_svn
151(SHELL PROMPT)$ svn import $HOME/foo file://$PWD/FOO_svn -m 'Initial import'
152Adding         FOO
153Adding         FOO/trunk
154Adding         FOO/trunk/doc
155Adding         FOO/trunk/doc/hello.html
156Adding         FOO/trunk/doc/world.html
157Adding         FOO/trunk/src
158Adding         FOO/trunk/src/bar
159Adding         FOO/trunk/src/bar/egg.f90
160Adding         FOO/trunk/src/bar/ham.f90
161Adding         FOO/branches
162Adding         FOO/tags
163
164Committed revision 1.
165</pre>
166      </td>
167    </tr>
168  </table>
169
170  <p>Note that the <tt>svnadmin</tt> command takes a PATH as an argument, as
171  opposed to a URL for the <tt>svn</tt> command.</p>
172
173  <p>For further details please see the section
174  <a href="http://svnbook.red-bean.com/en/1.2/svn.reposadmin.projects.html#svn.reposadmin.projects.import">
175  Creating the Layout, and Importing Initial Data</a> from the Subversion book.</p> 
176
177  <h3><a name="svn_access">Access control</a></h3>
178
179  <p>You will not normally want to allow anonymous write access to your repository
180  since this means that changes do not get identified with a userid. Therefore
181  system managers need to provide the FCM team with a list of users who should
182  have write access to their repositories. You may also want to arrange passwords
183  for these users although this is only necessary if you need to prevent
184  malicious access. Further restrictions such as preventing anonymous read access
185  or restricting write access to the trunk to a limited set of users can be
186  arranged if necessary.</p>
187
188  <h3><a name="svn_hosting">Repository hosting</a></h3>
189
190  <p>The FCM team will organise the hosting of your repository. A number of
191  facilities will be set up for you as standard.</p>
192 
193  <ul>
194    <li>Your repository will be set up on a central FCM server and access will be
195    provided via <em>svnserve</em> (which we use in preference to <em>Apache</em>
196    for performance reasons). The FCM team will advise you of the URL.</li>
197       
198    <li>Each night a full backup of your repository will be taken. An integrity
199    check will also be performed using the <tt>svnadmin verify</tt> command.</li>
200       
201    <li>Standard hook scripts will be set up to handle the following post-commit
202    tasks:
203
204      <ul>
205        <li>Each time a changeset is successfully committed an incremental dump
206        of the new revision is taken. This would allow the repository to be
207        recovered should the live repository become corrupted for whatever
208        reason.</li>
209           
210        <li>A file is updated which records the latest revision of your
211        repository. This can be used by system managers to regularly check for
212        new commits in a cron job and perform any required actions (updating
213        files on a remote platform for instance). The FCM team can advise you
214        of the location of this file and show you some example scripts which
215        make use of it.</li>
216      </ul>
217    </li>
218
219    <li>Standard hook scripts will be set up to handle the following tasks for
220    changes in revision properties (pre-revprop-change/post-revprop-change):
221
222      <ul>
223        <li>If a user attempts to modify the log message of a changeset and
224        he/she is not the original author of the changeset, the
225        pre-revprop-change hook script will e-mail the original author. You can
226        also set up a watch facility to monitor changes of log messages that
227        affect particular paths in the repository. (See the <a href=
228        "#svn_watch">next sub-section</a> for details.)</li>
229
230        <li>The post-revprop-change hook script updates the Trac SQLite database
231        following a successful change in the log message.</li>
232      </ul>
233    </li>
234  </ul>
235
236  <p>Additional hook scripts can be put in place if you have a requirement. The
237  use of hook scripts is discussed in the section <a href=
238  "http://svnbook.red-bean.com/en/1.2/svn.reposadmin.create.html#svn.reposadmin.create.hooks">
239  Repository Creation and Configuration</a> from the Subversion book.</p>
240
241  <p>Note that if you want to use a Subversion repository for your own individual
242  use there is no need to get the FCM team to host it. You can simply create your
243  repository and then use a <tt>file://</tt> URL to access it.</p>
244
245  <h3><a name="svn_watch">Watching changes in log messages</a></h3>
246
247  <p>You can set up a watch facility to monitor changes in revision log messages
248  in your repository. An obvious use of this facility is for system managers to
249  monitor changes of log messages affecting the trunks of their projects. To set
250  up the facility, you will need to add a <tt>watch.cfg</tt> file to the root of
251  your repository. (To avoid checking out the whole repository, including every
252  branch, make sure that you checkout the root of your repository
253  non-recursively, i.e. <tt>fcm checkout -N URL</tt>.) The <tt>watch.cfg</tt>
254  file is an INI-type file, with the basic format:</p>
255
256  <pre>
257[repos_base]
258
259path/in/repos = list,of,watchers
260</pre>
261
262  <p>For example, if your repository is <tt>svn://fcm1/FCM_svn/</tt>, and you
263  want set up particular users to monitor changes of the log messages affecting
264  several areas within the repository, you may have something like this:</p>
265
266  <pre>
267[FCM_svn]
268
269FCM/trunk/src            = fred,bob
270FCM/trunk/doc            = fred,bob,alice
271FCM/branches/dev/*/*/src = fred
272</pre>
273
274  <p>Later on, if "dave" attempts to modify the log message of a changeset that
275  affects the path <tt>FCM/trunk/src</tt>, "fred" and "bob" will both be
276  notified by e-mail.</p>
277   
278  <h2><a name="trac">Trac</a></h2>
279
280  <h3><a name="trac_config">Trac configuration</a></h3>
281
282  <p>Normally the FCM team will set up your Trac system for you (using a script
283  which helps automate the configuration). This section describes some things you
284  may wish to be configured. This can be done when the Trac system is set up or
285  later if you are unsure what you will require at first.</p>
286
287  <h4><a name="trac_access">Access control</a></h4>
288
289  <p>You will not normally want to allow anonymous users to make changes to your
290  Trac system since this means that changes may not get identified with a userid.
291  The FCM team will set up your Trac system with accounts for all users who have
292  write access to your Subversion repository. You may also want to arrange
293  passwords for these users although this is only necessary if you need to prevent
294  malicious access. Further restrictions such as preventing anonymous read access
295  can be arranged if necessary.</p>
296
297  <p>An additional admin account will be set up for the system manager which will
298  be given TRAC_ADMIN privileges. This allows this account to do additional
299  things which normal users cannot do such as:</p>
300
301  <ul>
302    <li>Delete wiki pages (particular versions or the entire page).</li>
303       
304    <li>Add or modify milestones.</li>
305       
306    <li>Modify ticket descriptions and delete ticket attachments.</li>
307       
308    <li>Make wiki pages read-only.</li>
309  </ul>
310 
311  <p>We recommend that the system manager uses a normal account most of the time
312  and only uses their admin account when they require the additional privileges.
313  </p>
314
315  <p>For further details please see the section
316  <a href="http://projects.edgewall.com/trac/wiki/TracPermissions">
317  Trac Permissions</a> from the Trac documentation.</p> 
318
319  <h4><a name="trac_email">Email notification</a></h4>
320
321  <p>By default, each Trac system is configured such that the owner and reporter
322  and anyone on the "CC" list are notified whenever a change is made to a
323  ticket. If system mangers wish to be notified of all ticket changes then this
324  can also be configured. Alternatively, email notifications can be disabled if
325  they are not wanted.</p>
326
327  <h4><a name="trac_milestones">Milestones</a></h4>
328
329  <p>Milestones are useful for identifying when tickets need to be resolved.
330  Typically, milestones may be particular system releases or time periods. The
331  FCM team can configure milestones for you when they set up your Trac system.
332  However, this is not strictly necessary since milestones can be set up via the
333  web interface using the admin account (go to the <em>Roadmap</em> page).</p>
334
335  <h4><a name="trac_misc">Other configurable items</a></h4>
336
337  <p>There are lots of other things that can be configured in your Trac system
338  such as:</p>
339 
340  <ul>
341    <li>Components</li>
342       
343    <li>Versions</li>
344       
345    <li>Custom fields</li>
346       
347    <li>System icon</li>
348       
349    <li>Stylesheets</li>
350  </ul>
351
352  <p>For further details please see the sections
353  <a href="http://projects.edgewall.com/trac/wiki/TracIni">
354  The Trac Configuration File</a> and
355  <a href="http://projects.edgewall.com/trac/wiki/TracTickets">
356  The Trac Ticket System</a> from the Trac documentation.</p> 
357 
358  <h3><a name="trac_hosting">Trac hosting</a></h3>
359
360  <p>The FCM team will organise the hosting of your Trac system. It will be set
361  up on the same server that hosts your Subversion repository and access will be
362  provided via a web server. The FCM team will advise you of the URL. Each night
363  a backup of your Trac system will be taken in case the live system should
364  become corrupted for whatever reason.</p>
365 
366  <h2><a name="fcm-keywords">FCM keywords</a></h2>
367
368  <p>When you set up a repository for a new project, you will normally want the
369  FCM team to set up a set of URL keywords for it in the FCM central
370  configuration file. The name of the project should be a short string containing
371  only word characters. If your project is called "PROJ", the recommended set of
372  keywords are:</p>
373
374  <ul>
375    <li>fcm:proj - for the top level URL of the PROJ repository.</li>
376
377    <li>fcm:proj_tr - for the "trunk" sub-directory of PROJ repository.</li>
378
379    <li>fcm:proj_br - for the "branches" sub-directory of PROJ repository.</li>
380
381    <li>fcm:proj_tg - for the "tags" sub-directory of PROJ repository.</li>
382  </ul>
383
384  <p>If your project has an associated Trac browser, you should also declare the
385  Trac browser URL (that corresponds to the Subversion URL) in the central
386  configuration file. This allows FCM to associate the URL to the Trac browser
387  with a Subversion URL.</p>
388
389  <p>For more information on how to set up the keywords, please refer to <a
390  href="code_management.html#svn_basic_keywords">Repository &amp; Revision
391  Keywords</a> and the <a href="annex_fcm_cfg.html">Annex: Declarations in FCM
392  central/user configuration file</a>.</p>
393
394  <h2><a name="ext-bld-cfg">Extract and build configuration</a></h2>
395
396  <p>The extract and build systems are very flexibile and can be used in lots of
397  different ways. It is therefore difficult to give specific advice explaining
398  how to configure them. However, based on experience with a number of systems,
399  the following general advice can be offered.</p>
400
401  <ul>
402    <li>Standard extract configuration files should be defined and stored within
403    the repository. Users then include these files into their configurations,
404    before applying their local changes.</li>
405    <li>The files should be designed to include one another in a hierarchy. For
406    example, you may have one core file which defines all the repository and
407    source locations plus a series of platform/compiler specific files which
408    include the core file. More complex setups are also possible if you need to
409    cater for other options such as different optimisation levels, 32/64 bit,
410    etc.</li>
411    <li>When including other configuration files, always make use of the special
412    <tt>$HERE</tt> variable
413    (rather than, for instance, referring to a fixed repository location). When
414    your configuration file is parsed, this special variable is normally
415    expanded into the container directory of the current configuration file. This
416    means that the include statements should work correctly whether you are
417    referring to configuration files in the repository trunk, in a branch or in
418    a local working copy.</li>
419    <li>Make good use of user variables (e.g. <tt>%fred</tt>) to simplify
420    repetitive declarations and make your configuration files easier to
421    maintain.</li>
422    <li>Use continuation lines to split long lines and make them easier to read
423    (see the <a href="command_ref.html#fcm_config">FCM Command Reference</a>
424    section for further details about configuration files).</li>
425  </ul>
426
427  <p>Probably the best advice is to look at what has already been set up for
428  other systems. The FCM team can advise on the best systems to examine.</p>
429
430  <p>When you create a stable build you should keep an extract configuration file
431  that can reproduce the build. One easy way to do this is to create your build
432  using the standard configuration files and the latest versions of the code. You
433  can then save the expanded extract configuration file which is created when you
434  run the extraction. To help document your stable build you can use the command
435  <tt>fcm cmp-ext-cfg</tt> to show what has changed.</p>
436
437  <h2><a name="alternate_versions">Maintaining alternate versions of namelists
438  and data files</a></h2>
439
440  <p>Sometimes it is useful to be able to access particular revisions of some
441  directories from a FCM repository without having to go via Subversion. Typical
442  examples are namelist or data files used as inputs to a program. The script
443  <tt>fcm_update_version_dir.pl</tt> is designed to help with this. It can be
444  used to maintain a set of extracted version directories from a FCM repository.
445  The script has the following options and arguments:</p>
446
447  <ul>
448    <li>-f [--full]: specify the full mode, in which the versioned directories
449    of each specified item will be removed before being re-extracted.</li>
450
451    <li>-d [--dest] &lt;arg&gt;: specify a destination &lt;arg&gt; for the
452    extraction. If not specified, the current working directory will be used as
453    the base path.</li>
454
455    <li>-u [--url] &lt;arg&gt;: specify the source repository URL. This option
456    is compulsory.</li>
457  </ul>
458
459  <p>If an argument is specified, it must be the location of a configuration
460  file. Otherwise, the command reads its configuration from the standard input.
461  The configuration file is a line-based text file. Blank lines and lines
462  beginning with a "#" are ignored.</p>
463
464  <p>Each configuration line must contain the relative path of a sub-directory
465  under the specified source repository URL. If the path ends in "<tt>*</tt>"
466  then the path is expanded recursively and any sub-directories containing
467  regular files are added to the list of relative paths to extract.</p>
468
469  <p>Optionally, each relative path may be followed by a list of space separated
470  "conditions". Each condition is a conditional operator (&gt;, &gt;=, &lt;,
471  &lt;=, == or !=) followed by a revision number or the keyword HEAD.</p>
472
473  <p>The command uses the revision log to determine the revisions at which the
474  relative path has been updated in the source repository URL. If these
475  revisions also satisfy the "conditions" set by the user, they will be
476  considered in the extraction. In full mode, everything is re-extracted. In
477  incremental mode, the version directories are only updated if they do not
478  already exist.</p>
479
480  <p>Example:</p>
481
482  <table class="pad" summary="fcm_update_version_dir.pl example" border="1"
483  width="100%">
484    <tr>
485      <th>fcm_update_version_dir.pl example</th>
486    </tr>
487
488    <tr>
489      <td>
490        <pre>
491(SHELL PROMPT)$ fcm_update_version_dir.pl -u fcm:ver_tr &lt;&lt;EOF
492namelists/VerNL_AreaDefinition   &gt;1000 !=1234
493namelists/VerNL_GRIBToPPCode     &gt;=600 &lt;3000
494namelists/VerNL_StationList     
495elements/*                       &gt;1000
496EOF
497</pre>
498      </td>
499    </tr>
500  </table>
501
502  <p>N.B.</p>
503
504  <ol>
505    <li>Each time a sub-directory is revised, the script assigns a sequential
506    "v" number for the item. Each "v" number for a sub-directory, therefore, is
507    associated with a revision number. For each extracted revision directory,
508    there is a corresponding "v" number symbolic link pointing to it.</li>
509
510    <li>The system also creates a symbolic link "latest" to point to the latest
511    extracted revision directory.</li>
512  </ol>
513
514  <h2><a name="work-practise">Defining working practises and policies</a></h2>
515
516  <p>Some options on working practises and policies are defined in the chapter
517  on <a href="working_practices.html">Code Management Working Practices</a>.
518  Individual projects should document the approach they have adopted. In
519  addition, each project may also need to define its own working practices and
520  policies to suit its local need. For example each project may need to
521  specify:</p>
522
523  <ul>
524    <li>Whether changes are allowed directly on the trunk or whether branches
525    have to be used in all cases.</li>
526
527    <li>Whether all users are allowed to make changes to the trunk.</li>
528
529    <li>Whether Trac tickets have to be raised for all changes to the trunk.</li>
530       
531    <li>Whether Trac tickets should be raised for all support queries or whether
532    a Trac ticket should only be raised once there is an agreed "issue".</li>
533
534    <li>Whether branches should normally be made from the latest code or from a
535    stable release.</li>
536
537    <li>Whether a user is allowed to resolve conflicts directly when merging a
538    branch into the trunk or whether he/she should merge the trunk into the
539    branch and resolve the conflicts in the branch first.</li>
540
541    <li>Whether all code changes to the trunk need to be reviewed.</li>
542
543    <li>What testing is required before changes can be merged to the trunk.</li>
544
545    <li>Whether history entries are maintained in source files or whether
546    individual source files changes need to be described in the Subversion log
547    message.</li>
548
549    <li>Branch deletion policy.</li>
550
551    <li>Whether any files in the project require locking before being changed.
552    </li>
553  </ul>
554
555  <script type="text/javascript" src="maintain.js">
556  </script>
557</body>
558</html>
Note: See TracBrowser for help on using the repository browser.