Before running the SEA model, it is important to correctly describe the architecture being used, so that the correct compilation is performed. To do this, changes to the Makefile may be required: notably the ARCH and FORT makefile variables are used to specify CPP directives and compiler command line options, respectively.
The FORT variable allows various (machine-dependent) FORTRAN compiler command line options to be specified. The ARCH variable supports permutations of the following CPP directives:
ARCH = -Dparallel -Dno_spawn -Dmpi -Dcray -Dfs_opt -Dreal4
To run the model on a parallel platform, specify the -Dparallel option. This ensures that the high-level message passing routines will be compiled. (It will also be necessary to specify the location of the message passing header files and libraries, by specifying the MP_FLGS variable). If -Dparallel is not specified, then it is assumed that the model is to be run in sequential mode on a single processor.
By default, if SEA is to run in parallel mode, it will be assumed that PVM is to be used as the message passing environment. To opt for an MPI environment instead, the -Dmpi option should be set.
An incompatibility exists when using PVM on some parallel platforms. In particular, when using PVM on workstation clusters, a single process is started and then a (transparent!) start up procedure is invoked, to spawn the required number of other processes across the available processors. This procedure is not necessary on some MPP architectures, such as the Cray T3D which enforces a SPMD (Single Program Multiple Data) model, spawning ALL processes at once from the command line. To skip this PVM start-up phase on such architectures, specify the -Dno_spawn directive. Alternatively, use MPI, as this tends to be the least platform-dependent.
When running on the Cray T3D/T3E the -Dcray directive will cater for minor anomalies (mostly with respect to timing functions and I/O in the pre/post processing utilities). As yet no other MPP's have been considered in this way.
On some parallel platforms, the processor interconnect may not be particularly fast (especially true on workstation clusters), in which case the communications associated with the free surface calculation can become a substantial bottleneck. If this is likely, it can be optimised (by trading some of the communication for additional computation), by selecting the -Dfs_opt option.
The numerical precision of a machine is important to SEA for two reasons. In the case of inter-processor communications (when running in parallel) the message passing system needs to account for the size of the variables when generating messages. Similarly, when archiving data, the size of the variables is important to the HDF routines for storing. By default double precision (real*8) is assumed. To run using single precision (real*4), set the -Dreal4 option. (It will also be necessary to check the FORT makefile variable, to determine whether the -r8 or -r4 compiler options are required for the f77 command.
During a run, the SEA model will make use of direct access files. In doing so, it assumes that record lengths are specified in bytes. Some machines however may use words, by default. DEC Alphas, for example, have this default, which can be changed by the compiler option -assume byterecl, to force bytes to be used, as desired, and should therefore be specified in the FORT variable.
Two examples:
1. On a cluster of DEC workstations, which have PVM installed, the SEA model is to be run in parallel mode, with optimised free surface communications, using single precision. The ARCH and FORT parameters would be set as follows:
ARCH = -Dparallel -Dfs_opt -Dreal4 FORT = f77 -O -r4 -assume byterecl2. On a cluster of SUN workstations, which have MPI installed, the SEA model is to be run in parallel mode using double precision. The ARCH and FORT parameters would be set as follows:
ARCH = -Dparallel -Dmpi FORT = f77 -O -r8
It is the DOMAIN variable, within the Makefile, that needs to be amended at this stage, and two options may be specified.
The -Dsimple_world option prompts the model to automatically generate an idealised topography for a pseudo global model that extends from 70°S to 70°N (although these extents could be changed in include/domain.h). The topography generated will be dependent on the required horizontal resolution and the number of vertical levels (specified in include/domain.h).
If the -Dsimple_world option is not selected then the model will expect a topography file to exist (data/topog_kmt), describing the depth level to which the tracer-grid extends from each surface grid point. This should be in the expected file format, to be read in by utils/config.F.
The -Dcyclic indicates that cyclic periodic boundary conditions should be applied to the latitudinal boundaries. This is typically used for global models (such as the simple_world), but may also be required in case studies of some regional models. If it is not specified, the model will assume these boundaries to be closed. (Note: as yet, due to complications with the free surface, open boundaries are not considered in the SEA model).
Specify domain dimensions (edit include/domain.h)
Within this header file, the domain is described in terms of its starting location, its resolution and its overall size, as well as two additional parameters indicating the number of tracers and the maximum number of processors likely to be used. These are given by:
STLON longitude reference (in degrees) of first uv point STLAT latitude reference (in degrees) of first uv point DXDEG longitudinal grid resolution (in degrees) DYDEG latitudinal grid resolution (in degrees) IALL number of longitudinal grid boxes in domain JALL number of latitudinal grid boxes in domain KM number of vertical depth levels NT number of tracers MAXP maximum number of processors
The SEA model uses an Arakawa B Grid (Mesinger & Arakawa, 1976) representation for the model grid, which offsets the tracer values (T) from the velocities (uv). As a guide, Figure 1 illustrates how the six horizontal grid parameters correspond to the domain that is being modelled.
Specify vertical depth levels (edit include/levels.h)
The depth levels within the model may vary in number and size, and the thickness of each level is defined in the header file include/levels.h. The thickness (in centimetres) of each level is given by dz(k), which is initialised using a FORTRAN data statement. For example, a 15 level model might have the following thicknesses set:
data dz / & 30.00e2, 46.15e2, 68.93e2, 99.93e2, & 140.63e2, 192.11e2, 254.76e2, 327.95e2, & 409.81e2, 497.11e2, 585.36e2, 669.09e2, & 742.41e2, 799.65e2, 836.10e2 /
To generate the appropriate density coefficients, from the given level thicknesses, a routine (originally used in the MOM code) is provided and invoked by make dncoef. This will calculate the 9 coefficients of a third order polynomial approximation to the equation of state for sea water. The program yields coefficients that will compute density as a function of temperature and salinity, at pre-determined depths (taken from dz in include/levels.h and km in include/domain.h). More specifically, the densities calculated from the polynomial formula are in the form of sigma anomalies. The method is taken from that described by Bryan & Cox (1972). The resultant file is include/dncoef.h, to be used in the compilation of the main SEA executable.
As examples, three pairs of files are provided in the include directory, for 15, 24 and 32 depth level fields (called levels.15, dncoef.15, etc.). These may be used instead of creating your own. Depending on the number of levels that are required, the desired pair should be copied to their respective ".h" files. For example, if it is desired to run the model with 24 depth levels, then the following should be typed
cp include/levels.24 include/levels.h cp include/dncoef.24 include/dncoef.h
Configuration summary (include/gridres.h)
The file include/gridres.h is created. This declares a number of parameters, which are then used to dimension various arrays when the main SEA program is compiled:
EWSG number of sub-grids in the east-west direction NSSG number of sub-grids in the north-south direction IMAX maximum longitudinal grid points in any sub-domain JMAX maximum latitudinal grid points in any sub-domain BMAX maximum points in longest sub-domain boundary (*2)
In sequential mode there is just one sub-grid (the whole domain) and so the values are automatically set as follows:
EWSG = 1 NSSG = 1 IMAX = IALL JMAX = JALL BMAX = MAX(IMAX,JMAX) * 2
In parallel mode, the user is prompted, by the config program, to enter values for EWSG and NSSG). The maximum extents (IMAX, JMAX and BMAX) are then calculated accordingly.
If, in parallel mode, either EWSG or NSSG is set to 1 (but not both) then the option to load balance the sub-domains is offered. If selected, the extents of the sub-domains will be varied in an attempt to allocate equal numbers of sea points to each sub-domain, rather than simply an equal number of grid points (which can be either land or sea). This should result in a better parallel efficiency, but care must be taken to avoid memory problems (as array sizes will invariably rise).
Processor map (map.in)
So that each processor knows where its sub-domain is located within the whole domain, a file map.in is generated, which details the total number of processors and the relative offsets and extents of each and every sub-domain:
IOFF = grid point offset in the longitude direction JOFF = grid point offset in the latitude direction IMT = number of grid points in longitude direction JMT = number of grid points in latitude direction
The need for IMT and JMT, as well as IMAX and JMAX, is due to the fact that a domain decomposition may not naturally provide truly regular sub-domains (especially if one-dimensional load balancing has been selected). The model domain is divided up so that the arrays for each sub-domain are declared using the IMAX and JMAX constants, but processing only occurs over the sub-domain (IMT, JMT), such that IMT<=IMAX and JMT<=JMAX, as illustrated in figure 2, which results in IMAX=7 and JMAX=6 (note also that all extent values are +2, to allow for the outer boundary halo).
Topography Sub-files (data/model/topog.#)
From the supplied (or auto-generated) topography for the whole domain, the config program will generate a number of direct-access sub-files in the data/model directory, in a form ready to be read in by the main SEA executable. The number of sub-files generated will depend on the number of processors being used, and each will be suffixed with an appropriate process number, topog.# (where # = 0,NumProcs-1).
As well as the compile-time parameters that have been set up in previous sections, a run-time parameter file ocean.in is read in by the sea model at the start of a run. The Fortran NAMELIST command is used to read in the parameters from this file in the subroutine init_sea in the src/init.F module. (If your compiler does not support NAMELIST, then this part of the code may need modifying). Currently the parameters only apply to the base model. To add further run-time parameters for optional modules, it is recommended that these be added to the misc structure and the code changed accordingly in src/init.F.
An example ocean.in file:
&contrl
init = .true., restrt = .true.,
first_day = 0.0, last_day = 1.0,
archs = 0.0, snaps = 0.0,
tsi = 0.2, dpth_lev = 1
&end
&tsteps
dtts = 3600.0, dtuv = 3600.0,
dtbt = 72.0, nmix = 24,
eb = .true.
&end
&eddy
am = 2.0e8, fkpm = 1.0,
ah = 1.0e7, fkph = 0.5
&end
&misc
acor = 0.5, cdbot = 0.001,
aidif = 0.0, ionum = 0
&end
Most of the parameters are detailed, in context, in other sections, when describing various model aspects. Here, only a brief description of each parameter is given:
| init | start from an initial state (.true.) or from a restart file (.false.) |
| restrt | save a restart file at end of run (.true.) |
| first_day | the day number to start the model run from |
| last_day | the day number to run the model to |
| archs | frequency (days) to write out a restart file |
| snaps | frequency (days) to write out a snapshot file |
| tsi | frequency (days) to write out a diagnostic line |
| dpth_lev | depth level of horizontal slice of 3D data for snapshots |
| dtts | time interval (secs) of tracer timestep |
| dtuv | time interval (secs) of baroclinic timestep |
| dtbt | time interval (secs) of barotropic timestep |
| nmix | frequency of mixing timesteps |
| eb | use euler backward (.true) or forward (.false.) mixing timestep |
| am | lateral eddy viscosity (cm2/sec) |
| ah | lateral eddy diffusivity (cm2/sec) |
| fkpm | vertical viscosity coefficient (cm**2/sec) |
| fkph | vertical diffusion coefficient (cm2/sec) |
| acor | treatment of coriolis
(0.0=explicit, 1.0=implicit, >0/<1=bit of each) |
| cdbot | bottom drag coefficient |
| aidif | treatment of vertical diffusion
(0.0=explicit, 1.0=implicit, >0/<1=bit of each) |
| ionum | number of tokens to use to synchronise parallel I/O |