 be extended, as follows, to allow run-time linkage.
This implementation has been compiled and tested on the computer system detailed in Section 2.1 of the Installer's Note.
To run a native executable first you should log into the coprocessor you want to use. For example, ssh mic0 will log you into the first device in a multicard system, provided that you have been given an account on that device.
Note that the settings below assume that [INSTALL_DIR] is in a location that is mounted on the device being used. If this is not the case then it is also possible to transfer the libraries being linked to using the scp command, e.g. scp [INSTALL_DIR]/lib/mic/libnagsmp.so mic0:naglibs would transfer the shared library into a directory called naglibs in the user's home directory on the device. In this case LD_LIBRARY_PATH would need to point to ~naglibs instead. Note that scp may be used similarly to transfer your executable to the device if it is in a directory that is not mounted on the device. See http://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-developers-quick-start-guide for more details about using your Xeon Phi coprocessor with native executables.
to set the LD_LIBRARY_PATH when using an executable linked to the static NAG library.
Note that the above LD_LIBRARY_PATH settings will load the native Intel compiler run-time libraries and MKL libraries that are supplied by NAG. It should also be possible to set LD_LIBRARY_PATH to point to the locations of the corresponding libraries in your Intel compiler and MKL installation, assuming that its location is also mounted on the coprocessor. Once this environment variable has been set it is possible to set the number of OpenMP threads to use (see below) and run the native executable.
The example scripts demonstrate all of the different ways to link to the NAG library discussed in this section.
Note that the example scripts use the compiler's default optimization level (i.e. no -On flag is supplied). This is the case for all examples except the following examples as built for native execution: c05ndfe c05pdae c05pdfe c05qdfe c05rdfe. These examples require compilation at optimization level 1 (i.e. -O1) in order to avoid a compiler bug.
For heterogeneous executables by default environment variables set on the host are copied across to the Xeon Phi coprocessor during an offload. This is typically not what is required if e.g. OMP_NUM_THREADS is being used to control the number of OpenMP threads, since the number required on the Xeon Phi coprocessor will likely be much greater than needed on the host system. Thus, users should also set MIC_ENV_PREFIX=MIC to enable a separate environment variable, MIC_OMP_NUM_THREADS.
where N1 is the number of threads required on the host and N2 is the number of threads required on the Xeon Phi coprocessor. OMP_NUM_THREADS and MIC_OMP_NUM_THREADS may be re-set between each execution of the program, as desired.
The difference between scatter and balanced is what happens when more threads are being used than there are physical cores. In the case of scatter neighbouring threads do not share the same core; threads are assigned to cores in a round-robin manner. For balanced, the threads allocated to the same core are neighbours of one another, which may help with cache utilization in some cases. Note that the balanced option is available on the MIC architecture only.
where N is the number of threads required.
The recommended maximum number of threads for the host processor is equal to the number of logical cores, which is usually twice the number of physical cores. For the Xeon Phi coprocessor the recommended maximum number of threads for heterogeneous executables is 4*(number of physical cores - 1). This is because the MIC architecture is designed with hardware to support up to 4 threads per core. The -1 is because one core is used by the Operating System on the Xeon Phi coprocessor to handle offloads. For native executables this restriction does not apply, so the number of threads should not be greater than 4*(number of physical cores). Note that, on both the host and Xeon Phi coprocessor, the best performance may be achieved using fewer than the recommended maximum number of threads. Users are encouraged to experiment with different numbers of threads.
The number of physical cores in both the host and Xeon Phi coprocessor varies depending on the particular hardware being used. Please contact NAG if you would like help determining the specific details of your system.
The performance of offload regions can sometimes be improved when arrays are allocated using 2MB rather than the default 4KB pages on the Xeon Phi coprocessor. The compiler offload run-time selects to allocate using 2MB pages heap variables whose size is greater than the value of the environment variable MIC_USE_2MB_BUFFERS. For example, if MIC_USE_2MB_BUFFERS=64K arrays greater than 64KB in size will use 2MB pages. Users are advised to experiment with this setting.
Performance can also be improved by aligning arrays on 64 byte address boundaries. In Fortran this can be achieved using the -align array64byte compiler flag. It is recommended to always use this flag.
NAG_MIC_TARGET: this is a default-kind integer variable used to indicate which Xeon Phi coprocessor will be used for offloads within the library in a multi-device system. Its default value is 0.
NAG_HOST_ONLY: this is a default-kind integer variable used to indicate that offloading within the library should be completely disabled. Its default value is 0, which corresponds to false. Any value other than zero corresponds to true.
Routine-specific switches to control internal offloading. These are default-kind integer variables whose value determines what happens in routines with offload regions. If the value of the switch is 0 the routine never offloads. If the switch is 1 the routine always offloads. If the switch has any other value the decision whether or not to offload is based on problem size. The names of these switches are provided in Section 8 of the documentation for routines which are capable of offloading internally.
Users are encouraged to USE nag_performance_parameters and experiment with routine-specific switches to find out whether or not offloading is profitable.
In this implementation the decision whether or not to offload is based on problem size thresholds determined by NAG using a 61-core Xeon Phi coprocessor code-named "Knights Corner" with 8GB memory and a clock speed of 1.1GHz, attached to a host system with a single socket 8-core Xeon processor code-named "Sandy Bridge" with 20MB cache and a clock speed of 3.3GHz.
More information about nag_performance_parameters and the conditions for offloads to occur internally within NAG routine calls is given in "Introduction to the NAG Library for the Xeon Phi Coprocessor".
nagsmp_example shows how to link to the static NAG library for heterogeneous execution. LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH should include the settings made by sourcing the Intel environment setup scripts.
nagsmp_example_shar shows how to link to the shareable NAG library for heterogeneous execution. LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH should include the settings made by sourcing the Intel environment setup scripts.
nagsmp_example_native shows how to link to the static NAG library for native execution and how to run the native executable on the device. LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH need not be set when running this script. This script requires the installation directory to be mounted on the targeted device.
nagsmp_example_shar_native shows how to link to the shareable NAG library for native execution and how to run the native executable on the device. LD_LIBRARY_PATH and MIC_LD_LIBRARY_PATH need not be set when running this script. This script requires the installation directory to be mounted on the targeted device.
The example program concerned, the number of OpenMP threads to use on the host, and the number of OpenMP threads to use on the Xeon Phi coprocessor are specified by the arguments to the command, e.g.
will copy the example program and its data and options files (e04fcfe.f90, e04fcfe.d and e04fcfe.opt) into the current directory, compile the program and run it using 4 OpenMP threads on the host and 180 threads within any regions offloaded to the Xeon Phi coprocessor to produce the example program results in the file e04fcfe.r. The scripts for native execution additionally require the username and device name to use when running the example, e.g.
will copy the example program and its data and options files (e04fcfe.f90, e04fcfe.d and e04fcfe.opt) into the current directory, compile the program and run it using 120 OpenMP threads on the device mic0 as user micusr to produce the example program results in the file e04fcfe.r