IBM::LoadLeveler is a Perl interface to the IBM LoadLeveler work load management system. Documentation on the system can be found at http://publib.boulder.ibm.com/infocenter/clresctr/topic/com.ibm.cluster.loadl.doc/llbooks.html. The module is not for the faint hearted, the Data Access API can return several hundered parameters, I strongly advise you print out the IBM documentation, especially the very useful “LoadLeveler Job Object Model” diagram.
A simple example of using this module is shown below, it uses the Data Access API to get the submit time of all jobs in the system.:
use IBM::LoadLeveler; # Query Job information $query = ll_query(JOBS); # Ask for all data on all jobs $return=ll_set_request($query,QUERY_ALL,undef,ALL_DATA); if ($return != 0 ) { print STDERR "ll_set_request failed Return = $return\n"; } # Query the scheduler for information # $number will contain the number of objects returned $job=ll_get_objs($query,LL_CM,NULL,$number,$err); while ( $job) { # Get the Job submit time $SubmitTime=ll_get_data($job,LL_JobSubmitTime); $job=ll_next_obj($query); } # Free up space allocated by LoadLeveler to hold the job object ll_free_objs($job); # Free up space used by the Query Object ll_deallocate($query);
The module currently supports the following APIs:
This version has been “tested” with under quite a few versions of LoadLeveler from 3.1.0.16 to 3.5.1.2 under AIX, Linux and pLinux also thanks to SDSC it has been tested on Blue Gene. The coverage of the test suite however is not very good.
Since LoadLeveler does not change the version number in the llapi.h file a crude hack is used to work out what version is installed and only include the appropriate functions and enum values. If your compile starts spitting out errors like:
"LoadLeveler.xs", line 3459.22: 1506-045 (S) Undeclared identifier WCLIMIT_ADD_MIN. "LoadLeveler.xs", line 3459.22: 1506-051 (S) Case expression must be a valid integral constant.
Then the hack has probably not worked and you will either need to either fix the Makefile.PL file or hardcode a value for the LLVER variable:
3.1 : 3010000 3.2 : 3020000
The parameters to ll_get_data expand with most revisions, I believe this list is reasonably accurate, but there are problems with it. IBM does at times back port functionality, so some parameters seem to bounce in and out of the llapi.h at random. LL_StepDependency is a good example, it is in versions 3.1.0.31, 3.1.0.32, 3.2.0.16, 3.2.0.17, 3.2.0.18 & 3.3.0.3+.
The module relies on the llapi.h file supplied with LoadLeveler for definitions of constants. The make file automatically processes the llapi.h file into a llapi.ph file and installs it as part of the build process.
You might need to edit Makefile.PL to change the value of $LoadL to point to where LoadLeveler is installed
Standard build/installation supported by ExtUtils::MakeMaker(3)…
perl Makefile.PL make make test make install
To convert the pod documentation (what there is of it) to html:
make html
The module ships with a defs.h.in file which it process to get a defs.h file which contains all the Data Access API definitions. The llapi.h file on your system will be different from the ones used to generate the definitions file, the processing will remove definitions not found in the system llapi.h file and suggest adding new ones it has found. This results in output like:
> perl Makefile.PL Building for version 3040101 on Linux Building list of current ll_get_data specifications Generating new definitions array defs.h.in has the following definitions not seen in your system llapi.h file. They have been removed from defs.h: [LL_AdapterMinWindowSize] = LL_INT_STAR, [LL_MachineCpuList] = LL_INT_STAR, [LL_ColumnStepNames] = LL_CHAR_STAR_STAR_STAR, [LL_MatrixTimeSlice] = LL_INT_STAR, [LL_ClassNqsQuery] = LL_CHAR_STAR_STAR_STAR, [LL_MatrixColumnCount] = LL_INT_STAR, [LL_ColumnMachineName] = LL_CHAR_STAR_STAR, [LL_AdapterMaxWindowSize] = LL_INT_STAR, [LL_ClassExecutionFactor] = LL_INT_STAR, [LL_AdapterMemory] = LL_INT_STAR, [LL_StepExecutionFactor] = LL_INT_STAR, [LL_AdapterUsageWindowMemory] = LL_INT_STAR, [LL_ColumnProcessorNumber] = LL_INT_STAR, [LL_ClassNqsSubmit] = LL_CHAR_STAR_STAR, [LL_MatrixGetNextColumn] = LL_LL_ELEMENT_STAR, [LL_ColumnRowCount] = LL_INT_STAR, [LL_MatrixRowCount] = LL_INT_STAR, [LL_MatrixGetFirstColumn] = LL_LL_ELEMENT_STAR, [LL_ClassNqsClass] = LL_INT_STAR, Writing Makefile for IBM::LoadLeveler
The libllapi distributed for Linux on PowerPC architectures is 64 bit, typically the perl distributed is 32 bits. This isn't good, to use this module you will need to build a 64 bit version of perl. This has worked for me:
sh Configure -Accflags='-q64' -Dcc='xlc' \ -Aldflags='-q64' -Alddlflags='-G -q64' \ -Adefine:libs='-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc' \ -Doptimize='-O2' -des make make install PERLNAME=perl64
The libs definition was simply to remove the 32 bit -ldb that was discovered but couldn't be used. Several of the other libraries are not essential, if you don't have them don't include them, Perl just won't build modules that need them.
If you want or need to build it somewhere other than the default then add -Dprefix=/path to the Configure line.
ll_get_data has a whole set of 64 bit return types if you are in 32 bit mode then LoadLeveler will truncate them to 32 bits.
See APAR IY48329
This does not return some values, eg LL_StepState always returns 0. If your scheduler type is GANG then you don't get any results at all.
This module has been observed to crash when given a history file of >92MB and <132MB ( the killer value is probably 128MB ).
Workaround
The solution is to increase the bmaxdata value of the Perl executable. If you are using the installp version of perl it is recommended to copy the executable to another directory, and modify using ldedit to increase the number of data segments.
cp /usr/opt/perl/bin/perl /global/bin/llperl /usr/bin/ldedit -o bmaxdata:0x20000000 /global/bin/llperl
Then modify any scripts that have exhibited this behaviour to use the new executable. If this fails then increase the bmaxdata value until successful.
I would like to thank:
Mike Hawkins mike@pink-pit.com