IBM::LoadLeveler - Perl access to IBM LoadLeveler Workload Management System

Introduction

IBM::LoadLeveler is a Perl interface to the IBM LoadLeveler work load management system. Documentation on the system can be found at http://publib.boulder.ibm.com/infocenter/clresctr/topic/com.ibm.cluster.loadl.doc/llbooks.html. The module is not for the faint hearted, the Data Access API can return several hundered parameters, I strongly advise you print out the IBM documentation, especially the very useful “LoadLeveler Job Object Model” diagram.

A simple example of using this module is shown below, it uses the Data Access API to get the submit time of all jobs in the system.:

use IBM::LoadLeveler;
 
# Query Job information
$query = ll_query(JOBS);
 
# Ask for all data on all jobs
$return=ll_set_request($query,QUERY_ALL,undef,ALL_DATA);
if ($return != 0 )
{
    print STDERR "ll_set_request failed Return = $return\n";
}
 
# Query the scheduler for information
# $number will contain the number of objects returned
 
$job=ll_get_objs($query,LL_CM,NULL,$number,$err);
 
while ( $job)
{
	# Get the Job submit time
	$SubmitTime=ll_get_data($job,LL_JobSubmitTime);
	$job=ll_next_obj($query);
}
# Free up space allocated by LoadLeveler to hold the job object
ll_free_objs($job);
 
# Free up space used by the Query Object
ll_deallocate($query);

The module currently supports the following APIs:

LoadLeveler Versions

This version has been “tested” with under quite a few versions of LoadLeveler from 3.1.0.16 to 3.5.1.2 under AIX, Linux and pLinux also thanks to SDSC it has been tested on Blue Gene. The coverage of the test suite however is not very good.

Since LoadLeveler does not change the version number in the llapi.h file a crude hack is used to work out what version is installed and only include the appropriate functions and enum values. If your compile starts spitting out errors like:

"LoadLeveler.xs", line 3459.22: 1506-045 (S) Undeclared identifier WCLIMIT_ADD_MIN.
"LoadLeveler.xs", line 3459.22: 1506-051 (S) Case expression must be a valid integral constant.

Then the hack has probably not worked and you will either need to either fix the Makefile.PL file or hardcode a value for the LLVER variable:

3.1 : 3010000
3.2 : 3020000

The parameters to ll_get_data expand with most revisions, I believe this list is reasonably accurate, but there are problems with it. IBM does at times back port functionality, so some parameters seem to bounce in and out of the llapi.h at random. LL_StepDependency is a good example, it is in versions 3.1.0.31, 3.1.0.32, 3.2.0.16, 3.2.0.17, 3.2.0.18 & 3.3.0.3+.

Build/Installation

The module relies on the llapi.h file supplied with LoadLeveler for definitions of constants. The make file automatically processes the llapi.h file into a llapi.ph file and installs it as part of the build process.

You might need to edit Makefile.PL to change the value of $LoadL to point to where LoadLeveler is installed

Standard build/installation supported by ExtUtils::MakeMaker(3)…

perl Makefile.PL
make
make test
make install

To convert the pod documentation (what there is of it) to html:

make html

The module ships with a defs.h.in file which it process to get a defs.h file which contains all the Data Access API definitions. The llapi.h file on your system will be different from the ones used to generate the definitions file, the processing will remove definitions not found in the system llapi.h file and suggest adding new ones it has found. This results in output like:

> perl Makefile.PL
Building for version 3040101 on Linux
 
Building list of current ll_get_data specifications
Generating new definitions array
defs.h.in has the following definitions not seen in your system llapi.h file.  They have been removed from defs.h:
 
[LL_AdapterMinWindowSize] = LL_INT_STAR,
[LL_MachineCpuList] = LL_INT_STAR,
[LL_ColumnStepNames] = LL_CHAR_STAR_STAR_STAR,
[LL_MatrixTimeSlice] = LL_INT_STAR,
[LL_ClassNqsQuery] = LL_CHAR_STAR_STAR_STAR,
[LL_MatrixColumnCount] = LL_INT_STAR,
[LL_ColumnMachineName] = LL_CHAR_STAR_STAR,
[LL_AdapterMaxWindowSize] = LL_INT_STAR,
[LL_ClassExecutionFactor] = LL_INT_STAR,
[LL_AdapterMemory] = LL_INT_STAR,
[LL_StepExecutionFactor] = LL_INT_STAR,
[LL_AdapterUsageWindowMemory] = LL_INT_STAR,
[LL_ColumnProcessorNumber] = LL_INT_STAR,
[LL_ClassNqsSubmit] = LL_CHAR_STAR_STAR,
[LL_MatrixGetNextColumn] = LL_LL_ELEMENT_STAR,
[LL_ColumnRowCount] = LL_INT_STAR,
[LL_MatrixRowCount] = LL_INT_STAR,
[LL_MatrixGetFirstColumn] = LL_LL_ELEMENT_STAR,
[LL_ClassNqsClass] = LL_INT_STAR,
 
Writing Makefile for IBM::LoadLeveler

Building for Linux on PowerPC & Blue Gene

The libllapi distributed for Linux on PowerPC architectures is 64 bit, typically the perl distributed is 32 bits. This isn't good, to use this module you will need to build a 64 bit version of perl. This has worked for me:

sh Configure -Accflags='-q64' -Dcc='xlc' \ 
        -Aldflags='-q64' -Alddlflags='-G -q64' \
        -Adefine:libs='-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lc' \
        -Doptimize='-O2' -des
make
make install PERLNAME=perl64

The libs definition was simply to remove the 32 bit -ldb that was discovered but couldn't be used. Several of the other libraries are not essential, if you don't have them don't include them, Perl just won't build modules that need them.

If you want or need to build it somewhere other than the default then add -Dprefix=/path to the Configure line.

Known Problems

64 bit types and 32 bit perl

ll_get_data has a whole set of 64 bit return types if you are in 32 bit mode then LoadLeveler will truncate them to 32 bits.

QUERY_PROCID

See APAR IY48329

This does not return some values, eg LL_StepState always returns 0. If your scheduler type is GANG then you don't get any results at all.

Large History files

This module has been observed to crash when given a history file of >92MB and <132MB ( the killer value is probably 128MB ).

Workaround

The solution is to increase the bmaxdata value of the Perl executable. If you are using the installp version of perl it is recommended to copy the executable to another directory, and modify using ldedit to increase the number of data segments.

cp /usr/opt/perl/bin/perl /global/bin/llperl
/usr/bin/ldedit -o bmaxdata:0x20000000 /global/bin/llperl

Then modify any scripts that have exhibited this behaviour to use the new executable. If this fails then increase the bmaxdata value until successful.

External Links

  • The latest version of the module is 1.09 and can be downloaded from CPAN

Credits

I would like to thank:

  • Mark “Red Dwarf” Roberts for his extensive assistance in making this module and for the large number of example programs he has provided.
  • Patricia Kovatch at San Diego Supercomputer Center for an account on their Blue Gene system
  • and finally the long suffering Waiman Chan and Peter Mayes of IBM.

Author

Mike Hawkins mike@pink-pit.com

 
introduction.txt · Last modified: 2010/06/07 16:36 by mike
 
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki