The National Institute for Computational Sciences

Nautilus Job Accounting

Getting Accounting Information

For project allocation information and overall usage through the last job accounting posting (usually the previous night), use the showusage command.

For more detailed accounting information use the glsjob command, specifying Nautilus with the -m option:

glsjob -m nautilus.nics.teragrid -u <username>
Prints current accounting information for a particular user.
glsjob -m nautilus.nics.teragrid -J <jobid>.nautilus
Can be used to find information on a particular job.
glsjob -m nautilus.nics.teragrid -p <project>
Prints current accounting information for all jobs charged to a particular project account.
glsjob --man
Displays documentation for glsjob

Note: Without the -m option, glsjob will show information for jobs across all NICS resources.

The service unit charge for a job is calculated by multiplying the number of processors allocated to a job by its walltime in hours:

service units = ncpus * walltime

Understanding Nautilus Job Charges

It is important to distinguish between the number of CPUs that you request for a job and the number that is actually allocated (and that you will be charged for). On Nautilus, you may request any number of CPUs, up to 1024, with the -l ncpus PBS option. However, you are allocated (and charged for) CPUs only in multiples of 8. So for example, you may request 10 CPUs with -l ncpus=10, but you will be allocated 16 for your job.

If you do not specify a memory limit for your job, by default you will be given a memory limit of 4000 MB per CPU requested. For instance, if you request 24 CPUs then the corresponding memory limit will be 96000 MB. On the other hand, if you do request a memory limit and it is greater than 32000 MB per 8 CPUs requested,you will be allocated (and charged for) additional CPUs to accommodate the memory, to the nearest multiple of 8. For example if you request 8 CPUs and 100 GB of memory, then you will be allocated and charged for 32 CPUs.

Please note that 4000 MB per core is NOT the same as 4 GB per core. Although each node on Nautilus does have 32 GB of memory, some small portion of that memory needs to be used for the OS. Therefore, a user cannot use the entire 32 GB on a node. This is why, if you ask for one node (ncpus=8) you will see that you are allocated 32000 MB which is a little shy of the full 32 GB. Likewise, if you request ncpus=8,mem=32gb, you will get two nodes because you have requested more than the 32000 MB available on a single node.

Here are some additional examples of various allocation scenarios:

RequestCPUs Allocated and ChargedMemory Limit
-l ncpus=161664000 MB
-l ncpus=202480000 MB
-l ncpus=20,mem=10GB2410 GB (10240 MB)
-l ncpus=20,mem=100GB32100 GB (102400 MB)

Job Refund Policy

NICS will provide refunds for user jobs which are adversely impacted by system issues beyond the control of the user. Refunds requests must be made within two calendar weeks of a job’s completion date by submitting a ticket to help@xsede.org. Please provide: username, machine name, jobID, reason for refund request.

Examples of refund requests that will not be approved include: jobs run on projects that have a negative balance, jobs that started and completed after the project’s end date, and jobs that failed because they reached the user-specified wallclock limit.

NICS strongly encourages the use of application checkpoint restart files. Users should only request refunds from the time of the last successful checkpoint. Full refunds will probably not be approved for very large jobs that do not checkpoint and restart.