Monitor CLI application resource needs

When creating new workflows and optimizing the use of high throughput applications, it is important to know how much RAM and how many threads you will need over time. The two scripts presented here allow simple monitoring of cpu and ram usage during a test run and plot the results for review and reporting.

The the most current version of these tools is available from our github repository

Enjoy!

Example of a Bowtie2 run with 15million paired-end reads

The log file used to generate this plot (bowtie2_usage1401561413.log) and the command used to plot

log2plot.R -i bowtie2_usage1401561413.log \
    -r 16 \
    -m 6 \
    -f 1 \
    -t "bowtie2 monitoring" \
    -o bowtie2_monitoring

bash script to monitor the running application

This script works with most Unix except for OSX darwin where the top command lacks the '-b' parameters required for the code to work properly.

bash monitoring script

#!/bin/sh
 
# A script to log the cpu and memory usage of a linux processes 
# launch with APP's name as argument just before starting the APP
# scriptname: logmyapp
#
# REM: unix-only, will not work under OSX lacking top '-b' (sad!)
#
# inspired by http://www.unix.com/shell-programming-and-scripting/
#   223177-shell-script-logging-cpu-memory-usage-linux-process.html
#
# Stephane Plaisance VIB-BITS June-1-2014 v1.0
 
if [ $# -lt 1 ] ; then
	echo "Usage: logmyapp <name of the app to monitor> <interval (sec|default=5)>"
	exit
fi
 
# Change this as needed
# repeat loop every x sec
FREQ=${2:-5}
init=$(date +%s)
 
# log file name and header
LOG_FILE=${1}_usage"$(date +%s)".log
header='#time pid %cpu %MEM App'
echo ${header%/#\ /\t} > $LOG_FILE
 
echo "# logging to $LOG_FILE every $FREQ sec"
echo "# press <Ctrl>-C to stop logging"
 
# infinite loop will run until ctrl-C is hit
while :; do 
  top -b -n1 | awk -v runtime=$(echo $(date +%s) - ${init} | bc) -v app=${1} 'match($0, app) {print runtime, $1, $9, $10, $12}' >> ${LOG_FILE}
  sleep ${FREQ}
done
 
# when the app closes, no additional line will be added to the log file

[R] script to plot from the monitoring data

This scripts requires the [R] packages optparse (http://cran.r-project.org/web/packages/optparse) to handle command line arguments. Installing the dependencies is documented on the top of the code. Please read the respective package documentations if you wish to improve these scripts.

R script to generate the monitoring plot

#!/usr/bin/RScript
 
# Create nice plot from monitoring data saved by logmyapp
# usage: log2plot.R -h
#
# Stephane Plaisance VIB-BITS June-01-2014 v1.0
 
# required R-packages
# once only install.packages("optparse")
suppressPackageStartupMessages(library("optparse"))
 
#####################################
### Handle COMMAND LINE arguments ###
#####################################
 
# parameters
#  make_option(c("-h", "--help"), action="store_true", default=FALSE,
#              help="plots from logmyapp monitoring data")
 
option_list <- list(
  make_option(c("-i", "--infile"), type="character", 
              help="input file name"),
  make_option(c("-r", "--totram"), type="integer", default=4, 
            help="total RAM available on the computer [default: %default]"),
  make_option(c("-m", "--maxram"), type="integer", default=4, 
              help="max RAM value on y-scale [default: totram]"),
  make_option(c("-t", "--title"), type="character",
              help="Graph Main Title"),
  make_option(c("-o", "--outfile"), type="character", default="monitoring_plot",
              help="base name for output [default: %default]"),
  make_option(c("-f", "--outformat"), type="integer", default=1,
              help="file format for output 1:PNG, 2:PDF [default: %default]")
  )
 
## parse options
opt <- parse_args(OptionParser(option_list=option_list))
 
# check if arguments provided
if ( length(opt) > 1 ) {
 
	# check that infile exists
	if( file.access(opt$infile) == -1) {
	  stop(sprintf("Specified file ( %s ) does not exist", opt$infile))
	  }
 
	##### load data in
	log.data <- read.table(opt$infile, quote="\"", header=TRUE, comment.char = "@")
	attach(log.data)
 
	# define max values
	max.time <- ceiling(max(X.time)/100)*100
	max.ram <- max(X.MEM)
 
	# set colors
	cpu.col="black"
	ram.col="blue"
 
	# title
	main.title <- ifelse(!is.null(opt$title), opt$title, "") 
 
	# rescale RAM axis
	max.ram <- ifelse(!is.null(opt$maxram), opt$maxram, opt$totram)
 
	# output format
	if (opt$outformat==1){
	  # png
	  filename <- paste(opt$outfile, ".png", sep="")
	  png(file = filename, 
		  width = 600, 
		  height = 480, 
		  units = "px", 
		  pointsize = 12,
		  bg = "white")
 
	} else {
	  # pdf
	  filename <- paste(opt$outfile, ".pdf", sep="")
	  pdf(file = filename, 
		  width = 6, 
		  height = 5,
		  pointsize = 12,
		  bg = "white")
	}
 
	##### create plot in two steps (cpu then ram)
	par(mar = c(5,5,2,5))
	plot(X.cpu/100~X.time, 
		 xlab = "execution time (sec)",
		 xlim = c(0, max.time), 
		 ylab = "CPU", 
		 pch = 20,
		 col = cpu.col,
		 main = main.title)
	par(new = T)
	plot(opt$totram*X.MEM/100~X.time, 
		 xlim = c(0, max.time),
		 ylim = c(0, max.ram), 
		 col = ram.col, 
		 axes = F, 
		 xlab = NA, 
		 ylab = NA, 
		 pch = 4, 
		 cex = 0.5)
	axis(side = 4, col = ram.col, col.axis = ram.col)
	mtext(side = 4, line = 3, col = ram.col, "RAM (GB)")
 
	dev.off()	
	##### end  happily
 
}

---

References:

---

---

[ Main_Page ]