Introduction: ================ ``lbsh'' (also called #sh or just pound-shell) is a shell monitor. It stands for lab-book shell. It is designed to act as an automated lab-book for people that conduct experiments in their standard shell environments. This type of work includes processing files with R, using gnuplot, interactive perl/python, standard data processing with shell commands (awk, sort, join, etc.), etc. Pound -shell does not attempt to trace commands if it detects that a program has captured the screen. Examples include Vim, emacs, etc. In such cases, pound-shell waits for the sub-process to end and then resumes recording. In essence, you simply run ``lbsh'' and it automatically creates a new shell (the same type [bash | tcsh | etc.] that you already use) in a child process. Your session goes through pound-shell to the real shell underneath. When you decide you want to start logging commands for an experiment, you tell pound-shell to start logging. Then pound-shell simply records your commands (and also traces into the above mentioned tools). It does not record passwords, or anything else that is not output the screen. When you are done, you tell pound-shell to stop logging. Then you may do whatever and pound-shell will quietly wait. You may repeat as needed without having to restart pound-shell each time. Pound-shell is intended to provide the same utility as a scientist's lab-book. After processing and analyzing and producing data, it is often very desirable to know ``exactly'' what one did to arrive at their current state. Pound-shell allows users to log their sessions (even as they enter into other programs listed above) and recreate an entire session to arrive at the same results. Under the Hood: ================ The advance that pound-shell offers is that it is able to provide this utility without requiring the user to behave any differently. In a sense, users are free from needing to learn the diligence practiced by other scientists in their labs. Initial Configuration: ---------------- Pound-shell reads a user's ${HOME}/.lbshrc file (if it exists), when it starts up. All runtime dependent parameters are evaluated at that time. Users can run the utility ``lbshadmin -c'' to create their .lbshrc file. Starting an Experiment: ---------------- Pound-shell is designed to let someone process data as they normally would. At the user's behest (ctrl-b), pound-shell will begin logging the user's commands to a designated ``experiments'' directory. Upon completion of work, the user can signal the end of an experiment (ctrl-b) and pound -shell will stop logging. Pound-shell does not log anything until an experiment is running. Post-Processing: ---------------- After running experiments, users can examine their lab-book, update their .lbshrc file, view experiments, get the provenance, or even re-run experiments from the utility ``lbshadmin''. This utility has a menu interface so users can launch it and interactively post-process their work. Experiment Data: ---------------- Pound-shell's experiment-logging consists of 3 files; a script file that records the commands issued, a meta-data file that records system statistics (such as machine/OS type, PWD, etc.) and also per-command meta-data (such as timestamp and if the command spawned a program that pound-shell did, or did not, trace into), and a stat file that (if the user has specified a ``datadir'' in their .lbshrc file) will list which files were accessed (A) and which files were modified (M) in the datadir during the experiment. The files for an experiment will be stored in the directory that is specified in the their .lbshrc file. The names of these files are: ``experiment--script.out'' - and - ``experiment--meta.out'' - and - ``experiment--stat.out'' experiment--script.out: ---------------- This file contains just the commands issued by the user during this experiment. Its format is: one command per line. experiment--meta.out: ---------------- This file contains various pieces of meta-data about this experiment. The beginning of the file has `#' characters that act as comment indicators. Behind these comments are 4 initial pieces of meta-data: 1 - The start time of the experiment (in human readable form) 2 - The key ``NAME'' that specifies the user-supplied (or default) name for this experiment. 3 - The key ``UNAME'' that contains the uname information about the machine this experiment was run on: a - OS b - hostname c - OS Version d - Hardware platform. 4 - The key ``PWD'' that specifies the directory that the experiment was started in. Following these pieces of meta-data, there is 1 line for each line in the experiment--script.out file. These lines are of the form: |<[1|0]> The timestamp indicates when the command was issued, and the [1|0] boolean indicates if the command captured the screen and pound-shell stopped logging. This might have happened if the user entered an editor (such as Vim, emacs, etc.). The purpose of this datum is to allow one to re-run the previous session, but know which commands to skip (because they would capture the screen). experiment--stat.out: ---------------- This file contains a list of files with either an `A' or an `M' prepended (and separated by a tab): [A|M] The files are checked during experiments after each command issued. When the experiment is finished, all stat information is dumped to this file. Runtime Behavior: ---------------- If the user would like lbsh experiments to be able to reconstruct the provenance of files (the ``how did I get this file?'' answers) the .lbshrc file will have to define datadir. This variable will tell lbsh which directory to stat from (and recurse down from). IMPORTANT NOTE: if datadir is on an NFS share, this will likely not work. This is because the atime of files will not be modified when they are accessed. As a result, lbsh will not know what files were used to create new ones. In addition, file-systems with noatime will also fail to support this semantic. In order to programmatically detect if a shell is running under pound-shell, users can check the shell's environment for the key "LBSH_PID". When running in pound-shell, this variable will have the PID of the pound-shell (not the user's bash/tcsh/etc.). To make this easier, pound-shell comes with a script (that is installed by default) called ``lbshrunning.sh''. When executed the return value will be 1 if pound-shell is running, and 0 if it isn't. If the user specifies a ``-v'' flag, the script will output ``Yes'' or ``No'' respectively. In addition, in order to detect if pound-shell is currently running an experiment, users can look for a file called ``lbsh-exp.'' in their experiments directory. This file will use pound-shell's PID, and will only exist during a running experiment. To facilitate this query, pound-shell comes with a script called ``exprunning.sh'' that checks for this file. If it detects the file's existence, it returns 1, otherwise it returns 0. If the user runs the script with a ``-v'', the script will also output a ``Yes'' or ``No'', respectively. Example: ================ $ which lbsh /usr/local/bin/lbsh $ lbsh $  $ echo beginning beginning $ ls -la total 26208 drwxr-xr-x 17 eoster eoster 578 Aug 20 22:57 . drwxr-xr-x 6 eoster eoster 204 Aug 16 20:29 .. drwxr-xr-x 6 eoster eoster 204 Aug 13 21:33 CVS -rw-r--r-- 1 eoster eoster 1480 Aug 16 20:14 LICENSE -rw-r--r-- 1 eoster eoster 2441 Aug 15 23:31 Makefile.in -rw-r--r-- 1 eoster eoster 2057 Aug 20 22:57 README -rw-r--r-- 1 eoster eoster 13021700 Aug 13 21:35 boost_1_34_0.tar.bz2 drwxr-xr-x 4 eoster eoster 136 Aug 18 23:21 conf -rwxr-xr-x 1 eoster eoster 39715 Aug 13 21:38 config.guess -rwxr-xr-x 1 eoster eoster 29483 Aug 13 21:38 config.sub -rwxr-xr-x 1 eoster eoster 278892 Aug 13 23:11 configure -rw-r--r-- 1 eoster eoster 3458 Aug 15 23:31 configure.in -rwxr-xr-x 1 eoster eoster 5569 Aug 13 21:38 install-sh drwxr-xr-x 3 eoster eoster 102 Aug 18 23:21 scripts drwxr-xr-x 15 eoster eoster 510 Aug 18 23:25 src drwxr-xr-x 7 eoster eoster 238 Aug 18 23:17 tests $ gnuplot .... gnuplot> set title 'test' gnuplot> set xlabel 'blah' gnuplot> plot sin(x) gnuplot> quit $ echo done done $  The resulting experiment script (in the directory specified in your .lbshrc file, or the current dir by default) will contain: echo beginning ls -la gnuplot set title 'test' set xlabel 'blah' plot sin(x) quit echo done This can be seen by running ``lbshadmin''. Install: ================ To install: lbsh requires the boost_regex library to be installed. Instructions for this can be found at www.boost.org. Alternately, all major package systems that I have looked at have support for installing boost (apt-get, emerge, ports, darwinports, etc.). ./configure make make install Post-install configuration: ---------------- To create an initial .lbshrc file run the utility (installed by the above command): lbshadmin -c This command will walk you through the initial configuration and create your new .lbshrc file for you. All documentation (this file), the license, etc. is stored in the data-directory (/usr/local/share/doc by default) under a directory called ``lbsh''.