event scripts: add logging for low memory conditions

We should never enter swap; if we do, show the memory state of the machine and the process list.  This will help us diagnose what caused the condition before it's too late and the box starts OOM-killing processes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

diff --git a/config/events.d/00.ctdb b/config/events.d/00.ctdb
index 9e83d30d239ce12f55e33190e1c9ad6784fdc467..a576ce499c48bfee43ce3caf06c1a93d951db2bf 100755 (executable)
--- a/config/events.d/00.ctdb
+++ b/config/events.d/00.ctdb
@@ -29,11 +29,21 @@ case "$1" in
        ;;
 
     monitor)
+       # We should never enter swap, so SwapTotal == SwapFree.
+       if [ -n "`grep '^Swap\(Total\|Free\)' /proc/meminfo | uniq -s 10 -u`" ]; then
+           echo We are swapping:
+           cat /proc/meminfo
+           ps auxfww
+       fi
+
        # monitor that we are not running out of memory
        [ -z "$CTDB_MONITOR_FREE_MEMORY" ] || {
                FREE_MEM=`free -m | grep "buffers/cache" | while read A B C D ;do /bin/echo -n $D ; done`
                [ `expr "$FREE_MEM" "<" "$CTDB_MONITOR_FREE_MEMORY"` != "0" ] && {
                        echo "OOM. Free:$FREE_MEM while CTDB treshold is $CTDB_MONITOR_FREE_MEMORY"
+                       cat /proc/meminfo
+                       ps auxfww
+                       echo m > /proc/sysrq-trigger
                        ctdb disable
                        sleep 3
                        ctdb shutdown