Thanks to visit codestin.com
Credit goes to github.com

Skip to content

DevScripts

Marcel Weisgut edited this page Apr 24, 2023 · 9 revisions

Some scripts may be helpful more than once but are too specific to be placed into the scripts folder. Let's collect them here.

Running on a single NUMA node

When multiple users use a single machine, it is helpful to limit the compiler processes as well as the execution of Hyrise to a single NUMA node. That way, the risk of interfering with the other users' processes is reduced. This script prompts the user for a node number and calls numactl accordingly:

function bind_numa_node {
  if [ -z "$NESTED_BASH" ]; then
    NUMACTL_NODE=$1 NESTED_BASH=1 numactl -N $1 -m $1 bash
  fi
}

default_numa_node=0
# Login without promt. If a promt shall be required, remove the following two lines.
bind_numa_node $default_numa_node
skip_prompt=1

if [ -z "$NUMACTL_NODE" ] && [ -z "$skip_prompt"]; then
  target_numa_node=""
  # TODO(anyone) add script support for multiple nodes
  echo "Enter NUMA node index (default: $default_numa_node):"
  while [ -z "$target_numa_node" ]; do
    read input
    case $input in
      "") # enter as input
        echo "using default node"
        target_numa_node=$default_numa_node;;
      *[!0-9]*) # non-numeric input
        echo "Please enter an unsigned numeric node index.";;
      *) # numeric input
        node_size=$(numactl --hardware | grep available | cut -c12-12)
        if [ $input -lt $node_size ]; then
          target_numa_node=$input
        else
          echo "The entered node index exceeds the maximum index. $node_size nodes are available."
        fi;;
    esac
  done
  bind_numa_node $target_numa_node
else
  numactl -s
fi

If you add this to your ~/.bashrc, you will be prompted automatically. For iTerm users, you can also add the following in order to have a small badge reminding you of what node you are on:

function set_iterm_badge {
  printf "\e]1337;SetBadgeFormat=%s\a" $(echo -n "Node(s): $NUMACTL_NODE" | base64)
}
export PROMPT_COMMAND="set_iterm_badge; $PROMPT_COMMAND"

Checking who is using a given NUMA node

for pid in $(ps -a -o pid=); do cmdline=$(tr '\0' ' ' < /proc/$pid/cmdline 2>/dev/null); if [ "$cmdline" = "" ] || [ "$cmdline" = "zsh " ] || [ "$cmdline" = "bash " ]; then continue; fi; getent passwd $(tr '\0' ' ' < /proc/$pid/loginuid) | cut -d: -f1; echo $cmdline; taskset -pc $pid 2>/dev/null; echo; done
Jan.Kossmann
python3 rl_main.py 100000
pid 8187's current affinity list: 84-111,196-223

Search for CI failures across branches and builds

Sometimes you want to check how many builds were affected by a given issue. For example, #1954 discussed a data race in the aggregate operator:

SUMMARY: ThreadSanitizer: data race (/var/lib/jenkins/workspace/hyrise_hyrise_PR-1929/clang-release-thread-sanitizer/hyriseBenchmarkTPCH+0x42f31ab) in std::enable_if<(((((opossum::AggregateFunction)7) [...]

To check if this is still present, we want to see in which build logs AggregateFunction is mentioned. We can do this using:

$ for f in $(find /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches -name log); do grep -lE 'data race.*AggregateFunction' $f; done | xargs ls -l --full-time | awk '{print $6,$9}' | sort
2019-12-11 /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches/master/builds/349/log
2020-01-15 /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches/master/builds/382/log
2020-01-21 /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches/master/builds/387/log
2020-02-04 /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches/master/builds/416/log

Keep in mind that builds for non-master branches are deleted after a while. Have a look at the age of the folders in /var/lib/jenkins/jobs/hyrise/jobs/hyrise/branches/ to see how far back our history goes.

Clone this wiki locally