GCP Cloud GPU Quota Workaround

Photo by Taylor Vick on Unsplash

The search on GCP UI when creating new on-demand VM with GPU attached is not great and it’s very often a hit and trial on if your machine will successfully spin up due to regionwise GPU machine quota issue.

I have generated some useful bash functions to work with the same.

It gives you a bird eye view of what regions the VMs with specific GPUs are available in, then you can create your instance without hit and trial on the UI.

List GPU types

alias list_gpu_types="gcloud compute accelerator-types list --format=json | jq -r '.[].name' | sort | uniq"

List GPU Zones

list_gpu_zones() {
  local gpu_name=$1

  # Fetch the accelerator types in JSON format with fuzzy search and filter by the GPU name
  zones_json=$(gcloud compute accelerator-types list --filter="name~${gpu_name}" --format=json)

  # Extract the unique zones using jq
  unique_zones=$(echo "$zones_json" | jq -r '.[].zone' | awk -F/ '{print $NF}' | sort | uniq)

  # Print the unique zones
  if [ -z "$unique_zones" ]; then
    echo "No zones found for GPU: ${gpu_name}"
  else
    echo "Zones for GPUs matching '${gpu_name}':"
    echo "$unique_zones"
  fi
}

Get GPU Quota for a particular GPU type

Running this command takes time but you get a list of regions where you can create your GPU VM

get_gpu_quota() {
    local gpu_name=$1
    local region=$2

    if [ "$region" = "us-test" ]; then # ignore this region
        return
    fi

    # Fetch the region details in JSON format
    region_details=$(gcloud compute regions describe "$region" --format=json)

    # Lowercase the JSON content for case-insensitive search
    region_details_lower=$(echo "$region_details" | tr '[:upper:]' '[:lower:]')

    # Extract GPU quota information using jq and fuzzy search
    quota_info=$(echo "$region_details_lower" | jq -r --arg gpu_name "$gpu_name" '.quotas[] | select(.metric | test($gpu_name; "i")) | "\(.metric): \(.usage)/\(.limit)"')
    # Initialize an empty variable to store the modified lines
    modified_quota_info=""
    while IFS= read -r line; do
        modified_quota_info+="$(echo "$region $line")"$'\n'
    done <<< "$quota_info"

    standard_quota_info=$(echo "$modified_quota_info" | grep -v preemptible | grep -v committed | grep -v -v /0.0)

    # Print GPU quota information
    if [ ! -z "$standard_quota_info" ]; then
        echo -e "\033[32m$standard_quota_info\033[0m"
    fi
}


get_all_gpu_quotas() {
  local gpu_name=$1
  zones_json=$(gcloud compute accelerator-types list --filter="name~${gpu_name}" --format=json)
  unique_zones=$(echo "$zones_json" | jq -r '.[].zone' | awk -F/ '{print $NF}' | sed 's/-[a-z]$//' | sort | uniq)

#   echo $unique_zones
  for zone in $unique_zones; do
    # echo "Region: $zone"
    # echo "get_gpu_quota $gpu_name $zone"
    get_gpu_quota "$gpu_name" "$zone"
    # echo ""
  done
}

gpu_quota() {
  if [ -z "$1" ]; then
    get_all_gpu_quotas 80gb
  else
    get_all_gpu_quotas "$1"
  fi
}