Fixing GPG key conflicts in apt (Ubuntu)

Sometimes when working with Nvidia apt registry or other packages, we can encounter a conflict between signing keys.

It looks something like this:

E: Conflicting values set for option Signed-By regarding source https://nvidia.github.io/libnvidia-container/stable/deb/amd64/ /: /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg != /etc/apt/cloud-init.gpg.d/nvidia-docker-container.gpg E: The list of sources could not be read.

This happened for me when I added apt repository and updated source list using the following code from Nvidia Container Toolkit installation guide:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

This resulted in a creation of a new file /etc/apt/sources.list.d/nvidia-container-toolkit.list , content of which (keys and repo) conflicted with an existing apt source file.

Finding the conflict

grep -r ‘nvidia.github.io’ /etc/apt/

- /etc/apt/sources.list.d/nvidia-container-toolkit.list:deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /
- /etc/apt/sources.list.d/nvidia-container-toolkit.list:#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/experimental/deb/$(ARCH) /
- /etc/apt/sources.list.d/nvidia-docker-container.list:deb [signed-by=/etc/apt/cloud-init.gpg.d/nvidia-docker-container.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /

What it means

Content in the second file is commented out, so I can ignore it.

• The command I ran adds a new GPG key for the NVIDIA repository and saves it to /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg.
• It also modifies the source configuration (/etc/apt/sources.list.d/nvidia-container-toolkit.list) to explicitly use this new key ([signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg]).
• The conflict was that the apt repository url https://nvidia.github.io/libnvidia-container/stable/deb/amd64 had two conflicting public keys:
  - /etc/apt/cloud-init.gpg.d/nvidia-docker-container.gpg (existing)
  - /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg (new)
• So apt didn’t know which one to use to:
  1. Verify the updated list after running apt update
  2. Verify the installed package after apt install

Confirming and Fixing Problem

The first two files were newly created, which I confirmed using:

stat /etc/apt/sources.list.d/nvidia-container-toolkit.list

File: /etc/apt/sources.list.d/nvidia-container-toolkit.list
Size: 285 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 41620 Links: 1
Access: (0644/-rw-r — r — ) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2025–01–25 06:56:38.010471588 +0000
Modify: 2025–01–25 06:56:28.498259203 +0000
Change: 2025–01–25 06:56:28.498259203 +0000
Birth: 2025–01–25 06:56:28.444257998 +0000

Noticing that the file was created today, I felt it was safe to delete/deactivated the file by changing the extension from .list to .list.bak, which removed it from apt sources listing.

sudo mv /etc/apt/sources.list.d/nvidia-container-toolkit.list \
        /etc/apt/sources.list.d/nvidia-container-toolkit.list.bak

After these steps, the I was able to do sudo apt update successfully.

However, to completely undo the command I ran, earlier, I also deactivated

/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg after checking it’s creation date, which was again today.

sudo mv /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
        /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg.bak

What I learned

1. apt sources are listed here: /etc/apt/sources.list.d , and /etc/apt/sources.list
2. Public keys provided by repository managers are used to update apt list provided by the repository, and to verify packages after installation
3. Apparently, there is no standard place to save keyrings, or there is a gap in my knowledge, but it is out of scope for now.
4. Asking an llm to explain a command before running it is a good idea.