Panics on startup #10056

matthiasbeyer · 2023-08-06T08:22:58Z

Checklist

This is a bug report, not a question. Ask questions on discuss.ipfs.io.
I have searched on the issue tracker for my bug.
I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

Kubo version: 0.20.0                                                                                                                                                                                                                                                                                                         
Repo version: 13                                                                                                                                                                                                                                                                                                            
System version: amd64/linux                                                                                                                                                                                                                                                                                                  
Golang version: go1.20.6 

Newest version that ships with my distro, NixOS.

Config

Error: cannot connect to the api. Is the daemon running? To run as a standalone CLI command remove the api file in `$IPFS_PATH/api`

Description

Aug 06 10:19:35 takeshi ipfs[3421497]: Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiBPPId8_TywNuTlqMUKrDLB_BgvAIXhFuvQlBgcfVZ6zw/certhash/uEiAxgndE96iB-zJi-pNGMnT_iWziyQAc2HwrNvSiInHGrw                                                                                                               
Aug 06 10:19:35 takeshi ipfs[3421497]: panic: runtime error: index out of range [0] with length 0                                                                                                                                                                                                                              
Aug 06 10:19:35 takeshi ipfs[3421497]: goroutine 1 [running]:                                                                                                                                                                                                                                                                  
Aug 06 10:19:35 takeshi ipfs[3421497]: main.serveHTTPApi(0xc000684000, 0xc000698050)                                                                                                                                                                                                                                           
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/kubo/cmd/ipfs/daemon.go:717 +0x1010                                                                                                                                                                                                                             
Aug 06 10:19:35 takeshi ipfs[3421497]: main.daemonFunc(0xc000684000, {0x23d3b40?, 0x2c9f1c8?}, {0x25b2ac0?, 0xc000698050})                                                                                                                                                                                                     
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/kubo/cmd/ipfs/daemon.go:503 +0x1966                                                                                                                                                                                                                             
Aug 06 10:19:35 takeshi ipfs[3421497]: github.com/ipfs/go-ipfs-cmds.(*executor).Execute(0xc00069c440?, 0xc000684000, {0x7f2fec6515d0, 0xc0006a0240}, {0x25b2ac0, 0xc000698050})                                                                                                                                                
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/go-ipfs-cmds@v0.9.0/executor.go:88 +0x149                                                                                                                                                                                                                       
Aug 06 10:19:35 takeshi ipfs[3421497]: main.tracingWrappedExecutor.Execute({{0x2cac320?, 0xc00068c608?}}, 0xc000684000, {0x7f2fec6515d0, 0xc0006a0240}, {0x25b2ac0, 0xc000698050})                                                                                                                                             
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/kubo/cmd/ipfs/main.go:334 +0x3d4                                                                                                                                                                                                                                
Aug 06 10:19:35 takeshi ipfs[3421497]: github.com/ipfs/go-ipfs-cmds/cli.Run({0x2cc0418?, 0xc00021b810?}, 0x3e0fb40, {0xc000052040, 0x2, 0x2}, 0xc000099168?, 0x3?, 0xc000014020, 0x2a7b9e0, ...)                                                                                                                               
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/go-ipfs-cmds@v0.9.0/cli/run.go:137 +0x8f3                                                                                                                                                                                                                       
Aug 06 10:19:35 takeshi ipfs[3421497]: main.mainRet()                                                                                                                                                                                                                                                                          
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/kubo/cmd/ipfs/main.go:194 +0x59a                                                                                                                                                                                                                                
Aug 06 10:19:35 takeshi ipfs[3421497]: main.main()                                                                                                                                                                                                                                                                             
Aug 06 10:19:35 takeshi ipfs[3421497]:         github.com/ipfs/kubo/cmd/ipfs/main.go:78 +0x19                                                                                                                                                                                                                                  
Aug 06 10:19:35 takeshi systemd[1]: ipfs.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

The text was updated successfully, but these errors were encountered:

justinabrahms · 2023-08-10T23:14:25Z

Confirming that this is happening on v 0.20.0 through 0.22.0
There's something odd about the home directory of ipfs ( /var/lib/ipfs ) because I can start it just fine for my user.

Jorropo · 2023-08-10T23:34:19Z

Can one of your please post ipfs config show ?
If this also panics, your config json file (you would need to remove the private key manually or just post your private key and generate a new one).

justinabrahms · 2023-08-10T23:36:52Z

ipfs config show seems to break due to the daemon not running.

Error: cannot connect to the api. Is the daemon running? To run as a standalone CLI command remove the api file in $IPFS_PATH/api

There is no api file.

Here's the raw config. I'll regenerate it after we fix this:

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": null,
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWRr625evAwbyxk8t6xKWgN6adxBZ5fAkUHhqnuji7BWGc",
    "PrivKey": "CAESQL5orK97F7rgDTtu4yA6bDJOlucxKF6Vv19T+dksHIb37ivkbSDCZ1o3y1yjSdGMowLZZcZHsqmz3msK6HYk8K0="
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

justinabrahms · 2023-08-11T01:05:48Z

Thanks to @Jorropo, we uncovered the issue.

basically it seems nixos is doing something special (applying a custom config, ...) that is generating a config file that has a blank API field
Kubo assumes anyone would want to have an API endpoint so it panics since you have zero (which should be fixed)

Fix is to make it listen on an API for now:

  services.kubo = {
    enable = true;
    settings.Addresses.API = ["/ip4/127.0.0.1/tcp/5001"];
  };

Jorropo · 2023-08-11T01:09:42Z

So we could start like that, however this sounds like a nixos issue too:

    "API": null,

This looks wrong and make Kubo not usable to do anything, technically kubo is gonna continue make accessible whatever you had stored locally over the network as well as expose a gatewayh service, but I don't think this is what is intented.

If I understand NixOS/nixpkgs@f2be3ae correctly it should default to some unix domain path which makes sense as it's more secure in depth compared to localhost, it does not work with extensions AFAIK tho.

Fixes: #10056

Fixes: ipfs#10056

Fixes: #10056

This fixes a panic of the kubo daemon which could occur under certain conditions when the daemon was starting. It was caused by the `ipfs.service` unit not depending on the `ipfs-api.socket` and `ipfs-gateway.socket` units with `Wants=`. This allows the `ipfs.service` to be started manually or by `nixos-rebuild` without the sockets being set up before that. When that happens, the daemon won't know about these sockets and will only use what is set in `services.kubo.settings.Addresses.Gateway` and `services.kubo.settings.Addresses.API`. By default the `API` is an empty list in NixOS though. The daemon doesn't like this at all and panics on startup, see ipfs/kubo#10056. With this commit, starting `ipfs.service` will first set up the two sockets before starting the actual service. Adding the `Sockets=` option implicitly adds a `Wants=` for the sockets and this is exactly what we need. See https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Implicit%20Dependencies . This can be checked with `systemctl show ipfs.service`. This should probably be upstreamed to the unit file in the Kubo repo. The problem can be reproduced in the following way: - Add `services.kubo.enable = true` to `/etc/nixos/configuration.nix` - `sudo nixos-rebuild switch` (this may already fail, not sure why it's not deterministic for me) - `sudo systemctl stop ipfs-api.socket` - `sudo systemctl stop ipfs-gateway.socket` - `sudo systemctl stop ipfs.service` - `sudo systemctl start ipfs.service` Fixes NixOS#248447.

This fixes a panic of the kubo daemon which could occur under certain conditions when the daemon was starting. It was caused by the `ipfs.service` unit not depending on the `ipfs-api.socket` and `ipfs-gateway.socket` units with `Wants=`. This allows the `ipfs.service` to be started manually or by `nixos-rebuild` without the sockets being set up before that. When that happens, the daemon won't know about these sockets and will only use what is set in `services.kubo.settings.Addresses.Gateway` and `services.kubo.settings.Addresses.API`. By default the `API` is an empty list in NixOS though. The daemon doesn't like this at all and panics on startup, see ipfs/kubo#10056. With this commit, starting `ipfs.service` will first set up the two sockets before starting the actual service. Adding the `Sockets=` option implicitly adds a `Wants=` for the sockets and this is exactly what we need. See https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Implicit%20Dependencies . This can be checked with `systemctl show ipfs.service`. This should probably be upstreamed to the unit file in the Kubo repo. The problem can be reproduced in the following way: - Add `services.kubo.enable = true` to `/etc/nixos/configuration.nix` - `sudo nixos-rebuild switch` (this may already fail, not sure why it's not deterministic for me) - `sudo systemctl stop ipfs-api.socket` - `sudo systemctl stop ipfs-gateway.socket` - `sudo systemctl stop ipfs.service` - `sudo systemctl start ipfs.service` Fixes #248447. (cherry picked from commit d4fcb44)

This fixes a panic of the kubo daemon which could occur under certain conditions when the daemon was starting. It was caused by the `ipfs.service` unit not depending on the `ipfs-api.socket` and `ipfs-gateway.socket` units with `Wants=`. This allows the `ipfs.service` to be started manually or by `nixos-rebuild` without the sockets being set up before that. When that happens, the daemon won't know about these sockets and will only use what is set in `services.kubo.settings.Addresses.Gateway` and `services.kubo.settings.Addresses.API`. By default the `API` is an empty list in NixOS though. The daemon doesn't like this at all and panics on startup, see ipfs/kubo#10056. With this commit, starting `ipfs.service` will first set up the two sockets before starting the actual service. Adding the `Sockets=` option implicitly adds a `Wants=` for the sockets and this is exactly what we need. See https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Implicit%20Dependencies . This can be checked with `systemctl show ipfs.service`. This should probably be upstreamed to the unit file in the Kubo repo. The problem can be reproduced in the following way: - Add `services.kubo.enable = true` to `/etc/nixos/configuration.nix` - `sudo nixos-rebuild switch` (this may already fail, not sure why it's not deterministic for me) - `sudo systemctl stop ipfs-api.socket` - `sudo systemctl stop ipfs-gateway.socket` - `sudo systemctl stop ipfs.service` - `sudo systemctl start ipfs.service` Fixes NixOS#248447.

matthiasbeyer added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Aug 6, 2023

matthiasbeyer mentioned this issue Aug 6, 2023

kubo: 0.20.0 -> 0.21.0 NixOS/nixpkgs#241481

Merged

12 tasks

Jorropo mentioned this issue Aug 11, 2023

kubo: remove openssl from the build NixOS/nixpkgs#248441

Merged

12 tasks

Jorropo mentioned this issue Aug 11, 2023

kubo: default installation gives null API field instead of some unix domain socket NixOS/nixpkgs#248447

Closed

Jorropo added good first issue Good issue for new contributors dif/easy P2 Medium: Good to have, but can wait until someone steps up and removed need/triage Needs initial labeling and prioritization labels Aug 11, 2023

Jorropo added a commit that referenced this issue Aug 11, 2023

fix: allow daemon to start correctly if the API is null

a3c72d6

Fixes: #10056

Jorropo mentioned this issue Aug 11, 2023

fix: allow daemon to start correctly if the API is null #10062

Merged

Jorropo removed the good first issue Good issue for new contributors label Aug 11, 2023

Jorropo added a commit that referenced this issue Aug 11, 2023

fix: allow daemon to start correctly if the API is null

a45d3fe

Fixes: #10056

laurentsenta pushed a commit to laurentsenta/kubo that referenced this issue Aug 11, 2023

fix: allow daemon to start correctly if the API is null

c6817c1

Fixes: ipfs#10056

hacdias pushed a commit that referenced this issue Aug 11, 2023

fix: allow daemon to start correctly if the API is null

d4e4607

Fixes: #10056

Jorropo added a commit that referenced this issue Aug 15, 2023

fix: allow daemon to start correctly if the API is null

a986fe2

Fixes: #10056

Jorropo added a commit that referenced this issue Aug 15, 2023

fix: allow daemon to start correctly if the API is null

100ccfd

Fixes: #10056

hacdias pushed a commit that referenced this issue Aug 17, 2023

fix: allow daemon to start correctly if the API is null

ece4e14

Fixes: #10056

BigLep added this to IPFS Shipyard Team Nov 9, 2023

hacdias pushed a commit that referenced this issue Nov 20, 2023

fix: allow daemon to start correctly if the API is null

aced0fd

Fixes: #10056

Luflosi mentioned this issue Dec 4, 2023

nixos/kubo: fix potential panic on startup NixOS/nixpkgs#272041

Merged

13 tasks

hacdias pushed a commit that referenced this issue Dec 11, 2023

fix: allow daemon to start correctly if the API is null

351d27f

Fixes: #10056

hacdias closed this as completed in #10062 Dec 11, 2023

github-project-automation bot moved this to 🎉 Done in IPFS Shipyard Team Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Panics on startup #10056

Panics on startup #10056

matthiasbeyer commented Aug 6, 2023 •

edited by Jorropo

Loading

justinabrahms commented Aug 10, 2023

Jorropo commented Aug 10, 2023

justinabrahms commented Aug 10, 2023 •

edited

Loading

justinabrahms commented Aug 11, 2023

Jorropo commented Aug 11, 2023

Panics on startup #10056

Panics on startup #10056

Comments

matthiasbeyer commented Aug 6, 2023 • edited by Jorropo Loading

Checklist

Installation method

Version

Config

Description

justinabrahms commented Aug 10, 2023

Jorropo commented Aug 10, 2023

justinabrahms commented Aug 10, 2023 • edited Loading

justinabrahms commented Aug 11, 2023

Jorropo commented Aug 11, 2023

matthiasbeyer commented Aug 6, 2023 •

edited by Jorropo

Loading

justinabrahms commented Aug 10, 2023 •

edited

Loading