Identify interop with nim-libp2p (SPR field 8, listen addrs, strict decode)

[acul71]

Summary

When nim-libp2p (dialer) runs identify against a py-libp2p listener (e.g. transport-interop: WebSocket + Noise + Yamux), the Nim side can fail with:

Failed to finish outgoing upgrade in internalConnect: Incorrect message received!

That string comes from nim-libp2p’s identify decoder when decodeMsg does not return a valid IdentifyInfo—not from multistream or the WebSocket upgrade.

This discussion collects what we found, how Go and JavaScript handle the same signed peer record (SPR) field, and a practical workaround used in libp2p interop tests.

---

What Nim does (decodeMsg)

Nim parses the identify protobuf with a strict pipeline: failures while decoding individual fields can invalidate the entire message.

Relevant excerpt from nim-libp2p (libp2p/protocols/identify.nim):

proc decodeMsg*(buf: seq[byte]): Opt[IdentifyInfo] =
  var
    iinfo: IdentifyInfo
    pubkey: PublicKey
    oaddr: MultiAddress
    protoVersion: string
    agentVersion: string
    signedPeerRecord: SignedPeerRecord

  var pb = initProtoBuffer(buf)
  if ?pb.getField(1, pubkey).toOpt():
    iinfo.pubkey = Opt.some(pubkey)
    if ?pb.getField(8, signedPeerRecord).toOpt() and
        pubkey == signedPeerRecord.envelope.publicKey:
      iinfo.signedPeerRecord = Opt.some(signedPeerRecord.envelope)
  discard ?pb.getRepeatedField(2, iinfo.addrs).toOpt()
  discard ?pb.getRepeatedField(3, iinfo.protos).toOpt()
  if ?pb.getField(4, oaddr).toOpt():
    iinfo.observedAddr = Opt.some(oaddr)
  if ?pb.getField(5, protoVersion).toOpt():
    iinfo.protoVersion = Opt.some(protoVersion)
  if ?pb.getField(6, agentVersion).toOpt():
    iinfo.agentVersion = Opt.some(agentVersion)

  Opt.some(iinfo)

Field 8 is the signed peer record (optional bytes signedPeerRecord = 8 in implementation protos). If present, Nim tries to decode it as its SignedPeerRecord type; wire bytes that do not decode cleanly can contribute to decode failure (depending on error propagation in the ? / toOpt path).

Repeated field 2 (listenAddrs): Nim’s multiaddr-specific getRepeatedField requires that at least one entry parses as a MultiAddress. If every entry fails (e.g. multiaddrs Nim’s parser rejects), the repeated-field decode can return IncorrectBlob, and the whole identify decode fails.

---

What py-libp2p sends (identify + SPR)

Default identify builds a protobuf that includes signedPeerRecord from env_to_send_in_RPC (libp2p/identity/identify/identify.py):

def _mk_identify_protobuf(
    host: IHost, observed_multiaddr: Multiaddr | None
) -> Identify:
    ...
    envelope_bytes, _ = env_to_send_in_RPC(host)
    ...
    return Identify(
        ...
        signedPeerRecord=envelope_bytes,
    )

env_to_send_in_RPC (libp2p/peer/peerstore.py) creates or reuses a signed envelope with domain aligned to the rest of the ecosystem:

ENVELOPE_DOMAIN = "libp2p-peer-record"

def create_signed_peer_record(
    peer_id: ID, addrs: list[Multiaddr], pvt_key: PrivateKey
) -> Envelope:
    record = PeerRecord(peer_id, addrs)
    envelope = seal_record(record, pvt_key)
    return envelope

So intended format is the standard signed envelope + peer record (RFC-style), same domain string as Go.

---

go-libp2p (reference)

Proto (p2p/protocol/identify/pb/identify.proto):

optional bytes signedPeerRecord = 8;

Parsing (p2p/protocol/identify/id.go):

func signedPeerRecordFromMessage(msg *pb.Identify) (*record.Envelope, error) {
	if len(msg.SignedPeerRecord) == 0 {
		return nil, nil
	}
	env, _, err := record.ConsumeEnvelope(msg.SignedPeerRecord, peer.PeerRecordEnvelopeDomain)
	return env, err
}

Go uses PeerRecordEnvelopeDomain = "libp2p-peer-record" (core/peer/record.go), matching py-libp2p’s ENVELOPE_DOMAIN.

Sending: SPR is attached when not disabled and a certified peer record exists (getSignedRecord); hosts can opt out via DisableSignedPeerRecord.

---

js-libp2p (@libp2p/protocol-identify)

Sending (packages/protocol-identify/src/identify.ts): uses peerData.peerRecordEnvelope or seals a new PeerRecord with RecordEnvelope.seal and passes signedPeerRecord: envelope.marshal() (plus listenAddrs from addresses decapsulated from /p2p/).

Consuming (packages/protocol-identify/src/utils.ts):

if (message.signedPeerRecord != null) {
  const envelope = await RecordEnvelope.openAndCertify(peerRecordEnvelope, PeerRecord.DOMAIN)
  let peerRecord = PeerRecord.createFromProtobuf(envelope.payload)
  ...
}

Same conceptual pipeline: envelope bytes + peer-record domain + inner PeerRecord.

---

libp2p identify spec (v1.0.0 README)

The published identify doc lists the core message without field 8 (extensions added in implementations):

message Identify {
  optional string protocolVersion = 5;
  optional string agentVersion = 6;
  optional bytes publicKey = 1;
  repeated bytes listenAddrs = 2;
  optional bytes observedAddr = 4;
  repeated string protocols = 3;
}

Several fields are optional; identify/push explicitly says missing fields may be ignored for partial updates. That does not force every implementation to tolerate malformed optional fields on the pull side, but it supports sending minimal identify when interop requires it.

---

Workaround used in libp2p test-plans (Python interop image)

For nim × python transport-interop, the Python ping harness monkeypatches _mk_identify_protobuf to clear fields that were triggering Nim’s strict decoder on WS interop: listen_addrs, observed_addr, protocols, and signedPeerRecord, leaving pubkey + agent/protocol version strings.

That is a test-harness shim, not a claim that py-libp2p is “wrong”—it documents where byte-level interop with nim-libp2p still diverges.

---

Suggested follow-ups (for maintainers / community)

1. Interop tests: optional CI matrix nim-libp2p dialer × py-libp2p listener (or a small integration test) to catch identify regressions.
2. py-libp2p: consider whether identify should omit or normalize SPR / listen addrs when targeting strict peers, or document known limitations with nim-libp2p.
3. nim-libp2p: consider per-field decode resilience (skip bad optional fields) vs strict fail-fast—tradeoffs are real.

---

References

• nim-libp2p identify: decodeMsg, identify() reading readLp / decodeMsg
• py-libp2p: libp2p/identity/identify/identify.py, libp2p/peer/peerstore.py, libp2p/peer/envelope.py
• go-libp2p: p2p/protocol/identify/id.go, p2p/protocol/identify/pb/identify.proto
• js-libp2p: packages/protocol-identify/src/identify.ts, packages/protocol-identify/src/utils.ts
• Spec: specs/identify/README.md (identify v1.0.0)
• Signed envelopes / peer records: libp2p RFC 0002 (signed envelopes)

If useful, this can be turned into a tracking issue for code changes; opening as a Discussion first to align on whether the fix belongs in py-libp2p, nim-libp2p, or both.