Transport / format agnostic UCANs

[Gozala]

I'll provide more details later, but here are some quotes from my comments elsewhere

This is just a side note. We have been thinking about extending IPNS spec to allow UCAN based name publishing, so that publishing capability could be delegated without sharing private a key for the IPNS name.

In that setting JWT representation seems like an overkill and I'm starting to think that it would be great if UCANs were representation agnostic, that way JWT encoding would be a one common representation of them.

We did talk about this bit more since I wrote it, so I want to elaborate bit more on what I mean. I think it would be good if UCAN spec was transport agnostic and maybe refer to a JWT-UCAN spec which as well know and commonly used transport format.

I do not think that would imply that actors in the system are required to support all the transport formats, ultimately it's up to the service provider to decide which transports does it support.

That way in libp2p settings we could have alternative UCAN representation which would fit better it's constraints. That should not imply that in our HTTP service we would accept UCANs in that other format neither it would imply that JWT will be passed around in libp2p settings. Yet all of the UCAN implementation would be reusable aprt from transport format parsing and serialization.

[expede]

I've very supportive of this as a broad direction 👍 It could also help potentially converge a bunch of the related specs. Plus having first-class support for lookup in a CAR file would be pretty great.

I'm no fan of JWTs; they were chosen to build on top of because they're familiar to most web devs, and thus could drive adoption / lower barriers. (Strictly speaking, we don't need the JOSE header at all, for example)

My main open question relates to serialization being another degree of freedom. Especially when UCANs may be valid between multiple, non-coordinating applications, you can get into a situation where some of your witnesses are in different formats, and their witnesses can also be in more formats. You can end up in a situation where you need to ask an issuer for a UCAN in a particular format so that you can access another service that doesn't support your serialization format.

Degrees of freedom:

• DID resolution / signature schemes
• Arbitrary resources
• Arbitrary capabilities
• UCAN versions (backwards compatibility)
• Inline witnesses vs content addressing
• Proposed: signature payload serialization format

Options

0. JWT Everywhere

Continue to use JWT everywhere and give up on other use cases. I don't love this one, but 🤷‍♀️ we're still moving the needle

1. Switch to an Efficient Serialization

Drop JWT and define a very tight serialization format in CBOR/PB/Avro/etc. The JWT strategy has been successful in overcoming objections in many conversations, so this may be counterproductive (I could be wrong).

2. Closed-System UCANs

Per @Gozala

In that setting JWT representation seems like an overkill and I'm starting to think that it would be great if UCANs were representation agnostic, that way JWT encoding would be a one common representation of them.

Make JWT the "official" inter-operation format as the lowest common denominator (like how RSA gets used as the most common crypto for many handshakes). In closed systems, you can use whatever serialization that you like, but talking across applications you must switch to JWT (or some other common format that would be defined in the spec).

3. Codec Shortlist

We pick some number of "officially" supported codecs that we expect everyone to implement. This is a higher burden on implementer, but easier on users.

4. Codec Freedom

Use whatever codec you want! If the recipient can't read one, then your validation fails. You need to talk to the service to accept your format, or ask for your token to be reissued (may be recursive in the chain).

5. Autocodec

As part of other work, the idea of an Autocodec for arbitrary IPLD has been proposed. This is essentially referencing a Wasm module in the header that says how to deserialize the token into a common format.

As @matheus23 pointed out in a side conversation, Autocodec big hammer for this problem unless it's broadly available.

[johnandersen777]

Personally I tend to lean towards custom formats at first under similar circumstances as well. However, I am increasingly more wary custom formats. Not because I don't think it they don't offer a cleaner approach anti-redundancy wise, but due to the trade off being interoperability within the existing ecosystem. Personally I'd say with this use case where broad adoption is tied up with design, is it really worth saving a few bytes for performance in the aggregate in a world where bandwidth is increasingly cheaper? (yes there was the discussion on non-networked approaches today, but physical storage size is even less of an issue then if you have no network).

Sorry for the hasty response. I hope to participate more on this soon but am swamped with work right now!

[johnandersen777]

Even looking at JWT, community support cross language has only recently become mature within the past few years. As I understand it, we'll need to bridge the gap between web2 and web3 somehow, and making the bridge come in transports/formats people already use at least at first would probably help that transition be smoother than if we were to introduce something custom in parallel.

[expede]

we'll need to bridge the gap between web2 and web3 somehow

Exactly 👏

I am increasingly more wary custom formats

Same

Sorry for the hasty response. I hope to participate more on this soon but am swamped with work right now!

No worries at all. (I can relate). This is already really useful signal 👍 When you have more time, I'd love to pick your brain, but there's no rush!

[expede]

Thanks for the comments today :)

[johnandersen777]

Just noticed Peer DIDs allow for multiple codec options via numalgo: https://identity.foundation/peer-did-method-spec/index.html#method-specific-identifier

[expede]

@Gozala I converted this to a discussion. I haven't used this feature much before, so we can convert back if needed. I anticipate that this will become a longer thread, so having more tools for long form can't hurt.

[QuinnWilton]

That way in libp2p settings we could have alternative UCAN representation which would fit better it's constraints.

To clarify, are the constraints in question to do with minimizing the data on the wire and scanning time? If so, there may be a middle ground between using JSON + JWT everywhere, and switching to a compact binary serialization format.

For example, an encoding that offers dual-format interoperability, like UBJSON, would preserve the simplicity and familiarity benefits of JSON encoded JWT, while supporting both JSON and binary encodings of the token using the same schema.

There's still implementation overhead in terms of supporting UBJSON, but since the format maps 1:1 with JSON, the risk of divergence between both representations is minimized. Since parsing the format is trivial, there's also a clear path toward supporting it with an autocodec down the road.

UBJSON hasn't seen much adoption, so I mainly bring it up as an example, but it seems like an elegant approach to driving adoption without completely closing the door on performance, and without taking on the complexity of similar formats like BSON. The idea also has precedent at Amazon, through their ION project, which offers a similar dual-format approach to UBJSON, albeit with more complexity than is introduced by UBJSON.

The 1:1 mapping between JSON and UBJSON also means that it's viable for use in closed-systems without introducing much maintenance overhead and without changing the specification: the same schema can be used for both internal communication of binary encoded UCANs, and JSON encoded UCANs that are shared with external systems.

[bgins]

Yep, seeing UCAN in the context of JWT reveals why it is magical, and makes it easier to compare with other ways that JWTs are used.

[QuinnWilton]

The main challenge with anything other than a literal RFC 7519 compliant JWT is that we don't get to pull that marketing stunt to drive adoption.

This all makes sense, thanks!

I was imagining that the benefit had more to do with reusing existing tooling for working with the decoded data, but I can see how changing the encoding of the Envelope.Claims.Signature representation would get in the way of interop with existing tools + make the tokens harder to work with.

[expede]

Yeah, it's absolutely interop with existing tools 💯

Talking with folks in other orgs this afternoon, there's also a derisking angle, since it's much easier to sell "special kind of JWT" to leadership, than "fast but totally new token format".

So far there seems to be a web2/web3 split on what their tolerance to newness is like. I'm going to wait for all the data to come in first, but I'm starting to warm up to Brian's "pick 2" approach

[QuinnWilton]

Thanks for clarifying, that gives me a better idea of where everyone is coming from!

Brian's approach makes sense, but I think James is onto something by asking how much more compact these tokens can get. If the concern has to do with encoding / decoding time (constrained or IoT devices?) then there's definitely improvements to be had, but I do suspect that the signatures will dwarf any marginal improvements that can be made by encoding the fields more compactly.

[expede]

💯

[bgins]

One possibility would be something between Closed-System UCANs and Codec Shortlist. Require JWT and one other codec that is more efficient, and require conversion utilities between the two. Other codecs would be permissible in closed systems.

This approach would keep the familiarity of JWT and provide a more efficient option. It would also maintain interoperability between implementations.

@QuinnWilton's suggestion would do all of that in one format, which would be great if that works!

[expede]

Hmm yeah, that makes sense! Thinking out loud, how about something like this:

We pick one really compact encoding. It's good on the wire, and contains no additional structure (like the whole JWT header vs payload separated by a .). We call this the "canonical encoding", and is required.

Then we also support the existing JWT encoding. We call this the "compatibility encoding", and is also required.

Implementers typically have an IR of some kind, which would likely map 1:1 cleanly to the canonical format.

If someone wants to write other codecs, they need to be able to be isomorphic to the canonical format, and cannot expect them to be used outside of their closed context.

---

Hmm I keep going back and forth here. Once we have to support one, why support the second one? People will be able to mix the two. I need to touch base with $MYSTERY_PARTNER about what they think.

[expede]

Okay I pinged a couple folks at different orgs. So much cloak and dagger! 😆

[expede]

Okay, so I'm starting to hear back from enough folks that I'm able to draw some very broad conclusions, though N=8 (i.e. not a TON, but also more than a lot of psych studies :P )

Very roughly it's:

• JWT is a selling feature
  • web 2.0 engineers
  • "Management" (both web 2.0 & web3)
• Want compact, new efficient encodings
  • web3 engineers

So that's a pretty even split on "decision makers", depending on who has the power in various orgs. This likely solidifies either Irakli's original "format agnostic" or Brian's "pick 2 / why not both" approaches.

Which is really interesting in its own right. I also have some calls scheduled this coming week as others are starting to have ideas on the web 2.0 side, so I'll report as I know more :)

[expede]

@walkah brings up a good point: how much more efficient do we think we can make these for the libp2p/IPNS use case? The heaviest part is signatures if you're stuck using RSA. At what size/performance improvement do we think it's worth the tradeoff against familiarity

[expede]

One thing that leaning into "a cool new format" would be very easily moving to CARv2 files to get around the inline vs CID duality mentioned in #19. I believe that CAR in particular constrains us to CBOR.

[matheus23]

One thing to note as well: We can totally write our own IPLD codec for UCANs and thus be able to reuse pinning & DAG walking logic for CAR file exports.
Of course that won't work in the broader IPFS ecosystem, because not everyone will have that additional codec added, but it's already nice to be able to re-use CAR file-packing logic.

Also, once autocodecs land, I assume those would be based on what IPLD codecs are today, just with a canonical interface across languages (WASM), so it could totally be upgraded to that then.

[matheus23]

I'm thinking something between (3) Codec Shortlist and (4) Codec Freedom: An optional spec addition that describes a more compact encoding.
I worry that by going (4), we'll get small variations in efficient encodings (CBOR vs. UBJSON vs. BSON vs. protocol buffers vs. avro, etc.), but at the same time we don't want to force everyone to implement (3), unless at some point it turns out it's so useful, it should become a spec requirement.

[Gozala]

I really like idea of canonical IR representation. That way UCAN library could parse / serialize between JWT <-> IR. Any other system could implement own parser / serializer for IR and interop with JWTs via standard UCAN library.

As long as verification / derivation happen on IR I think marketing aspect will remain intact and it would also enable domain specific UCAN transportation. New cool compact representation could also be developed separately and maybe added to standard UCAN library given enough demand.

[Gozala]

I just want to stress that new better and more compact representation is not a motivation. Motivation is an ability to choose encoding and transport that would make most sense in specific domain.

To that end I would really like if UCANs were not tied to any encoding, and were more agnostic of what is inlined vs referenced as per #19 (reply in thread)

That is to allow client to decide what needs to be transported and what can be omitted given the recipient without invalidating tokens. As far as I understand things are pretty tightly coupled right now, even to a JSON serializer (which as far as can recall can produce different results based on order in which fields were set).

[expede]

@Gozala do you have a concrete use case for arbitrary encoding? I'm hearing a lot of "pick one format and stick to it" from other folks.

[expede]

Motivation is an ability to choose encoding and transport that would make most sense in specific domain.

@Gozala I'm genuinely trying to understand the use case, but am struggling a bit. I'm open to this change if we have a good use case for it. My general thoughts are below, but I'm always happy to get on a call to discuss in higher bandwidth! It's been awesome having you asking great questions and pushing for various features, but this one is just not fully connecting for me yet (which is more on me than you!)

Generality is often a nice feature, but makes interop significantly more difficult. Don't get me wrong, I'm a huge fan of generality in library code (maybe too much if you look at the type constraints in Fission's Haskell codebase 😅) We had previously discussed the possibility of ucan and ucan-jwt/ucan-cbor/etc, but this isn't without complexity tradeoffs when interacting with lots of other clients.

The problem for generalizing directly in the spec is when you need to mix formats because you've been delegated in different formats from different sources. We see this kind of problem in IPLD when you don't have the correct codec at-hand for some data (including nested data). You may not be able to control the encoding of a deeply nested proof, and so are unable to use a UCAN with a collaborator until they implement the relevant codec. If a design goal is to be interoperable (maybe it doesn't need to be?), this creates a big problem between implementations and applications.

So, then what if we say that clients using ucan-jwt exclusively cannot accept ucan-cbor, and the two are just separate standards? Because now we're in a situation where you cannot cross some network boundaries based on how someone in your proof chain decided to encode your token.

None of this prevents "off label" use of UCAN in arbitrary encodings.

I would like signatures to come out the same regardless if proofs are nested or referenced by CID

I think that we definitely can achieve this! I agree with the title of #19 to get rid of inlining completely, and use a first-class mechanism rather than the facts field (likely a pointer machine.) This gets rid of the dependency on DHT availability. I'll put a sketch in that thread.

As far as I understand things are pretty tightly coupled right now, even to a JSON serializer (which as far as can recall can produce different results based on order in which fields were set).

Field order is unspecified in JSON. Some parsers generate different orders based on input, and oythers always sort them alphanumerically. It's annoying, but in practice this hasn't been a huge problem — though you do need to store the encoded signed payload.

It's probably worth noting that there's nothing preventing the spec from imposing a field order, even beyond a consistent alphanumeric sort that's relatively common. Is the core need here a need to start processing a UCAN while it's still coming over the wire?

[Gozala]

@expede let me provide a bit more context. In .storage services we have been thinking about representing mutable DAGs that clients can update via transactions (for more detailed read please take a look at storacha/specs#1)

We want to encode those transactions as DAGs serialized in CAR format. Each transaction is modeled as a set of blocks to be added to the IPNS / DID key & is supplied with UCAN proving write access to both IPNS / DID key.

It makes whole a lot of for us to represent these transactions as self contained IPLD DAGs as opposed to IPLD DAG + JWT header, as that would allow us to just submit a CAR serialized data block which will be self contained representation of the transaction and allow arbitrary nodes to act upon.

We could alternatively encode UCAN with a custom coded that parses serialized JWT, but that just complicates things and adds extra bytes to be send / stored. We also want to have a flexibility in deciding which blocks to include / exclude in the serialized transaction without affecting signatures.

Broader point here is, JWTs just introduce incidental complexity and an overhead that seems hard to justify. Interop in this context is also not really a concern, as there are quite a few extra layers others will need to support before getting to UCANs. Neither we really care for generalized UCAN proofs in these transaction DAGs.

[expede]

Okay some clarity from a call with @Gozala and a few others today (Irakli, feel free to tag them here if appropriate). The short version is that JWT is actually fine at the spec level. To make this dag-json compatible, we need to make inlining and CIDs equivalent. One way of doing this would be to adjust the signature algorithm to sign not the encoded data verbatim, but a special serialization that at minimum turns all inlined values into CIDs. Then the exact encoding is unrestricted from being broken up into DAG chunks if you want to. This breaks the common JWT signature algorithms... unless you make the canonical encoding CIDs (or vice versa, but CIDs are more graceful when you don't want to pull all data just to serialize). Then if you inline them, you can always get back (fully deterministically) to the version that was signed.

JSON also tends to have extremely good/fast codecs and wide support, so compute performance is less likely a major tardeoff here.

Will think on this more, but I think that it's a great direction! 🚀

[expede]

Copy/paste from #30 (request to add CBOR encoding)

The TL;DR on current thinking (and we can be convinced otherwise!) is that this spec is the one for interoperability between systems. If we introduce many other encodings, then we run into token incompatibilities, or larger implementations that support multiple codecs. If folks want to adapt the spec to an internal use case that uses a different encoding, that's a great idea and we're happy to help, but that we can't expect it to interop with other systems and so shouldn't be in the spec directly.

[Gozala]

For what it's worth I've implemented dag-ucan which uses DAG-CBOR as a primary encoding, which is more compact and has a better hash consistency than a secondary RAW JWT encoding (basically currently specified representation). This way primary representation is transport agnostic and can be formatted into valid JWT UCAN. Since not all UCANs will have a valid DAG-CBOR representation secondary RAW representation is used when parsing such UCANs or linking to them.

I think this provides a reasonable path for migration to transport / format agnostic UCANs that can have more compact representation, while retaining interop with UCANs that do not opt in.

[Gozala]

@expede what do you think about updating spec so that it recommends (but does not demand) use alphabetical key order and no white-spaces. Tokens issued by implementations that following this recommendation would then be re-presentable in a more compact DAG-CBOR encoding.

P.S.: I have already proposed to do this in ucan-wg/ts-ucan#73

[johnandersen777]

After syncing with the SCITT WG I can confirm that CBOR would be really good to have as an encoding option. We talked about convergence of both on demand remote attestation and stored attestations (transparency logs style) using the same underlying formats (remote attestation folks speak CBOR).

2022-07-25 Supply Chain Integrity, Transparency and Trust (SCITT) meeting notes

[expede]

@pdxjohnny good to know! Since this thread kicked off, we've found a way to do an network-encoding agnostic version of UCAN while retaining compatibility with the core JWT version. Essentially the plan is not that different from what's mentioned in the JWT spec: we're going to have (an IPLD-friendly) deterministic encoding, so if you know how to round trip e.g. CBOR you can do that.

Not all implementations would be required to do this (it's probably not in the core spec immedietly), but that's fine since an implementation that doesn't restrict key ordering and whitespace will validate deterministic UCANs encoded as JWT just fine.

The main restriction is that the signature would need to be over the JWT encoding.