Remove the need for an explicit Candid file for cdk-rs/custom canisters

[lastmjs]

Motoko does not require the user to produce a Candid file. This leads to a superior DX compared with cdk-rs, and thus Azle and Kybra, as a Candid file must either be created by hand, or generated automatically from the Rust code. This automatic generation is difficult to deal with, because you must compile and run the code before the Candid can be retrieved.

Compiling and running the code is difficult to do as part of an automatic build process (which is what we must do for Kybra/Azle to provide a nice DX), as we either have to deploy the canister and call a canister method, or we have to run the canister locally, which requires compiling to the local machine's architecture (which has caused us problems), and either run a main program or run a test to generate the canister.

This adds extra plumbing to our build process, an extra generated file to the developer's repo, and adds a few seconds to our build process.

All of this I would think is not necessary. The Rust code provides all of the information necessary to generate the Candid file, and so why can't dfx just retrieve it from the Rust code itself, or from the Wasm binary metadata section?

I've heard arguments for why it's a good idea to create a Candid file by hand, I don't really agree with that and none of our users have asked us for the ability to create a Candid file by hand, they're happy that it's generated for them. If we want to maintain that functionality, then let's allow a manual file to override if necessary.

[chenyan-dfinity]

we have to run the canister locally, which requires compiling to the local machine's architecture (which has caused us problems), and either run a main program or run a test to generate the canister.

Can you be more specific about the problems you are facing when targeting a local architecture? If users can provide a hand written did file, you can skip this build step.

so why can't dfx just retrieve it from the Rust code itself, or from the Wasm binary metadata section?

It's the other way around. We need to generate the did file in the build process before we can put it into the Wasm metadata. Currently, we can only emit Candid types at the runtime. That's why we need to either deploy the canister or compile them in a local environment to extract the did file. To do this statically, we need to modify the Rust compiler, which is not only a lot work, but also introduces a lot of security concerns.

I've heard arguments for why it's a good idea to create a Candid file by hand, I don't really agree with that and none of our users have asked us for the ability to create a Candid file by hand, they're happy that it's generated for them.

This is similar to how protobuf works in Rust. Users wrote proto file first, and the protobuf compiler generates the Rust bindings. From the implementation perspective, this is also easier to do than modifying the Rust compiler.

Overall, we would love to remove that intermediate build step, but we don't see a clear solution at the moment. If we go with the candid centric approach, i.e., generating Rust bindings from did file, we can probably provide a better UX from that direction.

[paulyoung]

I’m not sure if it fits your use case but you can create an empty foo.did file and point to that in dfx.json if you just want to build without anything related to Candid.

[lastmjs]

Can you be more specific about the problems you are facing when targeting a local architecture? If users can provide a hand written did file, you can skip this build step.

We are specifically trying to avoid forcing the user to write their own Candid file. We are generating this file for them from their TypeScript and Python code. Forcing them to write these files themselves I believe would be a major reduction in developer experience.

[lastmjs]

It's the other way around. We need to generate the did file in the build process before we can put it into the Wasm metadata. Currently, we can only emit Candid types at the runtime. That's why we need to either deploy the canister or compile them in a local environment to extract the did file. To do this statically, we need to modify the Rust compiler, which is not only a lot work, but also introduces a lot of security concerns.

I understand that these are issues, but I also know that Motoko doesn't force the user to write a Candid file by hand. Static analysis should be able to get the Candid out of the Rust canister without executing the canister correct?

All of the structs, enums, funcs, and canister methods are statically typed in Rust, so a tool should be able to create Candid from that without executing the Rust.