diff --git a/README.md b/README.md index 0a55a81..f6051b4 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ # `ic-oss` + 🗂 A decentralized Object Storage Service on the Internet Computer. 💝 This project received a **$25k Developer Grant** from the [DFINITY Foundation](https://dfinity.org/grants). @@ -9,10 +10,12 @@ In decentralized enterprise applications, `ic-oss` will be an essential infrastructure. -`ic-oss` is a file infrastructure service, not a user-facing file product, but it will provide a simple management interface. +`ic-oss` is a file infrastructure service, not a user-facing product, but it will provide a simple management interface. > [!NOTE] -> `ic-oss` is in development and is not suitable for production use yet. +> The main functions of `ic-oss` have been developed, and the cluster management function is still under development (which will be completed soon). It can be used in the production environment. + +![IC-OSS](./ic-oss.webp) ## Features @@ -25,107 +28,27 @@ In decentralized enterprise applications, `ic-oss` will be an essential infrastr - [ ] Implements file encryption storage using ICP's vetKeys mechanism. - [ ] Integrates with external storage, supporting file storage in decentralized file services like IPFS and Arweave, with `ic-oss` managing file metadata. -## Running the project locally - -If you want to test your project locally, you can use the following commands: - -Deploy the bucket canister: -```bash -dfx canister create --specified-id mmrxu-fqaaa-aaaap-ahhna-cai ic_oss_bucket - -dfx deploy ic_oss_bucket --argument "(opt variant {Init = - record { - name = \"LDC Labs\"; - file_id = 0; - max_file_size = 0; - max_folder_depth = 10; - max_children = 1000; - visibility = 0; - max_custom_data_size = 4096; - enable_hash_index = true; - } -})" -# Output: -# ... -# Installing code for canister ic_oss_bucket, with canister ID mmrxu-fqaaa-aaaap-ahhna-cai -# Deployed canisters. -# URLs: -# Backend canister via Candid interface: -# ic_oss_bucket: http://127.0.0.1:4943/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=mmrxu-fqaaa-aaaap-ahhna-cai -``` - -Build cli tool: -```bash -cargo build -p ic-oss-cli - -# Run the cli tool -./target/debug/ic-oss-cli --help -./target/debug/ic-oss-cli identity --help -./target/debug/ic-oss-cli upload --help - -# Generate a new identity -./target/debug/ic-oss-cli identity --new --file myid.pem -# Output: -# principal: lxph3-nvpsv-yrevd-im4ug-qywcl-5ir34-rpsbs-6olvf-qtugo-iy5ai-jqe -# new identity: myid.pem -``` - -Add a manager to the bucket canister: -```bash -dfx canister call mmrxu-fqaaa-aaaap-ahhna-cai admin_set_managers '(vec {principal "lxph3-nvpsv-yrevd-im4ug-qywcl-5ir34-rpsbs-6olvf-qtugo-iy5ai-jqe"})' -``` - -Upload a file to the bucket canister: -```bash -./target/debug/ic-oss-cli -i myid.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file test.tar.gz -# Output: -# ... -# 2024-05-18 18:42:38 uploaded: 99.48% -# 2024-05-18 18:42:38 uploaded: 99.66% -# 2024-05-18 18:42:38 uploaded: 99.82% -# 2024-05-18 18:42:38 uploaded: 100.00% -# upload success, file id: 1, size: 147832281, chunks: 564, retry: 0, time elapsed: PT69.149941S -``` - -List files: -```bash -dfx canister call ic_oss_bucket list_files '(0, null, null, null)' -``` - -Get file info: -```bash -dfx canister call ic_oss_bucket get_file_info '(1, null)' -# Output: -# ( -# variant { -# Ok = record { -# id = 1 : nat32; -# parent = 0 : nat32; -# status = 0 : int8; -# updated_at = 1_716_028_957_265 : nat; -# hash = opt blob "\b7\bb\90\40\d6\44\79\a7\ca\56\c8\e0\3a\e2\da\dd\c8\19\85\9f\7b\85\84\88\c0\b9\98\ee\de\d6\de\de"; -# name = "test.tar.gz"; -# size = 147_832_281 : nat; -# content_type = "application/gzip"; -# created_at = 1_716_028_890_649 : nat; -# filled = 147_832_281 : nat; -# chunks = 564 : nat32; -# ert = null; -# } -# }, -# ) - -dfx canister call ic_oss_bucket get_file_info_by_hash '(blob "\b7\bb\90\40\d6\44\79\a7\ca\56\c8\e0\3a\e2\da\dd\c8\19\85\9f\7b\85\84\88\c0\b9\98\ee\de\d6\de\de", null)' -``` - -Delete file: -```bash -dfx canister call ic_oss_bucket delete_file '(1, null)' -``` - -Download the file in browser: -- by file id: `http://mmrxu-fqaaa-aaaap-ahhna-cai.localhost:4943/f/1` -- by file hash: `http://mmrxu-fqaaa-aaaap-ahhna-cai.localhost:4943/h/b7bb9040d64479a7ca56c8e03ae2daddc819859f7b858488c0b998eeded6dede` +## Libraries + +| Library | Description | +| :--------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------- | +| [ic_oss_bucket](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket) | An ICP smart contract and a storage bucket in the ic-oss cluster for storing files and folders. | +| [ic_oss_cluster](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cluster) | An ICP smart contract and the manager of the ic-oss cluster. | +| [ic-oss-can](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can) | A Rust library for implementing large file storage in ICP canisters. | +| [ic-oss-types](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_types) | A Rust types library used for integrating with ic-oss cluster. | +| [ic-oss-cose](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cose) | A Rust library based on COSE (RFC9052) and CWT (RFC8392) for issuing and verifying access tokens for the ic-oss cluster. | +| [ic-oss](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss) | The Rust version of the client SDK for the ic-oss cluster. | +| [ic-oss-cli](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cli) | A command-line tool implemented in Rust for the ic-oss cluster. | +| [examples/ai_canister](https://github.com/ldclabs/ic-oss/tree/main/examples/ai_canister) | A demonstration project used to show how to implement large file storage in the ICP canister by using `ic-oss-can`. | + +## Integration Workflow + +![IC-OSS Sequence](./ic-oss-sequence.webp) + +How to integrate `ic-oss`: +1. The backend of the Dapp calls the API of `ic_oss_cluster` to add access control policies for the user. +2. The frontend of the Dapp uses the `ic-oss-ts` SDK to obtain the `access_token` of the target `ic_oss_bucket` from `ic_oss_cluster`. +3. The frontend of the Dapp uses the `access_token` to call the API of the target `ic_oss_bucket` to operate the authorized files and folders. ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). diff --git a/examples/ai_canister/README.md b/examples/ai_canister/README.md index 1492ae8..f6e98a1 100644 --- a/examples/ai_canister/README.md +++ b/examples/ai_canister/README.md @@ -1,5 +1,11 @@ # Example: `ai_canister` +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ai_canister` is a demonstration project used to show how to implement large file storage in the ICP canister. By using `ic-oss-can` to include the `ic_oss_fs!` macro in your canister, an `fs` module and a set of Candid file system APIs will be automatically generated. You can use the `ic-oss-cli` tool to upload files to the ICP canister. + +For more information about `ic-oss-can`, please refer to [ic-oss-can](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can). + ## Running the project locally If you want to test your project locally, you can use the following commands: @@ -39,4 +45,4 @@ dfx canister call ai_canister list_files '(0, null, null, null)' ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. diff --git a/ic-oss-sequence.webp b/ic-oss-sequence.webp new file mode 100644 index 0000000..7cd1800 Binary files /dev/null and b/ic-oss-sequence.webp differ diff --git a/ic-oss.webp b/ic-oss.webp new file mode 100644 index 0000000..7549237 Binary files /dev/null and b/ic-oss.webp differ diff --git a/src/ic_oss/Cargo.toml b/src/ic_oss/Cargo.toml index 4b8562b..cc3be42 100644 --- a/src/ic_oss/Cargo.toml +++ b/src/ic_oss/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic-oss" -description = "A rust client of ic-oss." +description = "The Rust version of the client SDK for the ic-oss cluster." publish = true repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss" version.workspace = true diff --git a/src/ic_oss/README.md b/src/ic_oss/README.md index 9525e99..7fc420c 100644 --- a/src/ic_oss/README.md +++ b/src/ic_oss/README.md @@ -1,15 +1,15 @@ # `ic-oss` - ![License](https://img.shields.io/crates/l/ic-oss.svg) [![Crates.io](https://img.shields.io/crates/d/ic-oss.svg)](https://crates.io/crates/ic-oss) -[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml) +[![Test](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml) [![Docs.rs](https://img.shields.io/docsrs/ic-oss?label=docs.rs)](https://docs.rs/ic-oss) [![Latest Version](https://img.shields.io/crates/v/ic-oss.svg)](https://crates.io/crates/ic-oss) -A rust client of [ic-oss](https://github.com/ldclabs/ic-oss). -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ic-oss` is the Rust version of the client SDK for the ic-oss cluster. ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_bucket/Cargo.toml b/src/ic_oss_bucket/Cargo.toml index 5473a4a..d0f9229 100644 --- a/src/ic_oss_bucket/Cargo.toml +++ b/src/ic_oss_bucket/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic_oss_bucket" -description = "A bucket canister of ic-oss" +description = "An ICP smart contract and a storage bucket in the ic-oss cluster for storing files and folders." publish = false repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket" version.workspace = true diff --git a/src/ic_oss_bucket/README.md b/src/ic_oss_bucket/README.md index 4cf3dac..9ddb618 100644 --- a/src/ic_oss_bucket/README.md +++ b/src/ic_oss_bucket/README.md @@ -1,9 +1,48 @@ # `ic_oss_bucket` -[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml) +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ic_oss_bucket` is an ICP smart contract and a storage bucket in the `ic-oss` cluster for storing files and folders. The `ic-oss` cluster can deploy one or more buckets for horizontal file storage scaling, managed by `ic_oss_cluster`. + +## Features + +- [x] Supports large file uploads and downloads through file sharding, concurrent high-speed uploads, resumable uploads, and segmented downloads. +- [x] Provides data verification based on ICP's verification mechanisms to ensure file integrity during reading. +- [x] Supports file directory tree. +- [x] Access control with permissions for public, private, read-only, and write-only for files, folders, and buckets. + +## Candid API + +```shell +admin_set_auditors : (vec principal) -> (Result); +admin_set_managers : (vec principal) -> (Result); +admin_update_bucket : (UpdateBucketInput) -> (Result); +batch_delete_subfiles : (nat32, vec nat32, opt blob) -> (Result_1); +create_file : (CreateFileInput, opt blob) -> (Result_2); +create_folder : (CreateFolderInput, opt blob) -> (Result_2); +delete_file : (nat32, opt blob) -> (Result_3); +delete_folder : (nat32, opt blob) -> (Result_3); +get_bucket_info : (opt blob) -> (Result_4) query; +get_file_ancestors : (nat32, opt blob) -> (Result_5) query; +get_file_chunks : (nat32, nat32, opt nat32, opt blob) -> (Result_6) query; +get_file_info : (nat32, opt blob) -> (Result_7) query; +get_file_info_by_hash : (blob, opt blob) -> (Result_7) query; +get_folder_ancestors : (nat32, opt blob) -> (Result_5) query; +get_folder_info : (nat32, opt blob) -> (Result_8) query; +http_request : (HttpRequest) -> (HttpStreamingResponse) query; +list_files : (nat32, opt nat32, opt nat32, opt blob) -> (Result_9) query; +list_folders : (nat32, opt blob) -> (Result_10) query; +move_file : (MoveInput, opt blob) -> (Result_11); +move_folder : (MoveInput, opt blob) -> (Result_11); +update_file_chunk : (UpdateFileChunkInput, opt blob) -> (Result_12); +update_file_info : (UpdateFileInput, opt blob) -> (Result_11); +update_folder_info : (UpdateFolderInput, opt blob) -> (Result_11); +validate_admin_set_auditors : (vec principal) -> (Result) query; +validate_admin_set_managers : (vec principal) -> (Result) query; +validate_admin_update_bucket : (UpdateBucketInput) -> (Result) query; +``` -A bucket canister of [ic-oss](https://github.com/ldclabs/ic-oss). -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +The complete Candid API definition can be found in the [ic_oss_bucket.did](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket/ic_oss_bucket.did) file. ## Running locally @@ -28,8 +67,10 @@ MYID=$(dfx identity get-principal) ic-oss-cli -i debug/uploader.pem identity # principal: nprym-ylvyz-ig3fr-lgcmn-zzzt4-tyuix-3v6bm-fsel7-6lq6x-zh2w7-zqe +# add managers dfx canister call ic_oss_bucket admin_set_managers "(vec {principal \"$MYID\"; principal \"nprym-ylvyz-ig3fr-lgcmn-zzzt4-tyuix-3v6bm-fsel7-6lq6x-zh2w7-zqe\"})" +# add public keys to verify the access tokens dfx canister call ic_oss_bucket admin_update_bucket '(record { name = null; max_file_size = null; @@ -43,18 +84,25 @@ dfx canister call ic_oss_bucket admin_update_bucket '(record { trusted_eddsa_pub_keys = opt vec {vec {19; 152; 246; 44; 109; 26; 69; 124; 81; 186; 106; 75; 95; 61; 189; 47; 105; 252; 169; 50; 22; 33; 141; 200; 153; 126; 65; 107; 209; 125; 147; 202}}; }, null)' +# list files in the folder 2, with a access token dfx canister call ic_oss_bucket list_files '(2, null, null, opt blob "\84\44\a1\01\38\2e\a0\58\ac\a7\01\78\1b\61\6a\75\71\34\2d\72\75\61\61\61\2d\61\61\61\61\61\2d\71\61\61\67\61\2d\63\61\69\02\78\3f\7a\37\77\6a\70\2d\76\36\66\65\33\2d\6b\6b\73\75\35\2d\32\36\66\36\34\2d\64\65\64\74\77\2d\6a\37\6e\64\6a\2d\35\37\6f\6e\78\2d\71\67\61\36\63\2d\65\74\35\65\33\2d\6e\6a\78\35\33\2d\74\61\65\03\78\1b\6d\6d\72\78\75\2d\66\71\61\61\61\2d\61\61\61\61\70\2d\61\68\68\6e\61\2d\63\61\69\04\1a\66\8f\ce\68\05\1a\66\8f\c0\58\06\1a\66\8f\c0\58\09\78\18\46\6f\6c\64\65\72\2e\2a\3a\31\20\42\75\63\6b\65\74\2e\52\65\61\64\2e\2a\58\40\52\66\3e\e7\55\7e\99\2c\66\6d\65\56\54\9f\30\a1\2e\aa\56\69\66\b6\c6\e9\75\d7\c9\02\4c\24\1d\5d\7e\83\7d\c1\13\c6\00\91\56\d9\6a\ae\34\c3\a5\c9\b4\99\b3\47\b7\68\54\8d\dd\9c\9a\9b\a0\f9\1a\f5")' + +# list folders in the root folder 0 dfx canister call ic_oss_bucket list_folders '(0, null)' +# upload a file to the bucket ic-oss-cli -i debug/uploader.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file README.md +# read the file info dfx canister call ic_oss_bucket get_file_info '(1, null)' +# update the file 1's status to 0 dfx canister call ic_oss_bucket update_file_info "(record { id = 1; status = opt 0; }, null)" +# create a folder in the root folder dfx canister call ic_oss_bucket create_folder "(record { parent = 0; name = \"home\"; @@ -66,6 +114,7 @@ dfx canister call ic_oss_bucket create_folder "(record { name = \"jarvis\"; }, null)" +# move the file 1 from the root folder 0 to the folder 2 dfx canister call ic_oss_bucket move_file "(record { id = 1; from = 0; @@ -74,6 +123,7 @@ dfx canister call ic_oss_bucket move_file "(record { dfx canister call ic_oss_bucket list_files '(2, null, null, null)' +# delete the file 1 dfx canister call ic_oss_bucket delete_file '(1, null)' ``` @@ -85,4 +135,4 @@ Download the file in browser: ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_bucket/src/api_init.rs b/src/ic_oss_bucket/src/api_init.rs index 97d6a2a..261a8a8 100644 --- a/src/ic_oss_bucket/src/api_init.rs +++ b/src/ic_oss_bucket/src/api_init.rs @@ -12,14 +12,14 @@ pub enum CanisterArgs { #[derive(Clone, Debug, CandidType, Deserialize)] pub struct InitArgs { - name: String, - file_id: u32, - max_file_size: u64, - max_folder_depth: u8, - max_children: u16, - max_custom_data_size: u16, - enable_hash_index: bool, - visibility: u8, // 0: private; 1: public + name: String, // bucket name + file_id: u32, // the first file id, default is 0 + max_file_size: u64, // in bytes, default is 384GB + max_folder_depth: u8, // default is 10 + max_children: u16, // maximum number of subfolders and subfiles in a folder., default is 1000 + max_custom_data_size: u16, // in bytes, default is 4KB + enable_hash_index: bool, // if enabled, indexing will be built using file hash, allowing files to be read by their hash and preventing duplicate hash for files. default is false + visibility: u8, // 0: private; 1: public, can be accessed by anyone, default is 0 } #[derive(Clone, Debug, CandidType, Deserialize)] @@ -97,6 +97,10 @@ fn init(args: Option) { args.max_custom_data_size }; b.enable_hash_index = args.enable_hash_index; + + // The root folder 0 is created by default + b.folder_id = 1; + b.folder_count = 1; }); } CanisterArgs::Upgrade(_) => { diff --git a/src/ic_oss_bucket/src/api_query.rs b/src/ic_oss_bucket/src/api_query.rs index fd0baf7..78978e0 100644 --- a/src/ic_oss_bucket/src/api_query.rs +++ b/src/ic_oss_bucket/src/api_query.rs @@ -160,7 +160,7 @@ fn list_files( take: Option, access_token: Option, ) -> Result, String> { - let max_prev = store::state::with(|s| s.file_id).saturating_add(1); + let max_prev = store::state::with(|s| s.file_id); let prev = prev.unwrap_or(max_prev).min(max_prev); let take = take.unwrap_or(10).min(100); let canister = ic_cdk::id(); diff --git a/src/ic_oss_bucket/src/store.rs b/src/ic_oss_bucket/src/store.rs index 70a5dca..e5a63a6 100644 --- a/src/ic_oss_bucket/src/store.rs +++ b/src/ic_oss_bucket/src/store.rs @@ -815,7 +815,7 @@ pub mod fs { pub fn add_folder(metadata: FolderMetadata) -> Result { state::with_mut(|s| { FOLDERS.with(|r| { - let id = s.folder_id.saturating_add(1); + let id = s.folder_id; if id == u32::MAX { Err("folder id overflow".to_string())?; } @@ -828,7 +828,7 @@ pub mod fs { s.max_children as usize, )?; - s.folder_id = id; + s.folder_id = s.folder_id.saturating_add(1); s.folder_count += 1; Ok(id) }) @@ -838,7 +838,7 @@ pub mod fs { pub fn add_file(metadata: FileMetadata) -> Result { state::with_mut(|s| { FOLDERS.with(|r| { - let id = s.file_id.saturating_add(1); + let id = s.file_id; if id == u32::MAX { Err("file id overflow".to_string())?; } @@ -860,7 +860,7 @@ pub mod fs { } } - s.file_id = id; + s.file_id = s.file_id.saturating_add(1); s.file_count += 1; parent.files.insert(id); FS_METADATA_STORE.with(|r| r.borrow_mut().insert(id, metadata)); diff --git a/src/ic_oss_can/Cargo.toml b/src/ic_oss_can/Cargo.toml index 5d629ba..28e3a92 100644 --- a/src/ic_oss_can/Cargo.toml +++ b/src/ic_oss_can/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic-oss-can" -description = "A rust library of ic-oss for ICP canisters." +description = "A Rust library for implementing large file storage in ICP canisters" publish = true repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can" version.workspace = true diff --git a/src/ic_oss_can/README.md b/src/ic_oss_can/README.md index cb8a942..881f8d8 100644 --- a/src/ic_oss_can/README.md +++ b/src/ic_oss_can/README.md @@ -6,10 +6,81 @@ [![Docs.rs](https://img.shields.io/docsrs/ic-oss?label=docs.rs)](https://docs.rs/ic-oss) [![Latest Version](https://img.shields.io/crates/v/ic-oss.svg)](https://crates.io/crates/ic-oss) -A rust library of [ic-oss](https://github.com/ldclabs/ic-oss) for ICP canisters. -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ic-oss-can` is a Rust library for implementing large file storage in ICP canisters. By including the `ic_oss_fs!` macro in your canister, a `fs` module and a set of Candid filesystem APIs will be automatically generated. You can use the `ic-oss-cli` tool to upload files to the ICP canister. + +## Usage + +The following example is a minimal version using the `ic_oss_fs!` macro. Its only dependency is a thread-local constant named `FS_CHUNKS_STORE` of type `RefCell>`. + +```rust +use ic_stable_structures::{ + memory_manager::{MemoryId, MemoryManager, VirtualMemory}, + DefaultMemoryImpl, StableBTreeMap, +}; +use std::cell::RefCell; + +use ic_oss_can::ic_oss_fs; +use ic_oss_can::types::{Chunk, FileId, FileMetadata}; + +type Memory = VirtualMemory; + +const FS_DATA_MEMORY_ID: MemoryId = MemoryId::new(0); + +thread_local! { + + static MEMORY_MANAGER: RefCell> = + RefCell::new(MemoryManager::init(DefaultMemoryImpl::default())); + + + // `FS_CHUNKS_STORE`` is needed by `ic_oss_can::ic_oss_fs!` macro + static FS_CHUNKS_STORE: RefCell> = RefCell::new( + StableBTreeMap::init( + MEMORY_MANAGER.with_borrow(|m| m.get(FS_DATA_MEMORY_ID)), + ) + ); +} + +// need to define `FS_CHUNKS_STORE` before `ic_oss_can::ic_oss_fs!()` +ic_oss_fs!(); +``` + +For a more complete example, refer to [examples/ai_canister](https://github.com/ldclabs/ic-oss/tree/main/examples/ai_canister). + +### FS Module + +```rust +fs::set_max_file_size(size: u64); +fs::set_visibility(visibility: u8); +fs::set_managers(managers: BTreeSet); +fs::is_manager(caller: &Principal) -> bool; +fs::with(f: impl FnOnce(&Files) -> R) -> R; +fs::load(); +fs::save(); +fs::get_file(id: u32) -> Option; +fs::list_files(prev: u32, take: u32) -> Vec; +fs::add_file(file: FileMetadata) -> Result; +fs::update_file(change: UpdateFileInput, now_ms: u64) -> Result<(), String>; +fs::get_chunk(id: u32, chunk_index: u32) -> Option; +fs::get_full_chunks(id: u32) -> Result, String>; +fs::update_chunk(id: u32, chunk_index: u32, now_ms: u64, chunk: Vec) -> Result; +fs::delete_file(id: u32) -> Result; +``` + +### FS Candid API + +```shell +create_file : (CreateFileInput, opt blob) -> (Result_2); +delete_file : (nat32, opt blob) -> (Result_3); +list_files : (nat32, opt nat32, opt nat32, opt blob) -> (Result_4) query; +update_file_chunk : (UpdateFileChunkInput, opt blob) -> (Result_6); +update_file_info : (UpdateFileInput, opt blob) -> (Result_7); +``` + +The complete module API Candid API definition can be found in the [store.rs](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_can/src/store.rs) file. ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. diff --git a/src/ic_oss_can/src/lib.rs b/src/ic_oss_can/src/lib.rs index e4bc569..05e2619 100644 --- a/src/ic_oss_can/src/lib.rs +++ b/src/ic_oss_can/src/lib.rs @@ -1,2 +1,87 @@ pub mod store; pub mod types; + +#[cfg(test)] +mod test { + + use ic_stable_structures::{ + memory_manager::{MemoryId, MemoryManager, VirtualMemory}, + DefaultMemoryImpl, StableBTreeMap, + }; + use std::cell::RefCell; + + use crate::ic_oss_fs; + use crate::types::{Chunk, FileId, FileMetadata}; + + type Memory = VirtualMemory; + + const FS_DATA_MEMORY_ID: MemoryId = MemoryId::new(0); + + thread_local! { + + static MEMORY_MANAGER: RefCell> = + RefCell::new(MemoryManager::init(DefaultMemoryImpl::default())); + + + // `FS_CHUNKS_STORE`` is needed by `ic_oss_can::ic_oss_fs` macro + static FS_CHUNKS_STORE: RefCell> = RefCell::new( + StableBTreeMap::init( + MEMORY_MANAGER.with_borrow(|m| m.get(FS_DATA_MEMORY_ID)), + ) + ); + } + + // need to define `FS_CHUNKS_STORE` before `ic_oss_can::ic_oss_fs!()` + ic_oss_fs!(); + + #[test] + fn test_ic_oss_fs() { + fs::set_max_file_size(1024 * 1024); + + let files = fs::list_files(u32::MAX, 2); + assert!(files.is_empty()); + + fs::add_file(FileMetadata { + name: "f1".to_string(), + size: 100, + ..Default::default() + }) + .unwrap(); + + assert!(fs::get_file(0).is_none()); + assert_eq!(fs::get_file(1).unwrap().name, "f1"); + + fs::add_file(FileMetadata { + name: "f2".to_string(), + size: 100, + ..Default::default() + }) + .unwrap(); + + fs::add_file(FileMetadata { + name: "f3".to_string(), + size: 100, + ..Default::default() + }) + .unwrap(); + + fs::add_file(FileMetadata { + name: "f4".to_string(), + size: 100, + ..Default::default() + }) + .unwrap(); + + let files = fs::list_files(u32::MAX, 2); + assert_eq!( + files.iter().map(|f| f.name.clone()).collect::>(), + vec!["f4", "f3"] + ); + + let files = fs::list_files(files.last().unwrap().id, 10); + assert_eq!( + files.iter().map(|f| f.name.clone()).collect::>(), + vec!["f2", "f1"] + ); + } +} diff --git a/src/ic_oss_can/src/store.rs b/src/ic_oss_can/src/store.rs index b1e808c..3b872fe 100644 --- a/src/ic_oss_can/src/store.rs +++ b/src/ic_oss_can/src/store.rs @@ -7,10 +7,7 @@ macro_rules! ic_oss_fs { pub mod fs { use candid::Principal; use ciborium::{from_reader, into_writer}; - use ic_oss_types::{ - crc32, - file::{FileChunk, FileInfo, UpdateFileInput, CHUNK_SIZE}, - }; + use ic_oss_types::file::{FileChunk, FileInfo, UpdateFileInput, CHUNK_SIZE}; use serde_bytes::ByteBuf; use std::{cell::RefCell, collections::BTreeSet, ops}; @@ -69,25 +66,14 @@ macro_rules! ic_oss_fs { } pub fn get_file(id: u32) -> Option { + if id == 0 { + return None; + } FS_METADATA.with(|r| r.borrow().files.get(&id).cloned()) } pub fn list_files(prev: u32, take: u32) -> Vec { - FS_METADATA.with(|r| { - let m = r.borrow(); - let mut res = Vec::with_capacity(take as usize); - let mut id = prev.saturating_sub(1); - while id > 0 { - if let Some(file) = m.files.get(&id) { - res.push(file.clone().into_info(id)); - if res.len() >= take as usize { - break; - } - } - id = id.saturating_sub(1); - } - res - }) + FS_METADATA.with(|r| r.borrow().list_files(prev, take)) } pub fn add_file(file: FileMetadata) -> Result { @@ -109,6 +95,9 @@ macro_rules! ic_oss_fs { } pub fn update_file(change: UpdateFileInput, now_ms: u64) -> Result<(), String> { + if change.id == 0 { + Err("invalid file id".to_string())?; + } with_mut(|r| match r.files.get_mut(&change.id) { None => Err(format!("file not found: {}", change.id)), Some(file) => { @@ -132,6 +121,9 @@ macro_rules! ic_oss_fs { } pub fn get_chunk(id: u32, chunk_index: u32) -> Option { + if id == 0 { + return None; + } FS_CHUNKS_STORE.with(|r| { r.borrow() .get(&FileId(id, chunk_index)) @@ -140,6 +132,9 @@ macro_rules! ic_oss_fs { } pub fn get_full_chunks(id: u32) -> Result, String> { + if id == 0 { + Err("invalid file id".to_string())?; + } let (size, chunks) = with(|r| match r.files.get(&id) { None => Err(format!("file not found: {}", id)), Some(file) => { @@ -181,15 +176,19 @@ macro_rules! ic_oss_fs { now_ms: u64, chunk: Vec, ) -> Result { + if file_id == 0 { + Err("invalid file id".to_string())?; + } + if chunk.is_empty() { - return Err("empty chunk".to_string()); + Err("empty chunk".to_string())?; } if chunk.len() > CHUNK_SIZE as usize { - return Err(format!( + Err(format!( "chunk size too large, max size is {} bytes", CHUNK_SIZE - )); + ))?; } with_mut(|r| match r.files.get_mut(&file_id) { @@ -226,6 +225,10 @@ macro_rules! ic_oss_fs { } pub fn delete_file(id: u32) -> Result { + if id == 0 { + Err("invalid file id".to_string())?; + } + with_mut(|r| match r.files.remove(&id) { Some(file) => { FS_CHUNKS_STORE.with(|r| { @@ -244,7 +247,6 @@ macro_rules! ic_oss_fs { pub mod api { use ic_oss_types::{crc32, file::*}; use serde_bytes::ByteBuf; - use std::ops; use super::fs; use $crate::types::*; diff --git a/src/ic_oss_can/src/types.rs b/src/ic_oss_can/src/types.rs index 9a8f249..705488b 100644 --- a/src/ic_oss_can/src/types.rs +++ b/src/ic_oss_can/src/types.rs @@ -6,6 +6,7 @@ use serde::{Deserialize, Serialize}; use std::{ borrow::Cow, collections::{BTreeMap, BTreeSet}, + ops, }; pub const MILLISECONDS: u64 = 1_000_000_000; @@ -20,6 +21,26 @@ pub struct Files { pub files: BTreeMap, } +impl Files { + pub fn list_files(&self, prev: u32, take: u32) -> Vec { + let mut res = Vec::with_capacity(take as usize); + for (file_id, file) in self + .files + .range(ops::Range { + start: 1, + end: prev, + }) + .rev() + { + res.push(file.clone().into_info(*file_id)); + if res.len() >= take as usize { + break; + } + } + res + } +} + impl Storable for Files { const BOUND: Bound = Bound::Unbounded; diff --git a/src/ic_oss_cli/Cargo.toml b/src/ic_oss_cli/Cargo.toml index 5e61e1f..0d14964 100644 --- a/src/ic_oss_cli/Cargo.toml +++ b/src/ic_oss_cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic-oss-cli" -description = "A CLI tool of ic-oss" +description = "A command-line tool implemented in Rust for the ic-oss cluster." publish = true repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cli" version.workspace = true diff --git a/src/ic_oss_cli/README.md b/src/ic_oss_cli/README.md index 315612e..4a46415 100644 --- a/src/ic_oss_cli/README.md +++ b/src/ic_oss_cli/README.md @@ -1,7 +1,13 @@ # `ic-oss-cli` -> A CLI tool of ic-oss +![License](https://img.shields.io/crates/l/ic-oss-cli.svg) +[![Crates.io](https://img.shields.io/crates/d/ic-oss-cli.svg)](https://crates.io/crates/ic-oss-cli) +[![Test](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml) +[![Docs.rs](https://img.shields.io/docsrs/ic-oss-cli?label=docs.rs)](https://docs.rs/ic-oss-cli) +[![Latest Version](https://img.shields.io/crates/v/ic-oss-cli.svg)](https://crates.io/crates/ic-oss-cli) -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. It provides a simple and efficient way to store and retrieve files, supports large files, and offers unlimited horizontal scalability. It can serve as a reliable decentralized file infrastructure for NFT, chain blocks, verifiable credentials, blogs, documents, knowledge bases, games and other decentralized applications. +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ic-oss-cli` is a command-line tool implemented in Rust for the `ic-oss` cluster. ## Usage @@ -20,12 +26,27 @@ ic-oss-cli identity --new --file myid.pem # new identity: myid.pem ``` +Build from source: +```sh +git clone https://github.com/ldclabs/ic-oss.git +cd ic-oss +# build +cargo build -p ic-oss-cli --release +# get help info +target/release/ic-oss-cli --help +``` + Upload a file to the local canister: ```sh -ic-oss-cli -i myid.pem upload -b bkyz2-fmaaa-aaaaa-qaaaq-cai --file test.tar.gz +ic-oss-cli -i myid.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file test.tar.gz ``` Upload a file to the mainnet canister: ```sh -ic-oss-cli -i myid.pem upload -b bkyz2-fmaaa-aaaaa-qaaaq-cai --file test.tar.gz --ic -``` \ No newline at end of file +ic-oss-cli -i myid.pem upload -b mmrxu-fqaaa-aaaap-ahhna-cai --file test.tar.gz --ic +``` + +## License +Copyright © 2024 [LDC Labs](https://github.com/ldclabs). + +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_cluster/Cargo.toml b/src/ic_oss_cluster/Cargo.toml index ac7b4ed..03c8b49 100644 --- a/src/ic_oss_cluster/Cargo.toml +++ b/src/ic_oss_cluster/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic_oss_cluster" -description = "A cluster canister of ic-oss" +description = "An ICP smart contract and the manager of the ic-oss cluster." publish = false repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cluster" version.workspace = true diff --git a/src/ic_oss_cluster/README.md b/src/ic_oss_cluster/README.md index ed364d7..6f315d4 100644 --- a/src/ic_oss_cluster/README.md +++ b/src/ic_oss_cluster/README.md @@ -1,9 +1,29 @@ -# `ic_oss_cluster` +# `ic_oss_cluster` (WIP) -[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml) +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. -A cluster canister of [ic-oss](https://github.com/ldclabs/ic-oss). -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +`ic_oss_cluster` is an ICP smart contract and the manager of the `ic-oss` cluster. + +## Features + +- [x] Manages access control policies and issue access tokens for users. +- [ ] Manages `ic_oss_bucket` smart contract versions, including deploying and upgrading buckets. +- [ ] Manages associated keys for file data encryption. +- [ ] Manages extension schemas for integrating external file systems. + +## Candid API + +```shell +access_token : (principal) -> (Result); +admin_attach_policies : (Token) -> (Result_1); +admin_detach_policies : (Token) -> (Result_1); +admin_set_managers : (vec principal) -> (Result_1); +admin_sign_access_token : (Token) -> (Result); +get_cluster_info : () -> (Result_2) query; +validate_admin_set_managers : (vec principal) -> (Result_1) query; +``` + +The complete Candid API definition can be found in the [ic_oss_cluster.did](https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_bucket/ic_oss_cluster.did) file. ## Running locally @@ -11,24 +31,45 @@ Deploy to local network: ```bash dfx deploy ic_oss_cluster --argument "(opt variant {Init = record { + name = \"LDC Labs\"; ecdsa_key_name = \"dfx_test_key\"; + token_expiration = 3600; } })" -dfx canister call ic_oss_cluster get_state '(null)' +dfx canister call ic_oss_cluster get_cluster_info '()' MYID=$(dfx identity get-principal) +# add managers dfx canister call ic_oss_cluster admin_set_managers "(vec {principal \"$MYID\"})" +# sign a access token dfx canister call ic_oss_cluster admin_sign_access_token '(record { subject = principal "z7wjp-v6fe3-kksu5-26f64-dedtw-j7ndj-57onx-qga6c-et5e3-njx53-tae"; audience = principal "mmrxu-fqaaa-aaaap-ahhna-cai"; scope = "Folder.*:1 Bucket.Read.*"; })' + +# attach policies +dfx canister call ic_oss_cluster admin_attach_policies '(record { + subject = principal "z7wjp-v6fe3-kksu5-26f64-dedtw-j7ndj-57onx-qga6c-et5e3-njx53-tae"; + audience = principal "mmrxu-fqaaa-aaaap-ahhna-cai"; + scope = "Folder.* Bucket.List.*"; +})' + +# detach policies +dfx canister call ic_oss_cluster admin_detach_policies '(record { + subject = principal "z7wjp-v6fe3-kksu5-26f64-dedtw-j7ndj-57onx-qga6c-et5e3-njx53-tae"; + audience = principal "mmrxu-fqaaa-aaaap-ahhna-cai"; + scope = "Folder.*:1"; +})' + +# get access token for a audience +dfx canister call ic_oss_cluster access_token '(principal "mmrxu-fqaaa-aaaap-ahhna-cai")' ``` ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_cose/Cargo.toml b/src/ic_oss_cose/Cargo.toml index d5e1327..3ce482d 100644 --- a/src/ic_oss_cose/Cargo.toml +++ b/src/ic_oss_cose/Cargo.toml @@ -1,7 +1,7 @@ [package] name = "ic-oss-cose" -description = "cose of ic-oss" -publish = false # wait coset +description = "A Rust library based on COSE (RFC9052) and CWT (RFC8392) for issuing and verifying access tokens for the ic-oss cluster." +publish = false # wait coset repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_cose" version.workspace = true edition.workspace = true diff --git a/src/ic_oss_cose/README.md b/src/ic_oss_cose/README.md index fe83fe2..98f7f4c 100644 --- a/src/ic_oss_cose/README.md +++ b/src/ic_oss_cose/README.md @@ -1,15 +1,15 @@ # `ic-oss-cose` +![License](https://img.shields.io/crates/l/ic-oss-cose.svg) +[![Crates.io](https://img.shields.io/crates/d/ic-oss-cose.svg)](https://crates.io/crates/ic-oss-cose) +[![Test](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml) +[![Docs.rs](https://img.shields.io/docsrs/ic-oss-cose?label=docs.rs)](https://docs.rs/ic-oss-cose) +[![Latest Version](https://img.shields.io/crates/v/ic-oss-cose.svg)](https://crates.io/crates/ic-oss-cose) -![License](https://img.shields.io/crates/l/ic-oss-types.svg) -[![Crates.io](https://img.shields.io/crates/d/ic-oss-types.svg)](https://crates.io/crates/ic-oss-types) -[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml) -[![Docs.rs](https://img.shields.io/docsrs/ic-oss-types?label=docs.rs)](https://docs.rs/ic-oss-types) -[![Latest Version](https://img.shields.io/crates/v/ic-oss-types.svg)](https://crates.io/crates/ic-oss-types) +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. -Types of [ic-oss](https://github.com/ldclabs/ic-oss). -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +`ic-oss-cose` is a Rust library based on COSE (RFC9052) and CWT (RFC8392) for issuing and verifying access tokens for the `ic-oss` cluster. ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_types/Cargo.toml b/src/ic_oss_types/Cargo.toml index 7656e97..92afd1b 100644 --- a/src/ic_oss_types/Cargo.toml +++ b/src/ic_oss_types/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "ic-oss-types" -description = "types of ic-oss" +description = "A Rust types library used for integrating with ic-oss cluster." publish = true repository = "https://github.com/ldclabs/ic-oss/tree/main/src/ic_oss_types" version.workspace = true diff --git a/src/ic_oss_types/README.md b/src/ic_oss_types/README.md index 0eb6030..c344299 100644 --- a/src/ic_oss_types/README.md +++ b/src/ic_oss_types/README.md @@ -1,15 +1,15 @@ # `ic-oss-types` - ![License](https://img.shields.io/crates/l/ic-oss-types.svg) [![Crates.io](https://img.shields.io/crates/d/ic-oss-types.svg)](https://crates.io/crates/ic-oss-types) -[![CI](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/ci.yml) +[![Test](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml/badge.svg)](https://github.com/ldclabs/ic-oss/actions/workflows/test.yml) [![Docs.rs](https://img.shields.io/docsrs/ic-oss-types?label=docs.rs)](https://docs.rs/ic-oss-types) [![Latest Version](https://img.shields.io/crates/v/ic-oss-types.svg)](https://crates.io/crates/ic-oss-types) -Types of [ic-oss](https://github.com/ldclabs/ic-oss). -`ic-oss` is a fully open-source decentralized object storage service running on the Internet Computer. +[ic-oss](https://github.com/ldclabs/ic-oss) is a decentralized Object Storage Service on the Internet Computer. + +`ic-oss-type` is a Rust types library used for integrating with [ic-oss](https://github.com/ldclabs/ic-oss) cluster. ## License Copyright © 2024 [LDC Labs](https://github.com/ldclabs). -`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text. \ No newline at end of file +`ldclabs/ic-oss` is licensed under the MIT License. See [LICENSE](../../LICENSE-MIT) for the full license text. \ No newline at end of file diff --git a/src/ic_oss_types/src/file.rs b/src/ic_oss_types/src/file.rs index 4bed5d7..96f153f 100644 --- a/src/ic_oss_types/src/file.rs +++ b/src/ic_oss_types/src/file.rs @@ -8,8 +8,8 @@ use url::Url; use crate::{ByteN, MapValue}; pub const CHUNK_SIZE: u32 = 256 * 1024; -pub const MAX_FILE_SIZE: u64 = 384 * 1024 * 1024 * 1024; // 384G -pub const MAX_FILE_SIZE_PER_CALL: u64 = 1024 * 2000; // should less than 2M +pub const MAX_FILE_SIZE: u64 = 384 * 1024 * 1024 * 1024; // 384GB +pub const MAX_FILE_SIZE_PER_CALL: u64 = 1024 * 2000; // should less than 2MB #[derive(CandidType, Clone, Debug, Default, Deserialize, Serialize, PartialEq, Eq)] pub struct FileInfo {