- Contact: Michael.Akridge@NOAA.gov
File copy tool will copy files and directories from one place to another.
- It uses a subprocess to call a windows robust file copy command
- The app will skip any existing files in a destination directory
- It will also run multi-threaded for performance
- If a copy process is interrupted, then simply run again since it also has the ability to restart the transfer.
- NOTE: Multiplatform versions available
- Lightweight,python based, tool to gather folder stats like name, path, and size
- Exports a CSV of folder stats
- Tool to convert Garmin GPS files(GPX) to standard CSV/TXT file format
- Tool to batch convert HEIC/HEIF files to standard JPG file format
- Tool to Verify or Generate Archive Manifest Files
- Required when sending archive packages
- A separate manifest file is required for every file that is transferred
- A manifest file contains three text values comma delimited on one line with no spaces for each submitted file.
<file_name>,<file_md5_checksum>,<file_size_in_bytes>
- The manifest file name pattern is the name of the associated data file with an added '.mnf'.
<file_name>.mnf
- Send2NCEI (S2N) is an archiving tool that allows you to easily submit your data files and related documentation to the National Centers for Environmental Information for long term preservation, stewardship, and access.
- https://www.ncei.noaa.gov/archive/send2ncei/
- The Advanced Tracking and Resource tool for Archive Collections (ATRAC) provides a common interface for users to enter and display information on archiving projects at the NOAA National Centers for Environmental Information (NCEI).
- https://www.ncdc.noaa.gov/atrac/guidelines.html
- PHP based tools. Data packager, bagit data packager, and submission manifest tools.
- placeholder - link coming soon - placeholder
- GSUTIL backend: https://cloud.google.com/storage/docs/gsutil_install
- GCP Storage Bucket Permissions
- Python 3 and various packages
- syncs local data to Google Cloud Platform Storage using Google's gsutil backend
- Customizable gsutil command generation
- Logging and output message management
- Configurable parameters including dry run, multi-threading, and recursion
- A graphical user interface (GUI) is built using the Gooey library.
- Users can interact with the GUI to specify the source and destination paths, adjust threading, select a dry run, and decide whether to print or run the gsutil command.
- The main function fetches user input, configures the logger, and executes the copy process or prints the gsutil command based on user preference.
gsutil -m rsync -r C:\destination_folder_path gs://<bucket_name>/<destination_folder_path>
#!/bin/bash
# List of files to upload
source_files=(
"C:\example\of\file\path\file.txt"
[ADD LIST of FILES HERE]
)
# Destination Google Cloud Storage bucket path
destination_bucket="gs://bucket_name/destination_folder_path..."
# Loop through the source file list and upload each file
for source_file in "${source_files[@]}"; do
gsutil -o "GSUtil:parallel_thread_count=12" -m cp "$source_file" "$destination_bucket"
done
breakdown of what each flag and option does
- gsutil: This is the command-line tool for interacting with Google Cloud Storage. https://cloud.google.com/storage/docs/gsutil
- -o "GSUtil:parallel_thread_count=12" if use multi-threading, limit bandwidth usage with this flag
- -m: This flag enables multi-threading, which speeds up the transfer by using multiple connections.
- rsync: This is the command to synchronize files and directories between a local and a remote location.
- -r: This flag tells rsync to perform a recursive copy of the entire directory tree.
- -n: This flag performs a "dry run," which means that the synchronization will be simulated, and no actual data will be transferred. This is useful to preview what would happen if the command were executed without actually making any changes.
- -x: This flag excludes files and directories that match any of the specified patterns using regular expressions.
- local_dir C:\nodd\test: This is the local directory that will be synchronized.
- cloud_dir gs://nmfs_odp_pifsc/PIFSC/ESD/ARP/test: This is the Google Cloud Storage bucket and destination path where the data will be synchronized.
The -x excluded patterns in detail we use are:
- -x "(^.(?<!.JPG)$)|(._archive.|._YEAR.*|.ISLAND.|.SITE-ID.|.SITE_PHOTOS.|.uncorrected.|.MISC.|.DARK.|.Products.)"
- (^.*(?<!.JPG)$)
-
- Matches any files that do not end with the ".JPG" extension.
- (._archive.|._YEAR.|.ISLAND.|.SITE-ID.|.SITE_PHOTOS.|.uncorrected.|.MISC.|.DARK.|.Products.)
-
- Matches any files or directories that contain any of the specified strings.
gsutil -m rsync -r gs://<source_bucket_name>/<source_folder_path> C:\destination_folder_path
NODD
PIFSC NODD
- https://console.cloud.google.com/welcome?project=nmfs-trusted-images
- Cloud Console URL
- GSUtil URL
-
- gs://nmfs_odp_pifsc
- Bucket API
NODD for other NMFS Centers:
- https://console.cloud.google.com/storage/browser/nmfs_odp_afsc
- https://console.cloud.google.com/storage/browser/nmfs_odp_swfsc
- https://console.cloud.google.com/storage/browser/nmfs_odp_nefsc
- https://console.cloud.google.com/storage/browser/nmfs_odp_nwfsc
- https://console.cloud.google.com/storage/browser/noaa-passive-bioacoustic
- https://console.cloud.google.com/storage/browser/noaa-nidis-drought-gov-data
Google Cloud SDK Docs
- https://cloud.google.com/sdk/docs
- https://cloud.google.com/storage/docs/discover-object-storage-gsutil
This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project content is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
See the LICENSE.md for details