Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "merge --input-file-list NAME ..." as a way to exceed CLI limits #328

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,8 @@ Usage:

Options:
--input-files <input-files> Input BOM filenames (separate filenames with a space).
--input-files-list <input-files-list-files> One or more text file(s) with input BOM filenames (one per line).
--input-files-nul-list <input-files-list-files> One or more text-like file(s) with input BOM filenames (separated by 0x00 characters).
--output-file <output-file> Output BOM filename, will write to stdout if no value provided.
--input-format <autodetect|json|protobuf|xml> Specify input file format.
--output-format <autodetect|json|protobuf|xml> Specify output file format.
Expand All @@ -205,6 +207,24 @@ Options:
Note: To perform a hierarchical merge all BOMs need the subject of the BOM
described in the metadata component element.

The `--input-files-list` option can be useful if you have so many filenames to
merge that your shell interpreter command-line limit is exceeded if you list
them all as `--input-files`, or if your path names have spaces.

The related `--input-files-nul-list` is intended for lists prepared by commands
like `find ... -print0` and makes sense on filesystems where carriage-return
and/or line-feed characters may validly be present in a path name component.
Note: behavior with multi-byte encodings (Unicode family) where a 0x00 byte
can be part of a character may be undefined.

If you specify several of these options, the effective file lists will be
concatenated before the actual merge (first the individual `--input-files`,
then the contents of `--input-files-list`, and finally the contents of
`--input-files-nul-list`). If you have a document crafted to describe the
root of your product hierarchy tree, it is recommended to list it as the
first of individual `--input-files` (or otherwise on first line among used
lists).

### Examples

Merge two XML formatted BOMs:
Expand Down
54 changes: 53 additions & 1 deletion src/cyclonedx/Commands/MergeCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
using System.Threading.Tasks;
using CycloneDX.Models;
using CycloneDX.Utils;
using System.IO;
using System.Collections.Immutable;

namespace CycloneDX.Cli.Commands
{
Expand All @@ -32,6 +34,8 @@ public static void Configure(RootCommand rootCommand)
Contract.Requires(rootCommand != null);
var subCommand = new Command("merge", "Merge two or more BOMs");
subCommand.Add(new Option<List<string>>("--input-files", "Input BOM filenames (separate filenames with a space)."));
subCommand.Add(new Option<List<string>>("--input-files-list", "One or more text file(s) with input BOM filenames (one per line)."));
subCommand.Add(new Option<List<string>>("--input-files-nul-list", "One or more text-like file(s) with input BOM filenames (separated by 0x00 characters)."));
subCommand.Add(new Option<string>("--output-file", "Output BOM filename, will write to stdout if no value provided."));
subCommand.Add(new Option<CycloneDXBomFormat>("--input-format", "Specify input file format."));
subCommand.Add(new Option<CycloneDXBomFormat>("--output-format", "Specify output file format."));
Expand Down Expand Up @@ -61,7 +65,55 @@ public static async Task<int> Merge(MergeCommandOptions options)
return (int)ExitCode.ParameterValidationError;
}

var inputBoms = await InputBoms(options.InputFiles, options.InputFormat, outputToConsole).ConfigureAwait(false);
List<string> InputFiles;
jimklimov marked this conversation as resolved.
Show resolved Hide resolved
if (options.InputFiles != null)
{
InputFiles = (List<string>)options.InputFiles;
}
else
{
InputFiles = new List<string>();
}

Console.WriteLine($"Got " + InputFiles.Count + " individual input file name(s): ['" + string.Join("', '", InputFiles) + "']");
if (options.InputFilesList != null)
{
// For some reason, without an immutable list this claims
// modifications of the iterable during iteration and fails:
ImmutableList<string> InputFilesList = options.InputFilesList.ToImmutableList();
Console.WriteLine($"Got " + InputFilesList.Count + " file(s) with actual input file names: ['" + string.Join("', '", InputFilesList) + "']");
foreach (string OneInputFileList in InputFilesList)
{
Console.WriteLine($"Adding to input file list from " + OneInputFileList);
string[] lines = File.ReadAllLines(OneInputFileList);
InputFiles.AddRange(lines);
Console.WriteLine($"Got " + lines.Length + " entries from " + OneInputFileList);
}
}

if (options.InputFilesNulList != null)
{
ImmutableList<string> InputFilesNulList = options.InputFilesNulList.ToImmutableList();
Console.WriteLine($"Got " + InputFilesNulList.Count + " file(s) with NUL-separated actual input file names: ['" + string.Join("', '", InputFilesNulList) + "']");
foreach (string OneInputFileList in InputFilesNulList)
{
Console.WriteLine($"Adding to input file list from " + OneInputFileList);
string[] lines = File.ReadAllText(OneInputFileList).Split('\0');
InputFiles.AddRange(lines);
Console.WriteLine($"Got " + lines.Length + " entries from " + OneInputFileList);
}
}

// TODO: Consider InputFiles.Distinct().ToList() -
// but that requires C# 3.0 for extension method support,
// and .NET 3.5 to get the LINQ Enumerable class.
if (InputFiles.Count == 0)
{
// Revert to legacy (error-handling) behavior below
// in case the parameter was not passed
InputFiles = null;
}
var inputBoms = await InputBoms(InputFiles, options.InputFormat, outputToConsole).ConfigureAwait(false);

Component bomSubject = null;
if (options.Group != null || options.Name != null || options.Version != null)
Expand Down
4 changes: 3 additions & 1 deletion src/cyclonedx/Commands/MergeCommandOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ namespace CycloneDX.Cli.Commands
public class MergeCommandOptions
{
public IList<string> InputFiles { get; set; }
public IList<string> InputFilesList { get; set; }
public IList<string> InputFilesNulList { get; set; }
public string OutputFile { get; set; }
public CycloneDXBomFormat InputFormat { get; set; }
public CycloneDXBomFormat OutputFormat { get; set; }
Expand All @@ -29,4 +31,4 @@ public class MergeCommandOptions
public string Name { get; set; }
public string Version { get; set; }
}
}
}