Skip to content

Commit

Permalink
Apply standardrb, mandate Ruby 2.6
Browse files Browse the repository at this point in the history
standard is by now much more widespread than wetransfer_style.
The rubocop config of wetransfer_style was a consensus config and
has been bikeshedded over a lot, and changing it is unlikely in
the future. Using a standard (sic!) format will make the barrier to
contribution smaller. We also need to bump to 2.6 for running CI easier.

Installing older Rubies is hard, and this is, after all, a "new" gem.
  • Loading branch information
julik committed Feb 29, 2024
1 parent d2801df commit cfb053b
Show file tree
Hide file tree
Showing 58 changed files with 1,459 additions and 1,462 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ jobs:
matrix:
ruby:
- '3.3'
- '2.1'
- '2.5'
experimental: [false]
steps:
- name: Checkout
Expand Down
15 changes: 0 additions & 15 deletions .rubocop.yml

This file was deleted.

5 changes: 5 additions & 0 deletions .standard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ruby_version: 2.7
ignore:
- 'spec/**/*':
- Lint/ConstantDefinitionInBlock
- Style/GlobalVars
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

* Rebirth as zip_kit
* Adopt MIT license. The changes from 5.x get grandfathered in. The base for the fork is the 4.x version which was still MIT-licensed.
* Bump minimum Ruby version to 2.6
* Respond to `#write` in all objects that respond to `#<<`, because they should be usable with `IO.copy_stream`
* Allow the last file to be suppressed in the central directory via Streamer#rollback!
* Allow heuristic compression. Use `Streamer#write_file` to let zip_kit pick the right compression method for you. If a file will benefit from
Expand Down
2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
source 'https://rubygems.org'
source "https://rubygems.org"

# Specify your gem's dependencies in zip_kit.gemspec
gemspec
4 changes: 1 addition & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,7 @@ of unarchiving end-user applications and is well tested.

## Requirements

Ruby 2.1+ syntax support (keyword arguments with defaults) and a working zlib (all available to jRuby as well).
jRuby might experience problems when using the reader methods due to the argument of `IO#seek` being limited
to [32 bit sizes.](https://github.com/jruby/jruby/issues/3817)
Ruby 2.6+ syntax support is required, as well as a a working zlib (all available to jRuby as well).

## Diving in: send some large CSV reports from Rails

Expand Down
21 changes: 14 additions & 7 deletions Rakefile
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
require 'bundler/gem_tasks'
require 'rspec/core/rake_task'
require 'yard'
require 'rubocop/rake_task'
# frozen_string_literal: true

require "bundler/gem_tasks"
require "rspec/core/rake_task"
require "yard"
require "rubocop/rake_task"
require "standard/rake"

task :format do
`bundle exec standardrb --fix-unsafely`
`bundle exec magic_frozen_string_literal ./lib`
end

YARD::Rake::YardocTask.new(:doc) do |t|
# The dash has to be between the two to "divide" the source files and
# miscellaneous documentation files that contain no code
t.files = ['lib/**/*.rb', '-', 'LICENSE.txt', 'IMPLEMENTATION_DETAILS.md']
t.files = ["lib/**/*.rb", "-", "LICENSE.txt", "IMPLEMENTATION_DETAILS.md"]
end

RSpec::Core::RakeTask.new(:spec)
RuboCop::RakeTask.new(:rubocop)
task default: [:spec, :rubocop]
task default: [:spec, :standard]
10 changes: 4 additions & 6 deletions bench/buffered_crc32_bench.rb
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
require 'bundler'
require "bundler"
Bundler.setup

require 'benchmark'
require 'benchmark/ips'
require_relative '../lib/zip_kit'
require "benchmark"
require "benchmark/ips"
require_relative "../lib/zip_kit"

n_bytes = 5 * 1024 * 1024
r = Random.new
Expand All @@ -24,8 +24,6 @@
2 * 1024 * 1024
]

require 'benchmark/ips'

Benchmark.ips do |x|
x.config(time: 5, warmup: 2)
buffer_sizes.each do |buf_size|
Expand Down
10 changes: 5 additions & 5 deletions examples/archive_size_estimate.rb
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# frozen_string_literal: true

require_relative '../lib/zip_kit'
require_relative "../lib/zip_kit"

# Predict how large a ZIP file is going to be without having access to
# the actual file contents, but using just the filenames (influences the
# file size) and the size of the files
zip_archive_size_in_bytes = ZipKit::SizeEstimator.estimate do |zip|
zip_archive_size_in_bytes = ZipKit::SizeEstimator.estimate { |zip|
# Pretend we are going to make a ZIP file which contains a few
# MP4 files (those do not compress all too well)
zip.add_stored_entry(filename: 'MOV_1234.MP4', size: 898_090)
zip.add_stored_entry(filename: 'MOV_1235.MP4', size: 7_855_126)
end
zip.add_stored_entry(filename: "MOV_1234.MP4", size: 898_090)
zip.add_stored_entry(filename: "MOV_1235.MP4", size: 7_855_126)
}

puts zip_archive_size_in_bytes #=> 8_753_467
2 changes: 1 addition & 1 deletion examples/config.ru
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require File.dirname(__FILE__) + '/rack_application.rb'
require File.dirname(__FILE__) + "/rack_application.rb"

# Demonstrates a Rack app that can offer a ZIP download composed
# at runtime (see rack_application.rb)
Expand Down
20 changes: 10 additions & 10 deletions examples/deferred_write.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require_relative '../lib/zip_kit'
require_relative "../lib/zip_kit"

# Using deferred writes (when you want to "pull" from a Streamer)
# is also possible with ZipKit.
Expand All @@ -11,14 +11,14 @@
#
# Let's make a OutputEnumerator that writes a few files with random content. Note that when you create
# that body it does not immediately write the ZIP:
iterable = ZipKit::Streamer.output_enum do |zip|
iterable = ZipKit::Streamer.output_enum { |zip|
(1..5).each do |i|
zip.write_stored_file('random_%d04d.bin' % i) do |sink|
zip.write_stored_file("random_%d04d.bin" % i) do |sink|
warn "Starting on file #{i}...\n"
sink << Random.new.bytes(1024)
end
end
end
}

warn "\n\nOutput using #each"

Expand All @@ -29,7 +29,7 @@
# the output of the block within OutputEnumerator is interspersed with the stuff
# being yielded to each():
iterable.each do |_binary_string|
$stderr << '.'
$stderr << "."
end

warn "\n\nOutput Enumerator returned from #each"
Expand All @@ -40,19 +40,19 @@
# we find necessary:
enum = iterable.each
15.times do
_bin_str = enum.next # Obtain the subsequent chunk of the ZIP
$stderr << '*'
_bin_str = enum.next # Obtain the subsequent chunk of the ZIP
$stderr << "*"
end

# ... or a Fiber

warn "\n\nOutput using a Fiber"
fib = Fiber.new do
fib = Fiber.new {
iterable.each do |binary_string|
$stderr << '•'
$stderr << "•"
_next_iteration = Fiber.yield(binary_string)
end
end
}
15.times do
fib.resume # Process the subsequent chunk of the ZIP
end
26 changes: 13 additions & 13 deletions examples/parallel_compression_with_block_deflate.rb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# frozen_string_literal: true

require_relative '../lib/zip_kit'
require 'tempfile'
require_relative "../lib/zip_kit"
require "tempfile"

# This shows how to perform compression in parallel (a-la pigz, but in a less
# advanced fashion since the compression tables are not shared - to
Expand All @@ -19,9 +19,9 @@
# as well, in an independent deflate segment - the threads do not share
# anything. You could also multiplex this over multiple processes or
# even machines.
threads = (0..12).map do
threads = (0..12).map {
Thread.new do
source_tempfile = Tempfile.new 't'
source_tempfile = Tempfile.new "t"
source_tempfile.binmode

# Fill the part with random content
Expand All @@ -33,16 +33,16 @@
source_tempfile.rewind

# Create a compressed part
compressed_tempfile = Tempfile.new('tc')
compressed_tempfile = Tempfile.new("tc")
compressed_tempfile.binmode
ZipKit::BlockDeflate.deflate_in_blocks(source_tempfile,
compressed_tempfile)
compressed_tempfile)

source_tempfile.close!
# The data that the splicing process needs.
[compressed_tempfile, part_crc, source_tempfile.size]
end
end
}

# Threads return us a tuple with [compressed_tempfile, source_part_size,
# source_part_crc]
Expand Down Expand Up @@ -70,16 +70,16 @@
# We use a File as a destination here, but you can also use a socket or a
# non-rewindable IO. ZipKit never needs to rewind your output, since it is
# made for streaming.
output = File.open('zip_created_in_parallel.zip', 'wb')
output = File.open("zip_created_in_parallel.zip", "wb")

ZipKit::Streamer.open(output) do |zip|
zip.add_deflated_entry('parallel.bin',
size_of_uncompressed_file,
entire_file_crc.to_i,
size_of_deflated_segment)
zip.add_deflated_entry("parallel.bin",
size_of_uncompressed_file,
entire_file_crc.to_i,
size_of_deflated_segment)
compressed_part_files.each do |part_file|
part_file.rewind
while blob = part_file.read(2048)
while (blob = part_file.read(2048))
zip << blob
end
end
Expand Down
36 changes: 17 additions & 19 deletions examples/rack_application.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require_relative '../lib/zip_kit'
require_relative "../lib/zip_kit"

# An example of how you can create a Rack endpoint for your ZIP downloads.
# NEVER run this in production - it is a huge security risk.
Expand All @@ -12,10 +12,10 @@
# course (WEBrick or Thin).
class ZipDownload
def call(env)
file_path = env['PATH_INFO'] # Should be the absolute path on the filesystem
file_path = env["PATH_INFO"] # Should be the absolute path on the filesystem

# Open the file for binary reading
f = File.open(file_path, 'rb')
f = File.open(file_path, "rb")
filename = File.basename(file_path)

# Compute the CRC32 upfront. We do not use local footers for post-computing
Expand All @@ -30,35 +30,33 @@ def call(env)
# the user that the download stalled or was aborted in-flight.
# Note that using the size estimator here does _not_ read or compress
# your original file, so it is very fast.
size = ZipKit::SizeEstimator.estimate do |ar|
size = ZipKit::SizeEstimator.estimate { |ar|
ar.add_stored_entry(filename, f.size)
end
}

# Create a suitable Rack response body, that will support each(),
# close() and all the other methods. We can then return it up the stack.
zip_response_body = ZipKit::Streamer.output_enum do |zip|
begin
# We are adding only one file to the ZIP here, but you could do that
# with an arbitrary number of files of course.
zip.add_stored_entry(filename: filename, size: f.size, crc32: crc32)
# Write the contents of the file. It is stored, so the writes go
# directly to the Rack output, bypassing any RubyZip
# deflaters/compressors. In fact you are yielding the "blob" string
# here directly to the Rack server handler.
IO.copy_stream(f, zip)
ensure
f.close # Make sure the opened file we read from gets closed
end
# We are adding only one file to the ZIP here, but you could do that
# with an arbitrary number of files of course.
zip.add_stored_entry(filename: filename, size: f.size, crc32: crc32)
# Write the contents of the file. It is stored, so the writes go
# directly to the Rack output, bypassing any RubyZip
# deflaters/compressors. In fact you are yielding the "blob" string
# here directly to the Rack server handler.
IO.copy_stream(f, zip)
ensure
f.close # Make sure the opened file we read from gets closed
end

# Add a Content-Disposition so that the download has a .zip extension
# (this will not work well with UTF-8 filenames on Windows, but hey!)
content_disposition = 'attachment; filename=%<filename>s.zip' % {filename: filename}
content_disposition = "attachment; filename=%<filename>s.zip" % {filename: filename}

# and return the response, adding the Content-Length we have computed earlier
[
200,
{'Content-Length' => size.to_s, 'Content-Disposition' => content_disposition},
{"Content-Length" => size.to_s, "Content-Disposition" => content_disposition},
zip_response_body
]
end
Expand Down
2 changes: 1 addition & 1 deletion examples/s3_upload.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require_relative '../lib/zip_kit'
require_relative "../lib/zip_kit"

# Any writable object can be used as a destination for the Streamer.
# For example, you can write to an S3 bucket. Newer versions of the S3 SDK
Expand Down
18 changes: 9 additions & 9 deletions lib/zip_kit/block_deflate.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# frozen_string_literal: true

require 'zlib'
require "zlib"

# Permits Deflate compression in independent blocks. The workflow is as follows:
#
Expand Down Expand Up @@ -49,7 +49,7 @@

class ZipKit::BlockDeflate
DEFAULT_BLOCKSIZE = 1_024 * 1024 * 5
END_MARKER = [3, 0].pack('C*')
END_MARKER = [3, 0].pack("C*")
# Zlib::NO_COMPRESSION..
VALID_COMPRESSIONS = (Zlib::DEFAULT_COMPRESSION..Zlib::BEST_COMPRESSION).to_a.freeze
# Write the end marker (\x3\x0) to the given IO.
Expand Down Expand Up @@ -96,9 +96,9 @@ def self.deflate_chunk(bytes, level: Zlib::DEFAULT_COMPRESSION)
# @param block_size [Fixnum] The block size to use (defaults to `DEFAULT_BLOCKSIZE`)
# @return [Fixnum] number of bytes written to `output_io`
def self.deflate_in_blocks_and_terminate(input_io,
output_io,
level: Zlib::DEFAULT_COMPRESSION,
block_size: DEFAULT_BLOCKSIZE)
output_io,
level: Zlib::DEFAULT_COMPRESSION,
block_size: DEFAULT_BLOCKSIZE)
bytes_written = deflate_in_blocks(input_io, output_io, level: level, block_size: block_size)
bytes_written + write_terminator(output_io)
end
Expand All @@ -116,11 +116,11 @@ def self.deflate_in_blocks_and_terminate(input_io,
# @param block_size [Fixnum] The block size to use (defaults to `DEFAULT_BLOCKSIZE`)
# @return [Fixnum] number of bytes written to `output_io`
def self.deflate_in_blocks(input_io,
output_io,
level: Zlib::DEFAULT_COMPRESSION,
block_size: DEFAULT_BLOCKSIZE)
output_io,
level: Zlib::DEFAULT_COMPRESSION,
block_size: DEFAULT_BLOCKSIZE)
bytes_written = 0
while block = input_io.read(block_size)
while (block = input_io.read(block_size))
deflated = deflate_chunk(block, level: level)
output_io << deflated
bytes_written += deflated.bytesize
Expand Down
Loading

0 comments on commit cfb053b

Please sign in to comment.