We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
avg_record_size
I am concerned about significant fluctuations in `avg_record_size`. Here are a few suggestions for improvement:
let alpha = 0.2; // adjustable smoothing factor *this.avg_record_size = ((1.0 - alpha) * *this.avg_record_size as f64 + alpha * (fetch_bytes as f64 / records_and_offsets.len() as f64)) as usize;
let new_avg = fetch_bytes as usize / records_and_offsets.len(); let max_change = *this.avg_record_size / 10; // allow max 10% change *this.avg_record_size = (*this.avg_record_size + new_avg.clamp( this.avg_record_size.saturating_sub(max_change), this.avg_record_size.saturating_add(max_change) )) / 2;
this.record_sizes.push(fetch_bytes as usize / records_and_offsets.len()); if this.record_sizes.len() > 10 { // keep last 10 records this.record_sizes.remove(0); } *this.avg_record_size = this.record_sizes.iter().sum::<usize>() / this.record_sizes.len();
Originally posted by @fengjiachun in #4424 (comment)
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Set a maximum change magnitude to avoid large single-batch fluctuations.
Keep a record of the sizes of the last N batches and calculate their average.
Originally posted by @fengjiachun in #4424 (comment)
The text was updated successfully, but these errors were encountered: