Automatically degrade a feature when failure rate reaches a certain threshold. This is useful for the following if a new feature causes error rates to spike:
- Toggling off feature flags
- Disabling a service entirely
- Notifying the team when any of these has happened
Note: this is not a replacement for a monitoring/alerting solution, but can augment your alerts by automatically disabling the functionality that is firing them off.
-
Add the dependency to your
shard.yml
:dependencies: degradable: github: jgaskins/degradable
-
Run
shards install
Degradable comes out of the box with two adapters, one for operating in application memory and one for coordinating across multiple instances of the application via Redis.
In the following example, we're using Pennant
for feature flags and Slack for notifications. When the failure rate reaches 20% (and a minimum number of total checks to avoid wild swings with a low sample size), we will notify our team via the #degradations
Slack channel and automatically disable the feature flag.
This adapter is for features that can be disabled based on failures within a single process
require "degradable"
require "slack"
require "pennant"
# Degrade MY_FEATURE when failure rate reaches 20%
MY_FEATURE = Degradable::Memory.new(failure_threshold: 0.2) do
Slack.notify "#degradations", "Feature MY_FEATURE has reached 20% failure rate"
Pennant.disable "my_feature"
end
This adapter allows you to coordinate degradation of a feature across all instances of your entire application, as long as they can all talk to the same Redis instance. The only difference in how you use it is that you need to pass in a Redis instance to use as well as the name of the feature (which will be stored in Redis).
require "degradable"
require "slack"
require "pennant"
require "redis"
redis = Redis::Client.from_env("REDIS_URL")
# Degrade MY_FEATURE when failure rate reaches 20%
MY_FEATURE = Degradable::Redis.new("my-feature", failure_threshold: 0.2, redis: redis) do
Slack.notify "#degradations", "Feature MY_FEATURE has reached 20% failure rate"
Pennant.disable "my_feature"
end
Now that we have our feature's failure handler defined, we can invoke a check by calling MY_FEATURE.check
with a block:
if Pennant.enabled? "my_feature"
MY_FEATURE.check { feature_flag_enabled_behavior }
else
feature_flag_disabled_behavior
end
- Fork it (https://github.com/jgaskins/degradable/fork)
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
- Jamie Gaskins - creator and maintainer