Feature Flags as a Service Reviewed. The 2017 bakeoff!
vs vs vs "Doing it your damn self"
Feature flags should let you turn things on or off from a UI so you can put new dangerous code behind a flag and slowly roll it out. All the solutions here allow for turning a flag on, off, 64% on or off, or turning a feature on for a specific user or class of users. They differentiate themselves by just how advanced the "custom cohorting" is that they allow you to do. My .02 is that 80% of the value comes from the "on" and "off", 5% comes from the "turn this on for a specific user" and 14% comes from the percentage rollout. I'm leaving the last 1% of benefit for the "turn this on for females aged 35-44 who hail from Orlando" features.
- LaunchDarkly has a beautiful management dashboard, percentage rollouts, custom targeting rules, audit logging, & flexible admin roles and Optimizely and NewRelic integrations.
- Split.IO has some very fancy tools around custom targeting & rollouts. Detection of flags that are not being used, and a ton of inegrations like DataDog, NewRelic & Slack.
- RateLim.it has a basic UI, percentage rollouts, and a simple layer of "custom targeting".
- DIY has... well nothing, since you haven't built it yet.
None of these are much good until your code hits them. A good library should have some really intelligent caching so that you can use feature flags with abandon. Ideally you want to cache the definitions in a shared cache and then cache the actual lookups in process so that flags are something you can us in a tight loop. They all have an API under the covers so you're free to use them from Haskell etc if that's your game, you'll just have a bit more work to do.
- Ratelim.it has https://github.com/jdwyah/ratelimit-java and https://github.com/jdwyah/ratelimit-ruby. After that you're on your own to hit our API docs or you can try asking nicely to see if we'll port the client to the language of your choice.
- Build it yourself has.... nada. You're going to need to build your client in every language you want to support.
The four solutions we're looking at have hugely different pricing profiles which makes it a big challenging to compare. First let's take a look at the details of the four plans:
- LaunchDarkly has 3 paid plans and a "Call Us". $79/mo for 10k MAU, $299/mo for 25k MAU and $699/mo for 50k MAU. If you don't have many users, the only other ratchet in the plan is that there's no AuditLogging in the $79/mo plan.
- Split.IO is 100,000 free then $.50 per 1000 calls or you can use their "Call Us" plan.
- RateLim.it is $.0001 per feature flag request, but that only counts actual calls to the API. If you plug in memcached you'll actually call a maximum of about once a minute, so that's about $.14 per day or $4.20 / month for as essentially unlimitted usage.
- Running it yourself... well, all depends, but I can't imagine that you're actually going to increase the load on your system sufficient to require more resources. So I'd just bill this out as 8 hours of dev time to get a basic version from scratch. 40 hours to get a fancier version with nice caching, a decent UI, some logging etc.
Feature Flag Pricing in Practice
What does that mean in practice? Well let's take a hypothetical small / medium sized e-commerce as a simple example. Let's say we have about 400k uniques a month, everybody hits about 5 feature flags on average.
- LaunchDarkly that's easily in call-us territory so you have to imagine $1000/month minimum.
- Splito.io that's 2M flags. Minus 100k free... 1.9M flags.. divide by 1000 is 1900... $950/month.
- RateLim.it thats... 2M calls to your memcahced server... 43k calls to the ratelimit API... about $4.30.
- Running it yourself. Month one is about 40 hours times some consultant's exhorbitant rate of $150 is $6000 for the first month. Then close to free afterwards. After a year let's call it averaging out to $500 / month. LaunchDarkly wrote a post about Buying vs Building a Feature Flag System /shrug
Wow, I knew Ratelim.it would look good in comparison but that's ridiculous. I guess long story short: if you're doing feature flags for something e-commerc-y you really need to think twice about using a service (unless it's ratelim.it). Also I hate the idea that you're going to get crushed by bots and end up sending another 10k uniques and then are going to need to spend your time arguing for a refund. But perhaps they're great about that.
RateLim.it isn't actually focussed on feature flags (it's actually a RateLimit as a Service provider, weird, right?). But I added FeatureFlags to RateLim.it because I can't develop without it and it was just as easy to add it as a service vs build it for just us. That said it seems like it's a decent and much cheaper alternative to the existing heavyweights so it may well be worth checking out.
If you're really looking for the gold-plated version I think the toss up between Split.IO and LaunchDarkly will probably come down to your usage pattern & how that will work with their pricing model.
Building it yourself is a perfectly reasonable option, but like anything it will take you a bit longer than you first expect. The main thing to like about it is that you won't have to worry about a 3rd party SLA and for a simple app having feature flags share uptime with you main app may be worth it.