Storj is an innovative platform with a unique approach to cloud object storage that comes with cost and performance advantages that are particularly interesting to the SMB media and broadcast sector. In this short article, I want to break down Storj’s advantages for archive applications and go beyond the marketing straplines to see how much cost and workload Storj can save your business.
Technical advantages of multi-cloud erasure coding
And the award for the driest heading in an article goes to… But it is important to understand, at least at a mile-high level, what is fundamentally different about Storj’s platform to frame the points that follow.
The rest
All major cloud object store providers like AWS, Google, and Azure have the same basic model and store redundant (erasure encoded) data within a single region or data centre. If a storage node fails, recovery requires rebuilding missing data from a limited set of storage locations within that region.
To make sure there is enough parity data in the region to survive the loss of several storage nodes, many of the big cloud providers use triple replication (3 copies of your data) for redundancy.
So, let’s keep these points in our mind:
- Data is restricted to geographic regions, if you want it somewhere else you copy it there (even if you don’t realise, you’re doing it)
- Limited locations to re-construct your data if there is an outage
- Compute overhead can cause latency spikes for read or write operations
- For every byte of data you upload, you’re often using 3 bytes to store parity data.
Storj
Storj aren’t reinventing the wheel with their platform, but they are far more open-minded with the diameter of wheels, and by making one profound choice in their infrastructure, they have changed the way we can access and protect our data.
The core difference between the model of the hyperscale providers outlined above and Storj is that Storj has no concept of regions; they are a truly global filesystem, and as we will see, the design decisions this leads to are where the advantages for media users lie. Storj, like the centralised cloud providers, uses erasure encoding when writing your data but splits each file into 80 chunks, distributed globally on hundreds of nodes. These nodes are procured from spare capacity in global data centres, meaning they are low cost and vendor agnostic.
Only a 29 portion of these chunks is required to rebuild any file, meaning your data is incredibly secure. It also means that Storj can choose the fastest connections to your data from the hundreds of locations and download data in parallel, meaning much quicker access, not dependent on region.
Expanding the scale of nodes and breaking up files into more distributed chunks allows Storj to keep their data redundancy overhead down to 1.3x from the standard 3x in centralised cloud providers. So, in review, we see a much more flexible, performant, and economical platform.
So, in review we see a much more flexible, performant and economical platform:
- Data is NOT restricted to geographic regions, if you want it you can access it wherever you are.
- 80+ locations to re-construct your data if there is an outage, higher fault tolerance
- Parallel download of data chunks increases performance
- For every byte of data you upload, you’re using 1.3 bytes to store parity data, translating to be less cost overhead thus lower prices to end users.
Which is all to say that when compared to centralised providers, Storj is not only more resilient but also faster and simpler to use. And by using spare capacity to build their platform and vastly reducing data duplication, it is significantly cheaper and greener by design.
Archive Costs, the visible and the invisible
Let’s break with sales patter tradition and talk about cost upfront. For an archive, cost has always gone hand in hand with durability as the main driving factor, so why not cut to the chase?
With all cloud providers needing to meet each other’s standards, Storj included, of 11 9’s durability, we can avoid labouring that point. They are all, in their own way, fairly bulletproof in terms of data loss; however, data recovery is a little more complex, but more on that later.
As the leader in the media space, let’s pick on AWS’s published costs as a centralised cloud provider.
Let’s also throw in Wasabi as another popular but smaller cloud platform for our industry.
In my opinion, although even the most carefully curated LTFS archive is not going to get to 99.999999999% durable, we should still consider private LTO as a viable solution in this comparison. Who am I to argue with a 20+ year pedigree? It’s served us well, and there are very compelling systems out there on the later generations of tape.
So, let’s look at some numbers, or visible costs. These are what you are likely to see on a monthly bill. I will be the first to say that there is perhaps some unfairness here to AWS, as the price per TB does not take into consideration tiering (usually kicking in at around 150TB utilisation and moving from there) or pre-committed contracts which can be compelling if you’re a multinational digital rights holder but unattainable for the regular Joe. This is a good indicative guide for small to medium-sized media organisations looking to make their archive more available and cost-effective.
We can see that on the headline numbers, Storj blows AWS out of the water and, as already set out, has a more agile and available platform.
Where you might pause to consider your options would be on the ‘egress free’ providers such as Wasabi and the more traditional on-premise solutions based around tape architecture.
So, to address these providers, let’s talk about invisible costs, not necessarily on a consolidated monthly bill, but a very real cost to a business.
Escalating Infrastructure Costs
If you have a significant LTO estate, this one will feel familiar. Even if you are fastidious in maintaining your hardware, the speed of evolution in tape generations inevitably requires ongoing hardware refreshes, demands engineering staff, consumes rack space, power, and air-con, and is subject to escalating annual support costs, all of which become consolidated into the lower cost of Storj.
Operational Inefficiencies
How long do you keep a customer waiting for an asset in your archive? If you want to make AWS economical by using their glacier tiers, then you are talking hours or days. If you are on LTO and lucky enough to have the tapes to hand, you may be in the tens of minutes. If you have a well-organised Wasabi archive, you may be in a better position if you are in the right region. In all cases, with Storj’s parallel multi-node architecture, you will have better data speeds. Compared to Wasabi, you can expect a 2-5x increase with more efficiency on large data sets, and with half the first-byte latency, small file retrievals such as audio and file-per-frame video will see a large improvement. Even in comparison to AWS S3 standard and CDN architecture, Storj’s performance wins out. Plus, the time that you save during asset retrieval can mean your staff are moving onto the next task, your suites are working sooner, and your customers are happier.
Scalability Limitations
How’s your slot count? How many drives do you have? If you suddenly need to retrieve 4 tapes in your 2-drive library, how much longer is that conform the edit assist is working on going to take? Or how frustrated is your customer going to get waiting for those rushes? On-premise LTO is always finite, and in a world where data volumes are massively increasing and customer expectations are high.
Missed Revenue Opportunities
If your archive is a source of income, then the availability of your archived data needs to be at the heart of your operation. Poor performance and organisation have a direct impact on customer experience, and with the opportunity that AI analysis represents to make your assets more searchable and valuable, having simple and economical ways to integrate your archive into cloud services is more important than ever and can have a real impact on your P&L.
Conclusion
Why do you have an archive? If you have an insurance obligation to write your files once and only read or retrieve in an emergency, then your choice should really be based on the cost of ownership, maybe with a nod to user experience, but if it’s cheap and it works those two times a year you use it and you get your MD5 checksum, then who cares, right? Well, hopefully, I’ve given some context to the real cost of not only physical ownership but also the cost of inefficiency so often ignored that should make you think again.
And that cost of inefficiency truly becomes crucial if you have revenue hidden in your archive. Moving your archive into a cost-effective, highly available cloud platform like Storj should really be a no-brainer.
If the idea of moving your archive sends a cold chill down your spine, then fear not; my next article is going to focus on exactly that and how you can remove the pain from the process.
If you would like to learn more, see a demonstration of the power of Storj, and discuss what it can do to unlock your archive workflows, please get in touch with the Tyrell team.
Email us at sales@tyrellcct.com
