Today's Challenges for Post Production, Visual Effects and Playout Workflows
Today’s broadcast and post production environments are faced with a multitude of challenges, from complex and difficult to integrate ecosystems to increased pixel depth/frame rates to the evolving IP-video landscape. Chief engineers and IT administrators in many cases must integrate MAM, VFX, grading, transcoding, rendering/compositing workflows with make do cobbled together systems from disparate vendors, often resulting in expensive, unwieldy and nearly intractable piles of hardware and software. For instance, your MAM speaks NFS, your object store uses S3, your color artists all have Windows, and your render farm just isn’t fast enough. You bought storage for each one and you’re pondering how to integrate them all with limited staff. Adding to the debilitating stress level are the need for regional, national and global collaboration and fast turnaround project deadlines while keeping capex and opex costs in-line. All these facets of post-production to on-air and distribution workflows present one very challenging storage environment for companies in the M&E landscape.
What is needed to address this growing dilemma of not having to rely on monolithic or cobbled together hardware and software is a scalable, cost-effective, hardware agnostic storage software solution that delivers the IOPS, throughput, and resiliency to tame even the most formidable workflow needs. Further, it should be a system designed to ease a facility’s administrative burden, prioritize cost containment, provide seamless differing file system support, and afford broad connectivity choices in today’s highly responsive containerized world vs. previous years’ inflexible data center designs.
The Quobyte Solution for Post Production, Visual Effects and Playout Workflows
Enter Quobyte (www.quobyte.com), a Silicon Valley headquartered company founded by ex-Googlers Dr. Björn Kolbeck (CEO and Co-Founder) and Dr. Felix Hupfeld (CTO and Co-Founder). Quobyte’s unique and innovative parallel distributed file system with Google-style operation and functionality provides a single unified storage platform and namespace for the variety of file types in the workflow and gives post-production teams fast access to their files. Each person in the workflow, whether on a Mac®, Windows®, or Linux machine can access the same file. These same files can also be accessed via the company’s stateless S3 proxy for collaboration among everyone involved in the post-production process. The software supports parallel, sequential and small block random I/O workloads, and all data is available on any server or virtual machine. Quobyte thus eliminates silos of content and delivers a unified storage experience for productive collaboration.
From a 10,000 feet perspective, the benefits to a post production to VFX to playout workflows are numerous.
Post Production Workflow Benefits:
Aside from enterprise-grade provisioning, policy-based resource allocation and integrated monitoring and analytics, Quobyte’s solution also features built-in hardware management. Users benefit from automatic scrubbing, detection, and repair of corrupted data, and the software can drain a server with one click. Also built-in are monitoring and hardware watchdogs that can detect and initialize new devices and perform device lifecycle management for “lights-out” operations.
Enterprise Data Center Ready, Highly Configurable
Quobyte’s software storage solution is aimed at operational ease and flexibility. Users are free to deploy on any hardware, and cluster nodes don’t have to be identical. Features like user-definable data placement, policy-driven tiering rules, and data pinning give storage administrators the freedom to optimize for their environment with no vendor lock-in. And with full support of all modern Linux distributions with no kernel dependencies, the software is extremely flexible. Native Linux, Windows, and Mac drivers deliver high performance parallel I/O. NFS or S3 workloads can be connected through stateless proxies and storage is accessible through all interfaces simultaneously. Quobyte’s software also allows facilities multi-thousand node scalability from four nodes to entire data centers and beyond, affording additional nodes to be added to increase performance without downtime. In addition, Quobyte’s software solution offers unlimited data capacity and non-disruptive live updates.
Fast and Reliable Data Access with Full Fault Tolerance
Delivering hyper-performance storage while maintaining data availability despite disk or network failures takes a holistic design approach. Quobyte took ideas from high-performance computing technology and 10 to 15 years of research on fault tolerance with distributed systems, based on the Paxos algorithm. The result is a software solution purpose built to handle real- world environments, even node failures while keeping its operational integrity intact. True end-to-end checksums of every data block assures data integrity. Ultra-low latency and fast metadata operations keep up with the toughest small-file workloads. Hardware monitoring will offline an errant disk before it causes corruption, and self- healing heuristics route around node failures. The result – no single point of failure, partition tolerance and fully automated fault tolerance on commodity hardware.
Future Proof for Today’s and Tomorrow’s Workflows
Yesterday’s inflexible data center designs are quickly giving way to tomorrow’s highly responsive containerized world. Quobyte’s software provides persistent container storage and it can also be run inside of the containers. It delivers low-latency and parallel throughput performance to all containerized applications, so users can run databases, scale-out applications, and big data analytics on one single infrastructure.
The software works out of the box with a facility’s technology stack; combining it with Kubernetes, Mesosphere, or Docker allows users to set up an ultra-responsive and reliable storage infrastructure with ease to address current and future rapidly fluctuating workloads. Quobyte also supports OpenStack’s Cinder, Glance, Manila, and S3 interfaces, and is also compatible with Hadoop – resulting in a seamless, unified storage access architecture.
For broadcasters, studios and post production facilities facing continuing budget constraints and manpower reductions, the Quobyte solution offers a flexible, unified and collaborative storage experience that helps users cost-effectively scale out their storage performance and capacity requirements when needed, dramatically reduce administrative overhead while providing guaranteed and secure data integrity for the entire workflow chain.
From a 10,000 feet perspective, the benefits to a post production to VFX to playout workflows are numerous.
Post Production Workflow Benefits:
- Increased collaboration and productivity
- Eliminates needlessly copying, exporting, and uploading files between systems and platforms
- More efficient use of network bandwidth and storage capacity
- Eliminate management overhead and complexity
- Automatic file prefetching for jitter-free playback of 4K streams from applications such as NUKE or Autodesk’s Flame
- High-performance shared storage to support all aspects of a media and entertainment workflow
- A single storage platform eliminates data silos and promotes increased collaboration
- Easily scale storage performance and capacity
- Built-in reliability and data safety so data is always available
- No downtime for maintenance or when adding capacity or bandwidth
- Easy automated administration
- Software storage for a flexible and responsive infrastructure
- Built-in fault tolerance for “always-on” operation
- Cost effective, multi-petabyte content stores built on commodity hardware (HDD, SSD, NVMe, SMR)
- Easily scale storage performance and capacity
- Eliminate storage management overhead and complexity
- Support for Mac, Windows, Linux, and S3
Aside from enterprise-grade provisioning, policy-based resource allocation and integrated monitoring and analytics, Quobyte’s solution also features built-in hardware management. Users benefit from automatic scrubbing, detection, and repair of corrupted data, and the software can drain a server with one click. Also built-in are monitoring and hardware watchdogs that can detect and initialize new devices and perform device lifecycle management for “lights-out” operations.
Enterprise Data Center Ready, Highly Configurable
Quobyte’s software storage solution is aimed at operational ease and flexibility. Users are free to deploy on any hardware, and cluster nodes don’t have to be identical. Features like user-definable data placement, policy-driven tiering rules, and data pinning give storage administrators the freedom to optimize for their environment with no vendor lock-in. And with full support of all modern Linux distributions with no kernel dependencies, the software is extremely flexible. Native Linux, Windows, and Mac drivers deliver high performance parallel I/O. NFS or S3 workloads can be connected through stateless proxies and storage is accessible through all interfaces simultaneously. Quobyte’s software also allows facilities multi-thousand node scalability from four nodes to entire data centers and beyond, affording additional nodes to be added to increase performance without downtime. In addition, Quobyte’s software solution offers unlimited data capacity and non-disruptive live updates.
Fast and Reliable Data Access with Full Fault Tolerance
Delivering hyper-performance storage while maintaining data availability despite disk or network failures takes a holistic design approach. Quobyte took ideas from high-performance computing technology and 10 to 15 years of research on fault tolerance with distributed systems, based on the Paxos algorithm. The result is a software solution purpose built to handle real- world environments, even node failures while keeping its operational integrity intact. True end-to-end checksums of every data block assures data integrity. Ultra-low latency and fast metadata operations keep up with the toughest small-file workloads. Hardware monitoring will offline an errant disk before it causes corruption, and self- healing heuristics route around node failures. The result – no single point of failure, partition tolerance and fully automated fault tolerance on commodity hardware.
Future Proof for Today’s and Tomorrow’s Workflows
Yesterday’s inflexible data center designs are quickly giving way to tomorrow’s highly responsive containerized world. Quobyte’s software provides persistent container storage and it can also be run inside of the containers. It delivers low-latency and parallel throughput performance to all containerized applications, so users can run databases, scale-out applications, and big data analytics on one single infrastructure.
The software works out of the box with a facility’s technology stack; combining it with Kubernetes, Mesosphere, or Docker allows users to set up an ultra-responsive and reliable storage infrastructure with ease to address current and future rapidly fluctuating workloads. Quobyte also supports OpenStack’s Cinder, Glance, Manila, and S3 interfaces, and is also compatible with Hadoop – resulting in a seamless, unified storage access architecture.
For broadcasters, studios and post production facilities facing continuing budget constraints and manpower reductions, the Quobyte solution offers a flexible, unified and collaborative storage experience that helps users cost-effectively scale out their storage performance and capacity requirements when needed, dramatically reduce administrative overhead while providing guaranteed and secure data integrity for the entire workflow chain.