Graphic intensive editing and rendering workloads getting heavier and heavier following the rapid development in large resolution video formats. Professional raw and pro-res type video formats are now common across the field and the current vertically scaling on-prem render nodes are becoming economically unviable for this massive render jobs. With the economy of scale and the availability in the cloud, it makes more sense to move the intensive render jobs to the public cloud. This is a brief walkthrough of a project I designed and Imprinted for commercial production house to streamline their render process.
Client
Kandy Films Pvt Ltd.
Platform
Microsoft Azure
Type
Commericial
Previous Infarstructure.
The client was using a basic on-prem infrastructure, Consist of a private network, NAS file storage, subnet consist of a editing machines and another with the render engine machine.
Pitfalls and Procepects.
As you can see on the diagram, this infrastructure is vertically scaling. And since the render engine and the NAS server is disconnected from the internet, render jobs only can be executed within the network. Client`s future roadmap includes more edit labs in multiple locations and this make this infrastructure resource intensive and non compliance. While keeping the current edit nodes as it is, client`s requirement was to de-commissioning the on-prem render lab and move the entire render lab to cloud, so a render job can be pulled from anywhere at any time with the benefits of public cloud.
New Cloud Infarstructure
Simplified Architecture Diagram.
New cloud infrastructure consist of following components.
Within Azure VNet,
N Series VMs for render engine.
A bastion server in a different subnet for secure connection to the VM using RDP / SSH.
SMB file sharing using SMB 3.0 protocol. This can be also set up without a Virtual Network Gateway and a VPN.
Necessary networking components as shown in the diagram.
How it works
Initial Setup - Single Render Node
Deploy a N series or similar performance (Matching to your requirements) VM in a new private subnet within a new security group. This will be called Render VM hereafter.
Deploy a bastion server in another subnet and connect the Render VM through a RDP/SSH connection at port 3389.
Deploy a SMB file share within the azure to sync render ready project files between the on-prem & Cloud network. This can be set up with or without a Virtual Network gateway. But I strongly suggest using one.
Once the architecture is deployed and everything is connected, client or the sys admin can log in to the Render Engine through the bastion server from the Azure Portal. Once logged in, they can set up the "Adobe Media Encoder" within the VM.
Connect the SMB file share between the Render VM and the Client's local machine.
Watch folders set up in Azure SMB File Shares
Instead of running the render engine locally, we can use the power and freedom of the public cloud to run the render engine in cloud for a fraction of the price. We will be using adobe media encoder as the render engine and it will be installed to a N series or similar range Virtual Machine. And we will be also using a SMB file share to sync the render files between the cloud and on prem computers. Once a project is ready to render, the client can save the final project files with the media files in to a pre defined folder (called watch folders) in the SMB disk. As soon as the project files are successfully saved in the shared disk, Adobe media encoder in the azure VM will automatically pickup the render job and will run it. Upon finishing the render, the output file will be saved in the same SMB disk for easy access and the VM will automatically shutdown, minimizing the operational costs. Virtual machine can be turn on manually through the azure portal. But much better option would be running the whole Infrastructure as code. Thus, saving more time on setting up. This method is more suited when you are on production environment.
*This is a breif overview about the project. This is not the documentaion nor the white paper.