Azure Friday | Azure Batch Rendering Service

Azure Friday | Azure Batch Rendering Service


Hi, I’m Donovan with another
episode of Azure Friday. We’re here with Dave to talk
about Azure Batch Rendering. Welcome to the show, Dave.>>Thanks, Donovan,
good to be here.>>So what do you do
here at Microsoft?>>I’m an engineering manager
for a team in New Zealand called the Bit Compute team
as part of Asia.>>Awesome, so tell me all
about Azure Batch Rendering, what it is and
why do we need it?>>Yeah, so Azure Batch Rending,
it’s built on top of Azure Batch, Azure Batch
service which is designed for massive scale
parallel processing. Rendering is a great example of
one of those workloads where we can take Individual frames and send them to separate virtual
machines to be rendered.>>Okay, so rendered for what,
is this for video games, movies, what are we talking about?>>It could be any of those. And so the rendering process is
about taking the 3D model in the scene or the geometry and
textures and then calculating all the light in the fix and
producing that final image, whether it’s a still image or
an animation.>>Awesome, so tell me,
how does it work? So how do I get going? Do I have to interact directly
with the Azure Batch? Or do I use the same tools I
use today, like 3D Studio Max? How does that work?>>Yeah, there are a few
different options that customers can use to interact or
integrate with the service. And so we’ve got the Azure CLI,
and we’ve got our SDKs. But I want to actually show you
today is a bit of a walk-through some of the interactive
options that we have for running jobs and so forth.>>Love see it.>>Cool, so firstly,
I wanna flick over here. This is the Autodesk Maya
application, and this is really popular
in the field for creating these 3D models and
these things. And then this here is an example
of where we’ve got a plugin that sits right inside
of the application. So it’s a fully
integrated experience. And I think it’s a great example
of where we can bring the power of Azure right to the end users in the environment
that they’re used to.>>Yeah, I love the fact that
I don’t have to go learn something new. I can keep using the tool
that I love today.>>Totally, totally.
>>Awesome.>>And this will do things like
detect all of the assets that a scene requires. And then if you’re talking
about submitting a job, it’s gonna check with any of
those that need to be updated or uploaded to the cloud. So there’s a bunch of smarts in
there to make it really easy to do stuff end to end. But what I wanna primarily focus
on today is the new application that we released recently
called Batch Labs. And so Batch Labs is
a cross-platform desktop app for managing batch environments. And we’ve got a whole
bunch of features in here to help with learning
windowing jobs.>>Okay, cool.>>So first,
a couple of concepts. In batch we have
the notion of these pools, pools of virtual machines. And so here we have a CPU pool. It’s got D5 instances
which is 16-core machines. But what’s cool is we also
have a GPU pool here. And so these are NC6 instances
that I got in video K80 cards which are general purpose
processing units. And got 100 of those. And if we go into
create a new pool, I just wanna call out a couple
of options or features here. One thing is we have this notion
of dedicated nodes as well as low priority nodes. And so this is something we
announced a little bit earlier this year. I think it’s still
in public preview, where customers can get access
to unutilized capacity in Azure within a region, and
get that for a massive discount.>>I see.>>Up to 80% discount-
>>Wow.>>On the sticker price. And so the downside is that
if the demand increases in that region, then we might have
to take that capacity back. But the cool thing with
rendering workloads and other batch workloads is you
might lose a bit of work, but the service will schedule that
work to another virtual machine and get the task done.>>Okay, so even before I start
in the application of my choice, I would have had to come and first set up a pool to which I’m
gonna send my batch jobs to, and that’s what you’re
showing us now.>>That’s what this
interface is all about.>>Got it.
>>And you could do this from the
plugin inside Maya as well, so they’re fully featured. But just showing the example
of how to do it here.>>Another option.
>>Yeah, yeah, yeah, that’s right. And so if we do need to take
some of that capacity back for these low priority nodes, then
the Batch service is constantly gonna try and
seek to that target. So when the capacity
becomes available again, I can get those VMs back and
life is good. And so for workloads where you
don’t have to give them back, you don’t have an urgent time
line at the end, this is massive cost savings of THD available
here at a discount. So the other thing we have
here is this graphics and rendering option for the VM
image type that we wanna deploy. And so this was something
we put out there as a convenience thing. We have preprovisioned
an VM image with all of the latest versions of Maya,
3ds Max, Arnold, V-Ray, a bunch of the different
rendering apps. And so it’s a really easy way
to get started, cut the ties. If you need a specific
version or versions of these different apps, then you can
set up your own custom image. Or we have a container solution,
container-based solution which sort of offers the best
of both worlds. I think stability as well as
not having to do it yourself.>>So I provide you my image
that has all the rendering tools I want. And then you’re going to be
able to then scale that up and then do batch jobs
against my own image.>>Yeah, yep, got it. If we select one of these, this application licenses
feature shows up. And so this is really
the most exciting and defining feature of the whole
thing where we can actually do pay per use licensing with
these commercial apps. And it’s fully integrated into
all the Azure billing system. So when I use this, I would just
see additional line items on my Azure bill for
the software that I’m using.>>I was wondering.
That was my next question and since you answered that, so I don’t have to first purchase
3D Studio Max myself. I can actually just buy it on
a per need basis right here.>>Exactly, so
this is always the big blocker where the customers
using the cloud can say, okay, I can spin up another 100 or
1000 VMs, but I’ve got to have additional licenses to
own the software, so.>>Sure.
>>So the economics of it fell apart
pretty quickly, a fair model. So we’ll close out of here, because we’ve already got
our pools that we need. The next thing I want to show is
this data management capability we have in here. And so I can upload or
create these file groups and upload assets from my computer. And so this is an example here. We’ve got all of these
assets for this loft scene. And so that’s everything
I’m gonna need to render that particular scene. And then I’ll create another
file group here where I’m gonna have the outputs uploaded, too. And so these just resolve to
containers in Blob Storage.>>Okay, got it.>>So it’s just a convenient
way to upload the data. And the last option here is this
marketplace of applications that we’ve enabled. And so
this has sorta grown over time. And one example here is 3ds Max
which you mentioned earlier. And so if I click on there,
I can see a whole bunch of different job types or actions
particular for that application, whether it’s rendering
with CPU or GPU or using the distributor rendering
capability that it supports. And so if you’re wondering where
this list of application comes from, well, GitHub, of course.>>[LAUGH]
>>So we have a repository here
called BatchLabs-data. And so this has got a folder
here called NCJ, and that stands for no code jobs. That’s our nickname for
batch templates, which is what we’re using to
enable these applications. And so some guidance here on how
to enable an application using these templates. And then we’ve got all of
these ones that we saw inside the application. And so if we click into 3ds Max,
we can see all of the job types that we saw listed in
Batch Labs before. And if we click inside this
one here that we’re gonna run, we’ve got these three templates. The pool template is
a way to encapsulate all of the information about how to actually set up one of these
pools that we just saw before. And so it really simplifies
the whole creation experience.>>And is this open to
where I can contribute new jobs types and
things like that as well?>>Exactly, and so the idea
is that we wanna help sort of energize the community to easily
enable these applications and copy paste, and
modify it to what they need. And it’s not just for
rendering this. There’s other verticals like
financial services, and engineering, and
anything like that. So it’s very sort of open and
flexible. And so yeah, some of the
benefits are, because we know this is a GPU job that
we’re talking about here, we can restrict it to
a particular Set of VM types and sizes because we know these
are the particular ones that work well for the GPU rendering. And then other things like
installing in the video drivers and all that jazz. And so if we go back to
Batch Labs and click on this, We can create a pool here or
use an existing pool. So just to contrast with the
experience we saw before with all of those options about
how to create a pool, here we’ve got three inputs. What’s your pool name,
how many nodes do you want? And we’ve only got
a list of the NC->>Based on the JSON we just saw from.>>Exactly, so it really
simplifies the experience, because we know a bit more
about what we’re trying to do. So continue with one of
the existing pools we have, a GPU pool. We can see it’s got
a whole bunch of inputs to submit this job. And again, this is driven by the
JSON template, the job template. And so it defines the parameters
that we need for the job, the scene file that we need,
the file group and so forth, start frame, the end frame,
resolution, all these things.>>Everything you can imagine
that you’re gonna need to configure to make sure you get
what you need on the outside.>>Exactly, exactly, and there’s
a bit more detail about how we generate the tasks,
using these task factories. And that just sort of sweeps
over the start frame to the end frame, generates a task for
each one. And then this is a command
line that we’re gonna execute.>>It’s almost like a build job.>>Yeah, very,
very similar, yeah, defining what’s actually
gonna happen on the nodes. Cool, and so
we’ll skip through this. We can select our assets
file group, select the scene file that we want to render,
which is this job here. Save button’s lower
than it should be.>>[LAUGH]
>>[LAUGH]>>From frame 1 to frame 100. And then the resolution
like we saw. And then we want the outputs to
go into the outputs file group. And we could put the output
spec into the same->>If you wanted to, but this is gonna make it easier for
me to identify what’s been done. So this is going to go off and
run. I’m gonna be able
to monitor this. When it’s done I get to
download my final asset. Which might be, like you said,
an animation or still image of the 3D
rendered product.>>Yeah, most customers, they just want to get
the individual frames out, and then they might do some
post-processing on those frames before they combine them into
an animation, into a video file. But actually if you look back
at the marketplace there’s an FFmpeg job here, so you could actually stitch these
things together very easily.>>Using Batch as well.>>Yeah, create an animation from your images because the
data’s already in Blob Storage.>>This all kind of flows
together quite nicely. And obviously, you can do this
all programmatically through the SDKs as well.>>Cool, and so
with our job submitted, we can see we’ve got 100 tasks. And this little counts or
statuses at the top here, so they’re all running,
because I’ve got 100 GPU nodes.>>Sure, the true power
of the cloud is that I don’t have to do that any more. Infinite scale for
me right here.>>Exactly, you got it. And so I can click into
each of these tasks and see what is going on below. I’ll click to one of the jobs
that’s completed before, and click into one of the tasks. And we can see the files that
are actually on the virtual machine where that task ran. And so, for example,
I can see the final image, the rendered image
that was produced. And I can see the standard
out and standard error. So if the image didn’t
come out as I expected, maybe a texture is missing or
something, I can go and look in the standard error. I’ll extend it out and
should be able to see the [CROSSTALK]
>>Troubleshoot what’s going on.>>And find this particular
file or something like that. So it makes it pretty easy
to see what’s happening. Cool, and then if we switch
back to our pool here, we should be able to see that. Yeah, we’ve got this nice little
heat map and we can see all of our nodes are busy processing
that job we just submitted. The crosshatch lines indicate
the low priority nodes. So we’ll get a combination
of the low priority and dedicated here.
>>I see, gotcha.>>It’s pretty cool.
>>Awesome.>>And then the outputs, we
find them in our output folder. So the job name, and
then the [CROSSTALK] yeah, cool.>>Fantastic.>>So that is pretty much
how the service works. And then as I mentioned, you
wanna use your SDK to integrate this with your own
queuing system or your own on-premises plot find. That’s what a lot of customers
do that hopefully there will be options to go out and-
>>And it’s all available today?>>Yep, and you can use it.>>That’s right, and so,
it’s available today. V-ray is one of the renderers
that’s still in limited preview. And so if you wanna use that,
test it out, then get in contact with us, and we can whitelist
your Batch account.>>Okay.
>>But then, yeah, otherwise we’ve got,
Maya, 3ds Max, and Arnold, and GA,
General Availability. And we’re adding other
windows as we go along.>>One last question, so,
if I’m using one of those tools, how do I get that
plugin into the tool? Is that from a free marketplace? How do I get, for example,
the 3d Studio Max? How did I integrate
the Azure plugin into that?>>Yeah, so there’s,
On GitHub there’s, of course,
everything’s on GitHub. So azure_batch_maya,
there’s the plugin there. And there’s instructions here
for how to install it and get it going into Maya. There’s also a really good
walk-through article we’ve got here which sort of talks about
all of the different options for getting the applications or
the software into the system, as well as for
submitting the jobs as well. So hopefully there’s
enough guidance here for people to figure things out. And we’re gonna post those
links at the bottom of the show as well. Well, hopefully [LAUGH].
>>Awesome, thank you so much. So we’re learning all about
Azure Batch Rendering here on Azure Friday. [MUSIC]

2 thoughts on “Azure Friday | Azure Batch Rendering Service

  • Is there a more detailed demo of this video ? I'm looking at batch-rendering of Adobe Premiere Pro, After Effects projects and Adobe Media Encoder using RemoteFX via vGPU. Is there a way to move my Hyper-V VM into Azure without any pain-points ?

Leave a Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Copyright © 2019 Toneatronic. All rights reserved.